Information processing device, information processing method, and information processing program

Abstract

The present invention executes a process for displaying a drawing and an image differing in coordinate system from the drawing one on top of another, even when the drawing is of a general type.　The present invention generates a temporary superimposed diagram using first information that pertains to a drawing of an object having a plurality of components and that is related to components on the drawing, and second information that pertains to an image of the object and that is different in coordinate system from the drawing and includes information related to components on the image. The present invention comprises: a second determination unit that calculates an overlap index that represents the overlapping of components on the drawing and components on the image and determines components on the image that correspond to components on the drawing in the temporary superimposed diagram; a calculation unit that adds up overlap indices at the time the second determination unit determined components on the image that corresponded to components on the drawing in the temporary superimposed diagram and calculates the aggregate overlap index of the temporary superimposed diagram; and a third determination unit that determines, on the basis of the aggregate overlap index, a corresponding relationship between components on the drawing that a first acquisition unit has acquired and components on the image that a second acquisition unit has acquired.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.

[0002] Some information processing devices perform processing for superimposing a drawing and an image having a coordinate system different from that of the drawing and displaying the superimposed image.

[0003] When three-dimensional design data, which is a drawing of a product, is superimposed on a photographed image of the product, a virtual image is generated using the drawing of the product, and feature point pairs are searched for in which local features of the virtual image are similar to local features of the photographed image of the product. Then, based on the feature point pairs, external parameters of the camera that captured the image are estimated, and the estimated external parameters are used to perform processing for superimposing and displaying the photographed image of the product on the three-dimensional design data of the product (for example, see Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2020-17096

[0005] Such information processing devices have a problem in that, when the drawing is general, they cannot perform processing to superimpose and display the drawing on an image having a coordinate system different from that of the drawing. In order to search for feature point pairs in which the local features of a virtual image generated using the drawing are similar to the local features of the image, the drawing must have detailed data that allows the generation of a virtual image having local features similar to the local features of the image. However, this is because general drawings do not have such detailed data.

[0006] The present invention has been made to solve the problems described above, and aims to provide an information processing device that can process and display a drawing and an image that has a different coordinate system from the drawing, even if the drawing is a general one.

[0007] The information processing device of the present invention includes a first acquisition unit that acquires first information, which is information on a drawing of an object having a plurality of parts, and includes information on the parts on the drawing; a second acquisition unit that acquires second information, which is information on an image of the object, and which has a coordinate system different from that of the drawing and includes information on the parts on the image; a first determination unit that uses the first information and the second information to select parts on the drawing and parts on the image and determine a tentative correspondence between the parts on the drawing and the parts on the image; and a generation unit that aligns the coordinate system of the drawing and the coordinate system of the image based on the tentative correspondence, thereby overlaying the drawing and the image and generating a tentative overlaid drawing. The system includes a second determination unit that uses the first information and the second information to determine overlap indices in the temporary overlay drawing that indicate overlap between parts on the drawing and parts on the image, and determines parts on the image that correspond to parts on the drawing in the temporary overlay drawing based on the overlap indices; a calculation unit that calculates a total overlap indices for the temporary overlay drawing by adding up the overlap indices when the second determination unit determines parts on the image that correspond to parts on the drawing in the temporary overlay drawing; and a third determination unit that determines a correspondence relationship between the parts on the drawing acquired by the first acquisition unit and the parts on the image acquired by the second acquisition unit based on the total overlap indices.

[0008] According to the information processing device of the present invention, even if the drawing is a general one, it is possible to perform processing for overlaying and displaying a drawing and an image having a different coordinate system from the drawing. This is because a temporary overlay drawing is generated based on a temporary correspondence determined using first information and second information, a total overlap index of the temporary overlay drawing is calculated, and a correspondence between components on the drawing and components on the image is determined based on the total overlap index. Unlike the prior art, local features are not used to determine the correspondence, so there is no need for feature information to be included in the drawing, and drawings created by general CAD or the like can be used.

[0009] FIG. 1 is a block diagram showing a hardware configuration of a computer that can constitute an information processing device according to a first embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the information processing device according to the first embodiment of the present invention. FIG. 3 is a flowchart showing an information processing method using the information processing device according to the first embodiment of the present invention. FIG. 4 is a diagram explaining a third processing step and a fourth processing step of the information processing method using the information processing device according to the first embodiment of the present invention, showing an example of processing in the third processing step and the fourth processing step. FIG. 5 is a diagram explaining a third processing step and a fourth processing step of the information processing method using the information processing device according to the first embodiment of the present invention, showing another example of processing in the third processing step and the fourth processing step. FIG. 6 is a diagram explaining a fifth processing step of the information processing method using the information processing device according to the first embodiment of the present invention, showing a method of determining a component on an image that corresponds to a component on a drawing in a temporary overlay drawing. FIG. 7 is a diagram explaining a fifth processing step of the information processing method using the information processing device according to the first embodiment of the present invention, showing a method of calculating an overlap index. FIG. 8 is a block diagram showing a functional configuration of an information processing device according to a second embodiment of the present invention. FIG. 9 is a block diagram showing a functional configuration of an information processing device according to a third embodiment of the present invention. FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a fourth embodiment of the present invention. FIG. 11 is a block diagram showing a functional configuration of an information processing device according to a fifth embodiment of the present invention. FIG. 12 is a block diagram showing a functional configuration of an information processing device according to a sixth embodiment of the present invention. FIG. 13 is a block diagram showing a functional configuration of an information processing device according to a seventh embodiment of the present invention. FIG. 14 is a block diagram showing a functional configuration of an information processing device according to an eighth embodiment of the present invention.

[0010] First Embodiment First, a configuration of an information processing device 100 according to a first embodiment of the present invention will be described. The information processing device 100 according to the first embodiment of the present invention can be configured using a computer 1. FIG. 1 is a block diagram showing the hardware configuration of the computer 1 that can configure the information processing device 100 according to the first embodiment of the present invention. As shown in FIG. 1, the computer 1 includes a storage device 2, a CPU (Central Processing Unit) 3, an input device 4, an output device 5, an auxiliary storage device 6, a network connection device 7, and a bus 8. The bus 8 connects the storage device 2, the CPU 3, the input device 4, the output device 5, the auxiliary storage device 6, and the network connection device 7.

[0011] The storage device 2 can be configured using semiconductor memory such as a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), or a flash memory. The storage device 2 stores various programs and data, including an information processing program used by the CPU 3. The CPU 3 is an example of a processor, and executes various processes described below.

[0012] The input device 4 can be configured using, for example, a pointing device such as a mouse, a keyboard, etc. The input device 4 is used when a user inputs instructions, information, etc. to the computer 1. Here, the user includes not only a person who actually operates the computer 1, but also a person who operates the computer 1 remotely. The output device 5 can be configured using, for example, a display device such as a liquid crystal display. The output device 5 outputs the content being processed by the information processing device 100 according to this embodiment, the processing results, messages to the user, etc. The input device 4 and the output device 5 may be configured using, for example, a touch panel having the functions of both a keyboard and a display device. The input device 4 may also be a combination of a keyboard and a pointing device.

[0013] The auxiliary storage device 6 can be configured using semiconductor memory such as a magnetic disk device, an optical disk device, a magneto-optical disk device, or a magnetic tape device. The auxiliary storage device 6 stores various programs and data, including information processing programs used by the CPU 3. The programs and data stored in the auxiliary storage device 6 can also be used by the user by loading them into the storage device 2.

[0014] The network connection device 7 is an example of a communication interface that is connected to a communication network (not shown) such as a LAN (Local Area Network) or a WAN (Wide Area Network) and converts data involved in communication. For example, the computer 1 can receive various programs and data from an external device connectable to the communication network via the network connection device 7 and load them into the storage device 2 for use.

[0015] Note that the computer 1 capable of configuring the information processing device 100 according to this embodiment may have the hardware configuration shown in FIG. 1 modified as appropriate by omitting or adding some devices depending on the conditions of use of the computer 1. For example, if the computer 1 does not need to communicate with external devices, the network connection device 7 may be omitted. Additionally, if the computer 1 is a mobile terminal device with a communication function, the input device 4 may include a microphone, which is an example of a communication device, and the output device 5 may include a speaker, which is an example of a communication device. The input device 4 may also include an imaging device such as a camera.

[0016] Fig. 2 is a block diagram showing the functional configuration of the information processing device 100 according to the first embodiment of the present invention. Each function of the information processing device 100 according to the first embodiment of the present invention is realized by the CPU 3 of the computer 1 shown in Fig. 1. The information processing device 100 according to the first embodiment of the present invention performs processing for superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing.

[0017] The drawings in this embodiment are two-dimensional design data of an object having multiple components, and are drawn using CAD (Computer Aided Design). The coordinate system of the drawings in this embodiment is an XY Cartesian coordinate system with the horizontal axis as the X axis and the vertical axis as the Y axis, where, for example, mm is used as the unit of measurement, and one measurement can represent a length in the SI unit system. Any unit other than mm can be used as long as it can represent the physical length of an actual object, such as meters or inches. The images in this embodiment are images of an object having multiple components captured using an imaging device such as a camera. The coordinate system of the image in this embodiment is an XY Cartesian coordinate system with the horizontal axis as the X axis and the vertical axis as the Y axis, where px is used as the unit of measurement, and one measurement cannot represent a physical length.

[0018] Unlike the coordinate system of a drawing, in which one division is expressed as a length, one division in the coordinate system of an image cannot be expressed as a length, and therefore the units used in the coordinate system of the image cannot be directly converted into the units used in the coordinate system of the drawing. Of course, the units used in the coordinate system of the drawing cannot be directly converted into the units used in the coordinate system of the image, either. In this embodiment, the coordinate system of a drawing and the coordinate system of an image are different, meaning that the units of one division are different, and it is not possible to directly convert the units used in the coordinate system of the drawing into the units used in the coordinate system of the image, or the units used in the coordinate system of the image into the units used in the coordinate system of the drawing.

[0019] Here, the drawings are described as two-dimensional design data drawn using CAD, but the drawings in this embodiment are not limited to this. Furthermore, the images are described as images captured using an imaging device such as a camera, but the images in this embodiment are not limited to this. In this embodiment, the drawings and the images only need to have different coordinate systems. That is, the units of one scale are different, and it is not necessary to directly convert the units used in the coordinate system of the drawing to the units used in the coordinate system of the image, or vice versa.

[0020] In this embodiment, the drawings and images are of objects having multiple parts. Examples of parts include meters, lamps, switches, and nameplates. Examples of objects having multiple parts include measuring instruments with multiple meters, and distribution boards with multiple lamps and switches. However, the parts and objects according to this embodiment are not limited to these. In this embodiment, a part is something that has an area on a drawing and an image, and does not necessarily represent only points or lines, but may be anything that can be represented by a closed curve on a drawing and an image. In other words, in this embodiment, parts on a drawing and parts on an image have an outline. An outline is a line that forms the outer shape of an object, and the outline of a part is a line that shows the outer shape of the part.

[0021] As shown in FIG. 2 , the information processing device 100 according to the first embodiment of the present invention includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 60, a third determination unit 70, and a control unit 80. The first acquisition unit 10 acquires first information, which is information about a drawing of an object having a plurality of parts and includes information about the parts on the drawing. The second acquisition unit 20 acquires second information, which is information about an image of the object, the coordinate system of which is different from that of the drawing and includes information about the parts on the image. The first determination unit 30 uses the first information and the second information to select parts on the drawing and parts on the image and determine a provisional correspondence between the parts on the drawing and the parts on the image. The generation unit 40 overlays the drawing and the image by aligning the coordinate system of the drawing and the coordinate system of the image based on the provisional correspondence, thereby generating a provisional overlaid drawing.

[0022] The second determination unit 50 uses the first information and the second information to calculate overlap indices representing overlaps between components on the drawing and components on the image in the temporary overlay drawing, and determines components on the image corresponding to the components on the drawing in the temporary overlay drawing based on the overlap indices. The calculation unit 60 calculates a total overlap indices for the temporary overlay drawing by summing up the overlap indices used when the second determination unit 50 determined components on the image corresponding to the components on the drawing in the temporary overlay drawing. The third determination unit 70 determines a correspondence between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20 based on the total overlap indices. The control unit 80 controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0023] The first information, which is information about the drawing, includes information about parts on the drawing and information about the coordinate system of the drawing. The first information may also include information such as the purpose and operation of the object, information about the outline of the object, etc. The information about parts on the drawing includes information about the number of parts, the type of part, the coordinates of the center of the part, and the outline of the part. The type of part refers to a meter, lamp, switch, nameplate, etc. The information about the outline of the part includes the length of the outline, the area enclosed by the outline, and the coordinates of the corners if the outline forms a corner. In this embodiment, the information about parts on the drawing does not need to include all of the information such as the number of parts, the type of part, the coordinates of the center of the part, the length of the outline of the part, the area enclosed by the outline of the part, and the coordinates of the corners if the outline forms a corner. It is sufficient to include enough information to perform the operation of the information processing device 100 according to this embodiment and the information processing method using the information processing device 100 according to this embodiment, which will be described later. Furthermore, the outline of the part may have interruptions, and it is sufficient to perform the operation of the information processing device 100 according to this embodiment and the information processing method using the information processing device 100 according to this embodiment, which will be described later, by filling in the interruptions using a well-known method, etc.

[0024] The second information, which is image information, includes information about parts on the image and information about the coordinate system of the image. The second information may include information such as the purpose or operation of the object, information about the contour of the object, etc. The information about parts on the image includes information about the number of parts, the type of part, the coordinates of the center of the part, and the contour of the part. In this embodiment, like the information about parts on the drawing, the information about parts on the image does not need to include all of the following: the number of parts, the type of part, the coordinates of the center of the part, the length of the contour of the part, the area enclosed by the contour of the part, and, if the contour of the part forms a corner, the coordinates of the corner. It is sufficient if the information includes enough information to execute the operation of the information processing device 100 according to this embodiment and the information processing method using the information processing device 100 according to this embodiment, which will be described later.

[0025] Each function of the information processing device 100 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in Fig. 1 , and therefore the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, the third determination unit 70, and the control unit 80 shown in Fig. 2 are functions of the CPU 3. Here, the first information and the second information acquired by the first acquisition unit 10 and the second acquisition unit 20 are loaded into the storage device 2. Therefore, the operations of the first acquisition unit 10 and the second acquisition unit 20 of the information processing device 100 according to the present embodiment are realized by the CPU 3 acquiring the first information and the second information from the storage device 2.

[0026] Next, the operation of the information processing device 100 according to the first embodiment of the present invention will be described, and the configuration of the information processing device 100 according to the first embodiment of the present invention will be described in more detail. Fig. 3 is a flowchart showing an information processing method using the information processing device 100 according to the first embodiment of the present invention.

[0027] As shown in FIG. 3, the information processing method using the information processing device 100 according to the first embodiment of the present invention includes a first processing step S1, a second processing step S2, a third processing step S3, a fourth processing step S4, a fifth processing step S5, a sixth processing step S6, a determination step J1, and a seventh processing step S7.

[0028] First, the first processing step S1 of the information processing method using the information processing device 100 according to the present embodiment is a step of acquiring first information, which is information on a drawing of an object having a plurality of parts and includes information on the parts on the drawing. In the first processing step S1, the CPU 3 having the function of the first acquisition unit 10 of the information processing device 100 according to the present embodiment acquires the first information, which is information on a drawing of an object having a plurality of parts and includes information on the parts on the drawing, from the storage device 2.

[0029] The second processing step S2 of the information processing method using the information processing device 100 according to this embodiment is a process of acquiring second information, which is information on an image of an object, the coordinate system of which is different from that of the drawing, and which includes information on components on the image. In the second processing step S2, the CPU 3 having the function of the second acquisition unit 20 of the information processing device 100 according to this embodiment acquires from the storage device 2 the second information, which is information on an image of an object, the coordinate system of which is different from that of the drawing, and which includes information on components on the image.

[0030] In this embodiment, the information on parts in the image included in the second information is configured as a result of detecting parts included in the image using an object detection technique from an image capturing an object. Machine learning and deep learning are used as object detection techniques, and examples thereof include detection using YOLO (You Only Look Once). Other examples include detection using SVM (Support Vector Machine) and HOG (Histogram of Oriented Gradients) features. From the results of detecting parts included in the image using the object detection technique, information on the number of parts, type of parts, center coordinates of parts, and outlines of parts is obtained, and this information is loaded into the storage device 2 as information on parts in the image.

[0031] Here, it is assumed that the process of detecting parts included in an image using an object detection technique is performed by a device other than the computer 1 that can configure the information processing device 100 according to this embodiment, and information obtained from the result of the process is loaded into the storage device 2. However, this embodiment is not limited to this. For example, the process of detecting parts included in an image using an object detection technique may be performed by the CPU 3 of the computer 1 that can configure the information processing device 100 according to this embodiment, and the information obtained from the result of the process may be loaded into the storage device 2. In this embodiment, it is sufficient that the second acquisition unit 20 acquires information obtained from the result of detecting parts included in an image using the object detection technique.

[0032] The first acquisition unit 10 and the second acquisition unit 20 of the information processing device 100 according to this embodiment output the acquired first information and second information to the first determination unit 30 .

[0033] The third processing step S3 of the information processing method using the information processing device 100 according to the present embodiment is a step of selecting components on the drawing and components on the image using the first information and the second information, and determining a tentative correspondence between the components on the drawing and the components on the image. In the third processing step S3, the CPU 3 having the function of the first determination unit 30 of the information processing device 100 according to the present embodiment uses the first information and the second information to select components on the drawing and components on the image, and determines a tentative correspondence between the components on the drawing and the components on the image.

[0034] The operation of the first determination unit 30 and the third processing step S3 will be described in more detail with reference to Figures 4 and 5. Figures 4 and 5 are diagrams for explaining the third processing step S3 and the fourth processing step S4 of the information processing method using the information processing device 100 according to the first embodiment of the present invention, and show different examples of the processing in the third processing step S3 and the fourth processing step S4, respectively.

[0035] FIGS. 4(a) and 5(a) show diagrams representing drawings of an object having multiple parts, which are drawn using the first information acquired by the first acquisition unit 10 in the first processing step S1. The letters X, Y, Z, and W in FIG. 4(a) are symbols used in the following description. Similarly, the letters S, T, U, and V in FIG. 5(a) are symbols used in the following description. FIGS. 4(b) and 5(b) show diagrams representing images of an object having multiple parts, which are drawn using the second information acquired by the second acquisition unit 20 in the second processing step S2. The letters x, y, z, and w in FIG. 4(b) are symbols used in the following description. Similarly, the letters s, t, u, and v in FIG. 5(b) are symbols used in the following description. Note that, for the sake of clarity, the outlines of the objects are not drawn in FIGS. 4 and 5, but they may be drawn in this embodiment.

[0036] FIG. 4( a) and FIG. 4( b) have different coordinate systems. Similarly, FIG. 5( a) and FIG. 5( b) have different coordinate systems. FIG. 4( c) is a diagram showing an image in which the coordinate system of the image shown in FIG. 4( b) is converted into the coordinate system of the drawing shown in FIG. 4( a) and the coordinate system is aligned with the coordinate system of the drawing shown in FIG. 4( a). Aligning the coordinate system of the drawing and the coordinate system of the image means making the coordinate systems the same for the drawing and the image, meaning that one scale unit in the coordinate system of the drawing and one scale unit in the coordinate system of the image are the same. FIG. 4( d) is a diagram showing a provisional overlay diagram in which the drawing shown in FIG. 4( a) and the image in which the coordinate system has been transformed as shown in FIG. 4( c) are overlaid. In FIG. 4( d), the solid line indicates the drawing, and the dotted line indicates the image in which the coordinate system has been transformed.

[0037] Similarly, Fig. 5(c) is a diagram showing an image in which the coordinate system of the image shown in Fig. 5(b) is converted into the coordinate system of the drawing shown in Fig. 5(a) and the coordinate system is combined with the coordinate system of the drawing shown in Fig. 5(a). Fig. 5(d) is a diagram showing a temporary superimposition diagram in which the drawing shown in Fig. 5(a) and the image in which the coordinate system has been converted shown in Fig. 5(c) are superimposed. In Fig. 5(d), the solid line indicates the drawing, and the dotted line indicates the image in which the coordinate system has been converted.

[0038] As shown in Figures 4(a) and 4(b) and Figures 5(a) and 5(b), the drawings and images each show 17 parts. That is, the object shown in Figures 4(a) and 4(b) and Figures 5(a) and 5(b) has 17 parts. The 17 parts are broken down into 8 meters, 1 large nameplate, and 8 small nameplates. The drawings and images shown in Figures 4 and 5 are examples created to explain the third processing step S3, and the drawings and images to which the processing performed by the information processing device 100 according to this embodiment is applied are not limited to the drawings and images shown in Figures 4 and 5.

[0039] First, using FIG. 4 , a method in which the first determination unit 30 of the information processing device 100 according to this embodiment determines a tentative correspondence between components on a drawing and components on an image in the third processing step S3 will be described. The first determination unit 30 selects four components from the 17 components on the drawing and four components from the 17 components on the image. As an example, as shown in FIG. 4( a), the first determination unit 30 selects four components marked with the symbols X, Y, Z, and W from the 17 components on the drawing. Similarly, as an example, as shown in FIG. 4( b), the first determination unit 30 selects four components marked with the symbols x, y, z, and w from the 17 components on the image.

[0040] Then, the first determination unit 30 provisionally determines the parts on the image that correspond to the selected parts on the drawing. Here, it is provisionally determined that part X on the drawing corresponds to part x on the image, part Y on the drawing corresponds to part y on the image, part Z on the drawing corresponds to part z on the image, and part W on the drawing corresponds to part w on the image. This provisionally determined relationship between the parts on the drawing and the parts on the image is the provisional correspondence relationship in Figure 4. That is, in the provisional correspondence relationship in Figure 4, part X on the drawing corresponds to part x on the image, part Y on the drawing corresponds to part y on the image, part Z on the drawing corresponds to part z on the image, and part W on the drawing corresponds to part w on the image.

[0041] Note that a part on the drawing corresponds to a part on the image, meaning that the part on the drawing and the part on the image represent the same part of the object. In other words, in the tentative correspondence in Figure 4, part X on the drawing and part x on the image represent the same part of the object, meaning that they represent parts with the same position, type, size, and shape in the object. The same applies to the other three selected parts.

[0042] Next, a method for determining a tentative correspondence between components on a drawing and components on an image by the first determination unit 30 of the information processing device 100 according to this embodiment in the third processing step S3 will be described with reference to FIG. 5 . First, the first determination unit 30 selects four components from the 17 components on the drawing and four components from the 17 components on the image. As an example, as shown in FIG. 5( a), the first determination unit 30 selects four components marked with the symbols S, T, U, and V from the 17 components on the drawing. Similarly, as an example, as shown in FIG. 5( b), the first determination unit 30 selects four components marked with the symbols s, t, u, and v from the 17 components on the image.

[0043] Then, the first determination unit 30 provisionally determines the parts on the image that correspond to the selected parts on the drawing. Here, it is provisionally determined that part S on the drawing corresponds to part s on the image, part T on the drawing corresponds to part t on the image, part U on the drawing corresponds to part u on the image, and part V on the drawing corresponds to part v on the image. This provisionally determined relationship between the parts on the drawing and the parts on the image is the provisional correspondence relationship in Figure 5. That is, in the provisional correspondence relationship in Figure 5, part S on the drawing corresponds to part s on the image, part T on the drawing corresponds to part t on the image, part U on the drawing corresponds to part u on the image, and part V on the drawing corresponds to part v on the image.

[0044] The first determination unit 30 of the information processing device 100 according to this embodiment outputs the determined tentative correspondence, the first information, and the second information to the generation unit 40 .

[0045] A fourth processing step S4 of the information processing method using the information processing device 100 according to the present embodiment is a step of generating a provisional overlaid diagram by aligning the coordinate system of the drawing with the coordinate system of the image based on the provisional correspondence relationship. In the fourth processing step S4, the CPU 3 having the function of the generation unit 40 of the information processing device 100 according to the present embodiment aligns the coordinate system of the drawing with the coordinate system of the image based on the provisional correspondence relationship determined by the first determination unit 30, thereby overlaying the drawing with the image and generating a provisional overlaid diagram.

[0046] The operation of the generation unit 40 and the fourth processing step S4 will be described in more detail with reference to FIGS. 4 and 5. First, a method for generating a provisional overlay drawing by the generation unit 40 of the information processing device 100 according to this embodiment in the fourth processing step S4 will be described with reference to FIG. 4. The generation unit 40 calculates a projection transformation matrix that converts the coordinate system of the image into the coordinate system of the drawing based on the central coordinates of components X, Y, Z, and W on the drawing, the central coordinates of components x, y, z, and w on the image, and the provisional correspondence shown in FIG. 4. The generation unit 40 then transforms the image shown in FIG. 4(b) using the calculated projection transformation matrix to align the coordinate system of the image shown in FIG. 4(b) with the coordinate system of the drawing shown in FIG. 4(a). FIG. 4(c) shows an image obtained by transforming the image shown in FIG. 4(b) using the calculated projection transformation matrix. The generation unit 40 then generates a provisional overlay drawing by overlaying the drawing shown in FIG. 4(a) with the transformed image shown in FIG. 4(c), as shown in FIG. 4(d).

[0047] Next, a method for generating a provisional overlay diagram by the generation unit 40 of the information processing device 100 according to this embodiment in the fourth processing step S4 will be described with reference to FIG. 5 . The generation unit 40 calculates a projection transformation matrix that converts the coordinate system of the image into the coordinate system of the drawing based on the central coordinates of the components S, T, U, and V on the drawing, the central coordinates of the components s, t, u, and v on the image, and the provisional correspondences in FIG. 5 . The generation unit 40 then transforms the image shown in FIG. 5( b ) using the calculated projection transformation matrix to match the coordinate system of the image shown in FIG. 5( b ) with the coordinate system of the drawing shown in FIG. 5( a ). FIG. 5( c ) shows an image obtained by transforming the image shown in FIG. 5( b ) using the calculated projection transformation matrix. The generation unit 40 then generates a provisional overlay diagram by overlaying the drawing shown in FIG. 5( a ) with the transformed image shown in FIG. 5( c ), as shown in FIG. 5( d ).

[0048] In this embodiment, the tentative correspondences between parts on the drawing and parts on the image determined by the first determination unit 30 in the third processing step S3 are determined for all combinations of parts on the drawing and parts on the image. In this embodiment, the object has 17 parts. Furthermore, in order for the generation unit 40 to determine the projective transformation matrix in the fourth processing step S4, the first determination unit 30 selects four parts from the 17 parts on the drawing and four parts from the 17 parts on the image in the third processing step S3. Therefore, the total number of tentative correspondences determined by the first determination unit 30 in the third processing step S3 is １７ C ４ × １７ P ４ is.

[0049] Here, when the first determination unit 30 determines the provisional correspondence in the third processing step S3, it selects four of the 17 components on the drawing and four of the 17 components on the image. However, this embodiment is not limited to this. For example, the first determination unit 30 may select three of the 17 components on the drawing and three of the 17 components on the image in the third processing step S3. If the first determination unit 30 selects three components from each of the components on the drawing and the components on the image, the generation unit 40 may calculate an affine transformation matrix instead of a projective transformation matrix in the fourth processing step S4, and perform processing to align the coordinate system of the drawing with the coordinate system of the image. When the first determination unit 30 determines the provisional correspondence in the third processing step S3, the number of components selected from each of the components on the drawing and the components on the image may be any number necessary to calculate a transformation matrix that aligns the coordinate system of the drawing with the coordinate system of the image.

[0050] Furthermore, here, the generation unit 40 determines a projection transformation matrix that transforms the coordinate system of the image into the coordinate system of the drawing based on the central coordinates of the components X, Y, Z, and W on the drawing, the central coordinates of the components x, y, z, and w on the image, and the provisional correspondence relationship in Fig. 4 , but this embodiment is not limited to this. As long as a projection transformation matrix can be determined, for example, the projection transformation matrix may be determined based on the coordinates of the corners of the components X, Y, Z, and W on the drawing, the coordinates of the corners of the components x, y, z, and w on the image, and the provisional correspondence relationship in Fig. 4 . The same applies to the projection transformation matrix that transforms the coordinate system of the image into the coordinate system of the drawing based on the central coordinates of the components S, T, U, and V on the drawing, the central coordinates of the components s, t, u, and v on the image, and the provisional correspondence relationship in Fig. 5 .

[0051] Although the generation unit 40 converts the coordinate system of the image into the coordinate system of the drawing, the generation unit 40 of the information processing device 100 according to this embodiment may convert the coordinate system of the drawing into the coordinate system of the image to generate a provisional overlay diagram. That is, referring to FIG. 4 , a projection transformation matrix for converting the coordinate system of the drawing shown in FIG. 4( a) into the coordinate system of the image shown in FIG. 4( b) may be calculated based on a provisional correspondence between the central coordinates of components X, Y, Z, and W on the drawing and the central coordinates of components x, y, z, and w on the image, and the drawing shown in FIG. 4( a) may be transformed using the calculated projection transformation matrix. Then, the coordinate system of the drawing shown in FIG. 4( a) and the coordinate system of the image shown in FIG. 4( b) may be matched to generate a provisional overlay diagram.

[0052] The generation unit 40 of the information processing device 100 according to this embodiment outputs the generated temporary overlay image, the temporary correspondence determined by the first determination unit 30, the first information, and the second information to the second determination unit 50.

[0053] A fifth processing step S5 of the information processing method using the information processing device 100 according to this embodiment is a step of using the first information and the second information to obtain overlapping indices indicating overlapping between components on the drawing and components on the image in the temporary overlay drawing, and determining, based on the overlapping indices, components on the image that correspond to the components on the drawing in the temporary overlay drawing. In the fifth processing step S5, the CPU 3 having the function of the second determination unit 50 of the information processing device 100 according to this embodiment obtains overlapping indices indicating overlapping between components on the drawing and components on the image in the temporary overlay drawing generated by the generation unit 40, and determines, based on the overlapping indices, components on the image that correspond to the components on the drawing in the temporary overlay drawing.

[0054] The operation of the second determination unit 50 and the fifth processing step S5 will be described in more detail with reference to FIGS. 6 and 7. FIG. 6 is a diagram illustrating the fifth processing step S5 of the information processing method using the information processing device 100 according to the first embodiment of the present invention, and is a diagram illustrating how to determine components on an image corresponding to components on a drawing in a temporary overlay diagram. FIG. 6(a) shows an example of a temporary overlay diagram, with solid lines indicating the drawing and dotted lines indicating an image obtained by converting the coordinate system. The letters L, M, N, O, and P in FIG. 6(a) are symbols used in the following description. Similarly, the letters l, m, n, o, and p in FIG. 6(b) are symbols used in the following description. FIG. 7 is a diagram illustrating the fifth processing step S5 of the information processing method using the information processing device 100 according to the first embodiment of the present invention, and is a diagram illustrating how to determine an overlap index.

[0055] First, the second determining unit 50 selects one component on the drawing in the temporary overlay drawing shown in Fig. 6(a), which is an example of a temporary overlay drawing. Here, it is assumed that component L on the drawing is selected, as shown in Fig. 6(b). Next, the second determining unit 50 calculates overlap indices between component L on the drawing and components l, m, n, o, and p on the image.

[0056] Using FIG. 7 , a method for determining an overlap index representing the overlap between a component on a drawing and a component on an image will be described. The overlap index is determined for each component in a temporary overlay drawing. FIG. 7( a ) is a diagram showing the overlap between a component on the drawing and a component on a transformed image for a given component in the temporary overlay drawing. In FIG. 7( a ), the solid line indicates the drawing, and the dotted line indicates the image with the transformed coordinate system. FIG. 7( b ) is a diagram showing the overlapping surfaces (intersection) between the surfaces representing the component on the drawing and the surfaces representing the component on the transformed image for the component on the drawing and the component on the image shown in FIG. 7( a ). FIG. 7( c ) is a diagram showing the union of the surfaces of the component on the drawing and the surface of the component on the transformed image for the component on the drawing and the component on the image shown in FIG. 7( a ).

[0057] Fig. 7(d) is a diagram showing the state in which a part on the drawing and a part on the transformed image overlap for one part in another temporary overlay drawing. In Fig. 7(d), as in Fig. 7(a), the solid lines indicate the drawing, and the dotted lines indicate the image with the transformed coordinate system. Fig. 7(e) is a diagram showing the overlapping surfaces (intersection) between the surfaces representing the part on the drawing and the surfaces representing the part on the transformed image for the part on the drawing and the part on the image shown in Fig. 7(d). Fig. 7(f) is a diagram showing the union of the surfaces of the part on the drawing and the surface of the part on the transformed image for the part on the drawing and the part on the image shown in Fig. 7(d).

[0058] The overlap index for the part shown in Fig. 7(a) is the value obtained by dividing the area of ​​the face shown in Fig. 7(b) by the area of ​​the face shown in Fig. 7(c), and is called an Intersection over Union (IoU) value. Similarly, the overlap index for the part shown in Fig. 7(d) is the value obtained by dividing the area of ​​the face shown in Fig. 7(e) by the area of ​​the face shown in Fig. 7(f).

[0059] The diagram of the parts shown in Fig. 7(a) is a partial citation of the diagram shown in Fig. 4(d). The diagram of the parts shown in Fig. 7(d) is a partial citation of the diagram shown in Fig. 5(d). When the overlap index for the parts shown in Fig. 7(a) and the overlap index for the parts shown in Fig. 7(d) are actually calculated, the overlap index for the parts shown in Fig. 7(d) is larger than the overlap index for the parts shown in Fig. 7(a).

[0060] The overlap indexes between part L on the drawing shown in Fig. 6(b) and each of parts l, m, n, o, and p on the image are calculated using the method described above with reference to Fig. 7. The results are that the overlap index between part L on the drawing and part l on the image is 0.9, the overlap index between part L on the drawing and part m on the image is 0.0, the overlap index between part L on the drawing and part n on the image is 0.0, the overlap index between part L on the drawing and part o on the image is 0.0, and the overlap index between part L on the drawing and part p on the image is 0.0.

[0061] Based on this result, the second determination unit 50 determines a component on the image that corresponds to a component on the drawing in the temporary overlay diagram. When the overlap indices between component L on the drawing and components l, m, n, o, and p on the image are compared, the largest overlap indices are those between component L on the drawing and component l on the image. Therefore, in fifth processing step S5, the second determination unit 50 of the information processing device 100 according to this embodiment determines that the component on the image that corresponds to component L on the drawing in the temporary overlay diagram is component l on the image.

[0062] Next, the second determination unit 50 selects one of the components on the drawing for which a corresponding component on the image has not yet been determined in the temporary overlay diagram shown in FIG. 6( a), which is an example of a temporary overlay diagram. In this example, it is assumed that component M on the drawing is selected, as shown in FIG. 6( c). Then, the second determination unit 50 calculates the overlap index between component M on the drawing and the component on the image. At this time, component l on the image, for which a corresponding component on the drawing has already been determined, is excluded. In other words, the overlap index between component M on the drawing and each of components m, n, o, and p on the image is calculated.

[0063] Thereafter, the process is performed in the same manner as when determining the part on the image corresponding to part L on the drawing. That is, the overlap indexes of part M on the drawing and parts m, n, o, and p on the image are compared, and the part on the image with the largest overlap index is determined to be the part on the image corresponding to part M on the drawing. The same process is performed for parts O, N, and P on the drawing shown in FIG. 6A, and the parts on the image corresponding to each of them are determined. Referring to FIG. 6A, in the fifth processing step S5, the second determination unit 50 determines that part L on the drawing corresponds to part l on the image, part M on the drawing corresponds to part m on the image, part N on the drawing corresponds to part n on the image, part O on the drawing corresponds to part o on the image, and part P on the drawing corresponds to part p on the image. As described above, in this embodiment, the parts on the image corresponding to all parts on the drawing in the temporary overlay drawing are determined.

[0064] However, if the overlap index between the largest part on the image and the part on the drawing is 0.0, i.e., if there is no part on the image that overlaps with the part on the drawing, the part on the image that corresponds to the part on the drawing is left unknown, and processing continues. Even if the number of parts on the image is fewer than the number of parts on the drawing, the corresponding parts on the image for some of the parts on the drawing will remain unknown, but processing continues. Note that in Figure 6, the outline of the object is shown for convenience so that the number of parts that the object has can be seen.

[0065] Although corresponding components on the image are determined for all components on the drawings in the temporary overlay diagram, this embodiment is not limited to this. It is sufficient that no contradictions occur when the sixth processing step S6, the determination step J1, and the seventh processing step S7 described below are performed. For example, the second determination unit 50 may first select one component on the image in the temporary overlay diagram shown in FIG. 6( a), which is an example of a temporary overlay diagram, and then determine the overlap index between the selected component on the image and the components on each drawing. In other words, corresponding components on the drawings may be determined for all components on the image in the temporary overlay diagram.

[0066] The second determination unit 50 of the information processing device 100 according to this embodiment outputs the overlap index value when determining the component on the image corresponding to the component on the drawing, the tentative correspondence determined by the first determination unit 30, the first information, and the second information to the calculation unit 60. Here, the tentative correspondence determined by the first determination unit 30 is the tentative correspondence used when the tentative overlay drawing was generated.

[0067] A sixth processing step S6 of the information processing method using the information processing device 100 according to this embodiment is a step of calculating a total overlap index of the temporary overlay drawing by summing up the overlap indexes when the components on the drawing and the corresponding components on the image were determined in the fifth processing step S5 in the temporary overlay drawing. In the sixth processing step S6, the CPU 3 having the function of the calculation unit 60 of the information processing device 100 according to this embodiment calculates a total overlap index of the temporary overlay drawing by summing up the overlap indexes when the second determination unit 50 determined the components on the drawing and the corresponding components on the image in the temporary overlay drawing generated by the generation unit 40.

[0068] The operation of the calculation unit 60 and the sixth processing step S6 will be described in further detail. Referring to FIG. 6 , in the fifth processing step S5, the calculation unit 60 adds together the overlap index obtained when it is determined that the part on the image corresponding to part L on the drawing is part l on the image, the overlap index obtained when it is determined that the part on the image corresponding to part M on the drawing is part m on the image, the overlap index obtained when it is determined that the part on the image corresponding to part N on the drawing is part n on the image, the overlap index obtained when it is determined that the part on the image corresponding to part O on the drawing is part o on the image, and the overlap index obtained when it is determined that the part on the image corresponding to part P on the drawing is part p on the image. This sum is the total overlap index value of the provisional overlay diagram.

[0069] Note that when the total overlap index of the provisional overlay drawing shown in FIG. 4(d) and the total overlap index of the provisional overlay drawing shown in FIG. 5(d) are actually calculated, the overlap index for the part shown in FIG. 7(d) is larger than the overlap index for the part shown in FIG. 7(a). Taking this into consideration, the total overlap index of the provisional overlay drawing shown in FIG. 5(d) will be larger than the total overlap index of the provisional overlay drawing shown in FIG. 4(d).

[0070] The information processing method using the information processing device 100 according to the present embodiment includes a determination step J1, which is a process of determining whether or not the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all the tentative correspondences output in the third processing step S3. In the determination step J1, the CPU 3 having the function of the control unit 80 of the information processing device 100 according to the present embodiment determines whether or not the operations of the generation unit 40, the second determination unit 50, and the calculation unit 60 have been performed for all the tentative correspondences output by the first determination unit 30.

[0071] If the control unit 80 determines in judgment step J1 that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all the tentative correspondences output in the third processing step S3, the control unit 80 instructs the third determination unit 70 to operate. If the control unit 80 determines in judgment step J1 that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have not been performed for all the tentative correspondences output in the third processing step S3, the control unit 80 instructs the generation unit 40 to operate for the tentative correspondences for which no tentative overlay diagram has yet been generated. In other words, the fourth processing step S4 by the generation unit 40, the fifth processing step S5 by the second determination unit 50, and the sixth step S6 by the calculation unit 60 are repeatedly performed for all the tentative correspondences output by the first determination unit 30 in the third processing step S3.

[0072] The calculation unit 60 calculates the overlap index totals for all the tentative correspondences output by the first determination unit 30. If the control unit 80 determines in determination step J1 that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all the tentative correspondences output in the third processing step S3, the calculation unit 60 outputs all of the calculated overlap index totals, the tentative overlay diagrams from which the respective overlap index totals were calculated, the tentative correspondences used when the tentative overlay diagrams were generated, the first information, and the second information to the third determination unit 70. In other words, the third determination unit 70 of the information processing device 100 according to this embodiment receives from the calculation unit 60 the total overlap index values ​​calculated by the calculation unit 60 for all the provisional correspondence relationships output by the first determination unit 30, the provisional overlay diagrams from which the respective total overlap indexes have been calculated, the provisional correspondence relationships determined by the first determination unit 30 and used when the provisional overlay diagrams were generated by the generation unit 40, the first information, and the second information.

[0073] A seventh processing step S7 of the information processing method using the information processing device 100 according to this embodiment is a step of determining a correspondence relationship between the components on the drawing acquired in the first processing step S1 and the components on the image acquired in the second processing step S2 based on the overlap index total calculated in the sixth processing step S6. In the seventh processing step S7, the CPU 3 having the function of the third determination unit 70 of the information processing device 100 according to this embodiment determines a correspondence relationship between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20 based on the overlap index total calculated by the calculation unit 60.

[0074] First, the third determination unit 70 compares all of the overlap index totals transmitted from the calculation unit 60 and selects the temporary overlay diagram with the largest value. Then, the third determination unit 70 determines that the temporary correspondence relationship used when the temporary overlay diagram was generated is the correct correspondence relationship between the components on the drawing and the components on the image in this embodiment. In other words, the third determination unit 70 determines that the correspondence relationship between the components on the drawing and the components on the image in the temporary correspondence relationship used when the temporary overlay diagram with the largest overlap index total was generated is the correspondence relationship between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20.

[0075] The operation of the third determination unit 70 and the seventh processing step S7 will be described with reference to FIGS. 4 and 5. For ease of explanation, it is assumed that the third determination unit 70 receives two overlap index totals from the calculation unit 60: the total overlap index of the provisional overlay diagram shown in FIG. 4(d) and the total overlap index of the provisional overlay diagram shown in FIG. 5(d). The third determination unit 70 compares the total overlap index of the provisional overlay diagram shown in FIG. 4(d) with the total overlap index of the provisional overlay diagram shown in FIG. 5(d), and selects the provisional overlay diagram shown in FIG. 5(d) as the provisional overlay diagram having the largest value. This is because, as described above in the description of the sixth processing step S6, the total overlap index of the provisional overlay diagram shown in FIG. 5(d) is larger than the total overlap index of the provisional overlay diagram shown in FIG. 4(d).

[0076] Next, referring to the provisional correspondence relationships used when the provisional overlay drawing shown in Fig. 5(d) was generated, it can be seen that, as shown in Fig. 5(a) and Fig. 5(b), a part S on the drawing corresponds to a part s on the image, a part T on the drawing corresponds to a part t on the image, a part U on the drawing corresponds to a part u on the image, and a part V on the drawing corresponds to a part v on the image. Therefore, the third determination unit 70 determines that the correspondence relationships shown in Fig. 5(a) and Fig. 5(b), in which the part S on the drawing corresponds to the part s on the image, the part T on the drawing corresponds to the part t on the image, the part U on the drawing corresponds to the part u on the image, and the part V on the drawing corresponds to the part v on the image, are correspondence relationships between the parts on Fig. 5(a) showing the drawing acquired by the first acquisition unit 10 and the parts on Fig. 5(b) showing the image acquired by the second acquisition unit 20.

[0077] The information processing device 100 according to this embodiment is a device that performs processing for superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing, and the correspondence between components on the drawing and components on the image determined by the third determination unit 70 of the information processing device 100 according to this embodiment is used for superimposing and displaying the drawing and the image having a coordinate system different from that of the drawing. The information processing device 100 according to this embodiment may output only the correspondence between the components on the drawing and the components on the image, or may output an overlay diagram generated based on the correspondence between the components on the drawing and the components on the image determined by the third determination unit 70. Note that the overlay diagram generated based on the correspondence between the components on the drawing and the components on the image determined by the third determination unit 70 can be a provisional overlay diagram generated by the generation unit 40 and having the largest total overlap index value, so there is no need to newly generate it, and the information processing device 100 according to this embodiment can output it.

[0078] As described above, the information processing device 100 according to this embodiment can perform processing for overlaying and displaying a drawing and an image having a different coordinate system from that of the drawing, even if the drawing is a general drawing. This is because a temporary overlay drawing is generated based on a temporary correspondence determined using the first information and the second information, the total overlap index of the temporary overlay drawing is calculated, and the correspondence between the components on the drawing and the components on the image is determined based on the total overlap index. Unlike the prior art, local features are not used to determine the correspondence, so there is no need for the drawing to have feature information, and drawings created by general CAD or the like can be used.

[0079] Furthermore, the information processing device 100 according to the present embodiment can perform processing for accurately superimposing and displaying a drawing and an image having a different coordinate system from that of the drawing, even if the object has multiple components with the same appearance. Conventionally, when the object has multiple components with the same appearance, there are many points with the same feature values, so processing for accurately superimposing and displaying a drawing and an image having a different coordinate system from that of the drawing has not been possible. On the other hand, the information processing device 100 according to the present embodiment can perform processing for accurately superimposing and displaying a drawing and an image having a different coordinate system from that of the drawing. This is because a provisional overlay diagram is generated based on the provisional correspondence determined using the first information and the second information, the total overlap index of the provisional overlay diagram is calculated, and the correspondence between the components on the drawing and the components on the image is determined based on the total overlap index.

[0080] Furthermore, the information processing device 100 according to the present embodiment does not require manual work by the user, and therefore can easily perform the process of superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing. Furthermore, in the past, depending on the user's level of proficiency, the time required for the process of superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing could be long. On the other hand, the information processing device 100 according to the present embodiment does not require manual work by the user, and therefore can shorten the time required for the process of superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing.

[0081] Furthermore, the information processing device 100 according to the present embodiment can reduce the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing, even when the object has multiple components with the same appearance, and can also prevent the drawings from being superimposed with an incorrect correspondence. Conventionally, when the object has multiple components with the same appearance, a large number of feature points with similar local features exist, making it difficult to determine the correspondence between the feature points on the drawing and the feature points on the image. Therefore, depending on the processing method, the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing could be long. Furthermore, the presence of feature points with similar local features could lead to the determination of a correspondence between components that appear the same but are actually different, resulting in an incorrect superimposition.

[0082] On the other hand, the information processing device 100 according to the present embodiment can shorten the time required for processing to overlay and display a drawing and an image having a coordinate system different from that of the drawing, and can also prevent an erroneous overlay display result, because the information processing device 100 generates a temporary overlay drawing based on the temporary correspondence determined using the first information and the second information, calculates the total overlap index of the temporary overlay drawing, and determines the correspondence between the parts on the drawing and the parts on the image based on the total overlap index.

[0083] The correspondence between components on a drawing and components on an image output by the information processing device 100 according to the present embodiment, and the overlay diagram generated based on the correspondence between components on a drawing and components on an image, can be used for various purposes. For example, the correspondence between components on a drawing and components on an image determined by the third determination unit 70 of the information processing device 100 according to the present embodiment and the coordinates of the components on the drawing and the components on the image can be used to estimate the extrinsic parameters (also called the position and orientation) of the camera that captured the object, and the result can be used to overlay the drawing on an image with a different coordinate system from the drawing. Furthermore, for example, when displaying an augmented reality (AR), an image may be overlaid on a drawing with a different coordinate system from the image. By using the correspondence between components on a drawing and components on an image output by the information processing device 100 according to the present embodiment, the image can be overlaid on a drawing with a different coordinate system from the image.

[0084] Furthermore, the superimposition of a drawing and an image having a different coordinate system from that of the drawing, which is displayed based on the output of the information processing device 100 according to this embodiment, can be used for tasks such as checking whether a product has been manufactured according to the drawing, or for inspection tasks to remotely determine whether there are any abnormalities in equipment.

[0085] In the above description, the overlap index is an IoU value, but this is not limited to this in the present embodiment. For example, the overlap index may be the area itself where a component on a drawing overlaps a component on an image. Even if the overlap index is the area itself where a component on a drawing overlaps a component on an image, the operation of the information processing device 100 according to this embodiment and the information processing method using the information processing device 100 according to this embodiment remain unchanged.

[0086] Alternatively, for example, the length of the line where the outline of the component on the drawing and the outline of the component on the image overlap may be used as the overlap index. Even if the overlap index is the length of the line where the outline of the component on the drawing and the outline of the component on the image overlap, the operation of information processing device 100 according to this embodiment and the information processing method using information processing device 100 according to this embodiment do not change.

[0087] Alternatively, for example, the overlap index may be the reciprocal of the sum of the distances between the outlines of the components on the drawing and the outlines of the components on the image at locations where the outlines of the components on the drawing and the outlines of the components on the image do not overlap. Dots are evenly spaced on the outlines of the components on the drawing and the outlines of the components on the image, and the distances between each of the dots are calculated and summed. The reciprocal of this sum is then used as the overlap index. Even if the overlap index is set to the reciprocal of the sum of the distances between the outlines of the components on the drawing and the outlines of the components on the image at locations where the outlines of the components on the drawing and the outlines of the components on the image do not overlap, the operation of the information processing device 100 according to this embodiment and the information processing method using the information processing device 100 according to this embodiment do not change.

[0088] In the above description, the control unit 80 of the information processing device 100 according to the present embodiment instructs the generation unit 40 to perform the fourth processing step S4, the second determination unit 50 to perform the fifth processing step S5, and the calculation unit 60 to perform the sixth processing step S6 repeatedly for all the tentative correspondence relationships output by the first determination unit 30 in the third processing step S3. However, the present embodiment is not limited to this. For example, the following may be performed.

[0089] First, in a fourth processing step S4, the generation unit 40 generates a temporary overlay diagram for each of all the temporary correspondence relationships output by the first determination unit 30 in the third processing step S3. The generation unit 40 then outputs to the second determination unit 50 all of the temporary overlay diagrams generated for each of the temporary correspondence relationships determined by the first determination unit 30, the temporary correspondence relationships determined by the first determination unit 30, the first information, and the second information. The second determination unit 50 calculates overlap indices between components on the drawings and components on the images for each of the temporary overlay diagrams, and determines components on the images corresponding to the components on the drawings in the temporary overlay diagrams based on the overlap indices. The second determination unit 50 then outputs to the calculation unit 60 the overlap indices used when determining the components on the images corresponding to the components on the drawings for each of the temporary overlay diagrams, the temporary correspondence relationships determined by the first determination unit 30, the first information, and the second information. The calculation unit 60 calculates the total overlap index for all the provisional overlaid drawings and outputs the result to the third determination unit 70. The third determination unit 70 then compares all of the total overlap indexes transmitted from the calculation unit 60 and determines the correspondence between the components on the drawings acquired by the first acquisition unit 10 and the components on the images acquired by the second acquisition unit 20.

[0090] In this case, it can be said that the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70 of the information processing device 100 according to this embodiment each have the function of controlling themselves in addition to the functions described above.

[0091] The order of the first processing step S1 and the second processing step S2 may be reversed.

[0092] In the above description, the first information and the second information acquired by the first acquisition unit 10 and the second acquisition unit 20 are loaded into the storage device 2, but the present embodiment is not limited to this. The first information and the second information may be stored in an external device, and the first acquisition unit 10 and the second acquisition unit 20 may acquire the first information and the second information from the external device via the network connection device 7.

[0093] Furthermore, in the above description, the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70 of the information processing device 100 according to the present embodiment are described as functions provided in the CPU 3. However, they may be configured as follows: First, a program including a first processing step S1, a second processing step S2, a third processing step S3, a fourth processing step S4, a fifth processing step S5, a sixth processing step S6, a seventh processing step S7, and a determination step J1 is stored in the storage device 2. The CPU 3 then reads and executes the program from the storage device 2. Of course, the location where the program is stored is not limited to the storage device 2.

[0094] Second Embodiment The configuration of an information processing device 200 according to a second embodiment of the present invention will be described. The information processing device 200 according to the second embodiment of the present invention can also be configured using a computer 1, similar to the information processing device 100 according to the first embodiment of the present invention. In the second embodiment of the present invention, only the differences from the first embodiment of the present invention will be described, and a description of the parts that are the same as those in the first embodiment of the present invention will be omitted.

[0095] FIG. 8 is a block diagram showing a functional configuration of an information processing device 200 according to a second embodiment of the present invention. Each function of the information processing device 200 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 8, the information processing device 200 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 32, a generation unit 40, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The first determination unit 32 of the information processing device 200 according to the present embodiment includes a first determination unit 91. The information processing device 200 according to the present embodiment also includes a control unit 80, similar to the information processing device 100 according to the first embodiment of the present invention, but this control unit 80 is not shown in FIG. 8. The control unit 80 of the information processing device 200 according to the present embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 32, the generation unit 40, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0096] The information processing device 200 according to this embodiment differs from the first embodiment of the present invention in that the first determination unit 32 includes a first determination unit 91. The first determination unit 91 determines whether the type of a component on the image corresponding to a component on the drawing is the same as the type of the component on the drawing in the tentative correspondence. If the first determination unit 91 determines that the components are the same, the first determination unit 32 outputs the tentative correspondence to the generation unit 40.

[0097] The operation of the first determination unit 32 including the first judgment unit 91 and the third processing step included in the information processing method using the information processing device 200 according to this embodiment will be described in detail with reference to FIGS. 4 and 5 . In the third processing step, a method in which the first determination unit 32 of the information processing device 200 according to this embodiment determines a tentative correspondence between components on a drawing and components on an image will be described. As an example, as shown in FIG. 4A, the first determination unit 32 selects four components marked with the symbols X, Y, Z, and W from among the 17 components on the drawing. Similarly, as an example, as shown in FIG. 4B, the first determination unit 32 selects four components marked with the symbols x, y, z, and w from among the 17 components on the image.

[0098] The first determination unit 32 provisionally determines parts on the image that correspond to the selected parts on the drawing. Here, it is provisionally determined that part X on the drawing corresponds to part x on the image, part Y on the drawing corresponds to part y on the image, part Z on the drawing corresponds to part z on the image, and part W on the drawing corresponds to part w on the image. This provisionally determined relationship between the parts on the drawing and the parts on the image is the provisional correspondence relationship in FIG. 4.

[0099] The first determination unit 91 of the first determination unit 32 determines whether the type of a part on the image corresponding to a part on the drawing is the same as the type of the part on the drawing in the tentative correspondence, and therefore first determines whether the type of part x on the image corresponding to part X on the drawing is the same as the type of part X on the drawing. Next, the first determination unit 91 of the first determination unit 32 determines whether the type of part y on the image corresponding to part Y on the drawing is the same as the type of part Y on the drawing. Similarly, the first determination unit 91 determines whether the type of part z on the image corresponding to part Z on the drawing is the same as the type of part Z on the drawing, and determines whether the type of part w on the image corresponding to part W on the drawing is the same as the type of part W on the drawing.

[0100] If the first determination unit 91 determines that the parts are the same, the first determination unit 32 outputs this tentative correspondence to the generation unit 40. Here, as shown in FIG. 4 , the type of part x on the image corresponding to part X on the drawing is a meter, and the type of part X on the drawing is also a meter. Therefore, the first determination unit 91 determines that the type of part x on the image corresponding to part X on the drawing is the same as the type of part X on the drawing. Similarly, the first determination unit 91 determines that the type of part Y on the drawing is the same as the type of part y on the image, determines that the type of part Z on the drawing is the same as the type of part z on the image, and determines that the type of part W on the drawing is the same as the type of part w on the image. Therefore, the first determination unit 32 outputs to the generation unit 40 a tentative correspondence in which part X on the drawing corresponds to part x on the image, part Y on the drawing corresponds to part y on the image, part Z on the drawing corresponds to part z on the image, and part W on the drawing corresponds to part w on the image.

[0101] If the first determination unit 91 determines that the tentative correspondence relationship is not correct, the first determination unit 32 does not output this tentative correspondence relationship to the generation unit 40. For example, in the tentative correspondence relationship in FIG. 5 , assume that a part S on the drawing in FIG. 5( a) corresponds to a part t on the image in FIG. 5( b). The type of the part t on the image corresponding to the part S on the drawing is a nameplate, but the type of the part S on the drawing is a meter. Therefore, the first determination unit 91 determines that the type of the part t on the image corresponding to the part S on the drawing is not the same as the type of the part S on the drawing (no). As a result, the first determination unit 32 does not output the tentative correspondence relationship indicating that the part S on the drawing in FIG. 5( a) corresponds to the part t on the image in FIG. 5( b) to the generation unit 40. If the first determination unit 91 determines that the tentative correspondence relationship is not correct, the first determination unit 32 newly selects a part on the drawing and a part on the image to determine the tentative correspondence relationship.

[0102] As described above, the information processing device 200 according to the present embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing. This is because, when the first determination unit 91 determines that the types of parts on the image corresponding to the parts on the drawing are the same in the tentative correspondence, the first determination unit 32 outputs the tentative correspondence to the generation unit 40, thereby preventing the output of an erroneous correspondence.

[0103] Here, the first judgment unit 91 judges whether the type of part on the image corresponding to the part on the drawing is the same as the type of part on the drawing, but it may also be configured to judge whether the type of part on the drawing corresponding to the part on the image is the same as the type of part on the image.

[0104] Third Embodiment The configuration of an information processing device 300 according to a third embodiment of the present invention will be described. The information processing device 300 according to the third embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first and second embodiments of the present invention. In the third embodiment of the present invention, only the differences from the first and second embodiments of the present invention will be described, and descriptions of the parts that are the same as the first and second embodiments of the present invention will be omitted.

[0105] FIG. 9 is a block diagram showing a functional configuration of an information processing device 300 according to a third embodiment of the present invention. Each function of the information processing device 300 according to this embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 9, the information processing device 300 according to this embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 40, a second determination unit 53, a calculation unit 60, and a third determination unit 70. Like the information processing devices according to the first and second embodiments of the present invention, the information processing device 300 according to this embodiment also includes a control unit 80, which is not shown in FIG. 9. The control unit 80 of the information processing device 300 according to this embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 53, the generation unit 60, and the third determination unit 70.

[0106] The information processing device 300 according to this embodiment differs from the first and second embodiments of the present invention in the operation of the second determination unit 53 and the fifth processing step. The second determination unit 53 of the information processing device 300 according to this embodiment uses the first information and the second information in the temporary overlay drawing to obtain an overlap index indicating an overlap between a component on the drawing and a component on the image. Then, when determining a component on the image that corresponds to a component on the drawing in the temporary overlay drawing based on the overlap index, if the overlap index is equal to or greater than a certain value, the second determination unit 53 determines that the component on the image corresponds to the component on the drawing in the temporary overlay drawing.

[0107] The operation of the second determination unit 53 of the information processing device 300 according to this embodiment and the fifth processing step included in the information processing method using the information processing device 300 according to this embodiment will be described in detail with reference to Fig. 6. First, the second determination unit 53 of the information processing device 300 according to this embodiment selects one component on the drawing in the temporary overlay drawing shown in Fig. 6(a), which is an example of a temporary overlay drawing. Here, it is assumed that the component L on the drawing is selected, as shown in Fig. 6(b).

[0108] Next, an overlap index between the part L on the drawing and each of the parts l, m, n, o, and p on the image is calculated. Here, an IoU value is calculated as the overlap index. The second determination unit 53 then compares the overlap index between the part L on the drawing and each of the parts l, m, n, o, and p on the image. The overlap index with the largest overlap index is the overlap index between the part L on the drawing and the part l on the image. The second determination unit 53 determines whether the value of the overlap index between the part L on the drawing and the part l on the image with the largest value is equal to or greater than a certain value. If the value is equal to or greater than the certain value, the second determination unit 53 determines that the part on the image corresponding to the part L on the drawing in the temporary overlay diagram is the part l on the image. In other words, if the value of the overlap index between the part L on the drawing and the part l on the image with the largest value is less than a certain value, the second determination unit 53 does not determine that the part on the image corresponding to the part L on the drawing in the temporary overlay diagram is the part l on the image. Here, the certain value is, for example, 0.5.

[0109] If the value of the overlap index between the largest component L on the drawing and the component l on the image is less than a certain value, the second determination unit 53 cannot determine the component on the image that corresponds to the component on the drawing in the temporary overlay drawing, and therefore stops the processing. Since the second determination unit 53 stops the processing, it does not output anything to the calculation unit 60. If the second determination unit 53 stops the processing, i.e., if the fifth processing step is stopped, the control unit 80 skips the sixth processing step S6 and performs the judgment step J1.

[0110] Furthermore, even if the value of the overlap index between the part L on the drawing, which had the largest value, and the part l on the image is less than a certain value, the second determination unit 53 may continue processing, rather than terminating the processing as described above, by determining that the part on the image corresponding to this part on the drawing is unknown.

[0111] As described above, the information processing device 300 according to this embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing. This is because, when the second determination unit 53 determines a component on the image corresponding to a component on the drawing in the temporary overlay drawing based on the overlap index, if the overlap index is equal to or greater than a certain value, the second determination unit 53 determines that the component on the image corresponds to the component on the drawing in the temporary overlay drawing. In other words, if the overlap index is less than a certain value, the second determination unit 53 stops processing, and therefore does not output to the calculation unit 60 and the third determination unit 70 tentative correspondences that are unlikely to be correct, thereby reducing the number of objects for comparison by the third determination unit 70.

[0112] Furthermore, if the second determination unit 53 continues processing even if the value of the overlap index is less than a certain value, the information processing device 300 according to this embodiment can output the data by excluding inappropriate correspondences between parts on the drawing and parts on the image.

[0113] Fourth Embodiment The configuration of an information processing device 400 according to a fourth embodiment of the present invention will be described. The information processing device 400 according to the fourth embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to third embodiments of the present invention. In the fourth embodiment of the present invention, only the differences from the first to third embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to third embodiments of the present invention will be omitted.

[0114] FIG. 10 is a block diagram showing a functional configuration of an information processing device 400 according to a fourth embodiment of the present invention. Each function of the information processing device 400 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 10, the information processing device 400 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 44, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The generation unit 44 of the information processing device 400 according to the present embodiment includes a second determination unit 92. The information processing device 400 according to the present embodiment also includes a control unit 80, similar to the information processing devices according to the first to third embodiments of the present invention, but this control unit 80 is not shown in FIG. 10. The control unit 80 of the information processing device 400 according to the present embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 44, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0115] The information processing device 400 according to this embodiment differs from the first to third embodiments of the present invention in that the generation unit 44 includes a second determination unit 92. When generating a temporary overlay view, the second determination unit 92 determines whether the drawing or image with the changed coordinate system is a drawing or image obtained by rotating the drawing or image before the coordinate system change within a range of 90 degrees to 270 degrees, or whether the drawing or image with the changed coordinate system is a flipped drawing or image before the coordinate system change. If the second determination unit 92 determines that the rotation is not correct, the generation unit 44 generates a temporary overlay view and outputs the temporary overlay view to the second determination unit 50.

[0116] The operation of the generation unit 44 including the second determination unit 92 and the fourth processing step will be described in detail with reference to FIG. 4 . First, the generation unit 44 aligns the coordinate system of the drawing with the coordinate system of the image based on the provisional correspondence determined by the first determination unit 30. In the fourth processing step, the generation unit 44 calculates a projective transformation matrix in the same manner as the generation unit 40, and transforms the image shown in FIG. 4B using the calculated projective transformation matrix. The transformed image shown in FIG. 4B is shown in FIG. 4C. Then, in the fourth processing step, the second determination unit 92 of the generation unit 44 determines whether the image with the changed coordinate system is the image before the coordinate system change rotated by 90 degrees or more and 270 degrees or less, or whether the image with the changed coordinate system is the image before the coordinate system change inverted. That is, the second judgment unit 92 of the generation unit 44 judges whether or not Figure 4(c) showing the converted image is obtained by rotating Figure 4(b) showing the image before conversion within a range of 90 degrees or more and 270 degrees or less, or whether or not Figure 4(c) showing the converted image is obtained by inverting Figure 4(b) showing the image before conversion.

[0117] Here, the generation unit 44 converted the coordinate system of the image into the coordinate system of the drawing, but the generation unit 44 of the information processing device 400 according to this embodiment may convert the coordinate system of the drawing into the coordinate system of the image, similar to the generation unit 40 of the information processing device according to the first to third embodiments of the present invention. When the generation unit 44 converts the coordinate system of the drawing into the coordinate system of the image, the second determination unit 92 determines whether the drawing whose coordinate system has been changed is obtained by rotating the drawing before the coordinate system change within a range of 90 degrees to 270 degrees, or whether the drawing whose coordinate system has been changed is obtained by flipping the drawing before the coordinate system change.

[0118] Next, in the fourth processing step, if the second determination unit 92 determines that the image has been rotated within a range of 90 degrees to 270 degrees or inverted, the generation unit 44 generates a provisional superimposed drawing by superimposing the drawing and the image with the changed coordinate system. That is, if the second determination unit 92 determines that the image has been rotated within a range of 90 degrees to 270 degrees or inverted, the generation unit 44 does not generate a provisional superimposed drawing. Referring to FIG. 4 , if the second determination unit 92 determines that the image has been rotated within a range of 90 degrees to 270 degrees or inverted, the generation unit 44 superimposes the drawing shown in FIG. 4( a) with the drawing shown in FIG. 4( c) to generate the drawing shown in FIG. 4( d). On the other hand, if the second determination unit 92 determines that the image has been rotated within a range of 90 degrees to 270 degrees or inverted, the generation unit 44 does not generate the drawing shown in FIG. 4( d).

[0119] When an image with a changed coordinate system is obtained by rotating the image before changing the coordinate system within a range of 90 degrees to 270 degrees, this means that the top and bottom of the image are reversed before and after changing the coordinate system. Also, inversion includes both left-right and up-down inversion.

[0120] If the second determination unit 92 determines that the image has been rotated or flipped within a range of 90 degrees to 270 degrees, the generation unit 44 generates a temporary superimposed image and outputs the generated temporary superimposed image to the second determination unit 50. If the second determination unit 92 determines that the image has been rotated or flipped within a range of 90 degrees to 270 degrees, the generation unit 44 does not generate a temporary superimposed image and therefore stops processing. Since the generation unit 44 stops processing, it does not output anything to the second determination unit 50. If the generation unit 44 stops processing, i.e., if the fourth processing step is stopped, the control unit 80 skips the fifth processing step S5 and the sixth processing step S6 and performs determination step J1.

[0121] As described above, the information processing device 400 according to the present embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing. This is because, when the second determination unit 92 of the generation unit 44 determines that the drawing is not rotated or flipped within a range of 90 degrees to 270 degrees, the generation unit 44 stops processing. Therefore, the generation unit 44 does not output a provisional superimposed view based on a provisional correspondence that is unlikely to be a correct correspondence to the second determination unit 50, the calculation unit 60, and the third determination unit 70, thereby reducing the number of objects for comparison by the third determination unit 70.

[0122] Although the angle for determination is set to 90 degrees or more and 270 degrees or less in the above description, this angle is merely an example, and the determination may be made at a different angle, for example, 100 degrees or more and 260 degrees or less.

[0123] Fifth Embodiment The configuration of an information processing device 500 according to a fifth embodiment of the present invention will be described. The information processing device 500 according to the fifth embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to fourth embodiments of the present invention. In the fifth embodiment of the present invention, only the differences from the first to fourth embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to fourth embodiments of the present invention will be omitted.

[0124] FIG. 11 is a block diagram showing a functional configuration of an information processing device 500 according to a fifth embodiment of the present invention. Each function of the information processing device 500 according to this embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 11, the information processing device 500 according to this embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 35, a generation unit 40, a second determination unit 50, a calculation unit 60, and a third determination unit 70. Like the information processing devices according to the first to fourth embodiments of the present invention, the information processing device 500 according to this embodiment also includes a control unit 80, which is not shown in FIG. 11. The control unit 80 of the information processing device 500 according to this embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 35, the generation unit 40, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0125] The information processing device 500 according to this embodiment differs from the first to fourth embodiments of the present invention in the operation of the first determination unit 35 and the third processing step. When selecting components on the drawing and components on the image, the first determination unit 35 of the information processing device 500 according to this embodiment randomly selects components on the drawing and components on the image, and determines an arbitrary number of tentative correspondences between the components on the drawing and the components on the image.

[0126] In the first to fourth embodiments of the present invention, the provisional correspondence between the components on the drawing and the components on the image is determined for all combinations of the components on the drawing and the components on the image. In contrast, in the present embodiment, the provisional correspondence between the components on the drawing and the components on the image is not determined for all combinations of the components on the drawing and the components on the image, but rather, when selecting the components on the drawing and the components on the image, the components are selected randomly, and an arbitrary number of provisional correspondences between the components on the drawing and the components on the image are determined. In other words, the information processing device 500 according to the present embodiment employs a RANSAC (Random Sample Consensus) method when the first determination unit 35 determines the provisional correspondence between the components on the drawing and the components on the image.

[0127] The arbitrary number of provisional correspondences between parts on the drawing and parts on the image determined by the first determination unit 35 is determined, for example, as follows: When four parts are selected from nine parts on the drawing and four parts are selected from nine parts on the image, the total number of provisional correspondences is: ９ C ４ × ９ P ４ There are 381,024 possible correspondences. Here, the probability that a randomly selected provisional correspondence is correct is 0.033%. To find a correct correspondence with a 99% probability, 13,923 trials are sufficient. Therefore, if there are nine components on the drawing and nine components on the image, the first determination unit 35 of the information processing device 500 according to this embodiment selects components on the drawing and nine components on the image at random, and determines 13,923 pairs of provisional correspondences between components on the drawing and components on the image.

[0128] As described above, the information processing device 500 according to this embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing. This is because the first determination unit 35 does not determine tentative correspondences between components on the drawing and components on the image for all combinations of components on the drawing and components on the image, but randomly selects components on the drawing and components on the image, and determines an arbitrary number of tentative correspondences between components on the drawing and components on the image. In other words, this is because the information processing device 500 according to this embodiment employs the RANSAC method when the first determination unit 35 determines tentative correspondences between components on the drawing and components on the image.

[0129] Sixth Embodiment The configuration of an information processing device 600 according to a sixth embodiment of the present invention will be described. The information processing device 600 according to the sixth embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to fifth embodiments of the present invention. In the sixth embodiment of the present invention, only the differences from the first to fifth embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to fifth embodiments of the present invention will be omitted.

[0130] FIG. 12 is a block diagram showing a functional configuration of an information processing device 600 according to a sixth embodiment of the present invention. Each function of the information processing device 600 according to the sixth embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 12, the information processing device 600 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a third determination unit 93, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The information processing device 600 according to the present embodiment also includes a control unit 80, similar to the information processing devices according to the first to fifth embodiments of the present invention, but this control unit 80 is not shown in FIG. 12. The control unit 80 of the information processing device 600 according to the present embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the third determination unit 93, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0131] The information processing device 600 according to this embodiment differs from the first to fifth embodiments of the present invention in that it includes a third determination unit 93. The third determination unit 93 of the information processing device 600 according to this embodiment determines whether the number of components on the drawing included in the first information matches the number of components on the screen included in the second information. Alternatively, it determines whether the difference between the number of components on the drawing and the number of components on the image is equal to or greater than a certain value. For example, it determines whether the difference is three or more.

[0132] If the third determination unit 93 determines that the information is not the same, the information processing device 600 according to the present embodiment stops the processing, and the third determination unit 93 does not output anything to the first determination unit 30. If the third determination unit 93 determines that the information is the same, the third determination unit 93 outputs the first information and the second information to the first determination unit 30.

[0133] As described above, the information processing device 600 according to the present embodiment can prevent the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70 from performing processing based on erroneous first information or second information. This is because the information processing device 600 according to the present embodiment stops processing when the third determination unit 93 determines that the first information or second information is incorrect. Note that the number of components on the drawing included in the first information and the number of components on the screen included in the second information may not match for the following reasons: the objects indicated by the drawing and the image may be different, or a false detection may have occurred in the object detection results that formed the basis of the information on the components on the image included in the second information.

[0134] Seventh Embodiment The configuration of an information processing device 700 according to a seventh embodiment of the present invention will be described. The information processing device 700 according to the seventh embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to sixth embodiments of the present invention. In the seventh embodiment of the present invention, only the differences from the first to sixth embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to sixth embodiments of the present invention will be omitted.

[0135] Fig. 13 is a block diagram showing a functional configuration of an information processing device 700 according to the seventh embodiment of the present invention. Each function of the information processing device 700 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in Fig. 1. As shown in Fig. 13, the information processing device 700 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a third acquisition unit 85, a fourth determination unit 94, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The information processing device 700 according to the present embodiment also includes a control unit 80, as with the information processing devices according to the first to sixth embodiments of the present invention, but this control unit is not shown in Fig. 13. The control unit 80 of the information processing device 700 in this embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the third acquisition unit 85, the fourth judgment unit 94, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 60 and the third determination unit 70.

[0136] The information processing device 700 according to this embodiment differs from the information processing devices according to the first to sixth embodiments of the present invention in that it includes a third acquisition unit 85 and a fourth determination unit 94. The third acquisition unit 85 acquires information on the location where an image was captured. Specifically, the third acquisition unit 85 acquires the location of the image capturing device when the image indicated by the second information was captured by using a positioning method such as a Global Navigation Satellite System (GNSS) represented by a Global Positioning System (GPS), an Inertial Measurement Unit (IMU), an Inertial Navigation System (INS), an Ultra Wide Band (UWB), or another method. The fourth determination unit 94 determines whether the distance between the position of the object included in the first information and the position acquired by the third acquisition unit 85 is equal to or greater than a certain value.

[0137] Furthermore, the information processing method using the information processing device 700 according to this embodiment differs from the information processing methods using the information processing devices according to the first to sixth embodiments of the present invention in that it includes an auxiliary step before the third processing step S3. The auxiliary step is a process of acquiring information on the position where the image was captured, and determining whether the distance between the position of the object included in the first information and the position acquired by the third acquisition unit 85 is equal to or greater than a certain value.

[0138] If the fourth determination unit 94 determines that the distance is equal to or greater than the certain value, the information processing device 600 according to the present embodiment stops processing, and the fourth determination unit 94 does not output anything to the first determination unit 30. If the fourth determination unit 94 determines that the distance is not the certain value, the fourth determination unit 94 outputs the first information and the second information to the first determination unit 30. Here, the certain value is, for example, 5 meters.

[0139] As described above, the information processing device 700 according to this embodiment can prevent the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70 from performing processing based on erroneous first information or second information. This is because the information processing device 700 according to this embodiment stops processing when the fourth determination unit 94 determines that the distance is equal to or greater than a certain value. If the distance between the position of the object included in the first information and the position acquired by the third acquisition unit 85 is equal to or greater than a certain value, it is highly likely that the object shown in the drawing is different from the object shown in the image.

[0140] In this embodiment, the first information includes information about the position of the object.

[0141] Eighth Embodiment The configuration of an information processing device 800 according to an eighth embodiment of the present invention will be described. The information processing device 800 according to the eighth embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to seventh embodiments of the present invention. In the eighth embodiment of the present invention, only the differences from the first to seventh embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to seventh embodiments of the present invention will be omitted.

[0142] FIG. 14 is a block diagram showing a functional configuration of an information processing device 800 according to an eighth embodiment of the present invention. Each function of the information processing device 800 according to this embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 14, the information processing device 800 according to this embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 68, and a third determination unit 70. The calculation unit 68 of the information processing device 800 according to this embodiment includes a fifth determination unit 95. The information processing device 800 according to this embodiment also includes a control unit 80, similar to the information processing devices according to the first to seventh embodiments of the present invention, but this control unit 80 is not shown in FIG. 14. The control unit 80 of the information processing device 800 according to this embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 68, and the third determination unit 70.

[0143] The information processing device 800 according to the present embodiment differs from the first to seventh embodiments of the present invention in that the calculation unit 68 includes a fifth determination unit 95. The fifth determination unit 95 determines whether the calculated overlap index sum is equal to or greater than a certain value. If the fifth determination unit determines that the overlap index sum is equal to or greater than a certain value, the third determination unit 70 determines a correspondence between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20 based on the overlap index sum determined by the fifth determination unit to be equal to or greater than the certain value.

[0144] The operation of the calculation unit 68 including the fifth determination unit 95, as well as the sixth and seventh processing steps of the information processing method using the information processing device 800 according to this embodiment, will be described in detail. First, in the sixth processing step, the calculation unit 68 calculates the total overlap index of the temporary overlay drawing by summing the overlap indexes used when the second determination unit 50 determined the components on the image corresponding to the components on the drawing in the temporary overlay drawing generated by the generation unit 40. Here, the overlap index is an IoU value. Next, in the sixth processing step, the fifth determination unit 95 included in the calculation unit 68 determines whether the calculated total overlap index is equal to or greater than a certain value.

[0145] In the sixth processing step, if the fifth determination unit 95 included in the calculation unit 68 determines that the overlap index total is equal to or greater than the certain value, the calculation unit 68 outputs the tentative correspondence used when the tentative overlay image from which the overlap index total that was equal to or greater than the certain value was generated, as well as the first information and the second information, to the third determination unit 70. In the sixth processing step, if the fifth determination unit 95 included in the calculation unit 68 determines that the overlap index total is equal to or greater than the certain value, the control unit 80 instructs the control unit 80 to skip determination step J1 and perform a seventh processing step.

[0146] The third determination unit 70 determines that the provisional correspondence relationship used when the provisional overlay drawing, for which the total overlap index was determined to be equal to or greater than a certain value, was generated is the correct correspondence relationship between the components on the drawing and the components on the image in this embodiment, based on the total overlap index determined to be equal to or greater than a certain value by the fifth determination unit 95. In other words, the third determination unit 70 determines that the correspondence relationship between the components on the drawing and the components on the image in the provisional correspondence relationship used when the provisional overlay drawing, for which the total overlap index was determined to be equal to or greater than a certain value by the fifth determination unit 95, is the correspondence relationship between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20.

[0147] If the fifth determination unit 95 included in the calculation unit 68 determines "no" in the sixth processing step, the control unit 80 performs determination step J1, as in the first to seventh embodiments of the present invention. Here, the fixed value is, for example, the number of components on the drawing minus 1. If the number of components on the drawing is eight, the fixed value is seven.

[0148] As described above, the information processing device 800 according to this embodiment can shorten the time required for processing to overlay and display a drawing and an image having a coordinate system different from that of the drawing. This is because the third determination unit 70 determines the correspondence between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20 based on the overlap index total determined to be equal to or greater than a certain value by the fifth determination unit 95 included in the calculation unit 68. The provisional correspondence used when generating the provisional overlay drawing for which the overlap index total determined to be equal to or greater than a certain value by the fifth determination unit 95 is likely to be the correct correspondence. Therefore, the information processing device 800 according to this embodiment can terminate processing when the overlap index total of the provisional overlay drawing is determined to be based on the correct correspondence.

[0149] Within the scope of the present invention, the embodiments and modifications can be freely combined, and the embodiments can be modified or omitted as appropriate.

[0150] 1 Computer, 2 Storage device, 3 CPU, 4 Input device, 5 Output device, 6 Auxiliary storage device, 7 Network connection device, 8 Bus, 10 First acquisition unit, 20 Second acquisition unit, 30, 32, 35 First determination unit, 40, 44 Generation unit, 50, 53 Second determination unit, 60, 68 Calculation unit, 70 Third determination unit, 80 Control unit, 85 Third acquisition unit, 91 First judgment unit, 92 Second judgment unit, 93 Third judgment unit, 94 Fourth judgment unit, 95 Fifth judgment unit, 100, 200, 300, 400, 500, 600, 700, 800 Information processing device.

Claims

1. a first acquisition unit that acquires first information, the first information being information on a drawing of an object having a plurality of parts, the first information including information on the parts on the drawing; a second acquisition unit that acquires second information, the second information being information on an image of the object, the second information having a coordinate system different from that of the drawing and including information on a part on the image; a first determination unit that uses the first information and the second information to select a component on the drawing and a component on the image, and determines a tentative correspondence relationship between the component on the drawing and the component on the image; a generating unit that generates a provisional overlay drawing by matching a coordinate system of the drawing with a coordinate system of the image based on the provisional correspondence relationship and overlaying the drawing and the image; a second determination unit that uses the first information and the second information in the temporary overlay drawing to determine an overlap index representing an overlap between a part on the drawing and a part on the image, and determines a part on the image corresponding to the part on the drawing in the temporary overlay drawing based on the overlap index; a calculation unit that calculates a total overlap index of the provisional overlay drawing by summing up the overlap indexes when the second determination unit has determined a part on the image corresponding to a part on the drawing in the provisional overlay drawing; a third determination unit that determines a correspondence relationship between the components on the drawing acquired by the first acquisition unit and the components on the image acquired by the second acquisition unit based on the total overlap index; An information processing device comprising:

2. The overlap index uses an IoU (Intersection over Union) value. The information processing device according to claim 1 .

3. The first determination unit, a first determination unit that determines whether or not a type of a part on the image corresponding to a part on the drawing is the same as a type of a part on the drawing, or whether or not a type of a part on the drawing corresponding to a part on the image is the same as a type of a part on the image, in the tentative correspondence relationship; Equipped with If the first determination unit determines that the two images are the same, the tentative correspondence relationship is output to the generation unit.

3. The information processing device according to claim 1 or 2.

4. The second determination unit, When determining a part on the image corresponding to a part on the drawing in the provisional overlay drawing, if the overlap index is equal to or greater than a certain value, the part on the image is determined to be a part corresponding to the part on the drawing in the provisional overlay drawing.

3. The information processing device according to claim 1 or 2.

5. The generation unit is a second determination unit that, when generating the provisional overlaid drawing, determines whether the drawing or image whose coordinate system has been changed is obtained by rotating the drawing or image before the coordinate system has been changed within a range of 90 degrees to 270 degrees, or whether the drawing or image whose coordinate system has been changed is obtained by inverting the drawing or image before the coordinate system has been changed; Equipped with If the second determination unit determines that the determination is negative, the temporary superimposition diagram is generated, and the temporary superimposition diagram is output to the second determination unit.

3. The information processing device according to claim 1 or 2.

6. The first determination unit, When selecting the parts on the drawing and the parts on the image, the parts are selected randomly, and an arbitrary number of tentative correspondences between the parts on the drawing and the parts on the image are determined.

3. The information processing device according to claim 1 or 2.

7. a third determination unit that determines whether or not the number of parts on the drawing included in the first information matches the number of parts on the image included in the second information; Equipped with If the third determination unit determines that the detection is not successful, the process is terminated.

3. The information processing device according to claim 1 or 2.

8. A third acquisition unit that acquires information on a position where the image was captured; a fourth determination unit that determines whether or not a distance between a position of the object included in the first information and the position acquired by the third acquisition unit is equal to or greater than a certain value; Equipped with When the fourth determination unit determines that the value is equal to or greater than a certain value, the process is stopped.

3. The information processing device according to claim 1 or 2.

9. The calculation unit is a fifth determination unit that determines whether the calculated overlap index total is equal to or greater than a certain value; Equipped with The third determination unit, When the fifth determination unit determines that the overlap index total is equal to or greater than a certain value, a correspondence relationship between the parts on the drawing acquired by the first acquisition unit and the parts on the image acquired by the second acquisition unit is determined based on the overlap index total determined by the fifth determination unit to be equal to or greater than a certain value.

3. The information processing device according to claim 1 or 2.

10. a first processing step of obtaining first information of a drawing of an object having a plurality of parts, the first information including information about the parts on the drawing; a second processing step of obtaining second information of the image of the object, the second information having a coordinate system different from the drawing and including information about parts on the image; a third processing step of selecting a part on the drawing and a part on the image using the first information and the second information, and determining a tentative correspondence between the part on the drawing and the part on the image; a fourth processing step of superimposing the drawing and the image by matching a coordinate system of the drawing and a coordinate system of the image based on the provisional correspondence relationship to generate a provisional superimposed drawing; a fifth processing step of obtaining an overlap index representing an overlap between a part on the drawing and a part on the image in the temporary overlay drawing by using the first information and the second information, and determining a part on the image corresponding to the part on the drawing in the temporary overlay drawing based on the overlap index; a sixth processing step of summing up the overlap indices when the parts on the image corresponding to the parts on the drawing are determined in the fifth processing step in the provisional overlay drawing to obtain a total overlap indices of the provisional overlay drawing; a seventh processing step of determining a correspondence relationship between the parts on the drawing acquired in the first processing step and the parts on the image acquired in the second processing step based on the total overlap index; An information processing method comprising:

11. a first processing step of obtaining first information of a drawing of an object having a plurality of parts, the first information including information about the parts on the drawing; a second processing step of obtaining second information of the image of the object, the second information having a coordinate system different from the drawing and including information about parts on the image; a third processing step of selecting a part on the drawing and a part on the image using the first information and the second information, and determining a tentative correspondence between the part on the drawing and the part on the image; a fourth processing step of superimposing the drawing and the image by matching a coordinate system of the drawing and a coordinate system of the image based on the provisional correspondence relationship to generate a provisional superimposed drawing; a fifth processing step of obtaining an overlap index representing an overlap between a part on the drawing and a part on the image in the temporary overlay drawing by using the first information and the second information, and determining a part on the image corresponding to the part on the drawing in the temporary overlay drawing based on the overlap index; a sixth processing step of summing up the overlap indices when the parts on the image corresponding to the parts on the drawing are determined in the fifth processing step in the provisional overlay drawing to obtain a total overlap indices of the provisional overlay drawing; a seventh processing step of determining a correspondence relationship between the parts on the drawing acquired in the first processing step and the parts on the image acquired in the second processing step based on the total overlap index; An information processing program that causes a computer to execute the above.