Information processing device, information processing method, and information processing program

Abstract

The present invention executes a process for displaying a drawing and an image differing in coordinate system from the drawing one on top of another, even when the drawing is of a general type.　The present invention generates a temporary superimposed diagram using first information that pertains to a drawing of an object having a plurality of components and that is related to components on the drawing, and second information that pertains to an image of the object and that is different in coordinate system from the drawing and includes information related to components on the image. The present invention comprises: a second determination unit that calculates an overlap index that represents the overlapping of components on the drawing and components on the image and determines components on the image that correspond to components on the drawing in the temporary superimposed diagram; a calculation unit that adds up overlap indices at the time the second determination unit determined components on the image that corresponded to components on the drawing in the temporary superimposed diagram and calculates the aggregate overlap index of the temporary superimposed diagram; and a third determination unit that determines, on the basis of the aggregate overlap index, a corresponding relationship between components on the drawing that a first acquisition unit has acquired and components on the image that a second acquisition unit has acquired.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Some information processing devices perform processing for superimposing a drawing and an image having a coordinate system different from that of the drawing and displaying the superimposed image.

[0003] When three-dimensional design data, which is a drawing of a product, is superimposed on an image of the product, a virtual image is generated using the drawing of the product, and feature point pairs are searched for in which local features of the virtual image are similar to local features of the image of the product. Then, based on the feature point pairs, external parameters of the camera that captured the image are estimated, and the estimated external parameters are used to perform processing for superimposing and displaying the image of the product on the three-dimensional design data of the product (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-17096 Summary of the Invention [Problem to be solved by the invention]

[0005] Such information processing devices have a problem in that, when the drawing is general, they cannot perform processing to superimpose and display the drawing on an image having a coordinate system different from that of the drawing. In order to search for feature point pairs in which the local features of a virtual image generated using the drawing are similar to the local features of the image, the drawing must have detailed data that allows the generation of a virtual image having local features similar to the local features of the image. However, this is because general drawings do not have such detailed data.

[0006] The present invention has been made to solve the above-mentioned problems, and aims to provide an information processing device that can process and display a drawing and an image that has a different coordinate system from the drawing, even if the drawing is a general one. [Means for solving the problem]

[0007] The information processing device of the present invention includes a first acquisition unit that acquires first information, which is information on a drawing of an object having a plurality of parts, and includes information on the parts on the drawing; a second acquisition unit that acquires second information, which is information on an image of the object, and which has a coordinate system different from that of the drawing and includes information on the parts on the image; a first determination unit that uses the first information and the second information to select parts on the drawing and parts on the image and determine a tentative correspondence between the parts on the drawing and the parts on the image; and a generation unit that aligns the coordinate system of the drawing and the coordinate system of the image based on the tentative correspondence, thereby overlaying the drawing and the image and generating a tentative overlaid drawing. The system includes a second determination unit that uses the first information and the second information to determine overlap indices in the temporary overlay drawing that indicate overlap between parts on the drawing and parts on the image, and determines parts on the image that correspond to parts on the drawing in the temporary overlay drawing based on the overlap indices; a calculation unit that calculates a total overlap indices for the temporary overlay drawing by adding up the overlap indices when the second determination unit determines parts on the image that correspond to parts on the drawing in the temporary overlay drawing; and a third determination unit that determines a correspondence relationship between parts on the drawing acquired by the first acquisition unit and parts on the image acquired by the second acquisition unit based on the total overlap indices. [Effects of the Invention]

[0008] According to the information processing device of the present invention, even if the drawing is a general one, it is possible to perform processing for overlaying and displaying the drawing and an image having a different coordinate system from the drawing. This is because a temporary overlay drawing is generated based on a temporary correspondence determined using first information and second information, a total overlap index of the temporary overlay drawing is calculated, and a correspondence between components on the drawing and components on the image is determined based on the total overlap index. Unlike the prior art, local features are not used to determine the correspondence, so there is no need for feature information to be included in the drawing, and drawings created using general CAD, etc., can be used. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a hardware configuration of a computer that can constitute an information processing device according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing a functional configuration of an information processing device according to a first embodiment of the present invention; [Figure 3] 3 is a flowchart showing an information processing method using the information processing device according to the first embodiment of the present invention. [Figure 4] A figure explaining the third processing step and the fourth processing step of the information processing method using the information processing device of the first embodiment of the present invention, showing an example of the processing in the third processing step and the fourth processing step. [Figure 5] A figure explaining the third processing step and the fourth processing step of the information processing method using the information processing device of embodiment 1 of the present invention, showing another example of the processing in the third processing step and the fourth processing step. [Figure 6] FIG. 10 is a diagram illustrating the fifth processing step of the information processing method using the information processing device according to the first embodiment of the present invention, and is a diagram illustrating how to determine a component on an image that corresponds to a component on a drawing in a temporary overlay drawing. [Figure 7]FIG. 10 is a diagram illustrating a fifth processing step of the information processing method using the information processing device according to the first embodiment of the present invention, illustrating how to obtain an overlap index. [Figure 8] FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a second embodiment of the present invention. [Figure 9] FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a third embodiment of the present invention. [Figure 10] FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a fourth embodiment of the present invention. [Figure 11] FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a fifth embodiment of the present invention. [Figure 12] FIG. 13 is a block diagram showing a functional configuration of an information processing device according to a sixth embodiment of the present invention. [Figure 13] FIG. 13 is a block diagram showing a functional configuration of an information processing device according to a seventh embodiment of the present invention. [Figure 14] FIG. 13 is a block diagram showing a functional configuration of an information processing device according to an eighth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiment 1 First, the configuration of an information processing device 100 according to a first embodiment of the present invention will be described. The information processing device 100 according to the first embodiment of the present invention can be configured using a computer 1. FIG. 1 is a block diagram showing the hardware configuration of the computer 1 that can configure the information processing device 100 according to the first embodiment of the present invention. As shown in FIG. 1, the computer 1 includes a storage device 2, a CPU (Central Processing Unit) 3, an input device 4, an output device 5, an auxiliary storage device 6, a network connection device 7, and a bus 8. The bus 8 connects the storage device 2, the CPU 3, the input device 4, the output device 5, the auxiliary storage device 6, and the network connection device 7.

[0011] The storage device 2 can be configured using semiconductor memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), or a flash memory. The storage device 2 stores various programs and data, including an information processing program used by the CPU 3. The CPU 3 is an example of a processor, and executes various processes described below.

[0012] The input device 4 can be configured using, for example, a pointing device such as a mouse, a keyboard, etc. The input device 4 is used when a user inputs instructions, information, etc. to the computer 1. Here, the user includes not only a person who actually operates the computer 1 but also a person who operates the computer 1 remotely. The output device 5 can be configured using, for example, a display device such as a liquid crystal display. The output device 5 outputs the content being processed by the information processing device 100 according to this embodiment, the processing results, messages to the user, etc. The input device 4 and the output device 5 may be configured using, for example, a touch panel having the functions of both a keyboard and a display device. The input device 4 may also be a combination of a keyboard and a pointing device.

[0013] The auxiliary storage device 6 can be configured using semiconductor memory such as a magnetic disk device, an optical disk device, a magneto-optical disk device, or a magnetic tape device. The auxiliary storage device 6 stores various programs and data, including information processing programs used by the CPU 3. The programs and data stored in the auxiliary storage device 6 can also be loaded into the storage device 2 by the user for use.

[0014] The network connection device 7 is an example of a communication interface that is connected to a communication network (not shown) such as a LAN (Local Area Network) or a WAN (Wide Area Network) and converts data associated with communication. For example, the computer 1 can receive various programs and data from an external device connectable to the communication network via the network connection device 7 and load them into the storage device 2 for use.

[0015] Note that the computer 1 that can constitute the information processing device 100 according to this embodiment may have the hardware configuration shown in FIG. 1 modified as appropriate by omitting or adding some devices depending on the conditions of use of the computer 1. For example, if the computer 1 does not need to communicate with external devices, the network connection device 7 may be omitted. Additionally, if the computer 1 is a mobile terminal device with a communication function, the input device 4 may include a microphone, which is an example of a communication device, and the output device 5 may include a speaker, which is an example of a communication device. The input device 4 may also include an imaging device such as a camera.

[0016] Fig. 2 is a block diagram showing a functional configuration of the information processing device 100 according to the first embodiment of the present invention. Each function of the information processing device 100 according to the first embodiment of the present invention is realized by the CPU 3 of the computer 1 shown in Fig. 1. The information processing device 100 according to the first embodiment of the present invention performs processing for superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing.

[0017] The drawings in this embodiment are two-dimensional design data of an object having multiple parts, and are drawn using CAD (Computer Aided Design). The coordinate system of the drawings in this embodiment is an XY Cartesian coordinate system with the horizontal axis as the X axis and the vertical axis as the Y axis, where, for example, mm is used as the unit of measurement and one measurement can represent a length in the SI unit system. Any unit other than mm can be used as long as it can represent the physical length of an actual object, such as meters or inches. The images in this embodiment are images of an object having multiple parts captured using an imaging device such as a camera. The coordinate system of the image in this embodiment is an XY Cartesian coordinate system with the horizontal axis as the X axis and the vertical axis as the Y axis, where px is used as the unit of measurement and one measurement cannot represent a physical length.

[0018] Unlike the coordinate system of a drawing, in which one division is expressed as a length, one division in the coordinate system of an image cannot be expressed as a length, and therefore the units used in the coordinate system of the image cannot be directly converted into the units used in the coordinate system of the drawing. Of course, the units used in the coordinate system of the drawing cannot be directly converted into the units used in the coordinate system of the image, either. In this embodiment, the coordinate system of a drawing and the coordinate system of an image are different, meaning that the units of one division are different, and it is not possible to directly convert the units used in the coordinate system of the drawing into the units used in the coordinate system of the image, or the units used in the coordinate system of the image into the units used in the coordinate system of the drawing.

[0019] Here, the drawings are described as two-dimensional design data created using CAD, but the drawings in this embodiment are not limited to this. Furthermore, the images are described as images captured using an imaging device such as a camera, but the images in this embodiment are not limited to this. In this embodiment, the drawings and images only need to have different coordinate systems. That is, the units of one scale are different, and it is not necessary to directly convert the units used in the coordinate system of the drawing to the units used in the coordinate system of the image, or vice versa.

[0020] In this embodiment, the drawings and images are of objects having multiple parts. Examples of parts include meters, lamps, switches, and nameplates. Examples of objects having multiple parts include measuring instruments with multiple meters, and distribution boards with multiple lamps and switches. However, the parts and objects according to this embodiment are not limited to these. In this embodiment, a part is something that has an area on a drawing and an image, and does not necessarily represent only points or lines, but may be anything that can be represented by a closed curve on a drawing and an image. In other words, in this embodiment, parts on a drawing and parts on an image have an outline. An outline is a line that forms the outer shape of an object, and the outline of a part is a line that shows the outer shape of the part.

[0021] As shown in FIG. 2, the information processing device 100 according to the first embodiment of the present invention includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 60, a third determination unit 70, and a control unit 80. The first acquisition unit 10 acquires first information, which is information about a drawing of an object having a plurality of parts and includes information about the parts on the drawing. The second acquisition unit 20 acquires second information, which is information about an image of the object, the coordinate system of which is different from that of the drawing and includes information about the parts on the image. The first determination unit 30 uses the first information and the second information to select parts on the drawing and parts on the image and determine a tentative correspondence between the parts on the drawing and the parts on the image. The generation unit 40 overlays the drawing and the image by aligning the coordinate system of the drawing and the coordinate system of the image based on the tentative correspondence, thereby generating a tentative overlaid drawing.

[0022] The second determination unit 50 uses the first information and the second information to calculate overlap indices representing overlaps between components on the drawing and components on the image in the temporary overlay drawing, and determines components on the image corresponding to the components on the drawing in the temporary overlay drawing based on the overlap indices. The calculation unit 60 calculates a total overlap indices for the temporary overlay drawing by summing up the overlap indices used when the second determination unit 50 determined components on the image corresponding to the components on the drawing in the temporary overlay drawing. The third determination unit 70 determines a correspondence between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20 based on the total overlap indices. The control unit 80 controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0023] The first information, which is information about the drawing, includes information about the parts on the drawing and information about the coordinate system of the drawing. The first information may also include information about the purpose and operation of the object, information about the outline of the object, etc. The information about the parts on the drawing includes information about the number of parts, the type of part, the coordinates of the center of the part, and the outline of the part. The type of part refers to a meter, lamp, switch, nameplate, etc. The information about the outline of the part includes the length of the outline, the area enclosed by the outline, and the coordinates of the corners if the outline forms a corner. In this embodiment, the information about the parts on the drawing does not need to include all of the information about the number of parts, the type of part, the coordinates of the center of the part, the length of the outline of the part, the area enclosed by the outline of the part, and the coordinates of the corners if the outline forms a corner. It is sufficient to include enough information to perform the operation of the information processing device 100 according to this embodiment and the information processing method using the information processing device 100 according to this embodiment, which will be described later. Furthermore, the outline of the part may have interruptions, and it is sufficient to perform the operation of the information processing device 100 according to this embodiment and the information processing method using the information processing device 100 according to this embodiment, which will be described later, by filling in the interruptions using a well-known method, etc.

[0024] The second information, which is information about the image, includes information about parts on the image and information about the coordinate system of the image. The second information may include information such as the purpose or operation of the object, information about the contour of the object, etc. The information about parts on the image includes information about the number of parts, the type of part, the center coordinates of the parts, and the contour of the parts. In this embodiment, like the information about parts on the drawing, the information about parts on the image does not need to include all of the information about parts on the image, such as the number of parts, the type of part, the center coordinates of the parts, the length of the contour of the part, the area enclosed by the contour of the part, and, if the contour of the part forms a corner, the coordinates of the corner. It is sufficient if the information includes enough information to perform the operation of information processing device 100 according to this embodiment and an information processing method using information processing device 100 according to this embodiment, which will be described later.

[0025] Each function of the information processing device 100 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in Fig. 1 , and therefore the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, the third determination unit 70, and the control unit 80 shown in Fig. 2 are functions of the CPU 3. Here, the first information and the second information acquired by the first acquisition unit 10 and the second acquisition unit 20 are loaded into the storage device 2. Therefore, the operations of the first acquisition unit 10 and the second acquisition unit 20 of the information processing device 100 according to the present embodiment are realized by the CPU 3 acquiring the first information and the second information from the storage device 2.

[0026] Next, the operation of the information processing device 100 according to the first embodiment of the present invention will be described, and the configuration of the information processing device 100 according to the first embodiment of the present invention will be described in more detail. Fig. 3 is a flowchart showing an information processing method using the information processing device 100 according to the first embodiment of the present invention.

[0027] As shown in FIG. 3, the information processing method using the information processing device 100 according to the first embodiment of the present invention includes a first processing step S1, a second processing step S2, a third processing step S3, a fourth processing step S4, a fifth processing step S5, a sixth processing step S6, a determination step J1, and a seventh processing step S7.

[0028] First, the first processing step S1 of the information processing method using the information processing device 100 according to the present embodiment is a step of acquiring first information, which is information on a drawing of an object having a plurality of parts and includes information on the parts on the drawing. In the first processing step S1, the CPU 3 having the function of the first acquisition unit 10 of the information processing device 100 according to the present embodiment acquires the first information, which is information on a drawing of an object having a plurality of parts and includes information on the parts on the drawing, from the storage device 2.

[0029] The second processing step S2 of the information processing method using the information processing device 100 according to this embodiment is a step of acquiring second information, which is information on an image of an object, the coordinate system of which is different from that of the drawing, and which includes information on components on the image. In the second processing step S2, the CPU 3 having the function of the second acquisition unit 20 of the information processing device 100 according to this embodiment acquires from the storage device 2 the second information, which is information on an image of an object, the coordinate system of which is different from that of the drawing, and which includes information on components on the image.

[0030] In this embodiment, the information on the parts in the image included in the second information is configured as a result of detecting parts included in the image using an object detection technique from an image of a captured object. Object detection techniques use machine learning or deep learning, for example, detection using YOLO (You Only Look Once). Other examples include detection using SVM (Support Vector Machine) and HOG (Histogram of Oriented Gradients) features. From the result of detecting parts included in the image using the object detection technique, information on the number of parts, type of parts, center coordinates of parts, and outlines of parts is obtained, and is loaded into storage device 2 as information on the parts in the image.

[0031] Here, it is assumed that the process of detecting parts included in an image using an object detection technique is performed by a device other than the computer 1 that can configure the information processing device 100 according to this embodiment, and information obtained from the result of the process is loaded into the storage device 2. However, this embodiment is not limited to this. For example, the process of detecting parts included in an image using an object detection technique may be performed by the CPU 3 of the computer 1 that can configure the information processing device 100 according to this embodiment, and the information obtained from the result of the process may be loaded into the storage device 2. In this embodiment, it is sufficient that the second acquisition unit 20 acquires the information obtained from the result of detecting parts included in an image using the object detection technique.

[0032] The first acquisition unit 10 and the second acquisition unit 20 of the information processing device 100 according to this embodiment output the acquired first information and second information to the first determination unit 30.

[0033] The third processing step S3 of the information processing method using the information processing device 100 according to this embodiment is a step of selecting components on the drawing and components on the image using the first information and the second information, and determining a tentative correspondence between the components on the drawing and the components on the image. In the third processing step S3, the CPU 3 having the function of the first determination unit 30 of the information processing device 100 according to this embodiment uses the first information and the second information to select components on the drawing and components on the image, and determines a tentative correspondence between the components on the drawing and the components on the image.

[0034] The operation of the first determination unit 30 and the third processing step S3 will be described in more detail with reference to Fig. 4 and Fig. 5. Fig. 4 and Fig. 5 are diagrams for explaining the third processing step S3 and the fourth processing step S4 of the information processing method using the information processing device 100 according to the first embodiment of the present invention, and show different examples of the processing in the third processing step S3 and the fourth processing step S4, respectively.

[0035] 4(a) and 5(a) show diagrams representing drawings of an object having multiple parts, which are drawn based on the first information acquired by the first acquisition unit 10 in the first processing step S1. The letters X, Y, Z, and W in FIG. 4(a) are symbols used in the following description. Similarly, the letters S, T, U, and V in FIG. 5(a) are symbols used in the following description. FIGS. 4(b) and 5(b) show diagrams representing images of an object having multiple parts, which are drawn based on the second information acquired by the second acquisition unit 20 in the second processing step S2. The letters x, y, z, and w in FIG. 4(b) are symbols used in the following description. Similarly, the letters s, t, u, and v in FIG. 5(b) are symbols used in the following description. Note that, for the sake of clarity, the outlines of the objects are not drawn in FIGS. 4 and 5, but they may be drawn in this embodiment.

[0036] FIG. 4(a) and FIG. 4(b) have different coordinate systems. Similarly, FIG. 5(a) and FIG. 5(b) have different coordinate systems. FIG. 4(c) is a diagram showing an image in which the coordinate system of the image shown in FIG. 4(b) is converted into the coordinate system of the drawing shown in FIG. 4(a) and the coordinate system is aligned with the coordinate system of the drawing shown in FIG. 4(a). Aligning the coordinate system of the drawing and the coordinate system of the image means making the coordinate systems the same for the drawing and the image, meaning that the unit of one scale in the coordinate system of the drawing is the same as the unit of one scale in the coordinate system of the image. FIG. 4(d) is a diagram showing a temporary overlay diagram in which the drawing shown in FIG. 4(a) and the image in which the coordinate system has been transformed as shown in FIG. 4(c) are overlaid. In FIG. 4(d), the solid line indicates the drawing, and the dotted line indicates the image in which the coordinate system has been transformed.

[0037] Similarly, Fig. 5(c) is a diagram showing an image in which the coordinate system of the image shown in Fig. 5(b) is converted into the coordinate system of the drawing shown in Fig. 5(a) and the coordinate system is aligned with that of the drawing shown in Fig. 5(a). Fig. 5(d) is a diagram showing a temporary superimposition diagram in which the drawing shown in Fig. 5(a) and the image in which the coordinate system has been converted shown in Fig. 5(c) are superimposed. In Fig. 5(d), the solid line indicates the drawing, and the dotted line indicates the image in which the coordinate system has been converted.

[0038] As shown in FIGS. 4(a) and 4(b) and 5(a) and 5(b), the drawings and images each show 17 parts. That is, the object shown in FIGS. 4(a) and 4(b) and 5(a) and 5(b) has 17 parts. The 17 parts are broken down into 8 meters, 1 large nameplate, and 8 small nameplates. The drawings and images shown in FIGS. 4 and 5 are examples created to explain the third processing step S3, and the drawings and images to which the processing performed by the information processing device 100 according to this embodiment is applied are not limited to the drawings and images shown in FIGS. 4 and 5.

[0039] First, a method for determining a tentative correspondence between components on a drawing and components on an image by the first determination unit 30 of the information processing device 100 according to this embodiment in the third processing step S3 will be described with reference to FIG. 4. The first determination unit 30 selects four components from the 17 components on the drawing and four components from the 17 components on the image. As an example, as shown in FIG. 4(a), the first determination unit 30 selects four components marked with the symbols X, Y, Z, and W from the 17 components on the drawing. Similarly, as an example, as shown in FIG. 4(b), the first determination unit 30 selects four components marked with the symbols x, y, z, and w from the 17 components on the image.

[0040] Then, the first determination unit 30 provisionally determines the parts on the image that correspond to the selected parts on the drawing. Here, it is provisionally determined that part X on the drawing corresponds to part x on the image, part Y on the drawing corresponds to part y on the image, part Z on the drawing corresponds to part z on the image, and part W on the drawing corresponds to part w on the image. This provisionally determined relationship between the parts on the drawing and the parts on the image is the provisional correspondence relationship in FIG. 4. That is, in the provisional correspondence relationship in FIG. 4, part X on the drawing corresponds to part x on the image, part Y on the drawing corresponds to part y on the image, part Z on the drawing corresponds to part z on the image, and part W on the drawing corresponds to part w on the image.

[0041] Note that a part on the drawing corresponds to a part on the image, meaning that the part on the drawing and the part on the image indicate the same part of the object. In other words, in the tentative correspondence in Figure 4, part X on the drawing and part x on the image indicate the same part of the object, meaning that they indicate the same part in the object with the same position, type, size, and shape. The same is true for the other three selected parts.

[0042] Next, a method for determining a tentative correspondence between components on a drawing and components on an image by the first determination unit 30 of the information processing device 100 according to this embodiment in the third processing step S3 will be described with reference to FIG. 5. First, the first determination unit 30 selects four components from the 17 components on the drawing and four components from the 17 components on the image. As an example, as shown in FIG. 5(a), the first determination unit 30 selects four components marked with the symbols S, T, U, and V from the 17 components on the drawing. Similarly, as an example, as shown in FIG. 5(b), the first determination unit 30 selects four components marked with the symbols s, t, u, and v from the 17 components on the image.

[0043] Then, the first determination unit 30 provisionally determines the parts on the image that correspond to the selected parts on the drawing. Here, it is provisionally determined that part S on the drawing corresponds to part s on the image, part T on the drawing corresponds to part t on the image, part U on the drawing corresponds to part u on the image, and part V on the drawing corresponds to part v on the image. This provisionally determined relationship between the parts on the drawing and the parts on the image is the provisional correspondence relationship in FIG. 5. That is, in the provisional correspondence relationship in FIG. 5, part S on the drawing corresponds to part s on the image, part T on the drawing corresponds to part t on the image, part U on the drawing corresponds to part u on the image, and part V on the drawing corresponds to part v on the image.

[0044] The first determining unit 30 of the information processing device 100 according to this embodiment outputs the determined tentative correspondence, the first information, and the second information to the generating unit 40.

[0045] A fourth processing step S4 of the information processing method using the information processing device 100 according to this embodiment is a step of matching the coordinate system of the drawing with the coordinate system of the image based on the temporary correspondence relationship, thereby overlaying the drawing and the image, and generating a temporary overlaid diagram. In the fourth processing step S4, the CPU 3 having the function of the generation unit 40 of the information processing device 100 according to this embodiment matches the coordinate system of the drawing with the coordinate system of the image based on the temporary correspondence relationship determined by the first determination unit 30, thereby overlaying the drawing and the image, and generating a temporary overlaid diagram.

[0046] The operation of the generation unit 40 and the fourth processing step S4 will be described in more detail with reference to FIGS. 4 and 5. First, with reference to FIG. 4, a method in which the generation unit 40 of the information processing device 100 according to this embodiment generates a provisional overlay drawing in the fourth processing step S4 will be described. The generation unit 40 calculates a projective transformation matrix that converts the coordinate system of the image into the coordinate system of the drawing based on the central coordinates of components X, Y, Z, and W on the drawing, the central coordinates of components x, y, z, and w on the image, and the provisional correspondence in FIG. 4. The generation unit 40 then uses the calculated projective transformation matrix to transform the image shown in FIG. 4(b) to match the coordinate system of the image shown in FIG. 4(b) with the coordinate system of the drawing shown in FIG. 4(a). FIG. 4(c) shows an image obtained by transforming the image shown in FIG. 4(b) using the calculated projective transformation matrix. The generation unit 40 then generates a provisional overlay drawing by overlaying the drawing shown in FIG. 4(a) with the transformed image shown in FIG. 4(c), as shown in FIG. 4(d).

[0047] Next, a method for generating a provisional overlay diagram by the generation unit 40 of the information processing device 100 according to this embodiment in the fourth processing step S4 will be described with reference to FIG. 5. The generation unit 40 calculates a projective transformation matrix for converting the coordinate system of the image into the coordinate system of the drawing based on the central coordinates of the components S, T, U, and V on the drawing, the central coordinates of the components s, t, u, and v on the image, and the provisional correspondence shown in FIG. 5. The generation unit 40 then converts the image shown in FIG. 5(b) using the calculated projective transformation matrix to match the coordinate system of the image shown in FIG. 5(b) with the coordinate system of the drawing shown in FIG. 5(a). FIG. 5(c) shows an image obtained by converting the image shown in FIG. 5(b) using the calculated projective transformation matrix. The generation unit 40 then generates a provisional overlay diagram by overlaying the drawing shown in FIG. 5(a) with the converted image shown in FIG. 5(c), as shown in FIG. 5(d).

[0048] In this embodiment, the tentative correspondences between the parts on the drawing and the parts on the image that the first determination unit 30 determines in the third processing step S3 are determined for all combinations of the parts on the drawing and the parts on the image. In this embodiment, the object has 17 parts. In addition, in order for the generation unit 40 to determine the projective transformation matrix in the fourth processing step S4, the first determination unit 30 selects 4 parts from the 17 parts on the drawing and 4 parts from the 17 parts on the image in the third processing step S3. Therefore, the total number of tentative correspondences that the first determination unit 30 determines in the third processing step S3 is 17 C4× 17 It's P4.

[0049] Here, when the first determination unit 30 determines the provisional correspondence in the third processing step S3, it selects four of the 17 components on the drawing and four of the 17 components on the image. However, this embodiment is not limited to this. For example, the first determination unit 30 may select three of the 17 components on the drawing and three of the 17 components on the image in the third processing step S3. If the first determination unit 30 selects three components from each of the components on the drawing and the components on the image, the generation unit 40 may calculate an affine transformation matrix instead of a projective transformation matrix in the fourth processing step S4 to align the coordinate system of the drawing with the coordinate system of the image. When the first determination unit 30 determines the provisional correspondence in the third processing step S3, the number of components selected from each of the components on the drawing and the components on the image may be any number necessary to calculate a transformation matrix that aligns the coordinate system of the drawing with the coordinate system of the image.

[0050] Furthermore, here, the generation unit 40 determines a projection transformation matrix that transforms the coordinate system of the image into the coordinate system of the drawing based on the central coordinates of the components X, Y, Z, and W on the drawing, the central coordinates of the components x, y, z, and w on the image, and the provisional correspondence relationship in Fig. 4, but this embodiment is not limited to this. As long as a projection transformation matrix can be determined, for example, the projection transformation matrix may be determined based on the coordinates of the corners of the components X, Y, Z, and W on the drawing, the coordinates of the corners of the components x, y, z, and w on the image, and the provisional correspondence relationship in Fig. 4. The same applies to the projection transformation matrix that transforms the coordinate system of the image into the coordinate system of the drawing based on the central coordinates of the components S, T, U, and V on the drawing, the central coordinates of the components s, t, u, and v on the image, and the provisional correspondence relationship in Fig. 5.

[0051] Although the generation unit 40 converts the coordinate system of the image into the coordinate system of the drawing, the generation unit 40 of the information processing device 100 according to this embodiment may convert the coordinate system of the drawing into the coordinate system of the image to generate a provisional overlay diagram. That is, referring to FIG. 4, a projection transformation matrix for converting the coordinate system of the drawing shown in FIG. 4(a) into the coordinate system of the image shown in FIG. 4(b) may be calculated based on a provisional correspondence between the central coordinates of components X, Y, Z, and W on the drawing and the central coordinates of components x, y, z, and w on the image, and the drawing shown in FIG. 4(a) may be transformed using the calculated projection transformation matrix. Then, the coordinate system of the drawing shown in FIG. 4(a) and the coordinate system of the image shown in FIG. 4(b) may be combined to generate a provisional overlay diagram.

[0052] The generation unit 40 of the information processing device 100 according to this embodiment outputs the generated temporary overlay image, the temporary correspondence determined by the first determination unit 30, the first information, and the second information to the second determination unit 50.

[0053] A fifth processing step S5 of the information processing method using information processing device 100 according to this embodiment is a step of using the first information and the second information to obtain overlapping indices indicating overlapping between components on the drawing and components on the image in the temporary overlay drawing, and determining, based on the overlapping indices, components on the image that correspond to components on the drawing in the temporary overlay drawing. In fifth processing step S5, CPU 3 having the function of second determination unit 50 of information processing device 100 according to this embodiment obtains overlapping indices indicating overlapping between components on the drawing and components on the image in the temporary overlay drawing generated by generation unit 40, and determines, based on the overlapping indices, components on the image that correspond to components on the drawing in the temporary overlay drawing.

[0054] The operation of the second determination unit 50 and the fifth processing step S5 will be described in more detail with reference to FIGS. 6 and 7. FIG. 6 is a diagram for explaining the fifth processing step S5 of the information processing method using the information processing device 100 according to the first embodiment of the present invention, and is a diagram for explaining how to determine components on an image corresponding to components on a drawing in a temporary overlay diagram. FIG. 6(a) shows an example of a temporary overlay diagram, in which solid lines indicate the drawing and dotted lines indicate an image obtained by converting the coordinate system. The letters L, M, N, O, and P in FIG. 6(a) are symbols used in the following description. Similarly, the letters l, m, n, o, and p in FIG. 6(b) are symbols used in the following description. FIG. 7 is a diagram for explaining the fifth processing step S5 of the information processing method using the information processing device 100 according to the first embodiment of the present invention, and is a diagram for explaining how to determine an overlap index.

[0055] First, the second determining unit 50 selects one component on the drawing in the temporary overlay drawing shown in Fig. 6(a), which is an example of a temporary overlay drawing. Here, it is assumed that component L on the drawing is selected, as shown in Fig. 6(b). Next, it calculates overlap indices between component L on the drawing and each of components l, m, n, o, and p on the image.

[0056] Using FIG. 7, we will explain how to obtain an overlap index that indicates the overlap between a component on a drawing and a component on an image. The overlap index is obtained for each component in a temporary overlay drawing. FIG. 7(a) is a diagram showing the overlap between a component on the drawing and a component on a transformed image for one component in the temporary overlay drawing. In FIG. 7(a), the solid line indicates the drawing, and the dotted line indicates the image with a transformed coordinate system. FIG. 7(b) is a diagram showing the overlapping surfaces (intersection) between the surfaces representing the component on the drawing and the surfaces representing the component on the transformed image for the component on the drawing and the component on the image shown in FIG. 7(a). FIG. 7(c) is a diagram showing the surface (union) formed by joining the surfaces of the component on the drawing and the surface of the component on the transformed image for the component on the drawing and the component on the image shown in FIG. 7(a).

[0057] FIG. 7(d) is a diagram showing the overlapping state of a certain component in another temporary overlay drawing between a component on the drawing and a component on the transformed image. In FIG. 7(d), as in FIG. 7(a), the solid lines indicate the drawing and the dotted lines indicate the image with the transformed coordinate system. FIG. 7(e) is a diagram showing the overlapping surfaces (intersection) between the surfaces representing the component on the drawing and the surfaces representing the component on the transformed image for the component on the drawing and the component on the image shown in FIG. 7(d). FIG. 7(f) is a diagram showing the union of the surfaces of the component on the drawing and the surface of the component on the transformed image for the component on the drawing and the component on the image shown in FIG. 7(d).

[0058] The overlap index for the part shown in Figure 7(a) is the area of ​​the face shown in Figure 7(b) divided by the area of ​​the face shown in Figure 7(c), and is called the Intersection over Union (IoU) value. Similarly, the overlap index for the part shown in Figure 7(d) is the area of ​​the face shown in Figure 7(e) divided by the area of ​​the face shown in Figure 7(f).

[0059] The diagram of the parts shown in Fig. 7(a) is a partial citation of the diagram shown in Fig. 4(d). The diagram of the parts shown in Fig. 7(d) is a partial citation of the diagram shown in Fig. 5(d). When the overlap index for the parts shown in Fig. 7(a) and the overlap index for the parts shown in Fig. 7(d) are actually calculated, the overlap index for the parts shown in Fig. 7(d) is larger than the overlap index for the parts shown in Fig. 7(a).

[0060] The overlap index between part L on the drawing shown in Fig. 6(b) and each of parts l, m, n, o, and p on the image is calculated using the method described above with reference to Fig. 7. The results are that the overlap index between part L on the drawing and part l on the image is 0.9, the overlap index between part L on the drawing and part m on the image is 0.0, the overlap index between part L on the drawing and part n on the image is 0.0, the overlap index between part L on the drawing and part o on the image is 0.0, and the overlap index between part L on the drawing and part p on the image is 0.0.

[0061] Then, based on this result, the second determination unit 50 determines the component on the image that corresponds to the component on the drawing in the temporary overlay drawing. When the overlap indexes between the component L on the drawing and each of the components l, m, n, o, and p on the image are compared, the overlap index between the component L on the drawing and the component l on the image has the largest overlap index. Therefore, in the fifth processing step S5, the second determination unit 50 of the information processing device 100 according to this embodiment determines that the component on the image that corresponds to the component L on the drawing in the temporary overlay drawing is the component l on the image.

[0062] Next, in the temporary overlay diagram shown in FIG. 6(a), which is an example of a temporary overlay diagram, the second determination unit 50 selects one of the components on the drawing for which a corresponding component on the image has not yet been determined. Here, as shown in FIG. 6(c), it is assumed that component M on the drawing is selected. Then, the second determination unit 50 calculates the overlap index between component M on the drawing and the component on the image. At this time, component l on the image, for which a corresponding component on the drawing has already been determined, is excluded. In other words, the overlap index between component M on the drawing and each of components m, n, o, and p on the image is calculated.

[0063] The subsequent steps are performed in the same manner as when determining the part on the image corresponding to part L on the drawing. That is, the overlap indexes of part M on the drawing and parts m, n, o, and p on the image are compared, and the part on the image with the largest overlap index is determined to be the part on the image corresponding to part M on the drawing. The same process is performed for parts O, N, and P on the drawing shown in FIG. 6(a), and the parts on the image corresponding to each of them are determined. Referring to FIG. 6(a), in fifth processing step S5, the second determination unit 50 determines that part L on the drawing corresponds to part l on the image, part M on the drawing corresponds to part m on the image, part N on the drawing corresponds to part n on the image, part O on the drawing corresponds to part o on the image, and part P on the drawing corresponds to part p on the image. As described above, in this embodiment, corresponding parts on the image are determined for all parts on the drawing in the temporary overlay drawing.

[0064] However, if the overlap index between the largest part on the image and the part on the drawing is 0.0, i.e., if there is no part on the image that overlaps with the part on the drawing, the part on the image that corresponds to the part on the drawing is left unknown and processing continues. Even if the number of parts on the image is fewer than the number of parts on the drawing, the corresponding parts on the image for some parts on the drawing will remain unknown, but processing continues. Note that in Figure 6, the outline of the object is shown for convenience so that the number of parts the object has can be seen.

[0065] Although the corresponding components on the image are determined for all components on the drawings in the temporary overlay diagram, this embodiment is not limited to this. It is sufficient that no contradictions occur when the sixth processing step S6, the determination step J1, and the seventh processing step S7 described later are performed. For example, the second determination unit 50 may first select one component on the image in the temporary overlay diagram shown in FIG. 6(a), which is an example of a temporary overlay diagram, and then determine the overlap index between the selected component on the image and the components on each drawing. In other words, the corresponding components on the drawings may be determined for all components on the image in the temporary overlay diagram.

[0066] The second determination unit 50 of the information processing device 100 according to this embodiment outputs the value of the overlap index when determining the part on the image corresponding to the part on the drawing, the tentative correspondence determined by the first determination unit 30, the first information, and the second information to the calculation unit 60. Here, the tentative correspondence determined by the first determination unit 30 is the tentative correspondence used when the tentative overlaid drawing was generated.

[0067] A sixth processing step S6 of the information processing method using the information processing device 100 according to this embodiment is a step of summing up the overlap indices when the components on the image corresponding to the components on the drawing are determined in the fifth processing step S5 in the temporary overlay drawing to find the total overlap indices of the temporary overlay drawing. In the sixth processing step S6, the CPU 3 having the function of the calculation unit 60 of the information processing device 100 according to this embodiment sums up the overlap indices when the second determination unit 50 determines the components on the image corresponding to the components on the drawing in the temporary overlay drawing generated by the generation unit 40.

[0068] The operation of the calculation unit 60 and the sixth processing step S6 will be described in further detail. Referring to FIG. 6, in the fifth processing step S5, the calculation unit 60 adds together the overlap index obtained when it is determined that the part on the image corresponding to part L on the drawing is part l on the image, the overlap index obtained when it is determined that the part on the image corresponding to part M on the drawing is part m on the image, the overlap index obtained when it is determined that the part on the image corresponding to part N on the drawing is part n on the image, the overlap index obtained when it is determined that the part on the image corresponding to part O on the drawing is part o on the image, and the overlap index obtained when it is determined that the part on the image corresponding to part P on the drawing is part p on the image. This sum is then used as the total overlap index value for the provisional overlay drawing.

[0069] Furthermore, when the total overlap index of the provisional overlay drawing shown in Figure 4(d) and the total overlap index of the provisional overlay drawing shown in Figure 5(d) are actually calculated, the overlap index for the part shown in Figure 7(d) is larger than the overlap index for the part shown in Figure 7(a). Taking this into consideration, the total overlap index of the provisional overlay drawing shown in Figure 5(d) will be larger than the total overlap index of the provisional overlay drawing shown in Figure 4(d).

[0070] The information processing method using the information processing device 100 according to the present embodiment includes a determination step J1, which is a process of determining whether or not the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all the tentative correspondence relationships output in the third processing step S3. In the determination step J1, the CPU 3 having the function of the control unit 80 of the information processing device 100 according to the present embodiment determines whether or not the operations of the generation unit 40, the second determination unit 50, and the calculation unit 60 have been performed for all the tentative correspondence relationships output by the first determination unit 30.

[0071] If the control unit 80 determines in judgment step J1 that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all of the tentative correspondences output in the third processing step S3, the control unit 80 instructs the third determination unit 70 to operate. If the control unit 80 determines in judgment step J1 that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have not been performed for all of the tentative correspondences output in the third processing step S3, the control unit 80 instructs the generation unit 40 to operate for the tentative correspondences for which no tentative overlay diagram has yet been generated. In other words, the fourth processing step S4 by the generation unit 40, the fifth processing step S5 by the second determination unit 50, and the sixth step S6 by the calculation unit 60 are repeatedly performed for all of the tentative correspondences output by the first determination unit 30 in the third processing step S3.

[0072] The calculation unit 60 calculates the overlap index totals for all the tentative correspondences output by the first determination unit 30. If the control unit 80 determines in determination step J1 that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all the tentative correspondences output in the third processing step S3, the calculation unit 60 outputs all the calculated overlap index totals, the tentative overlay diagrams from which the respective overlap index totals were calculated, the tentative correspondences used when the tentative overlay diagrams were generated, the first information, and the second information to the third determination unit 70. In other words, the third determination unit 70 of the information processing device 100 according to this embodiment receives from the calculation unit 60 the total overlap index values ​​calculated by the calculation unit 60 for all the provisional correspondences output by the first determination unit 30, the provisional overlay diagrams from which the respective total overlap index values ​​have been calculated, the provisional correspondences determined by the first determination unit 30 and used when the provisional overlay diagrams were generated by the generation unit 40, the first information, and the second information.

[0073] The seventh processing step S7 of the information processing method using the information processing device 100 according to this embodiment is a step of determining a correspondence relationship between the components on the drawing acquired in the first processing step S1 and the components on the image acquired in the second processing step S2 based on the overlap index total calculated in the sixth processing step S6. In the seventh processing step S7, the CPU 3 having the function of the third determination unit 70 of the information processing device 100 according to this embodiment determines a correspondence relationship between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20 based on the overlap index total calculated by the calculation unit 60.

[0074] First, the third determination unit 70 compares all of the overlap index totals transmitted from the calculation unit 60 and selects the temporary overlay diagram with the largest value. Then, the third determination unit 70 determines that the temporary correspondence relationship used when the temporary overlay diagram was generated is the correct correspondence relationship between the components on the drawing and the components on the image in this embodiment. In other words, the third determination unit 70 determines that the correspondence relationship between the components on the drawing and the components on the image in the temporary correspondence relationship used when the temporary overlay diagram with the largest overlap index total was generated is the correspondence relationship between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20.

[0075] The operation of the third determination unit 70 and the seventh processing step S7 will be described with reference to FIGS. 4 and 5. For ease of explanation, it is assumed that the third determination unit 70 receives two overlap index totals from the calculation unit 60: the total overlap index of the provisional overlay diagram shown in FIG. 4(d) and the total overlap index of the provisional overlay diagram shown in FIG. 5(d). The third determination unit 70 compares the total overlap index of the provisional overlay diagram shown in FIG. 4(d) with the total overlap index of the provisional overlay diagram shown in FIG. 5(d), and selects the provisional overlay diagram shown in FIG. 5(d) as the provisional overlay diagram with the largest value. This is because, as described above in the description of the sixth processing step S6, the total overlap index of the provisional overlay diagram shown in FIG. 5(d) is larger than the total overlap index of the provisional overlay diagram shown in FIG. 4(d).

[0076] Next, referring to the provisional correspondence relationships used when the provisional overlay drawing shown in Fig. 5(d) was generated, it is found that, as shown in Fig. 5(a) and Fig. 5(b), part S on the drawing corresponds to part s on the image, part T on the drawing corresponds to part t on the image, part U on the drawing corresponds to part u on the image, and part V on the drawing corresponds to part v on the image. Therefore, the third determination unit 70 determines that the correspondence relationships shown in Fig. 5(a) and Fig. 5(b), in which part S on the drawing corresponds to part s on the image, part T on the drawing corresponds to part t on the image, part U on the drawing corresponds to part u on the image, and part V on the drawing corresponds to part v on the image, are correspondence relationships between the parts on Fig. 5(a) showing the drawing acquired by the first acquisition unit 10 and the parts on Fig. 5(b) showing the image acquired by the second acquisition unit 20.

[0077] The information processing device 100 according to this embodiment is a device that performs processing for superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing. The correspondence between components on the drawing and components on the image determined by the third determination unit 70 of the information processing device 100 according to this embodiment is used for superimposing and displaying the drawing and the image having a coordinate system different from that of the drawing. The information processing device 100 according to this embodiment may output only the correspondence between the components on the drawing and the components on the image, or may output an overlay diagram generated based on the correspondence between the components on the drawing and the components on the image determined by the third determination unit 70. Note that the overlay diagram generated based on the correspondence between the components on the drawing and the components on the image determined by the third determination unit 70 can be a provisional overlay diagram generated by the generation unit 40 and having the largest total overlap index value, so there is no need to newly generate it and the information processing device 100 according to this embodiment can output it.

[0078] As described above, the information processing device 100 according to this embodiment can perform processing for superimposing and displaying a drawing and an image having a different coordinate system from that of the drawing, even if the drawing is a general drawing. This is because a temporary overlay drawing is generated based on a temporary correspondence determined using the first information and the second information, the total overlap index of the temporary overlay drawing is calculated, and the correspondence between the components on the drawing and the components on the image is determined based on the total overlap index. Unlike the prior art, local features are not used to determine the correspondence, so there is no need for the drawing to have feature information, and drawings created using general CAD or the like can be used.

[0079] Furthermore, the information processing device 100 according to the present embodiment can perform processing for accurately superimposing and displaying a drawing and an image having a different coordinate system from that of the drawing, even if the object has multiple components with the same appearance. Conventionally, when the object has multiple components with the same appearance, there are many points with the same feature amount, so processing for accurately superimposing and displaying a drawing and an image having a different coordinate system from that of the drawing was not possible. On the other hand, the information processing device 100 according to the present embodiment can perform processing for accurately superimposing and displaying a drawing and an image having a different coordinate system from that of the drawing. This is because a provisional overlay diagram is generated based on the provisional correspondence determined using the first information and the second information, the total overlap index of the provisional overlay diagram is calculated, and the correspondence between the components on the drawing and the components on the image is determined based on the total overlap index.

[0080] Furthermore, the information processing device 100 according to the present embodiment does not require manual work by the user, and therefore can easily perform the process of superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing. Conventionally, depending on the user's level of proficiency, the time required for the process of superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing could be long. On the other hand, the information processing device 100 according to the present embodiment does not require manual work by the user, and therefore can shorten the time required for the process of superimposing and displaying a drawing and an image having a coordinate system different from that of the drawing.

[0081] Furthermore, the information processing device 100 according to the present embodiment can reduce the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing, even when the object has multiple components with the same appearance, and can also prevent the superimposition of the components with an incorrect correspondence. Conventionally, when the object has multiple components with the same appearance, a large number of feature points with similar local features exist, making it difficult to determine the correspondence between the feature points on the drawing and the feature points on the image. Therefore, depending on the processing method, the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing could be long. Furthermore, the presence of feature points with similar local features could lead to the determination of a correspondence between components that appear the same but are actually different, resulting in an incorrect superimposition.

[0082] On the other hand, the information processing device 100 according to the present embodiment can reduce the time required for processing to overlay and display a drawing and an image having a coordinate system different from that of the drawing, and can also prevent an erroneous overlay display result, because the information processing device 100 generates a temporary overlay drawing based on the temporary correspondence determined using the first information and the second information, calculates the overlap index sum of the temporary overlay drawing, and determines the correspondence between the components on the drawing and the components on the image based on the overlap index sum.

[0083] The correspondence between components on a drawing and components on an image output by the information processing device 100 according to the present embodiment, and the superimposed drawing generated based on the correspondence between components on a drawing and components on an image, can be used for various purposes. For example, the correspondence between components on a drawing and components on an image determined by the third determination unit 70 of the information processing device 100 according to the present embodiment and the coordinates of the components on the drawing and the components on the image can be used to estimate the extrinsic parameters (also called position and orientation) of the camera that captured the object, and the result can be used to superimpose the drawing on an image having a different coordinate system from that of the drawing. Also, for example, when displaying an augmented reality (AR), an image may be superimposed on a drawing having a different coordinate system from that of the image. However, the correspondence between components on a drawing and components on an image output by the information processing device 100 according to the present embodiment can be used to superimpose the image on a drawing having a different coordinate system from that of the image.

[0084] Furthermore, the superimposition of a drawing and an image having a different coordinate system from that of the drawing, which is displayed based on the output of the information processing device 100 according to this embodiment, can be used for tasks such as checking whether a product has been manufactured according to the drawing, or for inspection tasks to remotely determine whether there are any abnormalities in equipment.

[0085] In the above description, the overlap index is an IoU value, but this embodiment is not limited to this. For example, the overlap index may be the area itself where a component on a drawing and a component on an image overlap. Even if the overlap index is the area itself where a component on a drawing and a component on an image overlap, the operation of the information processing device 100 according to this embodiment and the information processing method using the information processing device 100 according to this embodiment remain unchanged.

[0086] Alternatively, for example, the length of the line where the outline of the component on the drawing and the outline of the component on the image overlap may be used as the overlap index. Even if the overlap index is the length of the line where the outline of the component on the drawing and the outline of the component on the image overlap, the operation of information processing device 100 according to this embodiment and the information processing method using information processing device 100 according to this embodiment remain unchanged.

[0087] Alternatively, for example, the overlap index may be the reciprocal of the sum of the distances between the contours of the components on the drawing and the contours of the components on the image at locations where the contours of the components on the drawing and the contours of the components on the image do not overlap. Points are placed evenly on the contours of the components on the drawing and the contours of the components on the image, and the distances between each point are calculated and summed. The reciprocal of this sum is then used as the overlap index. Even if the overlap index is the reciprocal of the sum of the distances between the contours of the components on the drawing and the contours of the components on the image at locations where the contours of the components on the drawing and the contours of the components on the image do not overlap, the operation of information processing device 100 according to this embodiment and the information processing method using information processing device 100 according to this embodiment remain unchanged.

[0088] In the above description, the control unit 80 of the information processing device 100 according to this embodiment instructs the generation unit 40 to perform the fourth processing step S4, the second determination unit 50 to perform the fifth processing step S5, and the calculation unit 60 to perform the sixth step S6 repeatedly for all the provisional correspondence relationships output by the first determination unit 30 in the third processing step S3. However, this embodiment is not limited to this. For example, the following may be performed.

[0089] First, in a fourth processing step S4, the generation unit 40 generates a temporary overlay diagram for each of the temporary correspondence relationships output by the first determination unit 30 in the third processing step S3. The generation unit 40 then outputs to the second determination unit 50 all of the temporary overlay diagrams generated for each of the temporary correspondence relationships determined by the first determination unit 30, the temporary correspondence relationships determined by the first determination unit 30, and the first and second information. The second determination unit 50 calculates overlap indices between components on the drawing and components on the image for each of the temporary overlay diagrams, and determines components on the image corresponding to the components on the drawing in the temporary overlay diagram based on the overlap indices. The second determination unit 50 then outputs to the calculation unit 60 the overlap indices used when determining the components on the image corresponding to the components on the drawing for each of the temporary overlay diagrams, the temporary correspondence relationships determined by the first determination unit 30, and the first and second information. The calculation unit 60 calculates the total overlap index for all the temporary overlaid drawings and outputs the result to the third determination unit 70. Then, the third determination unit 70 compares all of the total overlap indexes transmitted from the calculation unit 60 and determines the correspondence between the parts on the drawings acquired by the first acquisition unit 10 and the parts on the images acquired by the second acquisition unit 20.

[0090] In this case, it can be said that the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70 of the information processing device 100 according to this embodiment each have the function of controlling themselves in addition to the above-mentioned functions.

[0091] Furthermore, the order of the first processing step S1 and the second processing step S2 may be reversed.

[0092] In the above description, the first information and the second information acquired by the first acquisition unit 10 and the second acquisition unit 20 are loaded into the storage device 2, but this embodiment is not limited to this. The first information and the second information may be stored in an external device, and the first acquisition unit 10 and the second acquisition unit 20 may acquire the first information and the second information from the external device via the network connection device 7.

[0093] In the above description, the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70 of the information processing device 100 according to this embodiment are described as functions included in the CPU 3. However, they may be configured as follows. First, a program including a first processing step S1, a second processing step S2, a third processing step S3, a fourth processing step S4, a fifth processing step S5, a sixth processing step S6, a seventh processing step S7, and a determination step J1 is stored in the storage device 2. The CPU 3 then reads the program from the storage device 2 and executes it. Of course, the location where the program is stored is not limited to the storage device 2.

[0094] Embodiment 2 The configuration of an information processing device 200 according to a second embodiment of the present invention will be described. The information processing device 200 according to the second embodiment of the present invention can also be configured using a computer 1, similar to the information processing device 100 according to the first embodiment of the present invention. In the second embodiment of the present invention, only the differences from the first embodiment of the present invention will be described, and descriptions of the parts that are the same as those in the first embodiment of the present invention will be omitted.

[0095] FIG. 8 is a block diagram showing a functional configuration of an information processing device 200 according to a second embodiment of the present invention. Each function of the information processing device 200 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 8, the information processing device 200 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 32, a generation unit 40, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The first determination unit 32 of the information processing device 200 according to the present embodiment includes a first determination unit 91. Like the information processing device 100 according to the first embodiment of the present invention, the information processing device 200 according to the present embodiment also includes a control unit 80, which is not shown in FIG. 8. The control unit 80 of the information processing device 200 according to the present embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 32, the generation unit 40, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0096] The information processing device 200 according to this embodiment differs from the first embodiment of the present invention in that the first determination unit 32 includes a first judgment unit 91. The first judgment unit 91 judges whether the type of a component on the image corresponding to a component on the drawing is the same as the type of the component on the drawing in the tentative correspondence. If the first judgment unit 91 judges that the components are the same, the first determination unit 32 outputs the tentative correspondence to the generation unit 40.

[0097] The operation of the first determination unit 32 including the first judgment unit 91 and the third processing step of the information processing method using the information processing device 200 according to this embodiment will be described in detail with reference to FIGS. 4 and 5. In the third processing step, a method in which the first determination unit 32 of the information processing device 200 according to this embodiment determines a tentative correspondence between components on a drawing and components on an image will be described. As an example, as shown in FIG. 4(a), the first determination unit 32 selects four components marked with the symbols X, Y, Z, and W from 17 components on the drawing. Similarly, as an example, as shown in FIG. 4(b), the first determination unit 32 selects four components marked with the symbols x, y, z, and w from 17 components on an image.

[0098] The first determination unit 32 provisionally determines the parts on the image that correspond to the selected parts on the drawing. Here, it is provisionally determined that part X on the drawing corresponds to part x on the image, part Y on the drawing corresponds to part y on the image, part Z on the drawing corresponds to part z on the image, and part W on the drawing corresponds to part w on the image. This provisionally determined relationship between the parts on the drawing and the parts on the image is the provisional correspondence relationship in FIG. 4.

[0099] The first determination unit 91 of the first determination unit 32 determines whether the type of a part on the image corresponding to a part on the drawing is the same as the type of the part on the drawing in the tentative correspondence, so first, it determines whether the type of part x on the image corresponding to part X on the drawing is the same as the type of part X on the drawing. Next, the first determination unit 91 of the first determination unit 32 determines whether the type of part y on the image corresponding to part Y on the drawing is the same as the type of part Y on the drawing. Similarly, it determines whether the type of part z on the image corresponding to part Z on the drawing is the same as the type of part Z on the drawing, and determines whether the type of part w on the image corresponding to part W on the drawing is the same as the type of part W on the drawing.

[0100] If the first determination unit 91 determines that they are the same, the first determination unit 32 outputs this tentative correspondence to the generation unit 40. Here, as shown in FIG. 4 , the type of the part x on the image corresponding to the part X on the drawing is a meter, and the type of the part X on the drawing is also a meter. Therefore, the first determination unit 91 determines that the type of the part x on the image corresponding to the part X on the drawing is the same as the type of the part X on the drawing. Similarly, the first determination unit 91 determines that the type of the part Y on the drawing is the same as the type of the part y on the image, determines that the type of the part Z on the drawing is the same as the type of the part z on the image, and determines that the type of the part W on the drawing is the same as the type of the part w on the image. Therefore, the first determination unit 32 outputs to the generation unit 40 a tentative correspondence in which the part X on the drawing corresponds to the part x on the image, the part Y on the drawing corresponds to the part y on the image, the part Z on the drawing corresponds to the part z on the image, and the part W on the drawing corresponds to the part w on the image.

[0101] If the first determination unit 91 determines that the part S on the drawing in FIG. 5(a) corresponds to the part t on the image in FIG. 5(b), the first determination unit 32 does not output this provisional correspondence to the generation unit 40. For example, in the provisional correspondence in FIG. 5, it is assumed that the part S on the drawing in FIG. 5(a) corresponds to the part t on the image in FIG. 5(b). The type of the part t on the image corresponding to the part S on the drawing is a nameplate, but the type of the part S on the drawing is a meter. Therefore, the first determination unit 91 determines that the type of the part t on the image corresponding to the part S on the drawing is not the same as the type of the part S on the drawing (No). As a result, the first determination unit 32 does not output the provisional correspondence that the part S on the drawing in FIG. 5(a) corresponds to the part t on the image in FIG. 5(b) to the generation unit 40. If the first determination unit 91 determines that the part S on the drawing in FIG. 5(a) corresponds to the part t on the image in FIG. 5(b), the first determination unit 32 newly selects a part on the drawing and a part on the image to determine the provisional correspondence.

[0102] As described above, the information processing device 200 according to this embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing. This is because, when the first determination unit 91 determines that the types of parts on the image corresponding to the parts on the drawing are the same in the tentative correspondence, the first determination unit 32 outputs the tentative correspondence to the generation unit 40, thereby preventing the output of an erroneous correspondence.

[0103] Here, the first judgment unit 91 judges whether the type of part on the image corresponding to the part on the drawing is the same as the type of part on the drawing, but it may also be configured to judge whether the type of part on the drawing corresponding to the part on the image is the same as the type of part on the image.

[0104] Embodiment 3 The configuration of an information processing device 300 according to a third embodiment of the present invention will be described. The information processing device 300 according to the third embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first embodiment of the present invention and the second embodiment of the present invention. In the third embodiment of the present invention, only the differences from the first embodiment of the present invention and the second embodiment of the present invention will be described, and descriptions of the parts that are the same as the first embodiment of the present invention and the second embodiment of the present invention will be omitted.

[0105] FIG. 9 is a block diagram showing a functional configuration of an information processing device 300 according to a third embodiment of the present invention. Each function of the information processing device 300 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 9, the information processing device 300 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 40, a second determination unit 53, a calculation unit 60, and a third determination unit 70. Like the information processing devices according to the first and second embodiments of the present invention, the information processing device 300 according to the present embodiment also includes a control unit 80, which is not shown in FIG. 9. The control unit 80 of the information processing device 300 according to the present embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 53, the generation unit 60, and the third determination unit 70.

[0106] The information processing device 300 according to this embodiment differs from the first and second embodiments of the present invention in the operation of the second determination unit 53 and the fifth processing step. The second determination unit 53 of the information processing device 300 according to this embodiment uses the first information and the second information in the temporary overlay drawing to obtain an overlap index indicating an overlap between a component on the drawing and a component on the image. Then, when determining a component on the image corresponding to a component on the drawing in the temporary overlay drawing based on the overlap index, if the overlap index is equal to or greater than a certain value, the second determination unit 53 determines that the component on the image corresponds to the component on the drawing in the temporary overlay drawing.

[0107] The operation of the second determination unit 53 of the information processing device 300 according to this embodiment and the fifth processing step of the information processing method using the information processing device 300 according to this embodiment will be described in detail with reference to Fig. 6. First, the second determination unit 53 of the information processing device 300 according to this embodiment selects one component on the drawing in the temporary overlay drawing shown in Fig. 6(a), which is an example of a temporary overlay drawing. Here, it is assumed that the component L on the drawing is selected, as shown in Fig. 6(b).

[0108] Next, an overlap index between the part L on the drawing and each of the parts l, m, n, o, and p on the image is calculated. Here, an IoU value is calculated as the overlap index. The second determination unit 53 then compares the overlap index between the part L on the drawing and each of the parts l, m, n, o, and p on the image. The overlap index with the largest overlap index is the overlap index between the part L on the drawing and the part l on the image. The second determination unit 53 determines whether the largest overlap index between the part L on the drawing and the part l on the image is equal to or greater than a certain value. If the largest overlap index is equal to or greater than the certain value, the second determination unit 53 determines that the part on the image corresponding to the part L on the drawing in the temporary overlay diagram is the part l on the image. In other words, if the largest overlap index between the part L on the drawing and the part l on the image is less than a certain value, the second determination unit 53 does not determine that the part on the image corresponding to the part L on the drawing in the temporary overlay diagram is the part l on the image. Here, the certain value is, for example, 0.5.

[0109] If the value of the largest overlap index between the part L on the drawing and the part l on the image is less than a certain value, the second determination unit 53 cannot determine the part on the image that corresponds to the part on the drawing in the temporary overlay drawing, and therefore stops the processing. Since the second determination unit 53 stops the processing, it does not output anything to the calculation unit 60. If the second determination unit 53 stops the processing, that is, if the fifth processing step is stopped, the control unit 80 skips the sixth processing step S6 and performs the judgment step J1.

[0110] Furthermore, even if the value of the overlap index between the part L on the drawing with the largest value and the part l on the image is less than a certain value, the second determination unit 53 may continue processing by determining that the part on the image corresponding to this part on the drawing is unknown, rather than terminating the processing as described above.

[0111] As described above, the information processing device 300 according to this embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing. This is because, when the second determination unit 53 determines a component on the image corresponding to a component on the drawing in the temporary overlay drawing based on the overlap index, if the overlap index is equal to or greater than a certain value, the second determination unit 53 determines that the component on the image corresponds to the component on the drawing in the temporary overlay drawing. In other words, if the overlap index is less than a certain value, the second determination unit 53 stops processing, and therefore does not output to the calculation unit 60 and the third determination unit 70 tentative correspondences that are unlikely to be correct, thereby reducing the number of objects for comparison by the third determination unit 70.

[0112] Furthermore, if the second determination unit 53 continues processing even if the value of the overlap index is less than a certain value, the information processing device 300 according to this embodiment can output the data by excluding inappropriate correspondences between parts on the drawing and parts on the image.

[0113] Embodiment 4 The configuration of an information processing device 400 according to a fourth embodiment of the present invention will be described. The information processing device 400 according to the fourth embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to third embodiments of the present invention. In the fourth embodiment of the present invention, only the differences from the first to third embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to third embodiments of the present invention will be omitted.

[0114] FIG. 10 is a block diagram showing a functional configuration of an information processing device 400 according to a fourth embodiment of the present invention. Each function of the information processing device 400 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 10, the information processing device 400 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 44, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The generation unit 44 of the information processing device 400 according to the present embodiment includes a second determination unit 92. Like the information processing devices according to the first to third embodiments of the present invention, the information processing device 400 according to the present embodiment also includes a control unit 80, which is not shown in FIG. 10. The control unit 80 of the information processing device 400 according to the present embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 44, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0115] The information processing device 400 according to this embodiment differs from the first to third embodiments of the present invention in that the generation unit 44 includes a second determination unit 92. When generating a temporary superimposed view, the second determination unit 92 determines whether the drawing or image with the changed coordinate system is obtained by rotating the drawing or image before the coordinate system change within a range of 90 degrees to 270 degrees, or whether the drawing or image with the changed coordinate system is obtained by flipping the drawing or image before the coordinate system change. If the second determination unit 92 determines that the drawing or image is flipped, the generation unit 44 generates a temporary superimposed view and outputs the temporary superimposed view to the second determination unit 50.

[0116] The operation of the generation unit 44 including the second determination unit 92 and the fourth processing step will be described in detail with reference to FIG. 4. First, the generation unit 44 aligns the coordinate system of the drawing with the coordinate system of the image based on the provisional correspondence determined by the first determination unit 30. In the fourth processing step, the generation unit 44 calculates a projective transformation matrix in the same manner as the generation unit 40, and transforms the image shown in FIG. 4(b) using the calculated projective transformation matrix. The transformed image shown in FIG. 4(b) is shown in FIG. 4(c). Then, in the fourth processing step, the second determination unit 92 of the generation unit 44 determines whether the image with the changed coordinate system is the image before the coordinate system change rotated within a range of 90 degrees to 270 degrees, or whether the image with the changed coordinate system is the image before the coordinate system change inverted. That is, the second judgment unit 92 of the generation unit 44 judges whether or not Figure 4(c) showing the converted image is obtained by rotating Figure 4(b) showing the image before conversion within a range of 90 degrees or more and 270 degrees or less, or whether or not Figure 4(c) showing the converted image is obtained by inverting Figure 4(b) showing the image before conversion.

[0117] Here, the generation unit 44 converted the coordinate system of the image into the coordinate system of the drawing, but the generation unit 44 of the information processing device 400 according to this embodiment may convert the coordinate system of the drawing into the coordinate system of the image, similar to the generation unit 40 of the information processing device according to the first to third embodiments of the present invention. When the generation unit 44 converts the coordinate system of the drawing into the coordinate system of the image, the second determination unit 92 determines whether the drawing whose coordinate system has been changed is obtained by rotating the drawing before the coordinate system change within a range of 90 degrees to 270 degrees, or whether the drawing whose coordinate system has been changed is obtained by flipping the drawing before the coordinate system change.

[0118] Next, in the fourth processing step, if the second determination unit 92 determines that the image has been rotated within a range of 90 degrees to 270 degrees or inverted, the generation unit 44 generates a temporary superimposed drawing by superimposing the drawing and the image with the changed coordinate system. That is, if the second determination unit 92 determines that the image has been rotated within a range of 90 degrees to 270 degrees or inverted, the generation unit 44 does not generate a temporary superimposed drawing. Explaining with reference to FIG. 4, if the second determination unit 92 determines that the image has been rotated within a range of 90 degrees to 270 degrees or inverted, the generation unit 44 superimposes the drawing shown in FIG. 4(a) with the drawing shown in FIG. 4(c) to generate the drawing shown in FIG. 4(d). On the other hand, if the second determination unit 92 determines that the image has been rotated within a range of 90 degrees to 270 degrees or inverted, the generation unit 44 does not generate the drawing shown in FIG. 4(d).

[0119] When an image with a changed coordinate system is obtained by rotating the image before changing the coordinate system between 90 and 270 degrees, this means that the top and bottom of the image are reversed before and after changing the coordinate system. Also, "flipping" includes both left-right and top-down flipping.

[0120] If the second determination unit 92 determines that the image has been rotated or flipped within a range of 90 degrees to 270 degrees, the generation unit 44 generates a temporary superimposed image and outputs the generated temporary superimposed image to the second determination unit 50. If the second determination unit 92 determines that the image has been rotated or flipped within a range of 90 degrees to 270 degrees, the generation unit 44 does not generate a temporary superimposed image and therefore terminates the process. Since the generation unit 44 terminates the process, it does not output anything to the second determination unit 50. If the generation unit 44 terminates the process, i.e., if the fourth process step is terminated, the control unit 80 skips the fifth process step S5 and the sixth process step S6 and performs the determination step J1.

[0121] As described above, the information processing device 400 according to this embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing. This is because, when the second determination unit 92 of the generation unit 44 determines that the drawing is not rotated or flipped within a range of 90 degrees to 270 degrees, the generation unit 44 stops processing. Therefore, the generation unit 44 does not output a temporary superimposed view based on a temporary correspondence that is unlikely to be a correct correspondence to the second determination unit 50, the calculation unit 60, and the third determination unit 70, thereby reducing the number of objects for comparison by the third determination unit 70.

[0122] Although the angle for determination is set to between 90 degrees and 270 degrees inclusive in the above description, this angle is merely an example, and the determination may be made at a different angle, for example, between 100 degrees and 260 degrees inclusive.

[0123] Embodiment 5. The configuration of an information processing device 500 according to a fifth embodiment of the present invention will be described. The information processing device 500 according to the fifth embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to fourth embodiments of the present invention. In the fifth embodiment of the present invention, only the differences from the first to fourth embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to fourth embodiments of the present invention will be omitted.

[0124] FIG. 11 is a block diagram showing a functional configuration of an information processing device 500 according to a fifth embodiment of the present invention. Each function of the information processing device 500 according to the present embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 11, the information processing device 500 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 35, a generation unit 40, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The information processing device 500 according to the present embodiment also includes a control unit 80, similar to the information processing devices according to the first to fourth embodiments of the present invention, but this control unit 80 is not shown in FIG. 11. The control unit 80 of the information processing device 500 according to the present embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 35, the generation unit 40, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0125] The information processing device 500 according to this embodiment differs from the first to fourth embodiments of the present invention in the operation of the first determination unit 35 and the third processing step. When selecting components on the drawing and components on the image, the first determination unit 35 of the information processing device 500 according to this embodiment randomly selects components on the drawing and components on the image, and determines any number of tentative correspondences between the components on the drawing and the components on the image.

[0126] In the first to fourth embodiments of the present invention, the provisional correspondence between the components on the drawing and the components on the image is determined for all combinations of the components on the drawing and the components on the image. On the other hand, in the present embodiment, the provisional correspondence between the components on the drawing and the components on the image is not determined for all combinations of the components on the drawing and the components on the image, but rather, when selecting the components on the drawing and the components on the image, the components are selected randomly, and an arbitrary number of provisional correspondences between the components on the drawing and the components on the image are determined. In other words, the information processing device 500 according to the present embodiment employs a RANSAC (Random Sample Consensus) method when the first determination unit 35 determines the provisional correspondence between the components on the drawing and the components on the image.

[0127] The arbitrary number of provisional correspondences between components on the drawing and components on the image determined by the first determination unit 35 is determined, for example, as follows: If four components are selected from nine components on the drawing and four components are selected from nine components on the image, the total number of provisional correspondences is 9C4 × 9P4, or 381,024. Here, the probability that a randomly selected provisional correspondence is correct is 0.033%. To obtain a correct correspondence with a 99% probability, 13,923 trials are sufficient. Therefore, if there are nine components on the drawing and nine components on the image, the first determination unit 35 of the information processing device 500 according to this embodiment randomly selects the components on the drawing and the components on the image and determines 13,923 pairs of provisional correspondences between components on the drawing and components on the image.

[0128] As described above, the information processing device 500 according to this embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a coordinate system different from that of the drawing. This is because the first determination unit 35 does not determine tentative correspondences between components on the drawing and components on the image for all combinations of components on the drawing and components on the image, but randomly selects components on the drawing and components on the image, and determines an arbitrary number of tentative correspondences between components on the drawing and components on the image. In other words, this is because the information processing device 500 according to this embodiment employs the RANSAC method when the first determination unit 35 determines tentative correspondences between components on the drawing and components on the image.

[0129] Embodiment 6 The configuration of an information processing device 600 according to a sixth embodiment of the present invention will be described. The information processing device 600 according to the sixth embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to fifth embodiments of the present invention. In the sixth embodiment of the present invention, only the differences from the first to fifth embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to fifth embodiments of the present invention will be omitted.

[0130] FIG. 12 is a block diagram showing a functional configuration of an information processing device 600 according to a sixth embodiment of the present invention. Each function of the information processing device 600 according to the sixth embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 12, the information processing device 600 according to the present embodiment includes a first acquisition unit 10, a second acquisition unit 20, a third determination unit 93, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The information processing device 600 according to the present embodiment also includes a control unit 80, similar to the information processing devices according to the first to fifth embodiments of the present invention, but this control unit 80 is not shown in FIG. 12. The control unit 80 of the information processing device 600 according to the present embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the third determination unit 93, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 60, and the third determination unit 70.

[0131] The information processing device 600 according to this embodiment differs from the first to fifth embodiments of the present invention in that it includes a third determination unit 93. The third determination unit 93 of the information processing device 600 according to this embodiment determines the number of parts on the drawing included in the first information and the number of pixels included in the second information. image It determines whether the number of parts on the drawing matches the number on the image. Alternatively, it determines whether the difference between the number of parts on the drawing and the number of parts on the image is equal to or greater than a certain value. For example, it determines whether the difference is 3 or more.

[0132] If the third determination unit 93 determines that the information is not the same, the information processing device 600 according to this embodiment stops the processing, and the third determination unit 93 does not output anything to the first determination unit 30. If the third determination unit 93 determines that the information is the same, the third determination unit 93 outputs the first information and the second information to the first determination unit 30.

[0133] As described above, the information processing device 600 according to this embodiment can prevent the first determining unit 30, the generating unit 40, the second determining unit 50, the calculating unit 60, and the third determining unit 70 from performing processing based on erroneous first information or second information. This is because the information processing device 600 according to this embodiment stops processing when the third determining unit 93 determines that the first information or second information is incorrect. Note that the number of parts on the drawing included in the first information and the number of parts on the drawing included in the second information are different from each other. image Possible reasons for the discrepancy between the number of parts on the drawing and the image include the following: The objects shown in the drawing and the image may be different, or the object detection results that formed the basis for the information on the parts on the image included in the second information may have been erroneously detected.

[0134] Embodiment 7 The configuration of an information processing device 700 according to a seventh embodiment of the present invention will be described. The information processing device 700 according to the seventh embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to sixth embodiments of the present invention. In the seventh embodiment of the present invention, only the differences from the first to sixth embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to sixth embodiments of the present invention will be omitted.

[0135] Fig. 13 is a block diagram showing a functional configuration of an information processing device 700 according to a seventh embodiment of the present invention. Each function of the information processing device 700 according to this embodiment is realized by the CPU 3 of the computer 1 shown in Fig. 1. As shown in Fig. 13, the information processing device 700 according to this embodiment includes a first acquisition unit 10, a second acquisition unit 20, a third acquisition unit 85, a fourth determination unit 94, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 60, and a third determination unit 70. The information processing device 700 according to this embodiment also includes a control unit 80, similar to the information processing devices according to the first to sixth embodiments of the present invention, but this control unit is not shown in Fig. 13. The control unit 80 of the information processing device 700 according to this embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the third acquisition unit 85, the fourth judgment unit 94, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 60 and the third determination unit 70.

[0136] The information processing device 700 according to this embodiment differs from the information processing devices according to the first to sixth embodiments of the present invention in that it includes a third acquisition unit 85 and a fourth determination unit 94. The third acquisition unit 85 acquires information on the location where an image was captured. Specifically, the third acquisition unit 85 acquires the location of the image capturing device when the image indicated by the second information was captured using a global navigation satellite system (GNSS) represented by a global positioning system (GPS), an inertial measurement unit (IMU), an inertial navigation system (INS), an ultra wide band (UWB), or other positioning techniques. The fourth determination unit 94 determines whether the distance between the location of the object included in the first information and the location acquired by the third acquisition unit 85 is equal to or greater than a certain value.

[0137] Furthermore, the information processing method using the information processing device 700 according to this embodiment differs from the information processing methods using the information processing devices according to the first to sixth embodiments of the present invention in that it includes an auxiliary step before the third processing step S3. The auxiliary step is a process of acquiring information on the position where the image was captured, and determining whether the distance between the position of the object included in the first information and the position acquired by the third acquisition unit 85 is equal to or greater than a certain value.

[0138] If the fourth determination unit 94 determines that the distance is equal to or greater than the certain value, the information processing device 600 according to this embodiment stops processing, and the fourth determination unit 94 does not output anything to the first determination unit 30. If the fourth determination unit 94 determines that the distance is not the certain value, the fourth determination unit 94 outputs the first information and the second information to the first determination unit 30. Here, the certain value is, for example, 5 meters.

[0139] As described above, the information processing device 700 according to this embodiment can prevent the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70 from performing processing based on erroneous first information or second information. This is because the information processing device 700 according to this embodiment stops processing when the fourth determination unit 94 determines that the distance is equal to or greater than a certain value. If the distance between the position of the object included in the first information and the position acquired by the third acquisition unit 85 is equal to or greater than a certain value, it is highly likely that the object shown in the drawing is different from the object shown in the image.

[0140] In this embodiment, the first information includes information about the position of the object.

[0141] Embodiment 8 The configuration of an information processing device 800 according to an eighth embodiment of the present invention will be described. The information processing device 800 according to the eighth embodiment of the present invention can also be configured using a computer 1, similar to the information processing devices according to the first to seventh embodiments of the present invention. In the eighth embodiment of the present invention, only the differences from the first to seventh embodiments of the present invention will be described, and descriptions of the parts that are the same as the first to seventh embodiments of the present invention will be omitted.

[0142] FIG. 14 is a block diagram showing a functional configuration of an information processing device 800 according to an eighth embodiment of the present invention. Each function of the information processing device 800 according to this embodiment is realized by the CPU 3 of the computer 1 shown in FIG. 1. As shown in FIG. 14, the information processing device 800 according to this embodiment includes a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 68, and a third determination unit 70. The calculation unit 68 of the information processing device 800 according to this embodiment includes a fifth determination unit 95. The information processing device 800 according to this embodiment also includes a control unit 80, similar to the information processing devices according to the first to seventh embodiments of the present invention, but this control unit 80 is not shown in FIG. 14. The control unit 80 of the information processing device 800 according to this embodiment controls the operations of the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 68, and the third determination unit 70.

[0143] The information processing device 800 according to this embodiment differs from the first to seventh embodiments of the present invention in that the calculation unit 68 includes a fifth determination unit 95. The fifth determination unit 95 determines whether the calculated overlap index sum is equal to or greater than a certain value. If the fifth determination unit determines that the overlap index sum is equal to or greater than a certain value, the third determination unit 70 determines the correspondence between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20 based on the overlap index sum determined by the fifth determination unit to be equal to or greater than the certain value.

[0144] The operation of the calculation unit 68 including the fifth determination unit 95, as well as the sixth and seventh processing steps of the information processing method using the information processing device 800 according to this embodiment, will be described in detail. First, in the sixth processing step, the calculation unit 68 calculates the total overlap index of the temporary overlay drawing by summing up the overlap indexes used when the second determination unit 50 determined the components on the image corresponding to the components on the drawing in the temporary overlay drawing generated by the generation unit 40. Here, the overlap index is an IoU value. Next, in the sixth processing step, the fifth determination unit 95 included in the calculation unit 68 determines whether the calculated total overlap index is equal to or greater than a certain value.

[0145] In the sixth processing step, if the fifth determination unit 95 included in the calculation unit 68 determines that the overlap index total is equal to or greater than the certain value, the calculation unit 68 outputs the tentative correspondence used when the tentative overlay image in which the overlap index total that was equal to or greater than the certain value was generated, as well as the first information and the second information, to the third determination unit 70. In the sixth processing step, if the fifth determination unit 95 included in the calculation unit 68 determines that the overlap index total is equal to or greater than the certain value, the control unit 80 instructs the control unit 80 to skip determination step J1 and perform the seventh processing step.

[0146] The third determination unit 70 determines that the provisional correspondence relationship used when the provisional overlay drawing, in which the total overlap index was determined to be equal to or greater than a certain value, was generated is the correct correspondence relationship between the components on the drawing and the components on the image in this embodiment, based on the total overlap index determined to be equal to or greater than a certain value by the fifth determination unit 95. In other words, the third determination unit 70 determines that the correspondence relationship between the components on the drawing and the components on the image in the provisional correspondence relationship used when the provisional overlay drawing, in which the total overlap index was determined to be equal to or greater than a certain value by the fifth determination unit 95, was generated is the correspondence relationship between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20.

[0147] If the fifth determination unit 95 included in the calculation unit 68 determines "no" in the sixth processing step, the control unit 80 performs determination step J1, similarly to the first to seventh embodiments of the present invention. Here, the fixed value is, for example, the number of components on the drawing minus 1. If the number of components on the drawing is eight, the fixed value is seven.

[0148] As described above, the information processing device 800 according to this embodiment can shorten the time required for processing to superimpose and display a drawing and an image having a different coordinate system from the drawing. This is because the third determination unit 70 determines the correspondence between the components on the drawing acquired by the first acquisition unit 10 and the components on the image acquired by the second acquisition unit 20 based on the overlap index total determined by the fifth determination unit 95 of the calculation unit 68 to be equal to or greater than a certain value. The provisional correspondence used when generating the provisional overlay drawing for which the overlap index total determined by the fifth determination unit 95 to be equal to or greater than a certain value is likely to be the correct correspondence. Therefore, the information processing device 800 according to this embodiment can terminate processing when it determines the overlap index total of the provisional overlay drawing that is considered to be based on the correct correspondence.

[0149] Within the scope of the present invention, the embodiments and modifications can be freely combined, and the embodiments can be modified or omitted as appropriate. [Explanation of symbols]

[0150] 1 Computer, 2 Storage device, 3 CPU, 4 Input device, 5 Output device, 6 Auxiliary storage device, 7 Network connection device, 8 Bus, 10 First acquisition unit, 20 Second acquisition unit, 30, 32, 35 First determination unit, 40, 44 Generation unit, 50, 53 Second determination unit, 60, 68 Calculation unit, 70 Third determination unit, 80 Control unit, 85 Third acquisition unit, 91 First judgment unit, 92 Second judgment unit, 93 Third judgment unit, 94 Fourth judgment unit, 95 Fifth judgment unit, 100, 200, 300, 400, 500, 600, 700, 800 Information processing device.

Claims

1. a first acquisition unit that acquires first information, the first information being information on a drawing of an object having a plurality of parts, the first information including information on the parts on the drawing; a second acquisition unit that acquires second information, the second information being information on an image of the object, the second information having a coordinate system different from that of the drawing and including information on a part on the image; a first determination unit that uses the first information and the second information to select a component on the drawing and a component on the image, and determines a tentative correspondence relationship between the component on the drawing and the component on the image; a generating unit that generates a provisional overlay drawing by matching a coordinate system of the drawing with a coordinate system of the image based on the provisional correspondence relationship and overlaying the drawing and the image; a second determination unit that uses the first information and the second information in the temporary overlay drawing to determine an overlap index representing an overlap between a part on the drawing and a part on the image, and determines a part on the image corresponding to the part on the drawing in the temporary overlay drawing based on the overlap index; a calculation unit that calculates a total overlap index of the provisional overlay drawing by summing up the overlap indexes when the second determination unit has determined a part on the image corresponding to a part on the drawing in the provisional overlay drawing; a third determination unit that determines a correspondence relationship between the components on the drawing acquired by the first acquisition unit and the components on the image acquired by the second acquisition unit based on the total overlap index; An information processing device comprising:

2. The overlap index uses an IoU (Intersection over Union) value. The information processing device according to claim 1 .

3. The first determination unit, a first determination unit that determines whether or not a type of a part on the image corresponding to a part on the drawing is the same as a type of a part on the drawing, or whether or not a type of a part on the drawing corresponding to a part on the image is the same as a type of a part on the image, in the tentative correspondence relationship; Equipped with If the first determination unit determines that the two images are the same, the tentative correspondence relationship is output to the generation unit.

3. The information processing device according to claim 1 or 2.

4. The second determination unit, When determining a part on the image corresponding to a part on the drawing in the provisional overlay drawing, if the overlap index is equal to or greater than a certain value, the part on the image is determined to be a part corresponding to the part on the drawing in the provisional overlay drawing.

3. The information processing device according to claim 1 or 2.

5. The generation unit is a second determination unit that, when generating the provisional overlaid drawing, determines whether the drawing or image whose coordinate system has been changed is obtained by rotating the drawing or image before the coordinate system has been changed within a range of 90 degrees to 270 degrees, or whether the drawing or image whose coordinate system has been changed is obtained by inverting the drawing or image before the coordinate system has been changed; Equipped with If the second determination unit determines that the determination is negative, the temporary superimposition diagram is generated, and the temporary superimposition diagram is output to the second determination unit.

3. The information processing device according to claim 1 or 2.

6. The first determination unit, When selecting the parts on the drawing and the parts on the image, the parts are selected randomly, and an arbitrary number of tentative correspondences between the parts on the drawing and the parts on the image are determined.

3. The information processing device according to claim 1 or 2.

7. a third determination unit that determines whether or not the number of parts on the drawing included in the first information matches the number of parts on the image included in the second information; Equipped with If the third determination unit determines that the detection is not successful, the process is terminated.

3. The information processing device according to claim 1 or 2.

8. A third acquisition unit that acquires information on a position where the image was captured; a fourth determination unit that determines whether or not a distance between a position of the object included in the first information and the position acquired by the third acquisition unit is equal to or greater than a certain value; Equipped with When the fourth determination unit determines that the value is equal to or greater than a certain value, the process is stopped.

3. The information processing device according to claim 1 or 2.

9. The calculation unit is a fifth determination unit that determines whether the calculated overlap index total is equal to or greater than a certain value; Equipped with The third determination unit, When the fifth determination unit determines that the overlap index total is equal to or greater than a certain value, a correspondence relationship between the parts on the drawing acquired by the first acquisition unit and the parts on the image acquired by the second acquisition unit is determined based on the overlap index total determined by the fifth determination unit to be equal to or greater than a certain value.

3. The information processing device according to claim 1 or 2.

10. a first processing step of obtaining first information of a drawing of an object having a plurality of parts, the first information including information about the parts on the drawing; a second processing step of obtaining second information of the image of the object, the second information having a coordinate system different from the drawing and including information about parts on the image; a third processing step of selecting a part on the drawing and a part on the image using the first information and the second information, and determining a tentative correspondence between the part on the drawing and the part on the image; a fourth processing step of superimposing the drawing and the image by matching a coordinate system of the drawing and a coordinate system of the image based on the provisional correspondence relationship to generate a provisional superimposed drawing; a fifth processing step of obtaining an overlap index representing an overlap between a part on the drawing and a part on the image in the temporary overlay drawing by using the first information and the second information, and determining a part on the image corresponding to the part on the drawing in the temporary overlay drawing based on the overlap index; a sixth processing step of summing up the overlap indices when the parts on the image corresponding to the parts on the drawing are determined in the fifth processing step in the provisional overlay drawing to obtain a total overlap indices of the provisional overlay drawing; a seventh processing step of determining a correspondence relationship between the parts on the drawing acquired in the first processing step and the parts on the image acquired in the second processing step based on the total overlap index; An information processing method comprising:

11. a first processing step of obtaining first information of a drawing of an object having a plurality of parts, the first information including information about the parts on the drawing; a second processing step of obtaining second information of the image of the object, the second information having a coordinate system different from the drawing and including information about parts on the image; a third processing step of selecting a part on the drawing and a part on the image using the first information and the second information, and determining a tentative correspondence between the part on the drawing and the part on the image; a fourth processing step of superimposing the drawing and the image by matching a coordinate system of the drawing and a coordinate system of the image based on the provisional correspondence relationship to generate a provisional superimposed drawing; a fifth processing step of obtaining an overlap index representing an overlap between a part on the drawing and a part on the image in the temporary overlay drawing by using the first information and the second information, and determining a part on the image corresponding to the part on the drawing in the temporary overlay drawing based on the overlap index; a sixth processing step of summing up the overlap indices when the parts on the image corresponding to the parts on the drawing are determined in the fifth processing step in the provisional overlay drawing to obtain a total overlap indices of the provisional overlay drawing; a seventh processing step of determining a correspondence relationship between the parts on the drawing acquired in the first processing step and the parts on the image acquired in the second processing step based on the total overlap index; An information processing program that causes a computer to execute the above.

Images