INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING PROGRAM

Abstract

Even if it is a general drawing, processing is performed to display the drawing and an image having a different coordinate system than the drawing in superimposed form. A preliminary superimposed diagram is generated by first using information about a drawing of an object having a plurality of parts, which contains information about the parts in the drawing, and second information containing information about an image of the object based on a different coordinate system than that of the drawing and containing information about parts in the image. Provided are: a second determination unit for calculating overlap indices indicating overlaps between the parts in the drawing and the parts in the image, to determine the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram(s) based on the overlap indices;a calculation unit for calculating one or more aggregated overlap indices of the preliminary superimposed diagrams by summing the overlap indices used when the second determination unit determines the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagrams, and a third determination unit for determining a corresponding relationship between the parts in the drawing acquired by the first acquisition unit and the parts in the image acquired by the second acquisition unit based on the aggregated overlap index(es).

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method and an information processing program. [Background on the state of the art]

[0002] Some information processing devices perform processing to display a drawing and an image that have a different coordinate system than the drawing, superimposed on each other.

[0003] When three-dimensional design data in the form of a product drawing and an image obtained by photographing the product are superimposed, such an information processing device uses the product drawing to generate a virtual image and searches for a pair of feature point information where local features of the virtual image and local features of the image obtained by photographing the product are similar. Subsequently, based on the feature group pair, the extrinsic parameters of the camera that took the image are estimated, and the processing to display the image obtained by photographing the product and the three-dimensional design data of the product in superimposed form is performed using the estimated extrinsic parameters. (Patent document 1, for example) [Previous technical documents][Patent documents]

[0004] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2020-17096 [Summary of the invention][Problems to be solved by the invention]

[0005] With such an information processing device, the problem is that it is not possible to superimpose the processing for displaying the drawing and an image with a coordinate system different from that of the drawing if the drawing is a generic drawing. In order to search for the pair of feature point information where the local features of the virtual image generated from the drawing and the local features of the image are similar, the drawing must contain sufficiently detailed data to generate a virtual image capable of exhibiting local features similar to those of the image. However, generic drawings do not contain such detailed data.

[0006] The present invention was developed to solve the problem described above, and an objective of the present invention is to provide an information processing device that can perform a process for displaying a drawing and an image with a coordinate system that differs from that of the drawing in a superimposed manner, even if the drawing is a general drawing. [Means to solve the problem]

[0007] An information processing device according to the present invention, comprising: a first acquisition unit for acquiring first information, which is information about a drawing of an object with a plurality of parts and contains information about the plurality of parts in the drawing; a second acquisition unit for acquiring second information, which is information about an image of the object with a different coordinate system than that of the drawing and contains information about a plurality of parts in the image; a first determination unit for selecting parts in the drawing and parts in the image and for determining a preliminary corresponding relationship between the parts in the drawing and the parts in the image using the first information and the second information;a generating unit for producing a preliminary superimposed diagram by matching the coordinate system of the drawing and the coordinate system of the image and thereby superimposing the drawing and the image on the basis of the preliminary corresponding relationship; a second determining unit for calculating overlap indices that indicate overlaps between the parts in the drawing and the parts in the image in the preliminary superimposed diagram using the first and second information, and for determining the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram on the basis of the overlap indices;a calculation unit for calculating the overlap indices used when the second determination unit identifies the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram, in order to calculate an aggregate overlap index of the preliminary superimposed diagram; and a third determination unit for determining a corresponding relationship between the parts in the drawing acquired by the first acquisition unit and the parts in the image acquired by the second acquisition unit, based on the aggregate overlap index. [Effects of the invention]

[0008] According to the information processing device of the present invention, even when dealing with a general drawing, it is possible to perform processing to display the drawing and an image with a coordinate system that differs from that of the drawing superimposed on one another. This is because, based on a preliminary corresponding relationship generated from first and second information, one or more preliminary superimposed diagrams are created, one or more aggregated overlap indices of the preliminary superimposed diagrams are determined, and subsequently, based on the aggregated overlap indices, a corresponding relationship between parts in a drawing and parts in an image is determined.Since local features are not used as in the prior art to determine the relevant relationship, it is not necessary to specify information about features in a drawing, so a general drawing created with a CAD system or the like can be used. [Brief description of the drawings] Fig. Figure 1 is a block diagram showing a hardware configuration of a computer that can function as an information processing device according to embodiment 1 of the present invention. Fig. Figure 2 is a block diagram showing a functional configuration of the information processing device according to embodiment 1 of the present invention. Fig. Figure 3 is a flowchart showing an information processing procedure using the information processing device according to embodiment 1 of the present invention. Fig. Figure 4 is a representation describing a third processing step and a fourth processing step of the information processing method using the information processing device according to embodiment 1 of the present invention and shows an example of the processing carried out in the third processing step and in the fourth processing step. Fig. Figure 5 is a representation describing a third processing step and a fourth processing step of the information processing method using the information processing device according to embodiment 1 of the present invention and shows a further example of the processing carried out in the third processing step and in the fourth processing step. Fig. Figure 6 is a representation describing a fifth processing step of the information processing method using the information processing device according to embodiment 1 of the present invention and is a representation describing a method for determining a part in an image that corresponds to a part in a drawing in a preliminary superimposed diagram. Fig. Figure 7 is a representation describing the fifth processing step of the information processing method using the information processing device according to embodiment 1 of the present invention and is a representation describing the calculation of overlap indices. Fig. Figure 8 is a block diagram showing a functional configuration of an information processing device according to embodiment 2 of the present invention. Fig. Figure 9 is a block diagram showing a functional configuration of an information processing device according to embodiment 3 of the present invention. Fig. Figure 10 is a block diagram showing a functional configuration of an information processing device according to embodiment 4 of the present invention. Fig. Figure 11 is a block diagram showing a functional configuration of an information processing device according to embodiment 5 of the present invention. Fig. Figure 12 is a block diagram showing a functional configuration of an information processing device according to embodiment 6 of the present invention. Fig. Figure 13 is a block diagram showing a functional configuration of an information processing device according to embodiment 7 of the present invention. Fig. Figure 14 is a block diagram showing a functional configuration of an information processing device according to embodiment 8 of the present invention. [Embodiments of the invention]Embodiment 1

[0009] First, a configuration of an information processing device 100 according to embodiment 1 of the present invention is described. The information processing device 100 according to embodiment 1 of the present invention can be implemented by a computer 1. Fig. Figure 1 is a block diagram showing a hardware configuration of a computer 1 that can function as an information processing device 100 according to embodiment 1 of the present invention. As shown in Fig. As shown in Figure 1, the computer 1 comprises a storage device 2, a central processing unit (CPU) 3, an input device 4, an output device 5, an auxiliary storage device 6, a network connection device 7, and a bus 8. The bus 8 connects the storage device 2, the CPU 3, the input device 4, the output device 5, the auxiliary storage device 6, and the network connection device 7.

[0010] Storage device 2 can be implemented using semiconductor memory such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), or flash memory. Storage device 2 stores various programs and data, including an information processing program used by CPU 3. CPU 3 is a type of processor and executes various processes described later.

[0011] The input device 4 can be implemented, for example, by a pointing device such as a mouse, a keyboard, and the like. The input device 4 is used when a user enters an instruction, information, or the like into the computer 1. Here, the user includes both a person operating the computer 1 locally and a person operating the computer 1 remotely. The output device 5 can be implemented by a display device such as a liquid crystal display. The output device 5 outputs the content of the processing performed and a result of the processing obtained by the information processing device 100 according to the present embodiment, as well as a message to the user, and the like.The input device 4 and the output device 5 can, for example, be implemented by a touch panel or the like, which has the functions of both a keyboard and a display device. The input device 4 can be a combination of a keyboard and a pointing device.

[0012] The auxiliary storage device 6 can be implemented using semiconductor storage such as a magnetic disk device, an optical disk device, a magneto-optical disk device, or a magnetic tape device. The auxiliary storage device 6 stores various programs and data, including an information processing program used by the CPU 3. The programs and data stored in the auxiliary storage device 6 can be used by the user by loading them into the storage device 2.

[0013] The network connection device 7 is a type of communication interface that connects to a communication network (not shown), such as a local area network (LAN) or a wide area network (WAN), and performs the data conversion associated with the communication. For example, computer 1 can receive various programs and data from an external device connected to the communication network via the network connection device 7 and use the programs and data by loading them into storage device 2.

[0014] The in Fig. The hardware configuration of computer 1, which implements the information processing device 100 according to the present embodiment, as shown in Figure 1, can be modified appropriately by omitting or adding some of the components, depending on the conditions of use, etc., of computer 1. For example, if computer 1 does not need to communicate with an external device, the network connection device 7 can be omitted. If computer 1 is a portable terminal with communication capabilities, the input device 4 can also include a microphone, which is a type of communication device, and the output device 5 can include a loudspeaker, which is a type of communication device. The input device 4 can also include an imaging device, such as a camera.

[0015] Fig. Figure 2 is a block diagram showing a functional configuration of the information processing device 100 according to embodiment 1 of the present invention. Each of the functions of the information processing device 100 according to embodiment 1 of the present invention is performed by the CPU 3 of the [unclear text]. Fig. The information processing device 100 according to embodiment 1 of the present invention performs processing to display a drawing and an image, which have a different coordinate system than the drawing, superimposed on each other.

[0016] The drawing in the present embodiment is a two-dimensional design data of an object with a plurality of parts and was created using a computer-aided design (CAD) system. The coordinate system in the drawing of the present embodiment can represent a scale division as length based on the SI unit system in an orthogonal XY coordinate system, in which the horizontal axis is the X-axis and the vertical axis is the Y-axis, for example, using millimeters as the unit per scale division. The unit can be any unit other than millimeters, as long as the unit can represent the physical length of a real object, and can be, for example, meters, inches, or the like. The image in the present embodiment is obtained by capturing an image of an object with a plurality of parts using an image acquisition device such as a camera.The coordinate system of the image in the present embodiment uses px as the unit per scale division and therefore cannot represent a scale division as a physical length in the orthogonal XY coordinate system, in which the horizontal axis is the X-axis and the vertical axis is the Y-axis.

[0017] Since a scale division in the image's coordinate system cannot be represented as a length, unlike a scale division in the drawing's coordinate system which can be represented as a length, the unit used for the image's coordinate system cannot be directly converted to the unit used for the drawing's coordinate system. Of course, the unit used for the drawing's coordinate system cannot be directly converted to the unit used for the image's coordinate system.When the present embodiment describes that the coordinate system of the drawing and the coordinate system of the image are different, this means that, due to the different units per scale division, the unit used for the coordinate system of the drawing cannot be directly converted into the unit used for the coordinate system of the image, or vice versa.

[0018] As described above, the drawing consists of two-dimensional design data created using a CAD system, but the drawing in the present embodiment is not limited to this. Furthermore, it was described above that the image is captured with an image-capturing device such as a camera, but the image in the present embodiment is not limited to this. The present embodiment can handle cases where the drawing and the image have different coordinate systems.This means that the present embodiment works if, due to the difference in units per scale division, the unit used for the coordinate system of the drawing cannot be directly converted into the unit used for the coordinate system of the image, or vice versa.

[0019] In the present embodiment, the drawing and the image depict an object with a plurality of parts. The parts are, for example, a counter, an indicator lamp, a switch, a nameplate, and the like. The object with a plurality of parts is, for example, a measuring device with a plurality of counters, a distributor with a plurality of indicator lamps and switches, and the like. However, the parts and the object according to the present embodiment are not limited to these. In the present embodiment, a part is not merely a point or a line, but something that has an area in the drawing and the image; that is, a part is something that can be represented by enclosing it with a closed curve in the drawing and the image. In other words, in the present embodiment, a part in the drawing and a part in the image each have an outline.The outline is a line that forms the outer shape of an object, and the outline of a part is a line that shows the outer shape of the part.

[0020] As in Fig. As shown in Figure 2, the information processing device 100 according to embodiment 1 of the present invention comprises a first acquisition unit 10, a second acquisition unit 20, a first determination unit 30, a generation unit 40, a second determination unit 50, a calculation unit 60, a third determination unit 70, and a control unit 80. The first acquisition unit 10 acquires first information, which is information about a drawing of an object with a plurality of parts and includes information about the parts in the drawing. The second acquisition unit 20 acquires second information, which is information about an image of the object based on a coordinate system that differs from that of the drawing and includes information about the parts in the image.The first determination unit 30 selects the parts in the drawing and the parts in the image and, based on the first and second sets of information, determines a preliminary corresponding relationship between the parts in the drawing and the parts in the image. The generation unit 40 overlays the drawing and the image by adjusting the coordinate systems of the drawing and the image based on this preliminary corresponding relationship, and generates a preliminary overlaid diagram.

[0021] A second determination unit 50 is used to calculate overlap indices that indicate overlaps between the parts in the drawing and the parts in the image in the preliminary overlaid diagram, using the first information and the first information, and to determine the parts in the image that correspond to the parts in the drawing in the preliminary overlaid diagram, based on the overlap indices. The calculation unit 60 sums the overlap indices used when the second determination unit 50 determines the parts in the image that correspond to the parts in the drawing in the preliminary overlaid diagram, in order to calculate an aggregated overlap index of the preliminary overlaid diagram.The third unit of determination 70 establishes a corresponding relationship between the parts in the drawing acquired by the first acquisition unit 10 and the parts in the image acquired by the second acquisition unit 20, based on the aggregated overlap index(s). The control unit 80 controls the operation of the first acquisition unit 10, the second acquisition unit 20, the first unit of determination 30, the production unit 40, the second unit of determination 50, a production unit 60, and the third unit of determination 70.

[0022] The initial information, that is, the drawing information, includes details about the parts in the drawing and the drawing's coordinate system. This initial information might include, for example, details about the object's use and operation, a description of the object, and similar information. The information about the parts in the drawing includes details about the number of parts, the part types, the point coordinates of the parts, and the outlines of the parts. Part types are categories of parts such as counters, indicator lights, switches, nameplates, and the like. The information about the outlines of parts includes the length of the outlines, the areas enclosed by the outlines, and the coordinates of the corners, if the outlines form corners.In the present embodiment, the information about the parts in the drawing need not include all part numbers, part types, point coordinates of the parts, lengths of the outlines of the parts, areas enclosed by the outlines of the parts, and coordinates of the corners if the outlines of the parts form corners. However, the information that should be included at a minimum is information that enables the operation of the information processing device 100 according to the present embodiment and implements an information processing method using the information processing device 100 according to the present embodiment; both of which will be described later.The outlines of parts may have an interruption as long as the operation of the information processing device 100 according to the present embodiment can be carried out and the information processing method can be implemented using the information processing device 100 according to the present embodiment, both of which will be described later by compensating for the interruption using a known method.

[0023] The second piece of information, the information about the image, includes information about the parts in the image and information about the image's coordinate system. This second piece of information can include, for example, details about the object's use and operation, a description of the object, and similar information. The information about the parts in the drawing includes details about the number of parts, the part types, the point coordinates of the parts, and the outlines of the parts. In the present embodiment, the information about the parts in the image, similar to the information about the parts in the drawing, need not include all part numbers, part types, point coordinates of the parts, lengths of the outlines of the parts, areas enclosed by the outlines of the parts, and coordinates of the corners if the outlines of the parts form corners.However, the information that should be included at a minimum is information that enables the operation of the information processing device 100 according to the present embodiment and implements an information processing method using the information processing device 100 according to the present embodiment, both of which will be described later.

[0024] Since each of the functions of the information processing device 100 according to the present embodiment is performed by the CPU 3 of the in Fig. The computer 1 shown in Figure 1 is implemented as follows: the first acquisition unit is 10, the second acquisition unit is 20, the first destination unit is 30, the generation unit is 40, the second destination unit is 50, the calculation unit is 60, the third destination unit is 70, and the control unit is 80, which are in Fig. Figure 2 shows the functions of the CPU 3. The first and second pieces of information acquired by the first acquisition unit 10 and the second acquisition unit 20 are loaded into the storage device 2. Therefore, the operations of the first acquisition unit 10 and the second acquisition unit 20 of the information processing device 100, according to the present embodiment, are realized by the CPU 3 receiving the first and second pieces of information from the storage device 2.

[0025] Next, the operation of the information processing device 100 according to embodiment 1 of the present invention will be described and the configuration of the information processing device 100 according to embodiment 1 of the present invention will be explained in more detail. Fig. Figure 3 is a flowchart showing an information processing procedure using the information processing device 100 according to embodiment 1 of the present invention.

[0026] As in Fig. Figure 3 shows that the information processing method using the information processing device 100 according to embodiment 1 of the present invention comprises a first processing step S1, a second processing step S2, a third processing step S3, a fourth processing step S4, a fifth processing step S5, a sixth processing step S6, an evaluation step J1 and a seventh processing step S7.

[0027] The first processing step S1 of the information processing method using the information processing device 100 according to the present embodiment is a step for acquiring the first piece of information, which is information about the drawing of an object with a plurality of parts and contains information about the parts in the drawing. The CPU 3, which has the function of the first acquisition unit 10 of the information processing device 100 according to the present embodiment, acquires the first piece of information, which is information about the drawing of an object with a plurality of parts and contains information about the parts in the drawing, from the storage device 2 in the first processing step S1.

[0028] The second processing step S2 of the information processing method using the information processing device 100 according to the present embodiment is a step for acquiring the second piece of information, which includes information about the image of the object with a coordinate system that differs from that of the drawing and contains information about the parts in the image. In the second processing step S2, the CPU 3, which has the function of the second acquisition unit 20 of the information processing device 100 according to the present embodiment, acquires the second piece of information, which consists of information about the image of the object with a coordinate system that differs from that of the drawing and contains information about the parts in the image, from the storage device 2.

[0029] In the present embodiment, the information about the parts in the image, which is contained in the second set of information, comprises a result of capturing the parts contained in the image, obtained by mapping the object using an object capture method. Machine learning or deep learning is used for the object capture method. "You Only Look Once" (YOLO) is an example of an object capture method. Furthermore, there is, for example, a capture method using a support vector machine (SVM) with a feature obtained through an oriented gradient histogram (HOG).From the result of the acquisition of the parts contained in the image using the object acquisition method, information about the number of parts, the part types, the point coordinates of the parts and the outlines of the parts is obtained and loaded into the storage device 2 as information about the parts in the image.

[0030] Here, it is assumed that the process of capturing the parts contained in the image using the object capture method is performed by a system other than computer 1, which can function as the information processing device 100 according to the present embodiment, and that the information obtained from the result is loaded into the storage device 2. However, the present embodiment is not limited to this. For example, it is also possible that the process of capturing the parts contained in the image using the object capture method is performed by the CPU 3 of computer 1, which can function as the information processing device 100 according to the present embodiment, and that the information obtained from the result is loaded into the storage device 2.In the present embodiment, it is sufficient if the second acquisition unit 20 can acquire the information obtained from the result of capturing the parts contained in the image using the object capture method.

[0031] The first acquisition unit 10 and the second acquisition unit 20 of the information processing device 100 according to the present embodiment output the acquired first information and second information to the first determination unit 30.

[0032] The third processing step S3 of the information processing method using the information processing device 100 according to the present embodiment is a process for selecting the parts in the drawing and the parts in the image and for determining the preliminary corresponding relationship between the parts in the drawing and the parts in the image using the first and second information. The CPU 3, which has the function of the first determination unit 30 of the information processing device 100 according to the present embodiment, selects the parts in the drawing and the parts in the image and determines the preliminary corresponding relationship between the parts in the drawing and the parts in the image using the first and second information in the third processing step S3.

[0033] An operation of the first determination unit 30 and the third processing step S3 are performed with reference to the Fig. 4 and Fig. 5 described in detail. Fig. 4 and Fig. Figure 5 are illustrations describing the third processing step S3 and the fourth processing step S4 of the information processing method using the information processing device 100 according to the first embodiment of the present invention, and each shows a different example of the processing in the third processing step S3 and in the fourth processing step S4.

[0034] Fig. 4(a) and Fig. 5(a) show diagrams representing drawings of objects with multiple parts, created based on the initial information acquired by the first acquisition unit 10 in the first processing step S1. The letters X, Y, Z, and W in Fig. 4(a) are symbols used in the following description. Likewise, the letters S, T, U and V in Fig. 5(a) Symbols used in the following description. Fig. 4(a) and Fig. 5(a) show diagrams representing drawings of objects with multiple parts, created based on the second piece of information acquired by the second acquisition unit 20 in the second processing step S2. The letters x, y, z and w in Fig. 4(b) are symbols used in the following description. Likewise, the letters s, t, u and v in Fig. 5(b) Symbols used in the following description. Although the outlines of the objects are shown here in the Fig. 4 and Fig. 5 which are not shown for the sake of clarity, they can be shown in the present embodiment.

[0035] The Fig. 4(a) and 4(b) have different coordinate systems. Similarly, Fig. 5(a) and Fig. 5(b) different coordinate systems. Fig. 4(c) is a representation showing an image obtained by converting the coordinate system of the Fig. 4(b) image shown in the coordinate system of the in Fig. 4(a) was obtained in order to adapt it to the coordinate system of the in Fig. 4(a) to adapt the drawing shown. To align the coordinate system of the drawing and the coordinate system of the image means to make the coordinate systems of the drawing and the image identical, i.e., to make the unit per scale division in the coordinate system of the drawing and the unit per scale division in the coordinate system of the image the same. Fig. 4(d) is a representation showing a preliminary superimposed diagram obtained by superimposing the in Fig. 4(a) shown in the drawing and the conversion of the into Fig. The image obtained was obtained in the coordinate system shown in 4(c). Fig. 4(d) the solid line represents the drawing, while the dashed line represents the image obtained by converting the coordinate system.

[0036] Similarly, Fig. 5(c) a representation showing the image obtained by converting the coordinate system of the in Fig. 5(b) image shown into the coordinate system of the in Fig. 5(a) is obtained in order to compare it with the coordinate system of the drawing shown in Fig. to bring into conformity with the drawing shown in 5(a). Fig. 5(d) is a representation showing a preliminary superimposed diagram obtained by superimposing the in Fig. 5(a) shown in the drawing and the conversion of the into Fig. The image obtained was obtained using the coordinate system shown in 5(c). Fig. 5(d) the solid line represents the drawing, while the dashed line represents the image obtained by converting the coordinate system.

[0037] As in the Fig. 4(a) and Fig. 4(b) as well as in the Fig. 5(a) and Fig. As shown in 5(b), each of the drawings and images depicts 17 parts. That is to say, the parts shown in the Fig. 4(a) and Fig. 4(b) as well as in the Fig. 5(a) and Fig. The objects shown in 5(b) each comprise 17 parts. The 17 parts are divided into 8 counters, 1 large label, and 8 small labels. The items shown in the Fig. 4 and Fig. The drawings and images shown in Figure 5 are examples created to describe the third processing step S3, and the drawings and images to which the processing performed by the information processing device 100 according to the present embodiment is applied are not to be taken from the drawings and images shown in Figure 5. Fig. 4 and Fig. Limited to the 5 drawings and images shown.

[0038] First, with reference to Fig. 4 describes a method by which the first determination unit 30 of the information processing device 100, according to the present embodiment, determines a preliminary corresponding relationship between the parts in the drawing and the parts in the image in the third processing step S3. The first determination unit 30 selects four parts from the 17 parts in the drawing and four parts from the 17 parts in the image. For example, the first determination unit 30 selects, as in Fig. As shown in Figure 4(a), four parts with a symbol X, Y, Z, or W are selected from the 17 parts in the drawing. Similarly, for example, the first determining unit selects 30, as shown in Figure 4(a). Fig. As shown in 4(b), select four parts from the 17 parts in the image, each with a symbol x, y, z or w.

[0039] The first determination unit 30 then provisionally identifies the parts in the image that correspond to the selected parts in the drawing. Here, it is provisionally determined that part X in the drawing corresponds to part x in the image, part Y in the drawing corresponds to part y in the image, part Z in the drawing corresponds to part z in the image, and part W in the drawing corresponds to part w in the image. The provisionally determined relationship between the parts in the drawing and the parts in the image is the provisional corresponding relationship in Fig. 4. That is to say, in the preliminary corresponding relationship in Fig. 4. Part X in the drawing corresponds to part x in the image, part Y in the drawing corresponds to part y in the image, part Z in the drawing corresponds to part z in the image, and part W in the drawing corresponds to part w in the image.

[0040] If the parts in the drawing and the parts in the image correspond, this means that the corresponding parts in the drawing and the corresponding parts in the image are the same parts of the object. In other words, this means that in the preliminary corresponding relationship in Fig. 4. Part X in the drawing and part x in the image represent the same part of the object, specifically that they represent the same part of the object with regard to their position, type, size, and shape. The same applies to the other 3 selected parts.

[0041] Next, with reference to Fig. 5 describes the method by which the first determination unit 30 of the information processing device 100, according to the present embodiment, determines the preliminary corresponding relationship between the parts in the drawing and the parts in the image in the third processing step S3. First, the first determination unit 30 selects four parts from the 17 parts in the drawing and four parts from the 17 parts in the image. For example, the first determination unit 30 selects, as shown in Fig. As shown in Figure 5(a), four parts with a symbol S, T, U, or V are selected from the 17 parts in the drawing. Similarly, for example, the first determining unit selects 30, as shown in Figure 5(a). Fig. 5(b) shows, from the 17 parts in the image select four parts with a symbol s, t, u or v.

[0042] The first determination unit 30 then provisionally identifies the parts in the image that correspond to the selected parts in the drawing. Here, it is provisionally determined that part S in the drawing corresponds to part s in the image, part T in the drawing corresponds to part t in the image, part U in the drawing corresponds to part u in the image, and part V in the drawing corresponds to part v in the image. The provisionally determined relationship between the parts in the drawing and the parts in the image is the provisional corresponding relationship in Fig. 5. That is to say, in the preliminary corresponding relationship in Fig. 5. Part S in the drawing corresponds to part s in the image, part T in the drawing corresponds to part t in the image, part U in the drawing corresponds to part u in the image, and part V in the drawing corresponds to part v in the image.

[0043] The first determining unit 30 of the information processing device 100 according to the present embodiment outputs the determined preliminary corresponding relationship as well as the first information and the second information to the generating unit 40.

[0044] The fourth processing step S4 of the information processing method using the information processing device 100 according to the present embodiment is a step for generating the preliminary superimposed diagram by aligning the coordinate system of the drawing and the coordinate system of the image, and thus superimposing the drawing and the image based on the preliminary corresponding relationship. In the fourth processing step S4, the CPU 3, which has the function of the generation unit 40 of the information processing device 100 according to the present embodiment, generates the preliminary superimposed diagram by aligning the coordinate system of the drawing and the coordinate system of the image, and thus superimposing the drawing and the image based on the preliminary corresponding relationship determined by the first determination unit 30.

[0045] An operation of the generation unit 40 and the fourth processing step S4 are performed with reference to the Fig. 4 and Fig. 5 described in detail. First, with reference to Fig. 4 describes a method by which the generating unit 40 of the information processing device 100, according to the present embodiment, generates the preliminary superimposed diagram in the fourth processing step S4. The generating unit 40 calculates a projection transformation matrix for converting the coordinate system of the image into the coordinate systems of the drawing based on the point coordinates of parts X, Y, Z, and W in the drawing, the point coordinates of parts x, y, z, and w in the image, and the preliminary corresponding relationship in Fig. 4. Subsequently, the generating unit 40 aligns the coordinate system of the in Fig. 4(b) shown in the image with the coordinate system of the in Fig. 4(a) shown in the drawing by the Fig. 4(b) transforms the image shown using the calculated projection transformation matrix. Fig. 4(c) is a representation showing the image obtained by converting the image from Fig. 4(b) was obtained using the calculated projection transformation matrix. Subsequently, the generating unit 40 is produced, as in Fig. 4(d) shows the preliminary superimposed diagram by replacing the one in Fig. 4(a) the drawing shown and the one in Fig. 4(c) shown superimposed converted image.

[0046] First, with reference to Fig. 5 describes a method by which the generating unit 40 of the information processing device 100, according to the present embodiment, generates the preliminary superimposed diagram in the fourth processing step S4. The generating unit 40 calculates a projection transformation matrix for converting the coordinate system of the image into the coordinate systems of the drawing based on the point coordinates of parts S, T, U, and V in the drawing, the point coordinates of parts s, t, u, and v in the image, and the preliminary corresponding relationship in Fig. 5. Subsequently, the generating unit 40 aligns the coordinate system of the in Fig. 5(b) shown in the image with the coordinate system of the in Fig. 5(a) shown in the drawing by the Fig. 5(b) transforms the image shown using the calculated projection transformation matrix. Fig. 5(c) is a representation showing the image obtained by converting the image from Fig. 5(b) was obtained using the calculated projection transformation matrix. Subsequently, the generating unit 40 is produced, as in Fig. 5(d) shows the preliminary superimposed diagram by representing the in Fig. 5(a) the drawing shown and the one in Fig. 5(c) shown, converted image superimposed.

[0047] In the present embodiment, the preliminary corresponding relationship between the parts in the drawing and the parts in the image, determined by the first determining unit 30 in the third processing step S3, is determined for all combinations of the parts in the drawing and the parts in the image. In the present embodiment, the object comprises 17 parts. In order for the generating unit 40 to calculate the projection-transformation matrix in the fourth processing step S4, the first determining unit 30 selects four parts from the 17 parts in the drawing and four parts from the 17 parts in the image in the third processing step S3. Thus, the total number of preliminary corresponding relationships that the first determining unit 30 can determine in the third processing step S3 is 17 C4 × 17 P4.

[0048] If, in the third processing step S3, the first determining unit 30 determines the preliminary corresponding relationship, four parts from the 17 parts in the drawing and four parts from the 17 parts in the image are selected, but the present embodiment is not limited to this. For example, in the third processing step S3, the first determining unit 30 can select three parts from the 17 parts in the drawing and three parts from the 17 parts in the image. If the first determining unit 30 selects three parts in the drawing and three parts in the image, the generating unit 40 can adjust the coordinate system of the drawing and the coordinate system of the image not by calculating the projection transformation matrix, but by calculating an affine transformation matrix in the fourth processing step S4.If the first determination unit 30 determines the preliminary corresponding relationship in the third processing step S3, the number of parts to be selected from each of the parts in the drawing and the parts in the image must only be sufficient to calculate the conversion matrix for matching the coordinate system of the drawing and the coordinate system of the image.

[0049] The generating unit 40 calculates the projection conversion matrix for converting the coordinate system of the image into the coordinate system of the drawing based on the point coordinates of parts X, Y, Z and W in the drawing, the point coordinates of parts x, y, z and w in the image and the preliminary corresponding relationship in Fig. 4, although the present embodiment is not limited thereto. If possible, for example, the projection transformation matrix can be based on the coordinates of the corners of parts X, Y, Z and W in the drawing, the coordinates of the corners of parts x, y, z and w in the image, and the preliminary corresponding relationship in Fig. 4. The same applies to the calculation of the projection transformation matrix for converting the coordinate system of the image into the coordinate system of the drawing based on the point coordinates of parts S, T, U and V in the drawing, the point coordinates of parts s, t, u and v in the image and the preliminary corresponding relationship in Fig. 5.

[0050] The generating unit 40 of the information processing device 100 according to the present embodiment cannot generate the preliminary superimposed diagram not by converting the coordinate system of the image into the coordinate system of the drawing, but by converting the coordinate system of the drawing into the coordinate system of the image. To do this using Fig. To describe section 4, the projection transformation matrix for converting the coordinate system into Fig. 4(a) the drawing shown in the coordinate system of the in Fig. 4(b) shown in the image, the point coordinates of parts X, Y, Z and W in the drawing, the point coordinates of parts x, y, z and w in the image and the preliminary corresponding relationship are calculated, and then the in Fig. 4(a) The drawing shown is transformed using the calculated projection transformation matrix. The preliminary superimposed diagram is then generated by converting the coordinate system of the drawing into Fig. 4(a) and the coordinate system of the image in Fig. 4(b) be coordinated with each other.

[0051] The generating unit 40 of the information processing device 100 according to the present embodiment outputs the generated preliminary superimposed diagram, the preliminary corresponding relationship determined by the first determining unit 30, as well as the first information and the second information to the second determining unit 50.

[0052] The fifth processing step S5 of the information processing procedure using the information processing device 100 according to the present embodiment is a step for calculating the overlap indices that indicate overlaps between the parts in the drawing and the parts in the image using the first information and the second information in the preliminary superimposed diagram, and for determining the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram, based on the overlap indices.In the fifth processing step S5, the CPU 3, which has the function of the second determination unit 50 of the information processing device 100 according to the present embodiment, calculates the overlap indices, which indicate the overlaps between the parts in the drawing and the parts in the image in the preliminary superimposed diagram generated by the generation unit 40, and determines the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram, based on the overlap indices.

[0053] An operation of the second determination unit 50 and the fifth processing step S5 are performed with reference to the Fig. 6 and Fig. 7 described in detail. Fig. Figure 6 is a representation to describe the fifth processing step S5 of the information processing method using the information processing device 100 according to embodiment 1 of the present invention and to describe a method for determining the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram. Fig. Figure 6(a) is an example of the preliminary superimposed diagram, in which the solid line represents the drawing and the dashed line represents the image obtained by transforming the coordinate system. The letters L, M, N, O and P in Fig. 6(a) are symbols used in the following description. Likewise, the letters l, m, n, o and p in Fig. 6(b) Symbols used in the following description. Fig. Figure 7 is a representation to describe the fifth processing step S5 of the information processing method using the information processing device 100 according to embodiment 1 of the present invention and to describe the calculation of the overlap indices.

[0054] First, the second determining unit 50 selects a part in the drawing in which it is located. Fig. 6(a) shows the preliminary superimposed diagram, which is an example of the preliminary superimposed diagram. Here, it is assumed as in Fig. 6(b) shows that part L in the drawing is selected. Next, the overlap indices between part L in the drawing and each of parts l, m, n, o, and p in the image are calculated.

[0055] A method for calculating an overlap index, which indicates the overlap between a part in the drawing and a part in the image, is described with reference to Fig. 7 described. The overlap indices are calculated for each of the parts in the preliminary superimposed diagram. Fig. 7(a) is a representation showing a state in which a part in the drawing and a part in the transformed image overlap with respect to a certain part in the preliminary superimposed diagram. Fig. 7(a) the solid line represents the drawing, while the dashed line represents the image obtained by converting the coordinate system. Fig. 7(b) is a representation showing the area of ​​overlap (intersection) of an area representing the part in the drawing and an area representing the part in the converted image, with respect to the part in the drawing and the part in the image. Fig. 7(a) shows. Fig. 7(c) is a representation showing the combined area (union) of the area of ​​the part in the drawing and the area of ​​the part in the converted image with respect to the part in the drawing and the part in the image. Fig. 7(a) shows.

[0056] Fig. 7(d) is a representation showing a state in which a part in the drawing and a part in the transformed image overlap with respect to a certain part in the preliminary superimposed diagram. In Fig. 7(d) represents the solid line as in Fig. 7(a) represents the drawing, and the dashed line represents the image obtained by transforming the coordinate system. Fig. 7(e) is a representation showing the area of ​​overlap (intersection) of an area representing the part in the drawing and an area representing the part in the converted image, with respect to the part in the drawing and the part in the image. Fig. 7(d) shows. Fig. 7(f) is a representation that shows the combined area (union) of the area of ​​the part in the drawing and the area of ​​the part in the converted image with respect to the part in the drawing and the part in the Fig. 7(d) shows the image.

[0057] The overlap index for the in Fig. 7(a) The part shown is a value obtained by dividing the size of the in Fig. 7(b) area shown by the size of the area shown Fig. The area shown in 7(c) is obtained and is called the Intersection over Union (IoU value). Similarly, the overlap index for the area shown in Fig. 7(d) the part shown is a value obtained by dividing the size of the in Fig. 7(e) area shown by the size of the area shown Fig. 7(f) shown area is obtained.

[0058] The representation of the in Fig. 7(a) The part shown is partly taken from the representation in Fig. 4(d) adopted. The representation of the in Fig. Part shown in 7(d) is partly taken from the representation in Fig. 5(d) adopted. If the overlap index for the in Fig. 7(a) shown part and the overlap index for the in Fig. The part shown in 7(d) is actually calculated, and the overlap index for the part shown in Fig. 7(d) shown part larger than the overlap index for the in Fig. 7(a) shown part.

[0059] The overlap indices between part L in the drawing and each of parts l, m, n, o and p in the in Fig. The representation shown in 6(b) is carried out using the procedure described above with reference to Fig. 7. The calculated results are as follows: The overlap index between part L in the drawing and part l in the image is 0.9, the overlap index between part L in the drawing and part m in the image is 0.0, the overlap index between part L in the drawing and part n in the image is 0.0, the overlap index between part L in the drawing and part o in the image is 0.0, and the overlap index between part L in the drawing and part p in the image is 0.0.

[0060] Based on this result, the second determination unit 50 identifies the part in the image that corresponds to the part in the drawing in the preliminary superimposed diagram. When the overlap indices between part L in the drawing and each of the parts l, m, n, o, and p in the image are compared, the overlap index of part L in the drawing and part l in the image is the largest. Therefore, according to the present embodiment, the second determination unit 50 of the information processing device 100 determines in the fifth processing step S5 that the part in the image that corresponds to part L in the drawing in the preliminary superimposed diagram is part l in the image.

[0061] Next, the second determining unit 50 selects a part from the parts in the drawing for which the corresponding parts in the image are not yet in the Fig. 6(a) shown preliminary superimposed diagram, which is an example of the preliminary superimposed diagram. Here, as in Fig. Figure 6(c) shows that part M in the drawing is selected. Next, the overlap indices between part M in the drawing and each of the parts in the image are calculated. At this point, part l in the image, for which the corresponding part in the drawing has already been determined, is excluded. That is, the overlap indices between part M in the drawing and each of parts m, n, o, and p in the image are calculated.

[0062] The same procedure is then carried out as for determining the image part that corresponds to part L in the drawing. That is, the overlap indices between part M in the drawing and parts m, n, o, and p in the image are compared, and the part in the image with the highest overlap index is determined to be the part in the image that corresponds to part M in the drawing. The same procedure is also carried out for parts O, N, and P in the drawing. Fig. 6(a) is carried out, and the parts in the image that correspond to each of them are determined. To illustrate the above, by means of Fig. To describe 6(a), the second determination unit 50 in the fifth processing step S5 determines that the part in the drawing corresponds to part l in the image, part M in the drawing corresponds to part m in the image, part N in the drawing corresponds to part n in the image, part O in the drawing corresponds to part o in the image, and part P in the drawing corresponds to part p in the image. As described above, in the present embodiment, the corresponding parts in the image are determined for all parts in the drawing in the preliminary superimposed diagram.

[0063] With respect to a specific part in the drawing, however, the process continues if the largest overlap index among the overlap indices with the parts in the image is 0,0, i.e., if no part in the image overlaps with the specific part in the drawing, leaving the part in the image corresponding to the specific part in the drawing unknown. If the number of parts in the image is less than the number of parts in the drawing, the corresponding parts in the image are unknown for some of the parts in the drawing, but the process continues. Fig. Figure 6 shows the outline of the object to illustrate the number of parts contained in the object.

[0064] Here, the corresponding parts in the image are determined for all parts in the drawing in the preliminary superimposed diagram, but the present embodiment is not limited to this. It is sufficient if no contradiction arises when performing the sixth processing step S6, the evaluation step J1, and the seventh processing step S7 described later. For example, the second determination unit 50 can first select a part of the image and determine the overlap indices between the selected part of the image and each of the parts of the drawing in the Fig. Calculate the preliminary superimposed diagram shown in 6(a), which is an example of a preliminary superimposed diagram. That is, the corresponding parts in the drawing can be determined for all parts in the image in the preliminary superimposed diagram.

[0065] The second determination unit 50 of the information processing device 100 according to the present embodiment outputs the values ​​of the overlap indices used when determining the parts in the image that correspond to the parts in the drawing, the preliminary corresponding relationship determined by the first determination unit 30, as well as the first information and the second information to the calculation unit 60. Here, the preliminary corresponding relationship determined by the first determination unit 30 is the preliminary corresponding relationship used when generating the preliminary superimposed diagram.

[0066] The sixth processing step S6 of the information processing procedure using the information processing device 100 according to the present embodiment is a step for calculating the aggregated overlap index of the preliminary superimposed diagram by summing the overlap indices used when the parts in the image that correspond to the parts in the drawing are determined in the preliminary superimposed diagram in the fifth processing step S5.The CPU 3, which has the function of the computing unit 60 of the information processing device 100 according to the present embodiment, calculates the aggregated overlap index of the preliminary superimposed diagram by summing the overlap indices used when the second determining unit 50 determines the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram generated by the generating unit 40 in the sixth processing step S6.

[0067] An operation of the calculation unit 60 and the sixth processing step S6 are described in detail. With reference to Fig. In the fifth processing step S5, the calculation unit 60 sums the overlap index when the part in the image corresponding to part L in the drawing is identified as part l in the image; the overlap index when the part in the image corresponding to part M in the drawing is identified as part m in the image; the overlap index when the part in the image corresponding to part N in the drawing is identified as part n in the image; the overlap index when the part in the image corresponding to part O in the drawing is identified as part o in the image; and the overlap index when the part in the image corresponding to part P in the drawing is identified as part p in the image. The value of the sum is the value of the aggregated overlap index of the preliminary superimposed diagram.

[0068] If the aggregated overlap index of the in Fig. 4(d) shown preliminary superimposed diagram and the aggregated overlap index of the in Fig. The aggregated overlap index of the preliminary superimposed diagram shown in 5(d) is actually calculated. Fig. 5(d) shown preliminary superimposed diagram larger than the aggregated overlap index of the one in Fig. 4(d) shown preliminary superimposed diagram, since the overlap indices for the in Fig. 7(d) parts shown are larger than the overlap indices for the in Fig. 7(a) shown parts.

[0069] The evaluation step J1 of the information processing procedure using the information processing device 100 according to the present embodiment is a step to determine whether the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all preliminary corresponding relationships output in the third processing step S3. In evaluation step J1, the CPU 3, which has the function of the control unit 80 of the information processing device 100 according to the embodiment, determines whether the operations of the generating unit 40, the second determining unit 50, and the computational unit 60 have been performed for all preliminary corresponding relationships output by the first determining unit 30.

[0070] If, in assessment step J1, control unit 80 determines that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all preliminary corresponding relationships output in the third processing step S3, control unit 80 instructs the third determination unit 70 to operate. If, in assessment step J1, control unit 80 determines that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have not been performed for all preliminary corresponding relationships output in the third processing step S3, control unit 80 instructs the generation unit 40 to operate for the preliminary corresponding relationships for which no preliminary overlaid diagrams have yet been generated.This means that the fourth processing step S4 by the generation unit 40, the fifth processing step S5 by the second determination unit 50 and the sixth processing step S6 by the calculation unit 60 are repeated for all preliminary corresponding relationships that were output by the first determination unit 30 in the third processing step S3.

[0071] The calculation unit 60 calculates the aggregated overlap indices for all preliminary corresponding relationships output by the first determination unit 30. When the control unit 80 determines in the evaluation step J1 that the fourth processing step S4, the fifth processing step S5, and the sixth processing step S6 have been performed for all preliminary corresponding relationships output in the third processing step S3, the calculation unit 60 outputs all calculated values ​​of the aggregated overlap indices, the preliminary superimposed diagrams for which the aggregated overlap index was calculated, the preliminary corresponding relationships used in generating the preliminary superimposed diagrams, and the first and second information to the third determination unit 70.That is, the third determination unit 70 of the information processing device 100 according to the present embodiment receives the values ​​of the aggregated overlap indices calculated by the computation unit 60 for all preliminary corresponding relationships output by the first determination unit 30, for which the aggregated overlap indices were calculated, the preliminary assignments determined by the first determination unit 30, which were used when the preliminary superimposed diagrams were generated by the generation unit 40, as well as the first information and the second information from the computation unit 60.

[0072] The seventh processing step S7 of the information processing procedure using the information processing device 100 according to the present embodiment is a step to determine the corresponding relationship between the parts in the drawing that were captured in the first processing step S1 and the parts in the image that were captured in the second processing step S2, based on the aggregated overlap indices that were calculated in the sixth processing step S6.The CPU 3, which has the function of the third determination unit 70 of the information processing device 100 according to the present embodiment, determines the corresponding relationship between the parts in the drawing, which were acquired by the first acquisition unit 10, and the parts in the image, which were acquired by the second acquisition unit 20, on the basis of the aggregated overlap indices calculated by the computation unit 60 in the seventh processing step S7.

[0073] First, the third determining unit 70 compares all the values ​​of the aggregated overlap indices received from the computation unit 60 and selects the preliminary superimposed diagram with the highest value. Then, the third determining unit 70 determines the preliminary corresponding relationship used when the selected preliminary superimposed diagram was generated as the correct corresponding relationship between the parts in the drawing and the parts in the image in the present embodiment.That is, the third determination unit 70 determines the corresponding relationship between the parts in the drawing and the parts in the image in the preliminary corresponding relationship that is used when the preliminary superimposed diagram for which the aggregated overlap index with the largest value is calculated is generated, as the corresponding relationship between the parts in the drawing acquired by the first acquisition unit 10 and the parts in the image acquired by the second acquisition unit 20.

[0074] An operation of the third determination unit 70 and the seventh processing step S7 are performed with reference to the Fig. 4 and Fig. 5 described. To simplify the description, it is assumed that the third unit of determination 70 has received two aggregated overlap indices from the computational unit 60: the aggregated overlap index of the one described in Fig. 4(d) shown preliminary superimposed representation and the aggregated overlap index of the in Fig. 5(d) shown in the preliminary superimposed representation. The third unit of determination 70 compares the aggregated overlap index of the in Fig. 4(d) shown preliminary superimposed diagram with the aggregated overlap index of the in Fig. 5(d) shown preliminary superimposed diagram and selects the one in Fig. 5(d) shows the preliminary superimposed diagram as the preliminary superimposed diagram with the largest value. This is because, as described above in the description of the sixth processing step S6, the aggregated overlap index of the in Fig. 5(d) shown preliminary superimposed diagram is larger than the aggregated overlap index of the one shown in Fig. 4(d) shown preliminary superimposed diagram.

[0075] Next, with reference to the preliminary corresponding relationship established during the generation of the in Fig. 5(d) shown preliminary superimposed diagram, it can be determined that part S in the drawing and part s in the image correspond to each other, part T in the drawing and part t in the image correspond to each other, part U in the drawing and part u in the image correspond to each other, and part V in the drawing and part v in the image correspond to each other, as shown in the Fig. 5(a) and Fig. 5(b) shown. Therefore, the third unit of determination 70 determines that the corresponding relationship in which part S in the drawing and part s in the image correspond to each other, part T in the drawing and part t in the image correspond to each other, part U in the drawing and part u in the image correspond to each other, and part V in the drawing and part v in the image correspond to each other, is the corresponding relationship between the parts in Fig. 5(a), which shows the drawing covered by the first acquisition unit 10, and the parts in Fig. 5(b), which shows the image acquired by the second acquisition unit 20.

[0076] The information processing device 100 according to the present embodiment is a device that performs processing to display a drawing and an image with a coordinate system that differs from that of the drawing in a superimposed manner, and the corresponding relationship between the parts in the drawing and the parts in the image, which is determined by the third determining unit 70 of the information processing device 100 according to the present embodiment, is used to display the drawing and the image, which have a different coordinate system than that of the drawing, superimposed.The information processing device 100 according to the present embodiment can either output only the corresponding relationship between the parts in the drawing and the parts in the image or only the superimposed diagram that was generated on the basis of the corresponding relationship between the parts in the drawing and the parts in the image, which were determined by the third determining unit 70.For the superimposed diagram, which is described as being generated on the basis of the corresponding relationship between the parts in the drawing and the parts in the image determined by the third determining unit 70, the preliminary superimposed diagram with the largest value of the aggregated overlap index, which was generated as a preliminary superimposed diagram by the generating unit 40, can be used, so that the information processing device 100 according to the present embodiment can output the superimposed diagram without having to generate a new superimposed diagram.

[0077] As described above, the information processing device 100 according to the present embodiment can perform processing to display a drawing and an image with a coordinate system different from that of the drawing in a superimposed manner, even if the drawing is a general drawing. This is because the preliminary superimposed diagrams are generated based on the preliminary corresponding relationships determined from the first and second pieces of information, the aggregated overlap indices of the preliminary superimposed diagrams are calculated, and the corresponding relationship between the parts in the drawing and the parts in the image is determined based on the aggregated overlap indices.Since the local features are not used to calculate the corresponding relationship as in the prior art, it is not necessary to specify information about the features in the drawing, so a general drawing created with a CAD system or the like can be used.

[0078] The information processing device 100 according to the present embodiment can perform processing to display a drawing and an image with a coordinate system different from that of the drawing in a superimposed manner with high accuracy, even if the object comprises a large number of parts with the same appearance. Previously, it was not possible to display a drawing and an image with a different coordinate system than that of the drawing with high accuracy for objects comprising a large number of parts with the same appearance, due to the large number of points with identical characteristics. On the other hand, the information processing device 100 according to the present embodiment can perform processing to display a drawing and an image with a coordinate system different from that of the drawing in a superimposed manner with high accuracy.This is because the preliminary superimposed diagrams are generated based on the preliminary corresponding relationships determined from the first and second information, the aggregated overlap indices of the preliminary superimposed diagrams are calculated, and the corresponding relationship between the parts in the drawing and the parts in the image is determined based on the aggregated overlap indices.

[0079] Since the information processing device 100, according to the present embodiment, requires no manual input from the user, it can also easily perform the processing for the superimposed display of a drawing and an image with a different coordinate system than that of the drawing. Furthermore, the processing time for displaying a drawing and an image with a different coordinate system than that of the drawing in superimposed form can be lengthy, depending on the user's knowledge, etc. On the other hand, the information processing device 100, according to the present embodiment, requires no manual work by the user, so the time required for processing the superimposed display of a drawing and an image with a different coordinate system than that of the drawing can be reduced.

[0080] Furthermore, according to the present embodiment, the information processing device 100 can reduce the processing time required to display a drawing and an image with a coordinate system different from that of the drawing, even when the object comprises a large number of parts with the same appearance, and prevent them from being superimposed in an incorrect corresponding relationship.Traditionally, when the object comprises a large number of parts with the same appearance, the processing time required to display a drawing and an image with a coordinate system different from that of the drawing can be long in a superimposed manner, depending on the processing method, because the corresponding relationship between the feature point information in the drawing and the feature point information in the image cannot be easily determined due to the existence of a large number of feature point information of similar local features.Furthermore, if feature point information with similar local features is available, the corresponding relationship can be determined based on combinations of parts that look the same but are actually different, so that a drawing and an image may be incorrectly displayed on top of each other due to the mismatched mapping result.

[0081] On the other hand, the information processing device 100 according to the present embodiment can reduce the time required for processing in order to display a superimposed drawing and an image that have a different coordinate system than the drawing, and prevent a drawing and an image from being displayed superimposed based on a mismatched mapping result. This is because the preliminary superimposed diagrams are generated based on the preliminary corresponding relationships determined from the first and second pieces of information, the aggregated overlap indices of the preliminary superimposed diagrams are calculated, and the corresponding relationship between the parts in the drawing and the parts in the image is determined based on the aggregated overlap indices.

[0082] The corresponding relationship between the parts in the drawing and the parts in the image, as well as the superimposed diagram generated on the basis of the corresponding relationship between the parts in the drawing and the parts in the image and output by the information processing device 100 according to the present embodiment, can be used for various purposes.For example, extrinsic parameters (also called pose parameters) of the camera that recorded the object can be estimated based on the corresponding relationship between the parts in the drawing and the parts in the image, which were determined by the third determining unit 70 of the information processing device 100 according to the present embodiment, and the coordinates of the parts in the drawing and the parts in the image, and a drawing and an image with a different coordinate system than that of the drawing can be superimposed using the calculation result.For example, in some cases an augmented reality display (AR display) is performed by superimposing an image and a drawing with a coordinate system that differs from that of the image, whereby the superimposition of the image and the drawing can be achieved using the appropriate relationship between the parts in the drawing and the parts in the image output by the information processing device 100 according to the present embodiment.

[0083] Furthermore, the superimposed drawing and the image, which has a different coordinate system than that of the drawing displayed on the basis of the output of the information processing device 100 according to the present embodiment, can be used, for example, for work to check whether a product has been manufactured according to a drawing, as well as for inspection work to remotely determine whether a device has an anomaly.

[0084] The above description assumes that the loU value is the overlap index, but the present embodiment is not limited to this. For example, the size of the overlap area between a part in the drawing and a part in the image can be used as the overlap index. Even if the overlap index is set to the size of the overlap area between a part in the drawing and a part in the image, the operations of the information processing device 100 according to the present embodiment and the information processing method using the information processing device 100 according to the present embodiment remain unchanged.

[0085] Alternatively, for example, the length of an outline of the overlap between the area of ​​a part in the drawing and the area of ​​a part in the image can be used as the overlap index. Even if the overlap index is set to the length of an outline of the overlap between the area of ​​a part in the drawing and the area of ​​a part in the image, the operations of the information processing device 100 according to the present embodiment and the information processing method using the information processing device 100 according to the present embodiment remain unchanged.

[0086] Alternatively, for example, the reciprocal of the sum of the distances between the outline of a part in the drawing and the outline of a part in the image can be used as the overlap index in an area where the area of ​​the part in the drawing and the area of ​​the part in the image do not overlap. This can be done by first marking points at equal intervals on both the outline of the part in the drawing and the outline of the part in the image, then calculating the distances between the marked points on the outlines and the sum of the distances, and finally determining the reciprocal of the sum as the overlap index.Even if the overlap index is set to the inverse of the sum of the distances between the outline of a part in the drawing and the outline of a part in the image in an area where the area of ​​the part in the drawing and the area of ​​the part in the image do not overlap, the operations of the information processing device 100 according to the present embodiment and the information processing method using the information processing device 100 according to the present embodiment remain unchanged.

[0087] It is described above that the fourth processing step S4 by the generating unit 40, the fifth processing step S5 by the second determining unit 50, and the sixth processing step S6 by the computation unit 60 are repeatedly performed for all preliminary corresponding relationships issued by the first determining unit 30 in the third processing step S3 in response to the instruction of the control unit 80 of the information processing device 100 according to the present embodiment. However, the present embodiment is not limited to this. For example, the following could be adopted.

[0088] First, in the fourth processing step S4, generation unit 40 creates preliminary overlaid diagrams for all preliminary corresponding relationships output by the first determination unit 30 in the third processing step S3. Subsequently, generation unit 40 outputs all preliminary overlaid diagrams generated for all preliminary corresponding relationships determined by the first determination unit 30, along with the first and second information, to the second determination unit 50. The second determination unit 50 calculates the overlap indices between the parts in the drawing and the parts in the image in all preliminary overlaid diagrams and uses these overlap indices to determine the parts in the image that correspond to the parts in the drawing in the preliminary overlaid diagrams.The second determination unit 50 then outputs the values ​​of the overlap indices used when the parts in the image corresponding to the parts in the drawing were determined in all preliminary superimposed diagrams, the preliminary corresponding relationships determined by the first determination unit 30, as well as the first and second information, to the computation unit 60. The computation unit 60 calculates the aggregated overlap indices for all preliminary superimposed diagrams and outputs the result to the third determination unit 70. The third determination unit 70 compares all aggregated overlap indices received from the computation unit 60 and determines the corresponding relationship between the parts acquired by the first acquisition unit 10 in the drawing and the parts acquired by the second acquisition unit 20 in the image.

[0089] In such a configuration, it can be said that the first determination unit 30, the generation unit 40, the second determination unit 50, the computation unit 60 and the third determination unit 70 of the information processing device 100 according to the present embodiment each have a function for controlling themselves in addition to the functions described above.

[0090] The order of the first processing step S1 and the second processing step S2 can be reversed.

[0091] It was described above that the first and second information acquired by the first acquisition unit 10 and the second acquisition unit 20 are loaded into the storage device 2, but the present embodiment is not limited to this. The first and second information can be stored in an external device, and the first acquisition unit 10 and the second acquisition unit 20 can acquire the first and second information from the external device via the network connection device 7.

[0092] Although described above, the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, the generation unit 40, the second determination unit 50, the computation unit 60, and the third determination unit 70 of the information processing device 100 according to the present embodiment are functions of the CPU 3, the following is also possible. First, a program comprising the first processing step S1, the second processing step S2, the third processing step S3, the fourth processing step S4, the fifth processing step S5, the sixth processing step S6, the seventh processing step S7, and the evaluation step J1 is stored in the storage device 2. Then, the CPU 3 reads the program from the storage device 2 and executes the program. Of course, the memory location of the program is not limited to the storage device 2. Design 2

[0093] A configuration of an information processing device 200 according to embodiment 2 of the present invention is described. The information processing device 200 according to embodiment 2 of the present invention can be implemented using the computer 1 similarly to the information processing device 100 according to embodiment 1 of the present invention. In embodiment 2 of the present invention, only the differences from embodiment 1 of the present invention are described, and the description of the identical parts as in embodiment 1 of the present invention is omitted.

[0094] Fig. Figure 8 is a block diagram showing a functional configuration of the information processing device 200 according to embodiment 2 of the present invention. Each of the functions of the information processing device 200 according to the present embodiment is performed by the CPU 3 of the [unclear text]. Fig. The computer shown in section 1 is implemented as shown in section 1. Fig. As shown in Figure 8, the information processing device 200 according to the present embodiment comprises the first acquisition unit 10, the second acquisition unit 20, a first determination unit 32, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70. The first determination unit 32 of the information processing device 200 according to the present embodiment comprises a first evaluation unit 91. The information processing device 200 according to the present embodiment also comprises the control unit 80, similar to the information processing device 100 according to embodiment 1 of the present invention, but in Fig. Figure 8 is not shown. The control unit 80 of the information processing device 200 according to the present embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the first destination unit 32, the generation unit 40, the second destination unit 50, the generation unit 60 and the third destination unit 70.

[0095] The information processing device 200 according to the present embodiment differs from embodiment 1 of the present invention in that the first determination unit 32 comprises the first evaluation unit 91. The first evaluation unit 91 evaluates whether the types of parts in the image, which correspond to the parts in the drawing, correspond to the types of parts in the drawing in a preliminary corresponding relationship. If the first evaluation unit 91 evaluates that they are of the same type, the first determination unit 32 outputs the preliminary corresponding relationship to the generation unit 40.

[0096] An operation of the first determination unit 32 including the first evaluation unit 91 and the third processing step of an information processing procedure using the information processing device 200 according to the present embodiment are described with reference to the Fig. 4 and Fig. 5 is described in detail. A method is described by which the first determination unit 32 of the information processing device 200, according to the present embodiment, determines a preliminary corresponding relationship between the parts in the drawing and the parts in the image in the third processing step. For example, the first determination unit 32 selects, as in Fig. As shown in 4(a), from the 17 parts in the drawing, select 4 parts with a symbol X, Y, Z or W. Similarly, for example, the first determining unit selects 32, as shown in Fig. As shown in 4(b), from the 17 parts in the image, select 4 parts with a symbol x, y, z or w.

[0097] The first determination unit 32 provisionally determines the parts in the image that correspond to the selected parts in the drawing. Here, it is provisionally determined that part X in the drawing corresponds to part x in the image, part Y in the drawing corresponds to part y in the image, part Z in the drawing corresponds to part z in the image, and part W in the drawing corresponds to part w in the image. The provisionally determined relationship between the parts in the drawing and the parts in the image is the provisional corresponding relationship in Fig. 4.

[0098] Since the first assessment unit 91 of the first determination unit 32 assesses whether the types of parts in the image that correspond to the parts in the drawing correspond to the types of parts in the drawing in the preliminary corresponding relationship, it first assesses whether the type of part x in the image that corresponds to part X in the drawing corresponds to the type of part X in the drawing. Next, the first assessment unit 91 of the first determination unit 32 assesses whether the type of part y in the image that corresponds to part Y in the drawing corresponds to the type of part Y in the drawing. Similarly, it assesses whether the type of part z in the image that corresponds to part Z in the drawing corresponds to the type of part Z in the drawing, and whether the type of part w in the image that corresponds to part W in the drawing corresponds to the type of part W in the drawing.

[0099] If the types of the corresponding parts are assessed as equal by the first assessment unit 91, the first determination unit 32 outputs this preliminary corresponding relationship to the production unit 40. As in Fig. As shown in Figure 4, the type of part x in the image that corresponds to part X in the drawing is "counter," and the type of part X in the drawing is also "counter," so the first assessment unit 91 assesses that the type of part x in the image that corresponds to part X in the drawing is the same as the type of part X in the drawing. Similarly, the first assessment unit 91 assesses that the type of part Y in the drawing corresponds to the type of part y in the image, the type of part Z in the drawing corresponds to the type of part z in the image, and the type of part W in the drawing corresponds to the type of part w in the image.Therefore, the first determining unit 32 outputs the preliminary corresponding relationship, in which part X in the drawing corresponds to part x in the image, part Y in the drawing corresponds to part y in the image, part Z in the drawing corresponds to part z in the image, and part W in the drawing corresponds to part w in the image, to the generating unit 40.

[0100] If the first assessment unit 91 evaluates it as negative, the first determination unit 32 does not output this preliminary corresponding relationship to the production unit 40. Assume the preliminary corresponding relationship in Fig. 5 suggests that part S in the drawing is in Fig. 5(a) the part t in the image in Fig. 5(b). However, the type of part t in the image that corresponds to part S in the drawing is “label”, while the type of part S in the drawing is “counter”. Therefore, the first assessment unit 91 judges that the type of part t in the image that corresponds to part S in the drawing does not correspond to the type of part S in the drawing (negative determination). Consequently, the first determination unit 32 gives the preliminary corresponding relationship, which suggests that part S in the drawing is in Fig. 5(a) the part t in the image in Fig. 5(b) corresponds, not to the production unit 40. In the event of a negative assessment by the first assessment unit 91, the first determination unit 32 again selects the parts in the drawing and the parts in the image to determine another preliminary corresponding relationship.

[0101] As described above, the information processing device 200 according to the present embodiment can reduce the time required for processing in order to display a drawing and an image, which have a different coordinate system than the drawing, superimposed. This is because, when the first evaluation unit 91 judges that the types of parts in the image corresponding to the parts in the drawing have a preliminary corresponding relationship with the types of parts in the drawing, the first determination unit 32 outputs this preliminary corresponding relationship to the generation unit 40, thereby preventing the output of non-corresponding corresponding relationships.

[0102] It was described above that the first assessment unit 91 assesses whether the types of parts in the image that correspond to the parts in the drawing match the types of parts in the drawing. Alternatively, however, the first assessment unit 91 can assess whether the types of parts in the drawing that correspond to the parts in the image match the types of parts in the image. embodiment 3

[0103] A configuration of an information processing device 300 according to embodiment 3 of the present invention is described. Similar to the information processing devices according to embodiments 1 and 2 of the present invention, the information processing device 300 according to embodiment 3 of the present invention can be implemented by a computer 1. In embodiment 3 of the present invention, only the differences from embodiments 1 and 2 of the present invention are described, and the description of the parts that are the same as in embodiments 1 and 2 of the present invention is omitted.

[0104] Fig. Figure 9 is a block diagram showing a functional configuration of the information processing device 300 according to embodiment 3 of the present invention. Each of the functions of the information processing device 300 according to the present embodiment is performed by the CPU 3 of the [unclear text]. Fig. The computer shown in section 1 is implemented as shown in section 1. Fig. As shown in Figure 9, the information processing device 300 according to the present embodiment comprises the first acquisition unit 10, the second acquisition unit 20, the first destination unit 30, the generation unit 40, a second destination unit 53, the calculation unit 60, and the third destination unit 70. The information processing device 300 according to the present embodiment also comprises the control unit 80, similar to the information processing devices according to embodiments 1 and 2 of the present invention, but in Fig. 9 is not shown. The control unit 80 of the information processing device 300 according to the present embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the first destination unit 30, the generation unit 40, the second destination unit 53, the generation unit 60 and the third destination unit 70.

[0105] The information processing device 300 according to the present embodiment differs from embodiments 1 and 2 of the present invention in a function of the second determination unit 53 and the fifth processing step. The second determination unit 53 of the information processing device 300 according to the present embodiment calculates the overlap indices, which indicate the overlaps between the parts in the drawing and the parts in the image in the preliminary superimposed diagrams, using the first information and the second information.If the parts in the image that correspond to the parts in the drawing in a preliminary overlaid diagram are determined on the basis of the overlap indices, then if the overlap index of a part in the image with respect to a part in the drawing is equal to or greater than a predetermined value, the part in the image is determined to be the part that corresponds to the part in the drawing in the preliminary overlaid diagram.

[0106] An operation of the second determination unit 53 of the information processing device 300 according to the present embodiment and the fifth processing step of an information processing procedure using the information processing device 300 according to the present embodiment are described with reference to Fig. 6 described in detail. First, the second determining unit 53 of the information processing device 300, according to the present embodiment, selects a part in the drawing in which it is located. Fig. 6(a) shows the preliminary superimposed diagram, which is an example of the preliminary superimposed diagram. Here, as in Fig. 6(b) shows the selected part of the drawing.

[0107] Next, the overlap indices between part L in the drawing and each of parts l, m, n, o, and p in the image are calculated. Here, the loU values ​​are calculated as overlap indices. Subsequently, the second determination unit 53 compares the overlap indices between the part of the drawing and each of parts l, m, n, o, and p in the image. The overlap index between part L in the drawing and part I in the image is the largest. The second determination unit 53 determines whether the value of the overlap index between part L in the drawing and part I in the image, which is the largest, is equal to or greater than a predetermined value, and if the value is equal to or greater than the predetermined value, it determines that the part in the image corresponding to part L in the drawing in the preliminary superimposed diagram is part l in the image.This means that if the value of the overlap index between the largest part of the drawing and the largest part l in the image is less than the specified value, the second determination unit 53 does not determine that the part in the image corresponding to the part of the drawing in the preliminary superimposed diagram is part l in the image. Here, the specified value is, for example, 0.5.

[0108] If the overlap index value between part L in the drawing and the largest part l in the image is less than the specified value, the second determination unit 53 cannot determine the parts in the image that correspond to the parts in the drawing in the preliminary overlaid diagram, and therefore the process is stopped. Since the second determination unit 53 stops processing, nothing is output to the calculation unit 60. When the second determination unit 53 stops processing, i.e., when the fifth processing step is stopped, the control unit 80 skips the sixth processing step S6 and executes the evaluation step J1.

[0109] Even if the value of the overlap index between part L in the drawing and the largest part l in the image is less than the specified value, the second determining unit 53 can continue processing without stopping the processing as described above, assuming that the part in the image corresponding to this part in the drawing is unknown.

[0110] As described above, the information processing device 300 according to the present embodiment can reduce the processing time required to display a drawing and an image, which have a different coordinate system than the drawing, superimposed. This is because, when the second determination unit 53 identifies the parts in the image that correspond to the parts in the drawing in a preliminary superimposed diagram based on the overlap indices, if the overlap index of a part in the image with respect to a part in the drawing is equal to or greater than a predetermined value, the part in the image is identified as the part that corresponds to the part in the drawing in the preliminary superimposed diagram.In other words, this is because the second determination unit 53 stops the process if the overlap indices are smaller than a predetermined value, meaning that the preliminary corresponding relationships, which are likely not the correct corresponding relationships, are not output to the computation unit 60 and the third determination unit 70, thus reducing the number of comparisons that would otherwise have to be performed by the third determination unit 70.

[0111] If the second determination unit 53 is set up to continue the process even when the values ​​of the overlap indices are below the specified value, the information processing device 300 according to the present embodiment can output the results by excluding the non-matching corresponding relationships between the parts in the drawing and the parts in the image. Design 4

[0112] A configuration of an information processing device 400 according to embodiment 4 of the present invention is described. The information processing device 400 according to embodiment 4 of the present invention can be implemented using the computer 1 similarly to the information processing devices according to embodiments 1 to 3 of the present invention. In embodiment 4 of the present invention, only the differences from embodiments 1 and 3 of the present invention are described, and the description of the parts that are the same as in embodiments 1 and 3 of the present invention is omitted.

[0113] Fig. Figure 10 is a block diagram showing a functional configuration of the information processing device 400 according to embodiment 4 of the present invention. Each of the functions of the information processing device 400 according to the present embodiment is performed by the CPU 3 of the [unclear text]. Fig. The computer shown in section 1 is implemented as shown in section 1. Fig. As shown in Figure 10, the information processing device 400 according to the present embodiment comprises the first acquisition unit 10, the second acquisition unit 20, the first determination unit 30, a generation unit 44, the second determination unit 50, the calculation unit 60, and the third determination unit 70. The generation unit 44 of the information processing device 400 according to the present embodiment comprises a second evaluation unit 92. The information processing device 400 according to the present embodiment also comprises the control unit 80, similar to the information processing devices according to embodiments 1 to 3 of the present invention, but in Fig. 10 is not shown. The control unit 80 of the information processing device 400 according to the present embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the first destination unit 30, the generation unit 44, the second destination unit 50, the generation unit 60 and the third destination unit 70.

[0114] The information processing device 400 according to the present embodiment differs from embodiments 1 to 3 of the present invention in that the generating unit 44 comprises the second evaluation unit 92. When generating a preliminary superimposed diagram, the second evaluation unit 92 evaluates whether the drawing or image with its converted coordinate system was obtained either by rotating a drawing or image by 90 to 270 degrees before the conversion of the coordinate system or by mirroring a drawing or image before the conversion of the coordinate system. If the second evaluation unit 92 determines that neither case applies, the generating unit 44 generates a preliminary superimposed diagram for output to the second determination unit 50.

[0115] An operation of the production unit 44, including the second evaluation unit 92 and the fourth processing step, is performed with reference to Fig. 4 described in detail. First, the generation unit 44 aligns the coordinate system of the drawing and the coordinate system of the image based on the preliminary corresponding relationship determined by the first determination unit 30. In the fourth processing step, the generation unit 44 calculates a projection transformation matrix in the same way as the generation unit 40 and transforms the Fig. The image shown in 4(b) is transformed using the obtained projection transformation matrix. This transformation is achieved by converting the image into Fig. The image obtained from the image shown in 4(b) is the one shown in Fig. 4(c) shown image. In the fourth processing step, the second evaluation unit 92 of the generation unit 44 assesses whether the image with its converted coordinate system is obtained either by rotating an image between 90 degrees and 270 degrees before the conversion of the coordinate system or by mirroring an image before the conversion of the coordinate system. That is, the second evaluation unit 92 of the generation unit 44 assesses whether the converted image is Fig. 4(c) either by rotating the image into Fig. 4(b), which shows an image before conversion, by 90 degrees to 270 degrees or by mirroring the image in Fig. 4(b), which shows the image before the transformation.

[0116] Although it is described above that the generating unit 44 of the information processing device 400, according to the present embodiment, converts the coordinate system of the image into the coordinate system of the drawing, the generating unit 44 can convert the coordinate system of the drawing into the coordinate system of the image in the same way as is done by the generating unit 40 of the information processing devices according to embodiments 1 to 3 of the present invention. When the generating unit 44 converts the coordinate system of the drawing into the coordinate system of the image, the second evaluation unit 92 assesses whether the drawing with its converted coordinate system is obtained either by rotating a drawing by 90 to 270 degrees before the conversion of the coordinate system or by mirroring a drawing before the conversion of the coordinate system.

[0117] Next, in the fourth processing step, if the second evaluation unit 92 determines that none of the cases apply, the generation unit 44 creates a preliminary superimposed diagram by overlaying the drawing and the image obtained by transforming the coordinate system. That is, if the second evaluation unit 92 determines that the transformation of the coordinate system is performed either by rotation between 90 degrees and 270 degrees or by reflection, the generation unit 44 does not create a preliminary superimposed diagram. With reference to Fig. 4 describes that if the second assessment unit 92 determines in the fourth processing step that neither of the two cases applies, the production unit 44, which is in Fig. The representation shown in 4(d) is generated by using the representation shown in Fig. 4(a) representation and the one shown in Fig. 4(c) superimposed. However, if the second evaluation unit 92 evaluates that the transformation of the coordinate system is carried out either by a rotation between 90 degrees and 270 degrees or by a reflection, the generation unit 44 does not produce the representation shown in Fig. 4(d) shown image.

[0118] When it is said that the image with its converted coordinate system is an image obtained by rotating an image by 90 to 270 degrees before the coordinate system conversion, this means that the top and bottom of the image are swapped before and after the coordinate system conversion. Furthermore, reflection includes both horizontal and vertical reflection.

[0119] In the event of a negative assessment by the second assessment unit 92, the generation unit 44 creates a preliminary overlaid diagram and outputs it to the second determination unit 50. If the second assessment unit 92 decides that the coordinate system transformation must be performed either by a rotation between 90 degrees and 270 degrees or by a reflection, the generation unit 44 does not create a preliminary overlaid diagram, and the process is stopped. Since the second generation unit 44 stops processing, nothing is output to the computation unit 50. When the generation unit 44 stops processing, i.e., when the fourth processing step is stopped, the control unit 80 skips the fifth processing step S5 and the sixth processing step S6 and executes the assessment step J1.

[0120] As described above, the information processing device 400, according to the present embodiment, can reduce the processing time required to display a drawing and an image, which have a different coordinate system than the drawing, superimposed on each other. This is because, when the second evaluation unit 92 of the generation unit 44 issues a negative evaluation, the generation unit 44 produces a preliminary superimposed diagram and outputs this to the second determination unit 50.In other words, this is because the generating unit 44 stops the process when the second evaluation unit 92 of the generating unit 44 evaluates that the transformation of the coordinate system is carried out either by rotation between 90 degrees and 270 degrees or by mirroring, so that the preliminary superimposed diagrams, which are based on the preliminary corresponding relationships that are probably not the correct corresponding relationships, are not output to the second determination unit 50, the calculation unit 60 and the third determination unit 70, and thus the number of comparisons that the third determination unit 70 would otherwise have to perform can be reduced.

[0121] The angle setting for the assessment is given above as being between 90 degrees and 270 degrees, but this is only an example, and the assessment can also be carried out with a different angle setting. For example, the angle setting can be between 100 degrees and 260 degrees. Design 5

[0122] A configuration of an information processing device 500 according to embodiment 5 of the present invention is described. The information processing device 500 according to embodiment 5 of the present invention can be implemented using the computer 1 similarly to the information processing devices according to embodiments 1 to 4 of the present invention. In embodiment 5 of the present invention, only the differences from embodiments 1 and 4 of the present invention are described, and the description of the parts that are the same as in embodiments 1 and 4 of the present invention is omitted.

[0123] Fig. Figure 11 is a block diagram showing a functional configuration of the information processing device 500 according to embodiment 5 of the present invention. Each of the functions of the information processing device 500 according to the present embodiment is performed by the CPU 3 of the [unclear] Fig. The computer shown in section 1 is implemented as shown in section 1. Fig. As shown in Figure 11, the information processing device 500 according to the present embodiment comprises the first acquisition unit 10, the second acquisition unit 20, a first determination unit 35, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70. The information processing device 500 according to the present embodiment also comprises the control unit 80, similar to the information processing devices according to embodiments 1 to 4 of the present invention, but in Fig. Figure 11 is not shown. The control unit 80 of the information processing device 500 according to the present embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the first destination unit 35, the generation unit 40, the second destination unit 50, the generation unit 60 and the third destination unit 70.

[0124] The information processing device 500 according to the present embodiment differs from embodiments 1 and 4 of the present invention in the operation of the first selection unit 35 and the third processing step. When selecting the parts in the drawing and the parts in the image, the first selection unit 35 of the information processing device 500 according to the present embodiment selects them randomly and then determines an arbitrary number of preliminary assignments between the parts in the drawing and the parts in the image.

[0125] In embodiments 1 to 4 of the present invention, it is described that the preliminary corresponding relationships between the parts in the drawing and the parts in the image are determined for all combinations of the parts in the drawing and the parts in the image. On the other hand, in the present embodiment, the preliminary corresponding relationships between the parts in the drawing and the parts in the image are not determined for all combinations of the parts in the drawing and the parts in the image, but rather an arbitrary number of preliminary corresponding relationships are determined by randomly selecting parts in the drawing and parts in the image. That is, the information processing device 500 according to the present embodiment uses a RANSAC (Random Sample Consensus) method for the first determination unit 35 to determine the preliminary corresponding relationships between the parts in the drawing and the parts in the image.

[0126] The arbitrary number of preliminary correspondences between the parts in the drawing and the parts in the image, to be determined by the first determining unit 35, is determined, for example, as follows. If four parts are selected from the nine parts in the drawing and four parts from the nine parts in the image, the total number of preliminary correspondences is 381,024 according to 9C4 × 9P4. Here, the probability that a randomly selected preliminary correspondence is correct is 0.033%. To obtain the correct correspondence with a probability of 99%, 13,923 trials are required.Therefore, if there are 9 parts in both the drawing and the image, the first determining unit 35 of the information processing device 500 according to the present embodiment 13923 should determine combinations of the preliminary corresponding relationships between the parts in the drawing and the parts in the image by randomly selecting the parts in the drawing and the parts in the image.

[0127] As described above, the information processing device 500 according to the present embodiment can reduce the time required for processing in order to display a drawing and an image, which have a different coordinate system than the drawing, superimposed on each other. This is because the first determining unit 35 does not determine the preliminary corresponding relationships between the parts in the drawing and the parts in the image for all combinations of the parts in the drawing and the parts in the image, but rather determines any number of preliminary corresponding relationships by randomly selecting the parts in the drawing and the parts in the image.In other words, this is because, according to the present embodiment, the information processing device 500 uses the RANSAC method for the first determination unit 35 to determine the preliminary corresponding relationships between the parts in the drawing and the parts in the image. Design 6

[0128] A configuration of an information processing device 600 according to embodiment 6 of the present invention is described. The information processing device 600 according to embodiment 6 of the present invention can be implemented using the computer 1 similarly to the information processing devices according to embodiments 1 to 5 of the present invention. In embodiment 6 of the present invention, only the differences from embodiments 1 and 5 of the present invention are described, and the description of the parts that are the same as in embodiments 1 and 5 of the present invention is omitted.

[0129] Fig. Figure 12 is a block diagram showing a functional configuration of the information processing device 600 according to embodiment 6 of the present invention. Each of the functions of the information processing device 600 according to the present embodiment is performed by the CPU 3 of the Fig. The computer shown in section 1 is implemented as shown in section 1. Fig. As shown in Figure 12, the information processing device 600 according to the present embodiment comprises the first acquisition unit 10, the second acquisition unit 20, a third evaluation unit 93, the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70. The information processing device 600 according to the present embodiment also comprises the control unit 80, similar to the information processing devices according to embodiments 1 to 5 of the present invention, but with the following modifications: Fig. Figure 12 is not shown. The control unit 80 of the information processing device 600 according to the present embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the third evaluation unit 93, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 60 and the third determination unit 70.

[0130] The information processing device 600 according to the present embodiment differs from embodiments 1 to 5 of the present invention in that it includes the third evaluation unit 93. The third evaluation unit 93 of the information processing device 600 according to the present embodiment evaluates whether the number of parts in the drawing contained in the first piece of information and the number of parts in the image contained in the second piece of information are the same. Alternatively, it evaluates whether the difference between the number of parts in the drawing and the number of parts in the image is equal to or greater than a predetermined value. For example, it evaluates whether the difference is 3 or more.

[0131] If the third evaluation unit 93 issues a negative evaluation, the information processing device 600, according to the present embodiment, stops processing, and the third evaluation unit 93 does not output anything to the first determination unit 30. If the third evaluation unit 93 evaluates that the two numbers are equal, the third evaluation unit 93 outputs the first piece of information and the second piece of information to the first determination unit 30.

[0132] As described above, the information processing device 600, according to the present embodiment, can prevent the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70 from performing processing based on incorrect first or second information. This is because, according to the present embodiment, the information processing device 600 stops processing if the third determination unit 93 issues a negative assessment. The following reasons are considered to be the case: the number of parts in the drawing contained in the first piece of information and the number of parts in the image contained in the second piece of information do not match.For example, the reasons could be that the objects represented by the drawing and the image are different, or that there is an error in the result of the object capture on which the information about the parts in the image is based, which is contained in the second piece of information. Model 7

[0133] A configuration of an information processing device 700 according to embodiment 7 of the present invention is described. The information processing device 700 according to embodiment 7 of the present invention can be implemented using the computer 1 similarly to the information processing devices according to embodiments 1 to 6 of the present invention. In embodiment 7 of the present invention, only the differences from embodiments 1 and 6 of the present invention are described, and the description of the parts that are the same as in embodiments 1 and 6 of the present invention is omitted.

[0134] Fig. Figure 13 is a block diagram showing a functional configuration of the information processing device 700 according to embodiment 7 of the present invention. Each of the functions of the information processing device 700 according to the present embodiment is performed by the CPU 3 of the [unclear text]. Fig. The computer shown in section 1 is implemented as shown in section 1. Fig. As shown in Figure 13, the information processing device 700 according to the present embodiment comprises the first acquisition unit 10, the second acquisition unit 20, a third acquisition unit 85, a fourth evaluation unit 94, the first determination unit 30, the generation unit 40, the second determination unit 50, the calculation unit 60, and the third determination unit 70. The information processing device 700 according to the present embodiment also comprises the control unit 80, similar to the information processing devices according to embodiments 1 to 6 of the present invention, but with the following modifications: Fig. Figure 13 is not shown. The control unit 80 of the information processing device 700 according to the present embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the third acquisition unit 85, the fourth evaluation unit 94, the first determination unit 30, the generation unit 40, the second determination unit 50, the generation unit 60 and the third determination unit 70.

[0135] The information processing device 700 according to the present embodiment differs from the information processing devices according to embodiments 1 to 6 of the present invention in that it comprises the third acquisition unit 85 and the fourth evaluation unit 94. The third acquisition unit 85 acquires the information about the position from which the image is captured.In particular, the third acquisition unit 85 acquires the position of the image-generating device when the image represented by the second piece of information is acquired by the image-generating device using a positioning method such as a global navigation satellite system (GNSS), an inertial measurement unit (IMU), an inertial navigation system (INS), ultra-wideband (UWB), or any other positioning method represented by the Global Positioning System (GPS). The fourth evaluation unit 94 evaluates whether the distance between the position of the object contained in the first piece of information and the position acquired by the third acquisition unit 85 is equal to or greater than a predetermined value.

[0136] An information processing method using the information processing device 700 according to the present embodiment comprises an additional step prior to the third processing step S3, in contrast to the information processing methods using the information processing devices according to embodiments 1 to 6 of the present invention. The auxiliary step is a step for acquiring information about the position from which the image is taken and for assessing whether the distance between the position of the object contained in the first information and the position acquired by the third acquisition unit 85 is equal to or greater than a predetermined value.

[0137] If the fourth evaluation unit 94 determines that the distance is equal to or greater than the specified value, the information processing device 600, according to the present embodiment, stops processing, and the fourth evaluation unit 94 does not output anything to the first determination unit 30. If the fourth evaluation unit 94 issues a negative evaluation, it outputs the first and second pieces of information to the first determination unit 30. Here, the specified value is, for example, 5 meters.

[0138] As described above, the information processing device 700 according to the present embodiment can prevent the first determining unit 30, the generating unit 40, the second determining unit 50, the computational unit 60, and the third determining unit 70 from performing processing based on incorrect first or second information. This is because the information processing device 700 according to the present embodiment stops processing when the fourth determining unit 94 determines that the distance is equal to or greater than the specified value.This is because the fact that the distance between the position of the object contained in the first piece of information and the position acquired by the third acquisition unit 85 is equal to or greater than the specified value shows that the object represented by the drawing and the object represented by the picture are highly likely to be different objects.

[0139] In the present embodiment, it is assumed that the first information contains details about the position of the object. Example 8

[0140] A configuration of an information processing device 800 according to embodiment 8 of the present invention is described. The information processing device 800 according to embodiment 8 of the present invention can be implemented using the computer 1 similarly to the information processing devices according to embodiments 1 to 7 of the present invention. In embodiment 8 of the present invention, only the differences from embodiments 1 and 7 of the present invention are described, and the description of the parts that are the same as in embodiments 1 and 7 of the present invention is omitted.

[0141] Fig. Figure 14 is a block diagram showing a functional configuration of the information processing device 800 according to embodiment 8 of the present invention. Each of the functions of the information processing device 800 according to the present embodiment is performed by the CPU 3 of the [unclear text]. Fig. The computer shown in section 1 is implemented as shown in section 1. Fig. As shown in Figure 14, the information processing device 800 according to the present embodiment comprises the first acquisition unit 10, the second acquisition unit 20, a first determination unit 30, the generation unit 40, the second determination unit 50, a calculation unit 68, and the third determination unit 70. The calculation unit 68 of the information processing device 800 according to the present embodiment comprises a fifth evaluation unit 95. The information processing device 800 according to the present embodiment also comprises the control unit 80, similar to the information processing devices according to embodiments 1 to 7 of the present invention, but in Fig.Figure 14 is not shown. The control unit 80 of the information processing device 800 according to the present embodiment controls the operation of the first acquisition unit 10, the second acquisition unit 20, the first destination unit 30, the generation unit 40, the second destination unit 50, the generation unit 68 and the third destination unit 70.

[0142] The information processing device 800 according to the present embodiment differs from embodiments 1 to 7 of the present invention in that the calculation unit 68 comprises the fifth evaluation unit 95. The fifth evaluation unit 95 determines whether the calculated aggregated overlap index is equal to or greater than a predetermined value. If the fifth evaluation unit judges that the calculated aggregated overlap index is equal to or greater than the predetermined value, the third determination unit 70 determines the corresponding relationship between the parts in the drawing acquired by the first acquisition unit 10 and the parts in the image acquired by the second acquisition unit 20, based on the overall overlap index that the fifth evaluation unit judges to be equal to or greater than the predetermined value.

[0143] An operation of the computing unit 68, which includes the fifth evaluation unit 95, as well as the sixth and seventh processing steps, which are contained in the information processing procedure using the information processing device 800 according to the present embodiment, are described in detail below. First, in the sixth processing step, the computing unit 68 sums the overlap indices that are used when the second determination unit 50 determines the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram generated by the generation unit 40, in order to obtain the aggregated overlap index of the preliminary superimposed diagram. Here, it is assumed that the overlap indices are loU values.Next, in the sixth processing step, the fifth assessment unit 95 of the calculation unit 68 assesses whether the calculated aggregated overlap index is equal to or greater than a specified value.

[0144] In the sixth processing step, when the fifth evaluation unit 95 of the computation unit 68 determines that the aggregated overlap index is equal to or greater than the predetermined value, the computation unit 68 outputs the preliminary corresponding relationship used when generating the preliminary superimposed diagram for which the aggregated overlap index is calculated to be equal to or greater than the predetermined value, as well as the first and second information, to the third determination unit 70. If, in the sixth processing step, the fifth evaluation unit 95 of the computation unit 68 determines that the aggregated overlap index is equal to or greater than the predetermined value, the control unit 80 instructs the evaluation step J1 to be skipped and the seventh processing step to be executed.

[0145] Based on the aggregated overlap index, which was judged by the fifth evaluation unit 95 to be equal to or greater than the specified value, the third evaluation unit 70 determines the preliminary corresponding relationship, which is used when the preliminary superimposed diagram, for which the aggregated overlap index was calculated to be equal to or greater than the specified value, is generated as the correct corresponding relationship between the parts in the drawing and the parts in the image in the present embodiment.This means that the third determination unit 70 determines the corresponding relationship between the parts in the drawing and the parts in the image in the preliminary corresponding relationship, which is used when the preliminary superimposed diagram, for which the aggregate overlap index was judged by the fifth assessment unit 95 to be equal to or greater than the specified value, is calculated as the corresponding relationship between the parts in the drawing acquired by the first acquisition unit 10 and the parts in the image acquired by the second acquisition unit 20.

[0146] If the fifth evaluation unit 95 of the calculation unit 68 issues a negative evaluation in the sixth processing step, the control unit 80 performs evaluation step J1 in the same way as in embodiments 1 to 7 of the present invention. Here, the predetermined value is, for example, a value obtained by subtracting 1 from the number of parts in the drawing. If the number of parts in the drawing is 8, the predetermined value is 7.

[0147] As described above, the information processing device 800, according to the present embodiment, can reduce the time required for processing in order to display a drawing and an image, which have a different coordinate system than the drawing, superimposed. This is because the third determination unit 70 determines the corresponding relationship between the parts in the drawing, acquired by the first acquisition unit 10, and the parts in the image, acquired by the second acquisition unit 20, based on the aggregated overlap index, which the fifth evaluation unit 95 of the computation unit 68 judges to be equal to or greater than the predetermined value.The preliminary corresponding relationship used when calculating the preliminary superimposed diagram, for which the aggregated overlap index is judged by the fifth evaluation unit 95 to be equal to or greater than the predetermined value, is highly likely to be the correct corresponding relationship. Therefore, according to the present embodiment, the information processing device 800 can terminate the process at the time when the aggregated overlap index of the preliminary superimposed diagram is calculated, which is deemed to be based on the correct corresponding relationship.

[0148] The present invention allows any combination of the embodiments and modifications, as well as their suitable adaptation and omission within the scope of the invention. [List of reference symbols] 1 computer, 2 Storage device, 3 CPUs, 4 Input device, 5 Output device, 6 Auxiliary storage device, 7 Network connection device, 8 Bus, 10 first acquisition unit, 20 second acquisition unit, 30, 32, 35 first unit of determination, 40, 44 units of production, 50, 53 second unit of determination, 60, 68 units of calculation, 70 third unit of determination, 80 control unit, 85 third acquisition unit, 91 first assessment unit, 92 second assessment unit, 93 third assessment unit, 94 fourth assessment unit, 95 fifth assessment unit, 100, 200, 300, 400, 500, 600, 700, 800 Information processing device QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2020-17096

[0004]

Claims

[1] Information processing device, comprising: a first acquisition unit for acquiring initial information, which is information about a drawing of an object with a multitude of parts and contains information about the multitude of parts in the drawing; a second acquisition unit for acquiring second information, which is information about an image of the object with a different coordinate system than that of the drawing and which contains information about a multitude of parts in the image; a first unit of determination for selecting parts in the drawing and parts in the image and for determining a preliminary corresponding relationship between the parts in the drawing and the parts in the image using the first information and the second information; a generating unit for generating a preliminary superimposed diagram by aligning the coordinate system of the drawing and the coordinate system of the image and thereby superimposing the drawing and the image on the basis of the preliminary corresponding relationship; a second unit of determination for calculating overlap indices that indicate overlaps between the parts in the drawing and the parts in the image in the preliminary superimposed diagram using the first information and the second information, and for determining the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram, based on the overlap indices; a calculation unit for summing the overlap indices, which are used when the second determining unit identifies the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram, in order to calculate an aggregated overlap index of the preliminary superimposed diagram; and a third unit of determination for determining a corresponding relationship between the parts acquired by the first acquisition unit in the drawing and the parts acquired by the second acquisition unit in the image on the basis of the aggregated overlap index. [2] Information processing device according to claim 1, wherein a value for Intersection over Union (IoU) is used as each of the overlap indices. [3] Information processing device according to claim 1 or 2, where the first unit of determination a first assessment unit comprises to assess whether types of parts in the image corresponding to parts in the drawing correspond to types of parts in the drawing, or whether types of parts in the drawing corresponding to parts in the image correspond to types of parts in the image in the preliminary corresponding relationship, and outputs the preliminary corresponding relationship to the production unit if the first assessment unit determines that the types are identical. [4] Information processing device according to one of claims 1 to 3, wherein the second determining unit, when determining the parts in the image that correspond to the parts in the drawing in the preliminary superimposed diagram, determines that a part in the image corresponds to a part in the drawing in the preliminary superimposed diagram if the overlap index between the parts is equal to or greater than a predetermined value. [5] Information processing device according to any one of claims 1 to 4, where the generating unit a second evaluation unit is included to assess, when generating the preliminary superimposed diagram, whether the drawing or image with its transformed coordinate system is obtained either by rotating the drawing or image by 90 to 270 degrees before transforming the coordinate system, or by mirroring, and The preliminary overlaid diagram is generated to be output to the second determination unit if the second assessment unit gives a negative assessment. [6] Information processing device according to any one of claims 1 to 5, wherein, in selecting the parts in the drawing and the parts in the image, the first determining unit selects these randomly and determines any number of preliminary corresponding relationships between the parts in the drawing and the parts in the image. [7] Information processing device according to any one of claims 1 to 6, further comprising: a third assessment unit to assess whether a number of parts in the drawing contained in the first piece of information and a number of parts in the image contained in the second piece of information correspond, Processing is stopped if the third assessment unit gives a negative assessment. [8] Information processing device according to any one of claims 1 to 7, further comprising: a third acquisition unit for acquiring information about a position from which the image was taken; and The fourth assessment unit assesses whether the distance between the position of the object contained in the first piece of information and the position acquired by the third acquisition unit is equal to or greater than a predetermined value. Processing is stopped when the fourth assessment unit judges that the distance is equal to or greater than the specified value. [9] Information processing device according to any one of claims 1 to 8, wherein the calculation unit includes a fifth assessment unit to assess whether the calculated aggregated overlap index is equal to or greater than a predetermined value, and wherein, if the fifth assessment unit judges that the calculated overall overlap index is equal to or greater than the specified value, the third determination unit determines the corresponding relationship between the parts in the drawing acquired by the first determination unit and the parts in the image acquired by the second acquisition unit, on the basis of the aggregated overlap index which the fifth assessment unit judges to be equal to or greater than the specified value. [10] Information processing techniques, comprehensive: a first processing step to acquire initial information, which is information about a drawing of an object with a multitude of parts and the information about the multitude of parts is contained in the drawing; a second processing step to acquire second pieces of information, which include information about an image of the object with a coordinate system that differs from that of the drawing, and information about a multitude of parts in the image; a third processing step in which parts in the drawing and parts in the image are selected and a preliminary corresponding relationship between the parts in the drawing and the parts in the image is determined using the first information and the second information; a fourth processing step to generate a preliminary superimposed diagram by aligning the coordinate system of the drawing and the coordinate system of the image, and thus superimposing the drawing and the image on the basis of the preliminary corresponding relationship; a fifth processing step to calculate overlap indices that show overlaps between the parts in the drawing and the parts in the image in the preliminary overlaid diagram using the first information and the second information, and to determine the parts in the image that correspond to the parts in the drawing in the preliminary overlaid diagram based on the overlap indices; a sixth processing step for calculating the overlap indices used when the parts in the image corresponding to the parts in the drawing in the preliminary overlaid diagram are determined in the fifth processing step, and for calculating an aggregated overlap index of the preliminary overlaid diagram; and a seventh processing step to determine a corresponding relationship between the parts in the drawing acquired in the first processing step and the parts in the image acquired in the second processing step, based on the aggregated overlap index. [11] Information processing program that causes a computer to execute: a first processing step to acquire initial information, which is information about a drawing of an object with a multitude of parts and the information about the multitude of parts is contained in the drawing; a second processing step to acquire second pieces of information, which include information about an image of the object with a coordinate system that differs from that of the drawing, and information about a multitude of parts in the image; a third processing step in which parts in the drawing and parts in the image are selected and a preliminary corresponding relationship between the parts in the drawing and the parts in the image is determined using the first information and the second information; a fourth processing step to generate a preliminary superimposed diagram by aligning the coordinate system of the drawing and the coordinate system of the image, and thus superimposing the drawing and the image on the basis of the preliminary corresponding relationship; a fifth processing step to calculate overlap indices that show overlaps between the parts in the drawing and the parts in the image in the preliminary overlaid diagram using the first information and the second information, and to determine the parts in the image that correspond to the parts in the drawing in the preliminary overlaid diagram based on the overlap indices; a sixth processing step for calculating the overlap indices used when the parts in the image corresponding to the parts in the drawing in the preliminary overlaid diagram are determined in the fifth processing step, and for calculating an aggregated overlap index of the preliminary overlaid diagram; and a seventh processing step to determine a corresponding relationship between the parts in the drawing acquired in the first processing step and the parts in the image acquired in the second processing step, based on the aggregated overlap index.

Images