Image processing apparatus, image processing system, image processing method, and program

Abstract

To provide an image processing device capable of comparing a three-dimensional CAD model with an object by using a photographed image of the object photographed at a place different from a top of a predetermined stage.SOLUTION: An image processing device includes: an information acquisition part for acquiring posture information on an object based on a photographed image of the object; a reference image generation part for generating a reference image of the object using the three-dimensional data on the object and the attitude information; and an image matching part for outputting a matching image in which a comparison image based on the photographed image and the reference image are overlapped with each other with their corresponding points aligned.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program.

Background Art

[0002] In non-destructive inspection, in order to confirm the difference between an object to be inspected and a design value, a 3D (Three Dimensions) CAD (Computer Aided Design) model having design information is used to compare the difference with a photographed image of the object, thereby performing the inspection.

[0003] For example, a technique is known in which a 3D CAD model is oriented, a test object is placed on a stage in a direction substantially matching the direction of the CAD model, and the CAD model and the test object are superposed with corresponding points aligned (see, for example, Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique disclosed in Patent Document 1, there is a problem that the CAD model and the object cannot be compared unless the object is placed on a predetermined stage in a direction substantially matching the 3D CAD model.

[0005] One embodiment of the present invention has been made in view of the above problems, and provides an image processing apparatus capable of comparing a 3D CAD model with an object using a photographed image of the object taken at a location different from a predetermined stage.

Means for Solving the Problems

[0006] To solve the above problems, an image processing apparatus according to an embodiment is based on a photographed image of an object, and the Image feature amount and posture information and of the object is obtained by an information acquisition unit, and A comparison image generation unit that generates a comparison image based on the captured image using the posture information and the image feature amount;A reference image generation unit that generates a reference image of the object using the three-dimensional data of the object and the pose information; the An image matching unit that outputs a matching image in which the comparison image and the reference image are superimposed with corresponding points aligned.

Advantages of the Invention

[0007] According to an embodiment of the present invention, it is possible to provide an image processing apparatus that can compare a three-dimensional CAD model with an object using a captured image of the object captured at a location different from a predetermined stage.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. <System Configuration> FIG. 1 is a diagram showing an example of the system configuration of an image processing system according to an embodiment. As an example, the image processing system 1 includes, as shown in FIG. 1, a camera 110 that photographs an object 12 to be assembled by an assembly worker 11, an inspection terminal 120 used by an inspection person 13, and an image processing apparatus 100.

[0010] The camera 110 transmits an image of the object 12 photographed during the assembly operation to the image processing apparatus 100 via the communication network 10.

[0011] The image processing apparatus 100 is a system including a computer or a plurality of computers. The image processing apparatus 100 generates a matching image that compares a three-dimensional CAD model with the object 12 based on the input image received from the camera 110 via the communication network 10, and transmits the generated matching image to the inspection terminal 120 via the communication network 10.

[0012] The inspection terminal 120 displays the matching image received from the image processing apparatus 100 via the communication network 10 on, for example, a display. The inspection person 13 determines whether the object assembled by the assembly worker 11 is good or bad (good product or defective product) from the matching image displayed on the display of the inspection terminal 120.

[0013] As another example, the image processing apparatus 100 may digitize the difference between the three-dimensional CAD model and the object 12, compare it with a threshold value to determine the quality of the object 12, and transmit the determination result together with the matching image to the inspection terminal 120.

[0014] Note that the system configuration of the image processing system 1 shown in FIG. 1 is an example. For example, in a small-scale system, the image processing apparatus 100 and the inspection terminal 120 may be the same information processing apparatus.

[0015] <Hardware Configuration> The image processing apparatus 100 and the inspection terminal 120 in FIG. 1 have, for example, the hardware configuration of a computer 200 as shown in FIG. 2. Note that the image processing apparatus 100 may be configured by a plurality of computers 200.

[0016] FIG. 2 is a diagram showing an example of the hardware configuration of a computer according to an embodiment. The computer 200 includes, for example, a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an HD (Hard Disk) 204, an HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 207, a network I / F 208, a keyboard 209, a pointing device 210, a DVD-RW (Digital Versatile Disk Rewritable) drive 212, a media I / F 214, a GPU (Graphics Processing Unit) 215, and a bus line 216, etc.

[0017] Among these, the CPU 201 controls the operation of the entire computer 200. The ROM 202 stores programs used for starting the computer 200, such as an IPL (Initial Program Loader). The RAM 203 is used, for example, as a work area for the CPU 201. The HD 204 stores programs such as an OS (Operating System), applications, device drivers, etc., and various data. The HDD controller 205 controls, for example, the reading or writing of various data to and from the HD 204 according to the control of the CPU 201.

[0018] The display 206 displays various types of information such as, for example, a cursor, menu, window, characters, or images. Note that the display 206 may be provided outside the computer 200. The external device connection I / F 207 is an interface such as, for example, a USB (Universal Serial Bus) for connecting various external devices to the computer 200. The network I / F 208 is an interface for communicating with other devices using, for example, the communication network 10 or the like.

[0019] The keyboard 209 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 210 is a type of input means for performing selections and executions of various instructions, selections of processing targets, movements of the cursor, and the like. Note that the keyboard 209 and the pointing device 210 may be provided outside the computer 200. The DVD-RW drive 212 controls reading or writing of various data with respect to the DVD-RW 211 as an example of a removable storage medium. Note that the DVD-RW 211 is not limited to a DVD-RW and may be other storage media.

[0020] The media I / F 214 controls reading or writing (storage) of data with respect to the media 213 such as a flash memory. The GPU 215 is a processor that executes image processing such as, for example, 3D graphic processing or extraction processing of image feature amounts, faster than the CPU 201. The bus line 216 includes an address bus, a data bus, and various control signals for electrically connecting the above-described respective components.

[0021] Note that the hardware configuration of the computer 200 shown in FIG. 2 is an example. The computer 200 may have any other configuration as long as it has, for example, the CPU 201, the ROM 202, the RAM 203, the network I / F 208, and the bus line 216.

[0022] <Functional Configuration> FIG. 3 is a diagram showing an example of the functional configuration of an image processing apparatus according to an embodiment. The image processing apparatus 100 realizes, for example, an image acquisition unit 301, a CAD data management unit 302, a comparison image generation unit 303, a reference image generation unit 304, an image matching unit 305, a determination unit 306, etc. when the CPU 201 executes a predetermined program. Note that at least a part of each of the above functional configurations may be realized by hardware.

[0023] The image acquisition unit 301 executes an image acquisition process for acquiring a captured image of the object 12 and information on the object 12 from an input image received from the camera 110 via the communication network 10. Here, the captured image of the object 12 is, for example, an image obtained by trimming the area in the input image where the object 12 appears. The image acquisition unit 301 acquires a captured image of the object 12 by trimming, from the captured image, the area obtained by, for example, an object position type estimation technique, etc. from the input image, and transmits the acquired captured image of the object 12 to the comparison image generation unit 303. Note that the image acquisition unit 301 may acquire a captured image of the object 12 by specifying the area in the input image where the object 12 appears using, for example, a general pattern matching technique, etc. and trimming the specified area.

[0024] Also, the information on the object 12 is, for example, identification information for specifying the object 12 or information such as a name. The image acquisition unit 301 specifies the object 12 using, for example, an object position type estimation method using deep learning, etc. from the image of the object 12, and transmits the information on the specified object 12 to the CAD data management unit 302. Note that the image acquisition unit 301 may specify the object 12 using, for example, a general pattern matching technique, etc., or when the information on the object 12 is preset in the image processing apparatus 100, the process of acquiring the information on the object 12 may be omitted.

[0025] The CAD data management unit 302 stores and manages the three-dimensional CAD data (3D data) of the object 12, for example, in a storage device or the like provided in the image processing apparatus 100. Further, when the CAD data management unit 302 receives information on the object 12 from the image acquisition unit 301, it transmits the three-dimensional CAD data of the object 12 to, for example, the comparison image generation unit 303, the reference image generation unit 304, or the like. Note that the CAD data management unit 302 may acquire the three-dimensional CAD data of the object 12 from an external server or a cloud service that can communicate via the communication network 10 instead of the storage device provided in the image processing apparatus 100.

[0026] The comparison image generation unit 303 generates a comparison image of the object 12 and pose information of the object 12 based on the captured image of the object 12 received from the image acquisition unit 301. Here, as an example, the comparison image is an image obtained by correcting the imaging conditions such as the background and light reflection from the captured image of the object 12. The pose information is vector information indicating the three-dimensional orientation of the object 12 in the captured image of the object 12. Note that a plurality of embodiments of the method for generating the comparison image and the pose information of the object 12 will be exemplified and described later.

[0027] The reference image generation unit 304 executes a reference image generation process for generating a reference image of the object 12 using the three-dimensional CAD data (3D data) of the object 12 received from the CAD data management unit 302 and the pose information of the object 12 received from the comparison image generation unit 303. For example, the reference image generation unit 304 generates a reference image by rotating the three-dimensional CAD data using the pose information and performing rendering, and transmits the generated reference image to the image matching unit 305.

[0028] The image matching unit 305 performs an image matching process of outputting a matching image in which the comparison image of the object 12 received from the comparison image generation unit 303 and the reference image received from the reference image generation unit 304 are superimposed with corresponding points aligned. This matching image is an image representing the difference between the comparison image and the reference image. For example, it is an image obtained by calculating the difference between the pixel values of the comparison image and the reference image and representing the calculated difference between each pixel value as a heat map. The image matching unit 305 transmits (outputs) the matching image to the inspection terminal 120 via, for example, the communication network 10. Alternatively, the image matching unit 305 may display (output) the matching image on a display 206 or the like provided in the image processing apparatus.

[0029] The determination unit 306 performs a determination process of quantifying the difference between the comparison image and the reference image of the object 12 and comparing it with a threshold value to determine the quality of the object 12. Note that the determination unit 306 is optional, and the image processing apparatus 100 may not have the determination unit 306.

[0030] Subsequently, a plurality of embodiments of the comparison image generation unit 303 will be illustrated and described.

[0031] [First Embodiment] <Functional Configuration of Comparison Image Generation Unit> (Functional Configuration 1) FIG. 4 is a diagram (1) showing an example of the functional configuration of the comparison image generation unit according to the first embodiment. This figure shows the minimum configuration of the comparison image generation unit 303 according to the first embodiment. In the example of FIG. 4, the comparison image generation unit 303 includes an information acquisition unit 410, an information storage unit 401, an image generation unit 420, a comparison image storage unit 402, and the like.

[0032] The information acquisition unit 410 acquires the image feature amount and the pose information of the object 12 based on the captured image of the object 12 received from the image acquisition unit 301. For example, the information acquisition unit 410 inputs the captured image of the object 12 received from the image acquisition unit 301 into the learned first deep learning model 411 to acquire the image feature amount and the pose information of the object 12. Here, the first deep learning model 411 is a deep learning model that has been pre-learned using the 3D CAD data of the object 12, the learning images of the object 12 under a plurality of poses and shooting conditions, and the correct pose information of the object 12. Note that the learning process of the first deep learning model will be described later.

[0033] Here, the image feature amount is a vector indicating the features of the image extracted by the first deep learning model 411. For example, if the output of the first deep learning model 411 is a 256-dimensional vector, the first 3-dimensional vector is the pose information, and the remaining 253-dimensional vector is the image feature amount. The information acquisition unit 410 stores the acquired pose information and image feature amount of the object 12 in, for example, the information storage unit 401 or the like.

[0034] The information storage unit 401 is realized by a storage device or the like provided in the image processing apparatus 100, and stores the pose information and the image feature amount of the object 12. The comparison image generation unit 303 outputs the pose information of the object 12 stored in the information storage unit 401 to the reference image generation unit 304.

[0035] The image generation unit 420 generates a comparison image of the object 12 using the pose information and the image feature amount of the object 12 stored in the information storage unit 401. For example, the image generation unit 420 inputs the pose information and the image feature amount of the object 12 into the learned second deep learning model 421 to generate a comparison image of the object 12. Here, the second deep learning model 421 is a deep learning model that has been pre-learned so that the error between the generated comparison image and the rendering image created using the 3D CAD data of the object 12 is equal to or less than a threshold value.

[0036] Here, the rendering image is an image of the object 12 viewed from a plurality of poses, created using the three-dimensional CAD data of the object 12. Preferably, the rendering image includes images in which a part of the parts of the object 12 is missing or the attachment position is shifted. The image generation unit 420 stores the generated comparison image in, for example, the comparison image storage unit 402 or the like.

[0037] The comparison image storage unit 402 is realized by a storage device or the like provided in the image processing apparatus 100, and stores the comparison image of the object 12. The comparison image generation unit 303 outputs the comparison image of the object 12 stored in the comparison image storage unit 402 to the image matching unit 305 or the like.

[0038] <Flow of processing> Subsequently, the flow of processing of the image processing method according to the first embodiment will be described. (Image matching processing) FIG. 5 is a flowchart showing an example of the image matching processing according to the first embodiment. As shown in FIG. 1, this processing shows an example of image matching processing in which the image processing apparatus 100 generates a matching image for comparing the three-dimensional CAD model and the object 12 based on the input image received from the camera 110 and outputs it to the inspection terminal 120.

[0039] In step S501, the image acquisition unit 301 acquires the input image from the camera 110 via the communication network 10.

[0040] In step S502, the image acquisition unit 301 acquires a photographed image of the object 12 obtained by trimming the area in the input image where the object 12 is shown. Further, the image acquisition unit 301 acquires information of the object 12 (for example, identification information of the object 12) based on the acquired photographed image of the object 12.

[0041] The image acquisition unit 301 outputs the captured image of the object 12 it has acquired to the comparison image generation unit 303. Also, the image acquisition unit 301 outputs the information of the acquired object 12 to the CAD data management unit 302. As a result, the CAD data management unit 302 outputs the three-dimensional CAD data of the object 12 to the comparison image generation unit 303, the reference image generation unit 304, etc.

[0042] In step S503, the information acquisition unit 410 of the comparison image generation unit 303 uses the captured image of the object 12 received from the image acquisition unit 301 to acquire the pose information and image feature amount of the object 12. For example, the information acquisition unit 410 inputs the captured image of the object 12 into the learned first deep learning model 411, acquires the pose information and image feature amount output by the first deep learning model 411, and stores the acquired pose information and image feature amount in the information storage unit 401. As a result, the comparison image generation unit 303 outputs the pose information of the object 12 stored in the information storage unit 401 to the reference image generation unit 304.

[0043] In step S504, the image generation unit 420 of the comparison image generation unit 303 uses the pose information and image feature amount of the object 12 stored in the information storage unit 401 to generate a comparison image of the object 12. For example, the image generation unit 420 inputs the pose information and image feature amount of the object 12 into the learned second deep learning model 421, acquires the comparison image output by the second deep learning model 421, and stores the acquired comparison image in the comparison image storage unit 402. As a result, the comparison image generation unit 303 outputs the comparison image of the object 12 stored in the comparison image storage unit 402 to the image matching unit 305.

[0044] Also, in parallel with the process of step S504, in step S505, the reference image generation unit 304 generates a reference image of the object 12 using the three-dimensional CAD data received from the CAD data management unit 302 and the pose information of the object 12 received from the comparison image generation unit 303. For example, the reference image generation unit 304 generates a reference image by rotating the three-dimensional CAD data of the object 12 using the pose information of the object 12 and performing rendering, and outputs the generated reference image to the image matching unit 305.

[0045] In step S506, the image matching unit 305 generates a matching image by superimposing the comparison image received from the comparison image generation unit 303 and the reference image received from the reference image generation unit 304 with corresponding points aligned.

[0046] In step S507, the image matching unit 305 outputs (transmits) the generated matching image to, for example, the inspection terminal 120 or the like.

[0047] FIG. 6 is a diagram showing an image of each image according to the first embodiment. The input image 610 is an image obtained by the camera 110 photographing the object 12 and the periphery of the object 12. This input image 610 may include, for example, an object other than the object 12 such as a background 611, a workbench 612 for assembling the object 12, or an assembly worker. Further, in the input image 610, depending on the photographing conditions such as the position of the illumination, a shadow or reflected light may be reflected, and the color of the object 12 may not be uniform.

[0048] The photographed image 620 is an image obtained by trimming the portion of the input image 610 in which the object 12 is reflected. Therefore, the photographed image 620 still has the influence of the background 611 or the photographing conditions.

[0049] The comparison image 630 is an image generated by the comparison image generation unit 303 from the image feature amount and the pose information of the object 12 in step S504 of FIG. 5. Therefore, since the background of the comparison image 630 is removed and it is an image under certain illumination conditions, the comparison image 630 is an image with reduced color non-uniformity and the like due to the shooting conditions.

[0050] The reference image 640 is an image obtained by rendering the three-dimensional CAD data serving as a reference for the object 12 using the pose information of the object 12 by the reference image generation unit 304 in step S505 of FIG. 5. Therefore, the reference image 640 is an image without an unnecessary background and not affected by the illumination conditions.

[0051] The matching image 650 is an image representing the difference between the comparison image 630 and the reference image 640, which is obtained by the image matching unit 305 superposing the comparison image 630 and the reference image 640 with corresponding points aligned in step S506 of FIG. 5. In the example of FIG. 6, since the attachment positions of the parts 651 are different between the comparison image 630 and the reference image 640, the different parts 651 are highlighted (for example, color-coded, or hatched, etc.).

[0052] As described above, according to the first embodiment, it is possible to provide the image processing apparatus 100 that can compare the three-dimensional CAD model and the object 12 using the captured image of the object 12 captured at a location different from the predetermined stage.

[0053] (Functional Configuration 2) FIG. 7 is a diagram (2) showing an example of the functional configuration of the comparison image generation unit according to the first embodiment. The first deep learning model 411 and the second deep learning model 421 need to be learned before executing the image matching process shown in FIG. 5. FIG. 7 shows an example of the functional configuration of the comparison image generation unit 303 when the learning processes of the first deep learning model 411 and the second deep learning model 421 are executed.

[0054] When executing the learning processes of the first deep learning model 411 and the second deep learning model 421, the comparison image generation unit 303 has a learning control unit 700 in addition to the functional configuration of the comparison image generation unit 303 described with reference to FIG. 4. Note that the learning control unit 700 may be provided outside the comparison image generation unit 303.

[0055] The learning control unit 700 includes, for example, a learning data storage unit 701 and an error calculation unit 702. The learning data storage unit 701 is realized by, for example, a storage device included in the image processing apparatus 100, and stores learning data used in the learning processes of the first deep learning model 411 and the second deep learning model 421. The learning data stored in the learning data storage unit 701 includes, for example, rendering images, learning images, and correct pose information.

[0056] FIG. 8 is a diagram for explaining the learning data according to the first embodiment. The learning data storage unit 701 stores learning data 800 that stores a rendering image, a learning image, and correct pose information in association with a plurality of data IDs, as shown in FIG. 8, for example.

[0057] The rendering image is an image obtained by rotating three-dimensional CAD data of the object 12 using the correct pose information and performing rendering. This rendering image includes, for example, an image in which a part of a part is missing, an image in which the attachment position of a part is shifted, or an image in which an unnecessary part is attached. The learning image is an image obtained by changing the background or illumination conditions of the rendering image. The correct pose information is pose information representing the poses of the rendering image and the learning image.

[0058] Here, returning to FIG. 7, the description of the learning control unit 700 is continued. The learning control unit 700 inputs the learning image stored in the learning data storage unit 701 to the first deep learning model. Further, the information acquisition unit 410 stores the pose information and the image feature amount output by the first deep learning model 411 in the information storage unit 401.

[0059] The error calculation unit 702 calculates posture error information, which is the vector difference between the posture information stored in the information storage unit 401 by the information acquisition unit 410 and the correct posture information. The learning control unit 700 (or the information acquisition unit 410) updates the weights of the first deep learning model 411 so that the value of the posture error information becomes smaller.

[0060] In addition, the image generation unit 420 inputs the posture information stored in the information storage unit 401 and the image feature amount into the second deep learning model 421, acquires a comparison image output by the second deep learning model 421, and stores the acquired comparison image in the comparison image storage unit 402.

[0061] The error calculation unit 702 calculates image error information, which is the difference (for example, the difference in all pixel values) between the comparison image stored in the comparison image storage unit 402 and the rendering image. The learning control unit 700 (or the image generation unit 420) updates the weights of the second deep learning model so that the value of the image error information becomes smaller.

[0062] (Flow of learning process) FIG. 9 is a flowchart showing an example of the learning process according to the first embodiment. FIG. 9(A) shows an example of a learning process in which the image processing apparatus 100 having the learning control unit 700 described in FIG. 7 learns the first deep learning model 411 and the second deep learning model 421.

[0063] Note that the image processing apparatus 100 that executes the learning process shown in FIG. 9(A) may be the same image processing apparatus as the image processing apparatus 100 that executes the image matching process described in FIG. 5, or may be a different image processing apparatus. Therefore, the image processing apparatus 100 that executes the image matching process described in FIG. 5 may or may not have the learning control unit 700 described in FIG. 7.

[0064] In step S901, the learning control unit 700 outputs the learning data stored in the learning data storage unit 701 to a predetermined output destination. For example, the learning control unit 700 outputs the learning image stored in the learning data storage unit 701 to the information acquisition unit 410, and outputs the correct posture information and the rendering image stored in the learning data storage unit 701 to the error calculation unit 702.

[0065] In step S902, the information acquisition unit 410 acquires posture information and image feature amounts using the first deep learning model 411. For example, the information acquisition unit 410 inputs the learning image received from the learning control unit 700 into the first deep learning model 411, acquires the posture information and the image feature amounts output by the first deep learning model 411, and stores the acquired posture information and image feature amounts in the information storage unit 401.

[0066] In step S903, the image generation unit 420 generates a comparison image using the second deep learning model 421. For example, the image generation unit 420 inputs the posture information and the image feature amounts stored in the information storage unit 401 into the second deep learning model 421, acquires the comparison image output by the second deep learning model, and stores the acquired comparison image in the comparison image storage unit 402.

[0067] In step S904, the learning control unit 700 updates the weights of the first deep learning model 411 and the second deep learning model 421 based on the error information calculated by the error calculation unit 702 so that the error information becomes smaller. For example, the error calculation unit 702 calculates posture error information, which is the error between the posture information stored in the information storage unit 401 and the correct posture information, and the learning control unit 700 (or the information acquisition unit 410) updates the weights of the first deep learning model 411 so that the posture error information becomes smaller. Further, the error calculation unit 702 calculates image error information, which is the error between the comparison image stored in the comparison image storage unit 402 and the rendering image, and the learning control unit 700 (or the information acquisition unit 410) updates the weights of the second deep learning model 421 so that the image error information becomes smaller.

[0068] In step S905, the learning control unit 700 determines whether the number of repetitions n of the processes in steps S901 to S904 has reached a predetermined number of times N. If the number of repetitions n has not reached the predetermined number of times N, the learning control unit 700 returns the process to step S901 and executes the processes in steps S901 to S904 again. On the other hand, if the number of repetitions n has reached the predetermined number of times N, the learning control unit 700 shifts the process to step S906.

[0069] When shifting to step S906, the learning control unit 700 determines the weights of the first deep learning model 411 and the second deep learning model 421.

[0070] FIG. 9(B) shows another example of a learning process in which the image processing apparatus 100 having the learning control unit 700 described in FIG. 7 learns the first deep learning model 411 and the second deep learning model 421. Among the processes shown in FIG. 9(B), the processes in steps S901 to A904 and step S906 are the same as the processes described in FIG. 9(A), and thus the description thereof is omitted here.

[0071] In step S910, the learning control unit 700 determines whether the errors of the posture error information and the image error information have become equal to or less than a threshold value. For example, the learning control unit 700 determines that the error has become equal to or less than the threshold value when the error of the posture error information is equal to or less than a first preset threshold value and the error of the image error information is equal to or less than a second preset threshold value.

[0072] If the error has not become equal to or less than the threshold value, the learning control unit 700 returns the process to step S901 and executes the processes in steps S901 to S904 again. On the other hand, if the error has become equal to or less than the threshold value, the learning control unit 700 shifts the process to step S906.

[0073] Through the process of FIG. 9(A) or FIG. 9(B), the learning control unit 700 can learn the first deep learning model 411 and the second deep learning model 421.

[0074] [Second Embodiment] In the first embodiment, an example of the process for acquiring the pose information of the object 12 using a deep learning model was described. In the second embodiment, an example of the process for acquiring the pose information of the object 12 by comparing a wireframe image created from three-dimensional CAD data with line segments extracted from a captured image (or input image) of the object 12 will be described.

[0075] (Overview of the Process) FIG. 10 is a diagram for explaining the pose information acquisition process according to the second embodiment. Note that the image processing apparatus 100 according to the second embodiment has, for example, a functional configuration as shown in FIG. 3.

[0076] The comparison image generation unit 303 creates, for example, a wireframe image of the object 12 as viewed from a plurality of viewpoints 1, 2, 3,... as shown in FIG. 10(B) using the three-dimensional CAD data of the object 12 as shown in FIG. 10(A). Preferably, the comparison image generation unit 303 extracts a wireframe from the three-dimensional CAD data, removes hidden lines, and then creates a rendered wireframe image. Note that a method for creating a wireframe image with hidden lines removed is known and can be executed using, for example, general CAD software or the like.

[0077] Also, the comparison image generation unit 303 creates, for example, a first line segment angle table 1001 that records the angles between line segments at each vertex for each of the plurality of viewpoints 1, 2, 3,... as shown in FIG. 10(C).

[0078] Also, in parallel with the processes (A) to (C) above, the comparison image generation unit 303 extracts straight lines from, for example, an input image (or a captured image of the object 12) as shown in FIG. 10(D) and creates, for example, a line segment image as shown in FIG. 10(E). Note that a method for extracting straight lines from an image is known, and for example, a method for detecting straight lines by Hough transform is known.

[0079] Furthermore, as shown in (F) of FIG. 10, for example, the comparison image generation unit 303 creates a second line segment angle table 1002 that records the angles between line segments at each vertex of the line segment image.

[0080] Subsequently, the comparison image generation unit 303 estimates the viewpoint information V by comparing the angle distribution T_v{1,2,3,···} included in the first line segment angle table 1001 with the angle distribution T_input included in the second line segment angle table 1002 and narrowing down v. As a result, the comparison image generation unit 303 can obtain viewpoint information that is close to the way the object 12 appears on the input image among the plurality of viewpoints 1, 2, 3, ···.

[0081] Here, the viewpoint information is represented by the relative position and rotation between the reference point of the object on the electronic data such as an image or CAD data and the viewpoints (for example, a plurality of viewpoints 1, 2, 3, ··· etc.) on the electronic data.

[0082] Based on the obtained viewpoint information V, the comparison image generation unit 303 can obtain an image close to the line segment image P' by performing an affine transformation A such as translation, scaling, rotation, and shear on the wireframe image V' corresponding to V.

[0083] The pose information P corresponding to this line segment image P represents the relative pose of the 3D CAD data and the object 12 in the input image. Here, the pose information refers to the combination of the reference viewpoint information V and the affine transformation A, but the pose information can also be mutually converted with a single viewpoint information.

[0084] <Flow of processing> (Processing for obtaining pose information) FIG. 11 is a flowchart showing an example of the processing for obtaining pose information according to the second embodiment. Since the basic processing content is the same as the outline of the processing described in FIG. 10, a detailed description of the same processing content is omitted here.

[0085] In step S1001, the comparison image generation unit 303 acquires the three-dimensional CAD data of the object 12 from the CAD data management unit 302.

[0086] In step S1002, the comparison image generation unit 303 generates (extracts) a wireframe from the three-dimensional CAD data of the object 12.

[0087] In step S1003, the comparison image generation unit 303 may (or may not) perform decimation of the wireframe. For example, in a complex CAD model, the comparison image generation unit 303 may perform a decimation process of discriminating and extracting main line segments from the wireframe in consideration of the line segment length and the distribution in the three-dimensional space in order to accurately and efficiently estimate the viewpoints.

[0088] In step S1004, the comparison image generation unit 303 creates a wireframe image with hidden lines removed as seen from a plurality of viewpoints 1, 2, 3,... by projection calculation, for example, as described in FIG. 10(B).

[0089] In step S1005, the comparison image generation unit 303 creates, for example, as described in FIG. 10(C), the first line segment angle table 1001 and stores it in a database (DB).

[0090] In step S1006, the comparison image generation unit 303 stores the wireframe image created in step S1004 in the database.

[0091] In step S1011, the comparison image generation unit 303 acquires an input image from the camera 110 (or a photographed image of the object 12 from the image acquisition unit 301).

[0092] In step 1012, the comparison image generation unit 303 extracts a linear component from the input image (or the captured image of the object 12) and creates a line segment image, for example, as described with reference to FIG. 10(E). Further, based on the created line segment image, the comparison image generation unit 303 creates a second line segment angle table 1002, for example, as described with reference to FIG. 10(F).

[0093] In steps S1013 and S1014, the comparison image generation unit 303 compares the first line segment angle table 1001 acquired from the database with the created second line segment angle table 1002 and estimates the viewpoint, for example, as described with reference to FIG. 10(G).

[0094] In steps S1015 and S1016, the comparison image generation unit 303 acquires the wireframe image stored in step S1006 from the database and acquires the wireframe image of the estimated viewpoint therefrom.

[0095] In step S1017, the comparison image generation unit 303 performs an affine transformation A on the wireframe image V' acquired in step S1016 to obtain an affine transformation A that gives an image closest to the line segment image P' created in step S1012.

[0096] In step S1018, the comparison image generation unit 303 outputs, for example, the reference viewpoint information V and the posture information represented by the affine transformation A from the reference viewpoint to the reference image generation unit 304 and the like.

[0097] Through the above processing, the comparison image generation unit 303 can acquire the posture information of the object 12 without relying on the deep learning model.

[0098] (Image matching process) FIG. 12 is a flowchart showing an example of image matching processing according to the second embodiment. Among the processes shown in FIG. 12, the processes of steps S501, S502, S505 to S507 are the same as the image matching processing according to the first embodiment described with reference to FIG. 5, and thus the description thereof is omitted here.

[0099] In step S1201, the comparison image generation unit 303 executes, for example, the posture information acquisition process as shown in FIG. 11.

[0100] In step S1202, the comparison image generation unit 303 generates a comparison image of the object 12. In the second embodiment, the comparison image generation unit 303 may use, for example, the captured image 620 of the object 12 as shown in FIG. 6 as the comparison image without using a deep learning model. Alternatively, the comparison image generation unit 303 may generate a comparison image of the object 12 by performing various image processes (for example, background removal, noise removal, etc.) on the captured image 620 of the object 12 as shown in FIG. 6.

[0101] According to the second embodiment, by the above-described processing, the image processing apparatus 100 can execute the posture information acquisition process and the image matching process without depending on a deep learning model.

[0102] As another example, the image processing apparatus 100 may improve the acquisition accuracy of the generated information by performing the posture information acquisition process according to the second embodiment simultaneously or in advance in addition to the posture information acquisition process according to the first embodiment and comparing the results.

[0103] As described above, according to each embodiment of the present invention, it is possible to provide an image processing apparatus 100, an image processing system 1, an image processing method, etc., which can compare a three-dimensional CAD model with an object 12 using a captured image of the object 12 captured at a location different from a predetermined stage.

[0104] <Supplementary Note> Each function of each of the embodiments described above can be realized by one or more processing circuits. Here, the "processing circuit" in this specification refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit) designed to execute each function described above, a DSP (digital signal processor), an FPGA (field programmable gate array), and devices such as conventional circuit modules.

[0105] Also, the device group described in the embodiments merely shows one of a plurality of computing environments for implementing the embodiments disclosed in this specification. In one embodiment, the image processing apparatus 100 includes a plurality of computing devices such as a server cluster. The plurality of computing devices are configured to communicate with each other via an arbitrary type of communication link including a network or a shared memory, and perform the processing disclosed in this specification. Further, each element of the image processing apparatus 100 may be integrated into one information processing apparatus or divided into a plurality of information processing apparatuses.

[0106] Also, the image processing apparatus 100 shown in FIG. 3 can be configured to share the processing of the image processing apparatus 100 shown in FIGS. 5, 9, 11, and 12 in various combinations. For example, at least a part of the processing executed by the image processing apparatus 100 may be executed by the inspection terminal 120. Also, at least a part of the processing executed by the image processing apparatus 100 may be executed by, for example, an external server device or a cloud service.

Explanation of Reference Numerals

[0107] 1 Image processing system 12 Object 100 Image processing apparatus 110 Camera 120 Inspection terminal 301 Image acquisition unit 304 Reference image generation unit 305 Image matching unit 306 Judgment unit 410 Information acquisition unit 411 First deep learning model 420 Image generation unit 421 Second deep learning model 620 Captured image 640 Reference image 650 Matched image

Prior art documents

Patent documents

[0108]

Patent Document 1

Claims

1. An information acquisition unit that acquires the image feature amount and the pose information of the object based on the photographed image of the object; A comparison image generation unit that generates a comparison image based on the photographed image by using the pose information and the image feature amount; A reference image generation unit that generates a reference image of the object by using the three-dimensional data of the object and the pose information; An image matching unit that outputs a matching image in which the comparison image and the reference image are superimposed with corresponding points aligned; An image processing apparatus having the above.

2. The information acquisition unit inputs the photographed image into a learned first deep learning model to acquire the pose information and the image feature amount of the object. The image processing apparatus according to claim 1.

3. The first deep learning model is a learned deep learning model that is learned by using a plurality of poses of the object, learning images of the object under shooting conditions, and correct pose information indicating the pose of the object, using the three-dimensional data of the object. The image processing apparatus according to claim 2.

4. The comparison image generation unit generates the comparison image by inputting the pose information and the image feature amount into a learned second deep learning model. The image processing apparatus according to claim 2 or 3.

5. The second deep learning model is a learned deep learning model that is learned by using the pose information and the image feature amount output by the first deep learning model and a rendering image of the object. The image processing apparatus according to claim 4.

6. The information acquisition unit compares a wireframe image based on the three-dimensional data of the object with a line segment image extracted from the photographed image of the object to acquire the pose information. The image processing apparatus according to claim 1.

7. The image matching unit outputs the matching image representing the difference between the comparison image and the reference image. The image processing apparatus according to any one of claims 1 to 6.

8. The image processing apparatus according to any one of claims 1 to 7, further comprising a determination unit that quantifies the difference between the comparison image and the reference image and compares the result with a threshold value to determine whether the object is good or bad.

9. The image processing apparatus according to any one of claims 1 to 8, further comprising an image acquisition unit that acquires the object and an input image obtained by photographing the periphery of the object, extracts the photographed image from the input image, and acquires information on the object.

10. An image processing system including the image processing apparatus according to claim 2 or 3, having a learning control unit that controls learning of the first deep learning model using a plurality of poses created using the three-dimensional data of the object, learning images of the object under imaging conditions, and correct pose information indicating the pose of the object. An image processing system.

11. An image processing system including the image processing apparatus according to claim 4 or 5, having a learning control unit that controls learning of the second deep learning model using the pose information and the image feature amount output by the first deep learning model and a rendering image of the object. An image processing system.

12. An information acquisition unit that acquires an image feature amount and pose information of the object based on a captured image of the object, A comparison image generation unit that generates a comparison image based on the captured image using the pose information and the image feature amount, A reference image generation unit that generates a reference image of the object using the three-dimensional data of the object and the pose information, An image matching unit that outputs a matching image in which the comparison image and the reference image are superimposed with corresponding points aligned, An image processing system having the above.

13. The image processing apparatus, A process of acquiring an image feature amount and pose information of the object based on a captured image of the object, A process of generating a comparison image based on the captured image using the pose information and the image feature amount, A process of generating a reference image of the object using the three-dimensional data of the object and the pose information, A process of outputting a matching image in which the comparison image and the reference image are superimposed with corresponding points aligned, An image processing method for executing the above.

14. On a computer, A process of acquiring an image feature amount and pose information of the object based on a captured image of the object, A process of generating a comparison image based on the captured image using the pose information and the image feature amount, A process of generating a reference image of the object using the three-dimensional data of the object and the pose information, A process of outputting a matching image in which the comparison image and the reference image are superimposed with corresponding points aligned, A program for executing the above.

Images