IMAGE PROCESSING APPARATUS, IMAGE PROCESSING SYSTEM, IMAGE PROCESSING METHOD, AND PROGRAM

Abstract

To provide an image processing apparatus capable of easily determining whether or not an object to be inspected is correctly assembled even when the object has parts different from a model created by CAD or the like.SOLUTION: An image processing apparatus includes: an acquisition unit that acquires a photographed image in which an object including a part of interest and an adjacent part being adjacent to the part of interest is captured; a correct answer image generation unit which, based on a reference image in which the object serving as a reference is captured and three-dimensional data of the object, generates a correct answer image in which an orientation of the object or the adjacent part included in the reference image is matched to the photographed image; and a matching image generation unit which outputs a matching image representing a difference in position or shape of the part of interest using the photographed image and the correct answer image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing system, an image processing method, and a program. [Background technology]

[0002] At assembly sites and other locations, inspections are carried out by comparing the differences between the target object and its design values ​​using a 3D (Three Dimensions) CAD (Computer Aided Design) model that contains the design information and a photographed image of the target object.

[0003] For example, a technique is known in which a 3D CAD model is oriented, a test object is placed on a stage in a direction that approximately matches the direction of the CAD model, and the CAD model and test object are overlaid by aligning corresponding points (see, for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0004] However, even if an object assembled at an assembly site or the like is assembled correctly, it may have parts that differ from the CAD model. In such cases, with the conventional technology as shown in Patent Document 1, it is difficult to determine whether an object assembled at an assembly site or the like has been assembled correctly.

[0005] One embodiment of the present invention has been made in consideration of the above-mentioned problems, and provides an image processing device that can easily determine whether an object to be inspected has been assembled correctly, even if the object has parts that differ from a model created using CAD or the like. [Means for solving the problem]

[0006] In order to solve the above problems, an image processing device according to one embodiment has an acquisition unit that acquires a captured image of an object including a component of interest and an adjacent component adjacent to the component of interest, a correct image generation unit that generates a correct image by matching the posture of the object or the adjacent component included in the reference image to the captured image based on a reference image of the object that serves as a reference and three-dimensional data of the object, and a matching image generation unit that uses the captured image and the correct image to output a matching image that shows a difference in the position or shape of the component of interest. Effect of the Invention

[0007] According to one embodiment of the present invention, it is possible to provide an image processing device that can easily determine whether an object to be inspected has been assembled correctly, even if the object has parts that differ from a model created using CAD or the like. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a system configuration of an image processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to an embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing device according to an embodiment. [Figure 4] 5 is a flowchart showing an example of a generation process of a reference image (3D) according to the first embodiment. [Diagram 5] FIG. 1 is a diagram (1) showing an image of a generation process of a reference image (3D) according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an image of the generation process of the reference image (3D) according to the first embodiment; [Figure 7] 5 is a flowchart showing an example of image matching processing according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an image of image matching processing according to the first embodiment. [Figure 9]FIG. 13 is a diagram showing an example of a matching image according to an embodiment; [Figure 10] 13 is a flowchart showing an example of a generation process of a reference image (3D) according to the second embodiment. [Figure 11] 10 is a flowchart showing an example of a generation process of a reference image (3D) corresponding to an adjacent part according to the second embodiment. [Figure 12] 13 is a flowchart showing an example of image matching processing according to the second embodiment. [Figure 13] FIG. 11 is a diagram showing an image of an image matching process according to the second embodiment. [Figure 14] FIG. 1 is a sequence diagram (1) showing an example of processing of an image processing system according to an embodiment. [Figure 15] FIG. 2 is a diagram showing an example of information managed by an image processing system according to an embodiment; [Figure 16] FIG. 11 is a sequence diagram (2) showing an example of processing of the image processing system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. <System configuration> Fig. 1 is a diagram showing an example of a system configuration of an image processing system according to an embodiment. As an example, as shown in Fig. 1, the image processing system 1 includes a camera 110 that captures an image of an object that an assembly worker 11 assembles, an inspection terminal 120 used by an inspector 12, a registration terminal 130 used by a registrant 13, and an image processing device 100. The image processing system 1 may also include a storage server 140 that stores data or information required for image processing according to this embodiment. The image processing device 100, the camera 110, the inspection terminal 120, the registration terminal 130, and the storage server 140 are connected to each other so as to be able to communicate with each other via a communication network 2 such as the Internet or a LAN (Local Area Network).

[0010] The camera 110 photographs an object assembled by the assembly worker 11, and transmits the photographed image and identification information (hereinafter referred to as a target part ID) for identifying a part (hereinafter referred to as a target part) that is to be inspected among the parts constituting the object, to the image processing device 100. As another example, the camera 110 may transmit the photographed image to the storage server 140, and the image processing device 100 may acquire the photographed image from the storage server 140.

[0011] The image processing device 100 is an information processing device having a computer configuration, or a system including multiple computers. The image processing device 100 generates a matching image that compares a captured image received from the camera 110 (or a captured image acquired from the storage server 14) with a reference image capturing an object as a reference, and transmits the generated matching image to the inspection terminal 120.

[0012] Inspection terminal 120 is an information processing device having a computer configuration, and displays the matching image received from image processing device 100 on a display unit such as a display. Inspector 12 checks the difference between the object assembled by assembler 11 and a reference object using the matching image displayed on the display unit by inspection terminal 120, and judges whether the object assembled by assembler 11 is good or bad (whether it is a good product or a defective product).

[0013] The registration terminal 130 is an information processing device having a computer configuration. The registrant 13 uses the registration terminal 130 to register information or data required for inspection, such as reference information, three-dimensional CAD data, and target part information, in the image processing device 100. As another example, the registration terminal 130 may be configured to store the reference information, three-dimensional CAD data, and target part information in the storage server 140, and the image processing device 100 may acquire the reference information, three-dimensional CAD data, and target part information from the storage server 140.

[0014] The storage server 140 is an information processing device having a computer configuration or a system including multiple computers, and stores various information or data. Note that the storage server 140 may be an external storage server or a cloud service of the image processing system 1.

[0015] In the conventional technology disclosed in Patent Document 1, an object to be inspected is inspected by comparing the difference between a three-dimensional CAD model of the object to be inspected and the object to be inspected.

[0016] On the other hand, the image processing device 100 according to the present embodiment uses a reference image obtained by photographing a reference object and three-dimensional CAD data (three-dimensional data) of the object to output a matching image for comparing the difference between the reference object and the object to be inspected. Therefore, according to the present embodiment, even if the object to be inspected has a part different from a model created by CAD or the like, it is possible to provide the image processing device 100 and the image processing system 1 that can easily determine whether the object is assembled correctly or not.

[0017] 1 is an example. For example, the function of the storage server 140 may be included in the image processing device 100 or the like. Also, the function of the registration terminal 130 may be included in the inspection terminal 120 or the like. Furthermore, in a small-scale system, the image processing device 100 and the inspection terminal 120 may be the same information processing device.

[0018] <Hardware configuration> The image processing device 100, the inspection terminal 120, the registration terminal 130, the storage server 140, etc. in Fig. 1 have, for example, the hardware configuration of a computer 200 as shown in Fig. 2. Alternatively, the image processing device 100 and the storage server 140 may be configured by a plurality of computers 200.

[0019] Fig. 2 is a diagram showing an example of a hardware configuration of a computer according to an embodiment. For example, as shown in Fig. 2, the computer 200 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a HD (Hard Disk) 204, a HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 207, a network I / F 208, a keyboard 209, a pointing device 210, a DVD-RW (Digital Versatile Disk Rewritable) drive 212, a media I / F 214, a GPU (Graphics Processing Unit) 215, and a bus line 216.

[0020] Of these, the CPU 201 controls the overall operation of the computer 200. The ROM 202 stores programs used to start up the computer 200, such as an IPL (Initial Program Loader). The RAM 203 is used, for example, as a work area for the CPU 201. The HD 204 stores programs such as an OS (Operating System), applications, device drivers, and various data. The HDD controller 205 controls the reading and writing of various data from and to the HD 204 under the control of the CPU 201, for example.

[0021] The display 206 displays various types of information such as a cursor, a menu, a window, characters, or an image. The display 206 may be provided outside the computer 200. The external device connection I / F 207 is an interface such as a Universal Serial Bus (USB) that connects various external devices to the computer 200. The network I / F 208 is an interface for communicating with other devices using, for example, the communication network 2.

[0022] The keyboard 209 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 210 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The keyboard 209 and the pointing device 210 may be provided outside the computer 200. The DVD-RW drive 212 controls reading and writing of various data from a DVD-RW 211 as an example of a removable storage medium. The DVD-RW 211 is not limited to a DVD-RW, and may be another storage medium.

[0023] A media I / F 214 controls reading and writing (storing) data from and to a media 213 such as a flash memory. A GPU 215 is an image processing processor that executes image processing and the like faster than the CPU 201. A bus line 216 includes an address bus, a data bus, various control signals, and the like for electrically connecting the above-mentioned components.

[0024] 2 is an example of the hardware configuration of the computer 200. As long as the computer 200 has, for example, a CPU 201, a ROM 202, a RAM 203, a network I / F 208, and a bus line 216, the rest of the configuration may be arbitrary.

[0025] <Functional configuration> Fig. 3 is a diagram showing an example of a functional configuration of an image processing device according to an embodiment. The image processing device 100 realizes, for example, a functional configuration as shown in Fig. 3 by the CPU 201 executing a predetermined program. In the example of Fig. 3, the image processing device 100 has a data processing unit 320 including a posture adjustment unit 321, a component region analysis unit 322, a posture conversion unit 323, and a three-dimensional information generation unit 324, and an image matching unit 330 including an acquisition unit 331, a component region extraction unit 332, a correct image generation unit 333, and a matching image generation unit 334. At least a part of the above functional configurations may be realized by hardware.

[0026] Furthermore, the image processing device 100 realizes a plurality of storage units, such as a CAD data storage unit 301, a part information storage unit 302, a reference image (3D) storage unit 303, a photographed image storage unit 304, and a reference information storage unit 305, by using a storage device such as the HD 204. At least a part of the above-mentioned plurality of storage units may be realized by the storage server 140.

[0027] The CAD data storage unit 301 is a storage unit that stores three-dimensional CAD data (three-dimensional data) of an object to be inspected, which is registered from the registration terminal 130 or the like. The part information storage unit 302 is a storage unit that stores part information such as target part information or adjacent part information, which will be described later. In this embodiment, of the multiple parts that make up the object to be inspected, the part to be inspected is called a "target part", and the parts adjacent to the target part are called "adjacent parts".

[0028] Both the target part and adjacent parts may be a set of multiple parts. For example, the inspector 12 or the like may set a set of parts (two or more parts) to be inspected as the target part. The adjacent parts may be two or more parts adjacent (or in contact with) the target part.

[0029] The reference image (3D) storage unit 303 is a storage unit that stores a reference image (3D) in which three-dimensional information such as three-dimensional position coordinates or depth information (depth information) is added to a reference image (2D) in which a reference object is photographed. In this embodiment, a reference image in which a reference object is photographed and registered from the registration terminal 130 or the like is called a "reference image (2D)", and a reference image in which three-dimensional information is added to the reference image (2D) is called a "reference image (3D)". The reference information storage unit 305 is a storage unit that stores reference information including, for example, a reference image (2D) registered from the registration terminal 130 or the like, and reference posture information which is posture information of the reference image (2D).

[0030] The data processing unit 320 executes a reference image (3D) generation process in which a reference image (2D) obtained by photographing a reference object is added with three-dimensional information (two-dimensional coordinates, depth information, etc.) of multiple parts that make up the object to generate a reference image (3D).

[0031] The attitude adjustment unit 321 executes attitude adjustment processing to align the attitude of a CAD model (CAD image) of an object rendered using three-dimensional CAD data with the attitude of the object photographed in the reference image (2D) based on the reference attitude information stored in the reference information storage unit 305. Here, the reference attitude information is information indicating the attitude of the object photographed in the reference image (2D).

[0032] In this embodiment, the orientation information is a 3×3 rotation matrix or four-dimensional data, for example, a combination of nine values ​​[r 00 ,r 01 ,r 02 ,r 10 ,r 11 ,r 12 ,r 20 ,r 21 ,r 22 ], or a combination of four values ​​[q 0 ,q 1 ,q 2 ,q 3 In addition, when translation is included, the posture information is expressed as a combination of the values ​​of the translation matrix [t x , t y , t z ] further includes.

[0033] The part region analysis unit 322 executes a part region analysis process to extract (segment) part regions of a plurality of parts constituting an object captured in a reference image (2D) based on part information of 3D CAD data. The part region may be extracted by applying a known image segmentation technique, for example, by creating a segmentation model for each part in advance or by using machine learning.

[0034] The orientation conversion unit 323 executes an orientation conversion process for matching the orientation of the parts of the CAD model to the part regions of the multiple parts constituting the target object captured in the reference image (2D). In addition, the orientation conversion unit 323 stores the orientation information of the parts of the CAD model whose orientation has been matched in the part information storage unit 302 or the like.

[0035] The 3D information generating unit 324 generates 3D information of a plurality of parts constituting the object photographed in the reference image (2D) by acquiring 3D information (3D coordinates, depth information, etc.) from the parts of the CAD model whose orientation has been adjusted by the orientation conversion unit 323. In addition, the 3D information generating unit 324 generates a reference image (3D) by, for example, adding the generated 3D information to a plurality of parts constituting the object photographed in the reference image (2D), and stores the generated reference image (3D) in the reference image (3D) storage unit 303.

[0036] With the above configuration, the data processing unit 320 can execute a reference image (3D) generation process for generating a reference image (3D) by adding three-dimensional information of a plurality of parts constituting an object to the reference image (2D). Note that when the image processing device 100 executes the reference image (3D) generation process, the image processing device 100 does not need to include the image matching unit 330.

[0037] The image matching unit 330 executes an image matching process using a captured image of an object to be inspected and a correct image, which will be described later, to output a matching image that shows a difference in the position or shape of a component of interest.

[0038] The acquisition unit 331 executes an acquisition process for acquiring captured images of an object to be inspected from, for example, the camera 110, the storage server 140, the captured image storage unit 304, or the like.

[0039] The part area extraction unit 332 executes a part area extraction process for extracting (segmenting) part areas of a plurality of parts constituting an object based on three-dimensional CAD data from the captured image acquired by the acquisition unit 331. The part area may be extracted by applying a known image segmentation technique, for example, by creating a segmentation model for each part in advance or by using machine learning.

[0040] The correct image generation unit 333 executes a correct image generation process to generate a correct image by matching the posture of the object or adjacent parts contained in the reference image (2D) to the captured image, based on a reference image (2D) captured of a reference object and three-dimensional CAD data of the object.

[0041] For example, the correct image generation unit 333 acquires from the reference image (3D) memory unit 303 a reference image (3D) generated by the data processing unit 320 based on a reference image (2D) captured by photographing a reference object and three-dimensional CAD data of the object.

[0042] In addition, as an example, the correct image generation unit 333 uses three-dimensional information of multiple parts included in the reference image (3D) to generate a correct image that matches the posture of the object included in the reference image (3D) to the posture of the object included in the captured image.

[0043] As another example, the correct image generation unit 333 uses three-dimensional information of the target part and adjacent parts contained in the reference image (3D) to generate a correct image that matches the posture of the adjacent part contained in the reference image (3D) to the posture of the adjacent part contained in the captured image.

[0044] The matching image generation unit 334 uses the captured image acquired by the acquisition unit 331 and the correct image generated by the correct image generation unit 333 to generate a matching image that represents (visualizes) differences in the position or shape of the target part to be inspected, and outputs the matching image to an output destination such as the inspection terminal 120.

[0045] With the above configuration, the image matching unit 330 can execute image matching processing to generate a matching image for comparing the difference between a reference object and an object to be inspected, and output the matching image to a predetermined output destination. Note that when the image processing device 100 executes the image matching processing, the image processing device 100 does not need to include the data processing unit 320.

[0046] The functional configuration of the image processing device 100 shown in Fig. 3 is an example. For example, each functional configuration included in the image processing device 100 shown in Fig. 3 may be distributed among multiple image processing devices. In addition, multiple storage units may be provided outside the image processing device 100.

[0047] <Processing flow> Next, the flow of processing of the image processing method according to this embodiment will be described.

[0048] [First embodiment] (Reference image (3D) generation process) 4 is a flowchart showing an example of a reference image (3D) generation process according to the first embodiment. This process shows an example of a reference image (3D) generation process executed by the data processing unit 320 described in FIG.

[0049] In step S401, the orientation adjustment unit 321 adjusts the orientation of a three-dimensional CAD model (CAD image) of an object to the orientation of the object in a reference image (2D).

[0050] Fig. 5(A1) shows an image of a reference image (2D) 510 obtained by photographing a reference object. Fig. 5(A2) shows an image of a CAD model (CAD image) 500 obtained by rendering the object using three-dimensional CAD data. Here, as an example for explanation, the following explanation will be given assuming that the object is composed of three parts: a target part 501 to be inspected, an adjacent part 502 adjacent to the target part 501, and another part 503.

[0051] The posture adjustment unit 321 uses the reference posture information included in the reference information to adjust the posture (orientation) of the CAD model 500 to the posture of the reference image (2D) 510. Here, the reference object may be assembled correctly but may have a portion different from the CAD model 500. For example, in the CAD model 500 shown in Fig. 5(A2), there is no gap 504b between the adjacent part 502 and the other part 503, but in the reference image (2D) 510 shown in Fig. 5(A1), there is a gap (or distortion) or the like between the adjacent part 504a and the other part.

[0052] In step S402, based on the part information of the three-dimensional CAD data, part region analysis unit 322 extracts part regions of multiple parts constituting the target object from reference image (2D) 510. For example, as shown in Fig. 5(B), part region analysis unit 322 extracts part region 511 of a target part, part region 512 of an adjacent part, and part region 513 of another part from reference image (2D).

[0053] In step S403, component region analysis section 322 selects one unprocessed component region for which the processes in steps S404 to S408 have not been executed.

[0054] In step S404, the component area analysis unit 322 determines whether the selected component area is the component area 511 of the component of interest. If the selected component area is the component area 511 of the component of interest, for example, as shown in Fig. 5(C), the component area analysis unit 322 shifts the process to step S405. On the other hand, if the selected component area is not the component area 511 of the component of interest, for example, as shown in Fig. 5(A1), the component area analysis unit 322 shifts the process to step S406.

[0055] In step S405, the component region analysis unit 322 stores, in the component information storage unit 302 or the like, component region information of interest, which is information on the component region 511 of the component of interest.

[0056] When the process proceeds to step S406, the posture conversion unit 323 acquires a CAD model of the part corresponding to the selected part region from the three-dimensional CAD data. For example, as shown in Fig. 6(A1), when the selected part region is part region 512 of an adjacent part, the posture conversion unit 323 acquires a CAD model of adjacent part 502 as shown in Fig. 6(A2). The CAD model of adjacent part 502 includes three-dimensional coordinate information (an example of three-dimensional information) such as (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), etc.

[0057] In step S407, the orientation conversion unit 323 matches the orientation of the acquired CAD model of the part to the selected part area. For example, when the orientation conversion unit 323 obtains a CAD model of an adjacent part 502 as shown in Fig. 6(A2), the orientation conversion unit 323 matches the orientation of the CAD model of the adjacent part 502 to the orientation of the part area 512 of the adjacent part as shown in Fig. 6(B).

[0058] In step S408, the three-dimensional information generation unit 324 acquires three-dimensional information from the CAD model whose orientation has been adjusted by the orientation conversion unit 323. For example, the three-dimensional information generation unit 324 acquires three-dimensional coordinate information (an example of three-dimensional information) such as (x1', y1', z1'), (x2', y2', z2'), (x3', y3', z3'), etc. from the CAD model of the adjacent part 502 as shown in Fig. 6(B).

[0059] In addition, the three-dimensional information generator 324 adds the obtained three-dimensional information to the selected part region. For example, as shown in Fig. 6(C), the three-dimensional information generator 324 adds the obtained three-dimensional coordinate information (x1', y1', z1'), (x2', y2', z2'), (x3', y3', z3'), ... to the part region 512 of the selected adjacent part.

[0060] In step S409, the three-dimensional information generation unit 324 determines whether or not there is an unprocessed part area for which the processes in steps S404 to S408 have not been executed. If there is an unprocessed part area, the three-dimensional information generation unit 324 returns the process to step S403. On the other hand, if there is no unprocessed part area, the three-dimensional information generation unit 324 shifts the process to step S410.

[0061] In step S410, the three-dimensional information generating unit 324 stores the reference image (3D) to which the three-dimensional information has been added in the reference image (3D) storage unit 303 or the like.

[0062] Through the above processing, the data processing unit 320 can generate a reference image (3D) in which three-dimensional information is added to each of the part regions of multiple parts extracted from the reference image (2D) 510, as shown in, for example, FIG. 5(B).

[0063] (Image matching processing) 7 is a flowchart showing an example of the image matching process according to the first embodiment. This process shows an example of the image matching process executed by the image matching unit 330 described with reference to FIG.

[0064] In step S701, the acquisition unit 331 acquires a captured image of an object to be inspected. For example, the acquisition unit 331 acquires a captured image 810 of an object to be inspected, as shown in Fig. 8 (A2), from the captured image storage unit 304, the camera 110, the storage server 140, or the like.

[0065] In step S702, the correct image generating unit 333 generates a correct image by matching the posture of the reference image (3D) to the captured image. For example, the correct image generating unit 333 acquires a reference image (3D) 800 as shown in FIG. 8(A1) from the reference image (3D) storage unit 303, and matches the posture of the reference image (3D) 800 to the posture of the object captured in the captured image 810 as shown in FIG. 8(B). In this way, the correct image generating unit 333 generates a correct image 820 as shown in FIG. 8(C1), for example. Since the reference image (3D) has three-dimensional information, it is possible to perform image processing such as rotation, translation, and enlargement / reduction in the same manner as a CAD model.

[0066] 8(C1), the matching image generation unit 334 selects the region 801 of the component of interest from the correct answer image 820. For example, the matching image generation unit 334 determines the region 801 of the component of interest based on the component of interest region information stored in the component information storage unit 302.

[0067] In step S704, the matching image generation unit 334 extracts a region 802 of the component of interest from the captured image 810, for example, as shown in Fig. 8(C2). For example, the matching image generation unit 334 may extract the region 802 of the component of interest by segmentation using three-dimensional CAD data. Alternatively, the matching image generation unit 334 may extract the region 802 of the component of interest from the captured image 810 based on the region 801 of the component of interest acquired from the correct image 820 in step S703.

[0068] In step S705, the matching image generation unit 334 generates a matching image that represents the difference in position or shape between the part of interest included in the captured image 810 and the part of interest included in the correct image 820, and outputs the image to a specified output destination, such as the inspection terminal 120.

[0069] Fig. 9 is a diagram showing an example of a matching image according to an embodiment. In the example of Fig. 9, the matching image generating unit 334 generates a matching image 920 that represents a difference between a peripheral image 902 of a target part 901 in a correct answer image 820 and a peripheral image 912 of a target part 911 in a photographed image 810. This matching image 920 is, for example, an image in which the difference in each pixel value between the peripheral image 902 and the peripheral image 912 is calculated and the calculated difference in each pixel value is represented as a heat map.

[0070] 9, the inspector 12 can determine whether the target part 911 is misaligned, has a different shape, or is pass (fail) based on the matching image 920. As a preferred example, the matching image generating unit 334 may convert the difference between the peripheral image 902 and the peripheral image 912 into a numerical value, compare it with a threshold value to determine pass / fail, and display the result of the determination in the matching image 920.

[0071] In this way, according to the image processing system 1 of this embodiment, even if the object to be inspected has parts that differ from a model created by CAD or the like, it becomes possible to easily determine whether the object has been assembled correctly.

[0072] [Second embodiment] In the first embodiment, the image processing device 100 generates a correct image 820 by aligning (matching) the posture of an object in a reference image (3D) 800 with an object captured in a captured image 810. However, with this method, for example, if the gap 504a between an adjacent part and another part, or distortion, etc., described in Fig. 5(A1) is different, an error may become large.

[0073] In the second embodiment, an example of processing for matching an adjacent part adjacent to a part of interest between a captured image and a reference image (3D) will be described.

[0074] <Processing flow> (Generation process of reference image (3D)) Fig. 10 is a flowchart showing an example of the reference image (3D) generation process according to the second embodiment. This process shows another example of the reference image (3D) generation process executed by the data processing unit 320 described in Fig. 3. Among the processes shown in Fig. 10, the processes of steps S401 to S405 are similar to the reference image (3D) generation process according to the first embodiment described in Fig. 4, and therefore will not be described here.

[0075] In step S1001, the component area analysis unit 322 determines whether the selected component area is the component area 512 of an adjacent component. If the selected component area is the component area 512 of an adjacent component adjacent to the component of interest, the component area analysis unit 322 shifts the process to step S1002. On the other hand, if the selected component area is not the component area 512 of an adjacent component adjacent to the component of interest, the component area analysis unit 322 stops the processes of steps S1002 and S1003, and shifts the process to step S1004.

[0076] In step S1002, the component region analysis unit 322 stores adjacent component region information, which is information on the component region 512 of the adjacent component, in the component information storage unit 302 or the like.

[0077] In step S1003, the data processing unit 320 executes a process of creating a reference image (3D) corresponding to the adjacent part as shown in FIG.

[0078] In step S1004, the three-dimensional information generation unit 324 determines whether there are any unprocessed part areas for which the processes in steps S403 to S405 and steps S1001 to S1003 have not been executed. If there are any unprocessed part areas, the three-dimensional information generation unit 324 returns the process to step S403. On the other hand, if there are no unprocessed part areas, the three-dimensional information generation unit 324 ends the process in FIG. 10.

[0079] (Generation process of reference images (3D) corresponding to adjacent parts) Fig. 11 is a flowchart showing an example of a process for generating a reference image (3D) corresponding to an adjacent part according to the second embodiment. This process shows an example of a process for generating a reference image (3D) corresponding to an adjacent part, which is executed by the data processing unit 320 in step S1003 in Fig. 10. Note that a detailed description of the process contents similar to those of the first embodiment will be omitted here.

[0080] In step S1101, the posture conversion unit 323 acquires a CAD model of an adjacent part. For example, the posture conversion unit 323 acquires a CAD model of an adjacent part 502 as shown in FIG.

[0081] In step S1102, the orientation conversion unit 323 aligns the orientation of the acquired CAD model of the adjacent part to the adjacent part area in the reference image (2D). For example, as shown in FIG. 6B, the orientation conversion unit 323 aligns the orientation of the CAD model of the adjacent part 502 to the part area 512 of the adjacent part in the reference image (2D) 510.

[0082] In step S1103, the three-dimensional information generation unit 324 acquires three-dimensional information (such as three-dimensional coordinate information) from the CAD model whose orientation has been adjusted by the orientation conversion unit 323, and adds the acquired three-dimensional information to the adjacent part area of ​​the reference image (2D).

[0083] Furthermore, the data processing unit 320 executes the processes of steps S1111 to S1113 in parallel with the processes of steps S1101 to S1103 (or following the processes of steps S1101 to S1103).

[0084] In step S1111, the orientation conversion unit 323 obtains a CAD model of the part of interest that corresponds to the adjacent part.

[0085] In step S1112, the orientation conversion unit 323 matches the orientation of the acquired CAD model of the part of interest to the part of interest region of the reference image (2D).

[0086] In step S1113, the three-dimensional information generating unit 324 acquires three-dimensional information from the CAD model whose orientation has been adjusted by the orientation converting unit 323, and adds the acquired three-dimensional information to the component of interest region of the reference image (2D).

[0087] In step S1114, the orientation conversion unit 323 stores in the reference image (3D) storage unit 303 the reference image (3D) obtained by adding three-dimensional information to the adjacent parts and the part of interest in the reference image (2D).

[0088] In the second embodiment, the process of FIG. 11 is executed for each adjacent part to generate one or more reference images (3D) corresponding to the adjacent part.

[0089] (Image matching processing) Fig. 12 is a flowchart showing an example of image matching processing according to the second embodiment. This processing shows another example of the image matching processing executed by the image matching unit 330 described in Fig. 3. Among the processing shown in Fig. 12, the processing of steps S701 and S703 to S705 is similar to the image matching processing according to the first embodiment described in Fig. 7, and therefore description thereof will be omitted here.

[0090] In step S1201, the correct image generating unit 333 selects an adjacent part of the reference image (3D). For example, the correct image generating unit 333 acquires a reference image (3D) 1300 as shown in Fig. 13(A1) from the reference image (3D) storage unit 303. The correct image generating unit 333 also acquires adjacent part area information from the part information storage unit 302, and selects an adjacent part 1301 as shown in Fig. 13(B1).

[0091] In step S1202, the part area extraction unit 332 extracts the part area of ​​the adjacent part from the captured image acquired by the acquisition unit 331. For example, the part area extraction unit 332 extracts the part area 1311 of the adjacent part from the captured image 1310 as shown in Fig. 13(A1) based on three-dimensional CAD data or the like as shown in Fig. 13(B2).

[0092] 13C, for example. In the second embodiment, since three-dimensional information is added only to the adjacent part 1301 and the part of interest 1302 in the reference image (3D) 1300, the correct image generating unit 333 may match only the orientations of the adjacent part 1301 and the part of interest 1302. For example, the correct image 1320 generated by the correct image generating unit 333 according to the second embodiment may depict only the adjacent part 1301 and the part of interest 1302. Alternatively, the correct image 1320 generated by the correct image generating unit 333 may be a composite image of only the orientations of the adjacent part 1301 and the target part 1302 and an image of another part 1303 whose orientation has not been changed.

[0093] In steps S703 to S705, the matching image generating unit 334 generates, for example, the matching image 920 described in Fig. 9. At this time, as shown in Fig. 9, the matching image 920 does not need to include images of other parts 1303.

[0094] In the second embodiment, a matching image that shows the difference in the position or shape of a part of interest is generated based on the posture of an adjacent part, so that even if the fit, etc., of multiple parts that make up an object is different, it is possible to determine whether the part of interest has been assembled correctly.

[0095] <Overall processing> In each of the above embodiments, the processing of the image processing device 100 has been described in detail, but here, an example of the flow of processing of the entire image processing system will be described.

[0096] Fig. 14 is a sequence diagram (1) showing an example of processing of an image processing system according to an embodiment. This processing shows an example of the overall processing of the image processing system 1 corresponding to the generation processing of the reference image (3D) described in Fig. 10, for example.

[0097] In step S1401, the registrant 13 uses the registration terminal 130 to register three-dimensional CAD data (three-dimensional data) of an object to be inspected in the storage server 140. In addition, in step S1402, the storage server 140 stores the three-dimensional CAD data received from the registration terminal 130.

[0098] The processes in steps S1401 and S1402 are merely examples. For example, the registrant 13 may use the registration terminal 130 to register three-dimensional CAD data of an object to be inspected in the image processing device 100. In this case, the image processing device 100 stores the three-dimensional CAD data received from the registration terminal 130 in the CAD data storage unit 301.

[0099] In step S1403, the registrant 13 uses the registration terminal 130 to register reference information of the object to be inspected in the storage server 140. This reference information includes a reference image (2D), reference orientation information, and target component information, etc. The reference image (2D) is an image of a reference object. The reference orientation information is information indicating the orientation of the object photographed in the reference image (2D). The reference orientation information is, for example, a 3×3 rotation matrix or four-dimensional data, and is managed as a combination of nine values ​​or a combination of four values. Furthermore, when the reference orientation information includes translation, it is managed by further adding a combination of three values ​​representing the translation matrix. The target component information is information of the target component to be inspected.

[0100] Fig. 15(A) shows an image of an example of the target component information 1501. In the example of Fig. 15(A), the target component information 1501 includes information such as "target component ID", "target component position information", "target component CAD model", and "reference image (2D)" as items.

[0101] "Part of interest ID" is identification information for identifying the part of interest. "Part of interest position information" is information indicating the position of the part of interest in three-dimensional CAD data (e.g., the central coordinates of the part of interest, etc.). "Part of interest CAD model" is information specifying the CAD model of the part of interest. Note that the "part of interest CAD model" may also be information specifying the part of interest within the CAD model. "Reference image (2D)" is information specifying the reference image (2D) to be used when inspecting the part of interest. As shown in Figure 15 (A), multiple parts of interest can be inspected using one reference image (2D).

[0102] Image processing system 1 may be configured to inspect multiple parts of interest simultaneously by specifying multiple parts of interest as shown in FIG. 15(A), or may be configured to inspect multiple parts of interest sequentially.

[0103] In step S1404, the storage server 140 stores the reference information received from the registration terminal 130. Note that the registrant 13 may use the registration terminal 130 to register the reference information of the object to be inspected in the image processing device 100.

[0104] In step S1405, when the registrant 13 transmits a processing start instruction to the image processing device 100 using the registration terminal 130, the image processing system 1 executes the processing from step S1406 onwards.

[0105] In step S1406, the image processing apparatus 100 acquires the three-dimensional CAD data and the target part information from the storage server 140.

[0106] In steps S1407 and S1408, the image processing apparatus 100 acquires adjacent part information based on the acquired three-dimensional CAD data and target part information, and transmits the adjacent part information to the storage server 140.

[0107] Fig. 15(B) shows an image of an example of adjacent part information 1502. In the example of Fig. 15(B), the adjacent part information 1502 includes items such as "adjacent part ID", "target part ID", "adjacent part position information", "adjacent part CAD model", and "reference image (2D)".

[0108] The "adjacent part ID" is identification information that identifies an adjacent part adjacent to the part of interest identified by the "part of interest ID". When one adjacent part is adjacent to multiple parts of interest, multiple adjacent part IDs may refer to one part. The "part of interest ID" is the part of interest that corresponds to the adjacent part identified by the "adjacent part ID". The "adjacent part position information" is information that indicates the three-dimensional CAD data information position of the adjacent part (e.g., the center coordinates of the adjacent part, etc.).

[0109] "Adjacent part CAD data" is information indicating a CAD model of an adjacent part. Note that "adjacent part CAD model" may be information that identifies an adjacent part in a CAD model. "Reference image (2D)" is information that specifies a reference image (2D) to be used when inspecting a part of interest that corresponds to an adjacent part.

[0110] In step S 1409 , the storage server 140 stores the adjacent part information 1502 received from the image processing device 100 .

[0111] In step S1410, the image processing device 100 acquires reference information and the like, and in step S1411, the data processing unit 320 of the image processing device 100 executes the generation process of the reference image (3D) described with reference to FIGS.

[0112] In step S1412, the image processing apparatus 100 transmits to the storage server 140 the generated reference images (3D) corresponding to the one or more adjacent parts and the adjusted part posture information.

[0113] Fig. 15(C) shows an image of an example of adjacent part orientation information 1503. In the example of Fig. 15(C), the adjacent part orientation information 1503 includes items such as "adjacent part ID", "target part ID", and "adjacent part orientation information".

[0114] The "adjacent part ID" and "part of interest ID" correspond to the adjacent part ID and part of interest ID of adjacent part information 1502. The "adjusted part orientation information" is orientation information of the adjusted part. As described above, the orientation information is, for example, a 3×3 rotation matrix or four-dimensional data, and is a combination of nine values ​​or a combination of four values. Furthermore, if the orientation information includes a translation, it further includes a combination of three values ​​that represents the translation matrix.

[0115] In steps S1413 and S1414, the storage server 140 stores one or more reference images (3D) and the adjusted part posture information received from the image processing apparatus 100.

[0116] 16 is a sequence diagram (2) showing an example of processing of the image processing system according to an embodiment. This processing shows an example of the overall processing of the image processing system 1 corresponding to the image matching processing described in FIG.

[0117] In steps S1601 and S1602, the camera 110 transmits a captured image of an object to be inspected and a component of interest ID for identifying a component of interest to a predetermined destination. In the example of Fig. 16, the camera 110 transmits the component of interest ID to the image processing device 100 and transmits the captured image to the storage server 140. However, the processes of steps S1601 and S1602 are merely examples, and the camera 110 may transmit the component of interest ID and the captured image to the image processing device.

[0118] In step S1603, the storage server 140 stores the captured image received from the camera 110.

[0119] In steps S1604 and S1605, the image processing device 100 acquires data necessary for the image matching process (for example, the captured image, the reference image (3D), three-dimensional CAD data, target part information, adjacent part information, adjacent part posture information, etc.) from the storage server 140. Note that the processes shown in steps S1604 and S1605 are merely an example, and the image processing device 100 may store data necessary for the image matching process in advance in multiple storage units as shown in FIG.

[0120] In step S1606, the image matching unit 330 of the image processing device 100 executes, for example, the image matching process described in Fig. 12. When multiple components of interest are included in the inspection target, the image matching unit 330 may execute the image matching process as shown in Fig. 12 for each of the components of interest and output multiple matching images. Alternatively, the image matching unit 330 may combine the multiple matching images into one matching image and output the combined image.

[0121] As described above, according to each embodiment of the present invention, it is possible to provide an image processing device 100 and an image processing system 1 that can easily determine whether an object to be inspected has been assembled correctly, even if the object has parts that differ from a model created by CAD or the like.

[0122] <Supplementary Information> Each function of each embodiment described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0123] Moreover, the devices described in the examples are merely one of a number of computing environments for implementing the embodiments disclosed herein. In one embodiment, the image processing device 100 includes a number of computing devices, such as a server cluster. The computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, and the like, and perform the processes disclosed herein. Furthermore, each element of the image processing device 100 may be integrated into one information processing device or may be divided into multiple information processing devices.

[0124] 3 can be configured to share the processes of the image processing device 100 shown in FIGS. 4, 7, 10 to 12 in various combinations. For example, at least a part of the processes executed by the image processing device 100 may be executed by the inspection terminal 120. Also, at least a part of the processes executed by the image processing device 100 may be executed by, for example, an external server device, a cloud service, or the like. [Explanation of symbols]

[0125] 1. Image Processing System 100 Image processing device 324 3D information generation section 331 Acquisition Department 333 Correct Image Generation Unit 334 Matching Image Generation Unit 800 Reference Images (3D) 810 Images 820 Correct Images 911 Featured Parts 920 Matching Images [Prior art documents] [Patent documents]

[0126] [Patent Document 1] Special Publication No. 2020-509370

Claims

1. an acquisition unit that acquires a captured image of an object including a component of interest and an adjacent component adjacent to the component of interest; a correct image generating unit that generates a correct image by matching a posture of the object or the adjacent part included in the reference image to the photographed image, based on a reference image obtained by photographing the object as a reference and three-dimensional data of the object; a matching image generating unit that uses the captured image and the correct image to output a matching image that represents a difference in a position or a shape of the target component; The image processing device includes:

2. a three-dimensional information generating unit that generates three-dimensional information of the target component and the adjacent component included in the reference image by using the reference image and three-dimensional data of the object, the correct image generating unit aligns a posture of the adjacent part included in the reference image with a posture of the adjacent part included in the captured image by using the three-dimensional information. The image processing device according to claim 1 .

3. a three-dimensional information generating unit that generates three-dimensional information of a plurality of components of the object included in the reference image by using the reference image and three-dimensional data of the object, The correct image generating unit aligns a posture of the object included in the reference image with a posture of the object included in the captured image by using the three-dimensional information. The image processing device according to claim 1 .

4. 4. The image processing device according to claim 2, wherein the three-dimensional information generation unit aligns a posture of a three-dimensional CAD image of a part included in the reference image to the posture of the part, and acquires three-dimensional information of the part included in the reference image from the oriented CAD image.

5. The image processing device according to claim 2 , wherein the three-dimensional information includes three-dimensional coordinate information or depth information.

6. The image processing device according to claim 2 , wherein the three-dimensional information generating unit adds the generated three-dimensional information to the reference image.

7. 7. The image processing device according to claim 1, wherein the matching image generation unit generates the matching image that visualizes a difference in position or shape between the target part included in the captured image and the target part included in the correct image.

8. an acquisition unit that acquires a captured image of an object including a component of interest and an adjacent component adjacent to the component of interest; a correct image generating unit that generates a correct image by matching a posture of the object or the adjacent part included in the reference image to the photographed image, based on a reference image obtained by photographing the object as a reference and three-dimensional data of the object; a matching image generating unit that uses the captured image and the correct image to output a matching image that represents a difference in a position or a shape of the target component; An image processing system comprising:

9. The computer A process of acquiring a photographed image of an object including a component of interest and an adjacent component adjacent to the component of interest; A process of generating a correct image by matching a posture of the object or the adjacent part included in the reference image to the photographed image based on a reference image obtained by photographing the object as a reference and three-dimensional data of the object; A process of outputting a matching image that shows a difference in a position or a shape of the target component by using the captured image and the correct image; The image processing method according to claim 1,

10. A program causing a computer to execute the image processing method according to claim 9.

Images