Object recognition device and program

Abstract

To provide a technology which enhances recognition accuracy for objects using images.SOLUTION: An image acquisition unit 151 acquires an image taken by a camera. Parts are captured in the taken image. An external shape recognition unit 152 recognizes external shapes of the parts from the image taken by the camera. An internal shape recognition unit 153 recognizes internal shapes of the parts from the image taken by the camera. An identification unit 154 uses recognition results of the external shape recognition unit 152 and / or the internal shape recognition unit 153, for identifying part information of elements captured in the image by the camera. In an example of one embodiment, the part information includes orientations and rotation angles of the parts.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to image-based object recognition. [Background technology]

[0002] Conventionally, in order to recognize an object, an image of the object is used, and various techniques for recognizing an object using an image have been proposed.

[0003] For example, Japanese Patent Laid-Open Publication No. 09-245177 (Patent Document 1) discloses a technique for selecting optimal image recognition conditions for each part. More specifically, in Patent Document 1, multiple types of workpiece imaging conditions (workpiece angle and illumination intensity) and multiple types of processing conditions (pre-processing methods for captured images and recognition processing methods) are set as recognition conditions. In Patent Document 1, image recognition processing is performed for all combinations of each of the multiple types of imaging conditions and each of the multiple types of processing conditions, and the combination with the least error is set as the final recognition condition. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 09-245177 Summary of the Invention [Problem to be solved by the invention]

[0005] There is a constant demand for improved accuracy in the recognition of objects using images as described above.

[0006] The present disclosure has been devised in view of the above circumstances, and its purpose is to provide a technique for improving the accuracy of object recognition using images. [Means for solving the problem]

[0007] According to one aspect of the present disclosure, there is provided an object recognition device for recognizing the state of an object, the object recognition device including: an external shape recognition unit that recognizes the external shape of the object based on an image taken of the object; an internal shape recognition unit that recognizes the internal shape of the object based on an image taken of the object; and an identification unit that identifies the state of the object using the recognition results of the external shape recognition unit and the internal shape recognition unit.

[0008] The image used by the external shape recognition unit to recognize the external shape may be a first image captured of the object, and the image used by the internal shape recognition unit to recognize the internal shape may be a second image obtained by applying a given image processing to the first image.

[0009] The image used by the external shape recognition unit to recognize the external shape may be a first image captured of the object, and the image used by the internal shape recognition unit to recognize the internal shape may be a second image captured under different imaging conditions than the first image.

[0010] The state of the object may include the orientation and rotation angle of the object. Both the external shape recognition unit and the internal shape recognition unit may recognize both the orientation and rotation angle of the object.

[0011] The image processing apparatus may further include a storage device. The storage device may store an external shape library that associates image patterns of the external shape of the object with the orientation and rotation angle of the object, and an internal shape library that associates image patterns of the internal shape of the object with the orientation and rotation angle of the object. The external shape recognition unit may recognize the orientation and rotation angle of the object based on a match rate between elements included in a captured image of the object and the image patterns included in the external shape library. The internal shape recognition unit may recognize the orientation and rotation angle of the object based on a match rate between elements included in a captured image of the object and the image patterns included in the internal shape library.

[0012] The external shape recognition unit may recognize both the orientation and the rotation angle of the object. The orientation of the object recognized by the external shape recognition unit may include two or more orientations. The internal shape recognition unit may identify one of the two or more orientations.

[0013] The system may further include a storage device. The storage device may store an external shape library associating image patterns of the external shape of the object with the orientation and rotation angle of the object, and an internal shape library associating image patterns of the internal shape of the object with the orientation and rotation angle of the object. The external shape library may associate two or more orientations with one rotation angle. The external shape recognition unit may recognize the rotation angle of the object based on a match rate between elements included in an image taken of the object and the image patterns included in the external shape library. The internal shape recognition unit may recognize the orientation of the object based on a match rate between elements included in an image taken of the object and the image patterns included in the internal shape library.

[0014] The apparatus may further include a camera for capturing an image of the object, and the image captured by the camera may be an image captured of the object.

[0015] According to another aspect of the present disclosure, there is provided a program that, when executed by a computer, causes the computer to perform the steps of: recognizing the external shape of an object based on an image taken of the object; recognizing the internal shape of the object based on an image taken of the object; and identifying the state of the object using the recognition results of the external shape and the recognition results of the internal shape. [Effects of the Invention]

[0016] According to the present disclosure, the state of an object is identified using the recognition results of the external shape of the object and the recognition results of the internal shape of the object, which are recognized based on images captured of the object, thereby improving the accuracy of object recognition using images. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram schematically illustrating a configuration of a component recognition system. [Figure 2] FIG. 1 is a diagram illustrating a hardware configuration of a recognition device 100. [Figure 3] FIG. 1 is a diagram illustrating a functional configuration of a recognition device 100. [Figure 4] FIG. 10 is a diagram illustrating an example of a data structure of an external shape library. [Figure 5] FIG. 10 is a diagram illustrating an example of a data structure of an internal shape library. [Figure 6] 10 is a flowchart of a process performed by the recognition device 100 to output component information of the component 400 placed on the tray 300. [Figure 7] 7 is a flowchart of a first modified example of the process of FIG. 6. [Figure 8] 7 is a flowchart of a second modified example of the process of FIG. 6. [Figure 9] FIG. 10 is a diagram showing a modified example of the data structure of the external shape library. DETAILED DESCRIPTION OF THE INVENTION

[0018] An embodiment of a parts recognition system will be described below with reference to the drawings. The parts recognition system is an example of a system including an object recognition device. In the following description, identical parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, their description will not be repeated.

[0019] [1. Parts Recognition System] 1 is a diagram showing a schematic configuration of a component recognition system 1000. The component recognition system 1000 is a system for identifying the state of a component 400 placed on a tray 300.

[0020] In the part recognition system 1000, a recognition device 100 is attached to a robot arm 200. The recognition device 100 is an example of an object recognition device, and is capable of communicating with the robot arm 200.

[0021] The recognition device 100 includes a main body 110 and a camera 120. The main body 110 uses an image captured by the camera 120 to identify the state of the component 400 placed on the tray 300. The component 400 is an example of an object. An image captured of the component 400 on the tray 300 is an example of an image captured of an object.

[0022] The recognition device 100 does not necessarily have to include the camera 120. That is, the recognition device 100 may use an image of the component 400 on the tray 300 taken by an external device to identify the state of the component 400.

[0023] 1 shows three axes, X, Y, and Z. These three axes are used when referring to directions in the component recognition system 1000. The X and Y axes extend in a direction along the placement surface of the tray 300. The Z axis extends in a direction intersecting the placement surface of the tray 300 (vertical direction).

[0024] [2. Hardware configuration] 2 is a diagram showing the hardware configuration of the recognition device 100. A main body 110 of the recognition device 100 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a storage 103, a display 104, an input device 105, and an interface 106.

[0025] The CPU 101 controls the recognition device 100 by executing a given program. The RAM 102 functions as a work area for the CPU 101. The storage 103 is an example of a storage device. The storage 103 stores programs and / or data required for executing the programs. The CPU 101 may execute programs stored in a storage device external to the recognition device 100, or may use data stored in a storage device external to the recognition device 100.

[0026] The display 104 displays the results of calculations performed by the CPU 101. The input device 105 is, for example, a keyboard and / or a mouse. The interface 106 is a communication interface that allows the recognition apparatus 100 to communicate with external devices.

[0027] [3. Functional configuration] Fig. 3 is a diagram showing the functional configuration of recognition device 100. As shown in Fig. 3, recognition device 100 functions as image acquisition unit 151, external shape recognition unit 152, internal shape recognition unit 153, and identification unit 154. In one implementation example, image acquisition unit 151, external shape recognition unit 152, internal shape recognition unit 153, and identification unit 154 are realized by CPU 101 executing a given program (for example, an application program for component recognition).

[0028] The image acquisition unit 151 acquires an image of the component 400. In one implementation example, the camera 120 generates image data by capturing an image of the component 400 placed on the tray 300. The image acquisition unit 151 acquires the image by reading the image data generated by the camera 120.

[0029] External shape recognition unit 152 recognizes the external shape of component 400 from the image captured by camera 120. In one implementation example, external shape recognition unit 152 determines whether the shape of the outer edge of an element recognized from the image matches an image pattern registered in advance as the outer edge of the component, and if it determines that the shape matches, identifies the element shown in the image as the component.

[0030] Internal shape recognition unit 153 recognizes the internal shape of component 400 from the image captured by camera 120. In one implementation example, external shape recognition unit 152 determines whether the shape (structure) of the part located inside the outer edge of the element recognized from the image matches an image pattern registered in advance as the internal structure of the component, and if it determines that the shape matches, identifies the element shown in the image as the component.

[0031] The identification unit 154 uses the recognition results of the external shape recognition unit 152 and / or the internal shape recognition unit 153 to identify part information of the elements shown in the image captured by the camera 120. In one implementation example, the part information includes the orientation and rotation angle of the part.

[0032] The orientation of a part refers to the orientation of the part 400 on the tray 300 relative to the camera 120. In one implementation, the orientation of a part is expressed as the side (front, back, right, or left) that faces the camera 120 among two or more sides of the part. The orientation of a part may be defined as the rotational position of the part 400 on the tray 300 relative to the X-axis and / or Y-axis in FIG. 1 .

[0033] The rotation angle of a part may be defined as the rotational position of the part 400 on the tray 300 relative to the Z axis in FIG.

[0034] [4. External Shape Library] 4 is a diagram showing an example of the data structure of the external shape library, which is used by external shape recognition unit 152 to recognize external shapes.

[0035] As shown in FIG. 4, the external shape library includes the items "part ID," "orientation," and "rotation angle." The "part ID" identifies each part handled by the part recognition system 1000. In the example of FIG. 4, information on two types of parts (part IDs "0001" and "0002") is shown. The external shape library includes four types of values ​​(front, back, right, and left) for "orientation," and values ​​representing the rotation angle (0°, 15°, 30°, ...) for "rotation angle."

[0036] The external shape library contains information for each "part ID." The external shape library also contains image patterns representing the outer edges of parts, associated with combinations of "orientation" and "rotation angle" values.

[0037] For example, the external shape library includes an image pattern associated with the combination of part ID "0001", orientation "front", and rotation angle "0°". The external shape library also includes an image pattern associated with the combination of part ID "0001", orientation "front", and rotation angle "15°".

[0038] The image patterns registered in the external shape library mainly represent the outer edge shapes of the parts. [5. Internal Shape Library] Fig. 5 is a diagram showing an example of the data structure of the internal shape library. The internal shape library is used for internal shape recognition by the internal shape recognition unit 153. Due to space limitations, Fig. 5 shows only part of the information related to the part ID "0001."

[0039] Like the external shape library, the internal shape library also includes, for each "part ID," an image pattern representing the outer edge of a part, associated with a combination of values ​​for "orientation" and "rotation angle."

[0040] The image patterns registered in the internal shape library primarily represent the internal structure of the part. [6. Processing flow] 6 is a flowchart of a process performed by the recognition device 100 to output part information of the part 400 placed on the tray 300. In one implementation example, the recognition device 100 performs the process of FIG. 6 by causing the CPU 101 to execute a given program. In one implementation example, the recognition device 100 starts the process of FIG. 6 in a state where it has already acquired the designation of the part ID to be processed.

[0041] In step S100, the recognition device 100 acquires an image captured by the camera 120. In one implementation example, the captured image is generated by the camera 120 capturing an image of the component 400 on the tray 300, and is stored in the RAM 102 or the storage 103.

[0042] In step S110, the recognition apparatus 100 refers to an external shape library to identify a combination of the orientation and rotation angle of the part.

[0043] More specifically, recognition device 100 extracts elements corresponding to parts from the image acquired in step S100. Then, recognition device 100 identifies an image pattern that matches the extracted element by comparing the shape of the extracted element with image patterns registered in an external shape library. If the matching rate between the extracted element and an image pattern registered in the external shape library is equal to or greater than a given value, recognition device 100 may determine that the element corresponds to the image pattern. Then, recognition device 100 identifies the combination of orientation and rotation angle associated with the identified image pattern as the combination of orientation and rotation angle of the part appearing in the captured image.

[0044] The recognition device 100 may identify two or more combinations. For example, assume that the matching rate between the image pattern corresponding to the combination of the orientation "front" and the rotation angle "15°" and the above element is "80%," the matching rate between the image pattern corresponding to the combination of the orientation "back" and the rotation angle "15°" and the above element is "82%," and the given value is "75%." In this case, the recognition device 100 determines that the above element matches each of these two image patterns. Then, the recognition device 100 identifies the combinations of the orientations and rotation angles of these two image patterns, i.e., the two combinations, as combinations of the orientations and rotation angles of the part captured in the captured image.

[0045] In the following description, the combination identified in step S110 may also be referred to as a "first result."

[0046] In step S120, recognition device 100 determines whether the number of combinations identified in step S110 is 1. If the number of combinations identified is 1 (YES in step S120), recognition device 100 proceeds to step S130, and if not (NO in step S120), recognition device 100 proceeds to step S140.

[0047] In step S130, the recognition device 100 outputs the combination identified in step S110 as the final result of the part information, and ends the processing in Fig. 6. The output may be displayed on the display 104, or may be transmitted to an external device (for example, the robot arm 200).

[0048] In step S140, recognition device 100 performs image processing on the image read in step S100. In one implementation, this image processing is performed to convert an image captured to recognize the outer edge of a component into an image for recognizing the internal structure of the component. The image processing may, for example, involve applying gamma correction to the image.

[0049] In step S150, the recognition apparatus 100 refers to the internal shape library to identify a combination of the orientation and rotation angle of the part.

[0050] More specifically, in step S140, recognition device 100 extracts elements corresponding to parts from the image that has been subjected to image processing. Then, recognition device 100 identifies an image pattern that matches the extracted element by comparing the shape of the extracted element with image patterns registered in an internal shape library. Then, recognition device 100 identifies a combination of orientation and rotation angle associated with the identified image pattern as the combination of orientation and rotation angle of the part appearing in the captured image. In the following description, the combination identified in step S150 may also be referred to as a "second result."

[0051] In step S160, the recognition device 100 uses the first result and the second result to determine a final result for the part information. In one implementation example, the recognition device 100 identifies an image pattern that has the highest matching rate in both the first result and the second result, and determines the combination of orientation and rotation angle associated with that image pattern as the final result.

[0052] For example, suppose that in the first result, the matching rate between the element extracted from the captured image and the image pattern of "rotation angle: 0°, orientation: front" is 80%, and the matching rate between the element extracted from the captured image and the image pattern of "rotation angle: 0°, orientation: back" is 82%. On the other hand, in the second result, the matching rate between the element extracted from the captured image and the image pattern of "rotation angle: 0°, orientation: front" is 10%, and the matching rate between the element extracted from the captured image and the image pattern of "rotation angle: 0°, orientation: back" is 90%. In this case, the image pattern of "rotation angle: 0°, orientation: back" is the image pattern with the highest matching rate in both the first result and the second result. Therefore, the combination of a rotation angle of "0°" and an orientation of "back" is identified as the final result.

[0053] In step S170, similar to step S130, the recognition device 100 outputs the final result determined in step S160, and ends the processing of FIG.

[0054] [7. Variation (1)] Fig. 7 is a flowchart of a first modified example of the process of Fig. 6. In the process of Fig. 7, the captured image used in step S150 is captured under different imaging conditions from the captured image used in step S110. For this reason, the process of Fig. 7 includes steps S102 and S142 instead of steps S100 and S140.

[0055] More specifically, in step S102, the recognition device 100 reads out a captured image for an external shape. In step S110, the recognition device 100 uses the captured image read out in step S102.

[0056] In addition, in step S142, the recognition device 100 reads out the captured image for the internal shape. In step S150, the recognition device 100 uses the captured image read out in step S142.

[0057] The camera 120 may capture the image read out in step S102 (captured image for external shape) and the image read out in step S142 (captured image for internal shape) under different capture conditions. In one implementation example, the camera 120 captures the image for internal shape with a longer exposure time, higher gain, and / or higher illumination intensity (in a bright environment) than the image for external shape. This makes it possible to both clarify the contours of elements corresponding to the part in the image for external shape and clarify the internal structure of the part in the image for internal shape.

[0058] [8. Variation (2)] Fig. 8 is a flowchart of a second modified example of the process of Fig. 6. In the process of Fig. 8, steps S120 and S130 are omitted compared to the process of Fig. 6. That is, in the process of Fig. 8, the recognition device 100 performs the control of steps S140 to S170 even if the number of combinations identified in step S110 is one.

[0059] Note that in step S150, recognition device 100 may identify only the "orientation" of the part information. If the outer edge shape of the front surface and the outer edge shape of the back surface of the part are the same or very similar, the first result of step S110 may be two combinations: one that defines the front surface at a certain rotation angle, and another that defines the back surface at that rotation angle. In such a case, if recognition device 100 identifies at least the "orientation" of the part information in step S150, it can identify the final combination by selecting the combination that includes the identified "orientation" from the two combinations.

[0060] [9. Variation (3)] Fig. 9 is a diagram showing a modified example of the data structure of the external shape library. In the example of Fig. 9, compared to the example of Fig. 4, for all rotation angles of the part ID "0001," both the "front" and "back" orientations are associated with the same image pattern without distinction. This corresponds to the fact that the part identified by the part ID "0001" has the same outer edge shape on both the front and back sides.

[0061] If an element in the captured image has a high matching rate with image patterns associated with both the front and back sides in the external shape library, the recognition device 100 identifies two combinations associated with the image pattern in step S110. For example, the two combinations are "rotation angle: 0°, orientation: front" and "rotation angle: 0°, orientation: back." In step S150, the recognition device 100 selects, as the final result, one of the two combinations that has a higher matching rate in the second result.

[0062] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. Furthermore, the inventions described in the embodiments and modifications are intended to be practiced, as far as possible, either alone or in combination. [Explanation of symbols]

[0063] 100 Recognition device, 102 RAM, 103 Storage, 104 Display, 105 Input device, 106 Interface, 110 Main body, 120 Camera, 151 Image acquisition unit, 152 External shape recognition unit, 153 Internal shape recognition unit, 154 Identification unit, 200 Robot arm, 300 Tray, 400, ID part, 1000 Part recognition system.

Claims

1. An object recognition device for recognizing the state of an object, an image acquisition unit that reads an image of the object taken by a camera; an external shape recognition unit that recognizes an external shape of the object based on an image of the object captured by the camera; an internal shape recognition unit that recognizes an internal shape of the object based on an image of the object; an identification unit that identifies a state of the object by using the recognition result of the external shape recognition unit and the recognition result of the internal shape recognition unit; the state of the object includes an orientation and a rotation angle of the object; the orientation of the object includes at least a front-facing and a back-facing; the image used by the external shape recognition unit to recognize the external shape is a first image captured of the object, An object recognition device, wherein the image used by the internal shape recognition unit to recognize the internal shape is a second image captured under different imaging conditions than the first image.

2. An object recognition device as described in claim 1, wherein the second image is captured with a longer exposure time, higher gain, and / or higher lighting illuminance than the first image due to the different imaging conditions.

3. 3. The object recognition device according to claim 1, wherein both the external shape recognition unit and the internal shape recognition unit recognize both the orientation and rotation angle of the object.

4. further comprising a storage device; The storage device is an external shape library that associates image patterns of the external shape of the object with the orientation and rotation angle of the object; an internal shape library that associates image patterns of the internal shape of the object with the orientation and rotation angle of the object; Store the external shape recognition unit recognizes the orientation and rotation angle of the object based on a matching rate between elements included in an image captured of the object and image patterns included in the external shape library; The object recognition device according to claim 3 , wherein the internal shape recognition unit recognizes the orientation and rotation angle of the object based on a matching rate between elements included in an image captured of the object and image patterns included in the internal shape library.

5. the external shape recognition unit recognizes both the orientation and rotation angle of the object; the orientation of the object recognized by the external shape recognition unit includes two or more orientations, 5. The object recognition device according to claim 1, wherein the internal shape recognition unit identifies one of the two or more orientations.

6. Further comprising a storage device, The storage device is an external shape library that associates image patterns of the external shape of the object with the orientation and rotation angle of the object; an internal shape library that associates image patterns of the internal shape of the object with the orientation and rotation angle of the object; Store the external shape library associates two or more orientations with a rotation angle; the external shape recognition unit recognizes a rotation angle of the object based on a matching rate between an element included in an image captured of the object and an image pattern included in the external shape library; The object recognition device according to claim 5 , wherein the internal shape recognition unit recognizes the orientation of the object based on a matching rate between elements included in an image captured of the object and image patterns included in the internal shape library.

7. The object recognition device according to any one of claims 1 to 6, further comprising the camera.

8. When executed by a computer, the computer: Retrieving a first image of an object taken by a camera and a second image of the object taken by the camera under different imaging conditions relative to the first image; Recognizing an external shape of the object based on the first image; recognizing an internal shape of the object based on the second image; and identifying a state of the object using the external shape recognition result and the internal shape recognition result; the state of the object includes an orientation and a rotation angle of the object; The orientation of the object includes at least a front-facing and a back-facing.

9. The program described in Claim 8, wherein the different imaging conditions result in the second image being captured with a longer exposure time, higher gain, and / or higher illumination intensity than the first image.

Images