Object recognition device, object recognition method

Abstract

To provide an object recognition device detecting an object in an image that can accurately detect a target object even when the background or other objects appear.SOLUTION: The object recognition device comprises: an image acquisition part acquiring a first image composed of two-dimensional pixels; a three-dimensional shape approximation determination part determining whether image information of a predetermined rectangular region in the first image can be approximated to predetermined three-dimensional shape information; an image region estimation part cutting out the rectangular region as a first estimated region based on the determination result of the three-dimensional shape approximation determination part; and a region selection part selecting, among the multiple first estimated regions, a region where the area of the first estimated region is the smallest.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an object recognition device and an object recognition method for detecting an object in an image, and in particular to a technique that is effective when applied to object detection in an image in which a background or other objects are captured in the same image. [Background technology]

[0002] One of the tasks that industrial robots are used for is order picking in warehouses. Picking work involves retrieving and collecting (pickup) products stored in a warehouse according to a picking list or slip, and requires accuracy and speed. In order to improve the efficiency of picking work and eliminate human error, it is expected that the application of industrial robots equipped with object detection functions using image processing will expand.

[0003] Knowledge about object detection using image processing is being applied in many fields, such as facial recognition and autonomous driving, and accuracy has improved significantly with advances in deep learning technology; however, technical challenges specific to the field of object detection remain.

[0004] For example, when trying to use image processing to detect a target product from a pile of many different types of products, depending on the object's posture, there is a possibility that an object-containing area that includes the background or other objects may be estimated, raising concerns about a decrease in object detection accuracy.

[0005] Background art in this technical field includes, for example, technology such as that described in Patent Document 1. Patent Document 1 discloses "an authentication system that does not require processing of an authentication target, extracts unique information from image data that captures individual features of the authentication target using a digital imaging device with a simple configuration, and determines whether the authentication target is legitimate based on the extracted unique information."

[0006] Furthermore, Non-Patent Document 1 discloses object detection using an object detection network that also estimates the inclination angle of an object. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2019-139640 A [Non-patent literature]

[0008] [Non-Patent Document 1] Xingjia Pan et al., “Dynamic Refinement Network for Oriented and Densely Packed Object Detection”, 2020 Computer Vision and Pattern Recognition, 10 Jun 2020, pp 11207-11216 Summary of the Invention [Problem to be solved by the invention]

[0009] As described above, when performing picking operations using object detection through image processing, depending on conditions such as the object's posture, an object-containing area that includes the background or other objects may be estimated, and the expected object detection accuracy may not be achieved.

[0010] In the above-mentioned Patent Document 1, a template of the object is prepared, then the template is rotated to create a pseudo template, and the object region of the object with different poses is estimated by matching the object shown in the input image with the pseudo template.

[0011] However, it is necessary to prepare a template for each object, which is disadvantageous in terms of efficiency and cost of the picking work.

[0012] In the above-mentioned Non-Patent Document 1, the addition of items to be estimated makes learning more difficult. Also, in addition to the annotations required for a normal object detection network for learning, annotations such as for adjusting the tilt angle and the rectangle are required separately, which increases the annotation man-hours.

[0013] Therefore, an object of the present invention is to provide an object recognition device and an object recognition method that can detect a target object with high accuracy, even in an image that includes the background or other objects, in an object recognition device that detects objects in an image. [Means for solving the problem]

[0014] In order to solve the above problem, the present invention is characterized by comprising an image acquisition unit that acquires a first image composed of two-dimensional pixels, a three-dimensional shape approximation determination unit that determines whether image information of a predetermined rectangular area in the first image can approximate predetermined three-dimensional shape information, an image area estimation unit that cuts out the rectangular area as a first estimated area based on a determination result of the three-dimensional shape approximation determination unit, and an area selection unit that selects, from a plurality of the first estimated areas, the area of ​​the first estimated area that is the smallest.

[0015] The present invention is also characterized by comprising the steps of: (a) acquiring a first image composed of two-dimensional pixels; (b) determining whether image information of a predetermined rectangular area in the first image can approximate predetermined three-dimensional shape information; (c) extracting the rectangular area as a first estimated area based on the determination result of step (b); and (d) selecting, from a plurality of the first estimated areas, the area of ​​the first estimated area that is the smallest. Effect of the Invention

[0016] According to the present invention, in an object recognition device that detects an object in an image, it is possible to realize an object recognition device and an object recognition method that can detect a target object with high accuracy even in an image that includes the background or other objects.

[0017] This enables accurate and quick picking operations within the warehouse.

[0018] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0019] [Figure 1] 1 is a diagram showing a schematic configuration of an object recognition device according to a first embodiment of the present invention. [Diagram 2] 3 is a flowchart showing an object recognition method according to the first embodiment of the present invention. [Diagram 3] 2 is a diagram conceptually illustrating the function of a region selection unit in FIG. 1; [Figure 4] FIG. 1 is a diagram showing an example of work to which the present invention is applied; [Diagram 5] FIG. 1 is a diagram showing a problem with the conventional technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same components are designated by the same reference numerals, and detailed description of overlapping components will be omitted. EXAMPLES

[0021] First, an example of work to which the present invention is applied and the problems that arise from it will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram showing an example of work to which the present invention is applied, and shows order picking work in a warehouse by an industrial robot 9. Fig. 5 is a diagram showing the problems that arise from the work in Fig. 4.

[0022] For example, consider a case where many different types of products 7 are stored in a pile on a cart 8 in a warehouse, and the shape of each product 7 is unknown. In the conventional technology, as shown in Fig. 4, each product is identified from a color image of the group of products 7 and a point cloud obtained by image processing. An industrial robot 7 picks out products 7 stored in the warehouse according to a picking list or slip.

[0023] In conventional picking operations using such an industrial robot 7, when individual products are stored relatively orderly without overlapping, as shown in FIG. 5, individual products can be accurately identified through image recognition by an object detection unit attached to the industrial robot 7.

[0024] On the other hand, if the individual products are stored in a cluttered manner with overlapping, or if the products are of various shapes and sizes, the image recognition by the object detection unit may estimate an object-containing area that includes the background or other objects, which may reduce the accuracy of object detection. Therefore, depending on the storage state of the product group, the industrial robot 7 may not be able to perform the picking work well.

[0025] Next, an object recognition device and an object recognition method of the present embodiment will be described with reference to Fig. 1 to Fig. 3. Fig. 1 is a diagram showing a schematic configuration of the object recognition device of the present embodiment. Fig. 2 is a flowchart showing the object recognition method of the present embodiment. Fig. 3 is a diagram conceptually showing the function of the area selection unit 5 of Fig. 1.

[0026] 1 and 2, the object recognition device 1 of this embodiment is mainly configured with an image acquisition unit 2, a three-dimensional shape approximate determination unit 3, an image area estimation unit 4, and an area selection unit 5. In addition, the three-dimensional shape approximate determination unit 3 has a learning device 6.

[0027] The image acquisition unit 2 acquires a first image made up of two-dimensional pixels based on an image captured by an image sensor (not shown).

[0028] The three-dimensional shape approximation determination unit 3 determines whether image information of a predetermined rectangular area in the first image acquired by the image acquisition unit 2 can approximate predetermined three-dimensional shape information.

[0029] Based on the determination result of the three-dimensional shape approximate determination section 3, the image area estimation section 4 cuts out a rectangular area as a first estimated area.

[0030] The region selection unit 5 selects, from among the multiple first estimated regions cut out by the image region estimation unit 4, the first estimated region having the smallest area.

[0031] A method for selecting a region having the smallest area of ​​the first estimated region by region selection unit 5 will be described with reference to FIG.

[0032] When a single image is acquired as the first image, a predetermined rectangular area is defined, for example, with an area (2, 7, 10) as shown in the upper diagram of Figure 3. Then, the area with the smallest area ("2") is selected as the object (product) to be detected.

[0033] Furthermore, when multiple images are acquired as the first image, a predetermined rectangular area is defined, for example, by area (any value between 2 and 10), as shown in the lower diagram of Fig. 3. Then, the area with the smallest area ("2") is selected as the object (product) to be detected.

[0034] In addition, the lower diagram in Figure 3 shows an example in which the image acquisition unit 2 generates four rotated images by rotating the first image (θ: 0°, 25°, 45°, 89°), and uses the rotated images as the first images.

[0035] The object recognition device 1 of this embodiment is configured as described above, and can detect a target object with high accuracy even in an image in which the background or other objects are captured.

[0036] A specific configuration example and functions of the object recognition device 1 will be described below.

[0037] <<Examples of "3D shape information">> The three-dimensional shape information may be at least one of a box type, a cylinder type, a cone type, a triangular prism type, a triangular pyramid type, a square pyramid type, a sphere type, and a torus type, and may also include types that can be expressed by a combination of three-dimensional shape information.

[0038] <<Example 1 of "First Image": When rotating an image>> In the image acquisition unit 2, for example, at least one rotated image obtained by rotating a first image may be generated, and the rotated image may be used as the first image.

[0039] <<Example of how to generate a "rotated image">> The image acquisition unit 2 may generate n rotated images (1≦n≦N) for each fixed discrete rotation angle value.

[0040] <<Example 2 of the "First Image": When rotating the image sensor (camera)>> In the image acquisition unit 2, when acquiring a first image from an image sensor (camera), the image sensor may be rotated around an axis in the optical axis direction of the image sensor to acquire at least one first image.

[0041] <How to rotate the "image sensor"> In the image acquisition unit 2, the image sensor may be rotated n times (1≦n≦N) for each fixed discrete rotation angle value.

[0042] <<Example 1 of the "3D shape approximation determination unit": The part that determines whether it can approximate 3D shape information uses machine learning methods>> The three-dimensional shape approximation determination unit 3 may determine whether the predetermined three-dimensional shape information can be approximated by using a learning device 6 that has been trained using a second image including a general object that can be approximated to the predetermined three-dimensional shape information, information indicating a rectangular area including the general object in the second image, and the predetermined three-dimensional shape information as learning data.

[0043] <<Example 2 of the "3D shape approximation determination unit": The part that estimates the position of a specified rectangular area and determines whether it can be approximated to 3D shape information is a machine learning method>> The three-dimensional shape approximation determination unit 3 may use a learning device that has been trained using a second image including a general object that can be approximated to predetermined three-dimensional shape information, information indicating a rectangular area including a general object in the second image, and the predetermined three-dimensional shape information as learning data, to estimate the position of the predetermined rectangular area in the first image and determine whether it can be approximated to the predetermined three-dimensional shape information.

[0044] <<Examples of "Learning Data">> The learning data may be formed by placing a 3D model having a size randomly selected from a certain range based on predetermined 3D shape information in a random position and orientation within a virtual environment, and placing a virtual camera in a position and orientation that allows it to acquire an image of the surface of the 3D model, and capturing the image as the second image, and the second image may consist of information indicating a rectangular area containing the 3D model and the predetermined 3D shape information.

[0045] <Simultaneous estimation of multiple objects> In the region selection unit 5, rectangular regions containing the same object may be identified from all of the first estimated regions to generate at least one group of identical object rectangular regions, and a rectangular region with the smallest area may be selected for each group of identical object rectangular regions.

[0046] <<Procedure for generating "identical object rectangular area group">> generating an identical rectangle containing region group having as its element information obtained from a first estimated region cut out from the nth first image, and an identical object tracking list having as its element an identical object rectangular region group; estimating information of an expected existence region in the n+1th first image using the rotation angle at which the n+1th first image was generated and information obtained from the latest first estimated region of the identical object rectangular region group, which is each element of the identical object tracking list; and generating an identical object expected existence region group having as its element information obtained from the first estimated region or information of the expected existence region, using the information of the expected existence region and the identical object tracking list; It is also possible to generate an expected presence area list having the same object expected presence area group as its elements, and depending on the degree of match between the information obtained from the first estimated area cut out from the n+1th first image and the latest element of the same object expected presence area group, which is each element of the expected presence area list, determine whether to add the information obtained from the first estimated area cut out from the n+1th first image as an element of the same object rectangular area group and update the same object tracking list, or to update the same object tracking list as a new same object rectangular area group, thereby generating the same object rectangular area group.

[0047] ≪Specific example 1 of "Information obtained from the first estimated region"≫ An Axis Aligned Bounding Box or an Oriented Bounding Box may be used as the information obtained from the first estimated area and the information on the predicted existence area, and an Intersection over Union (IOU) may be used as the degree of agreement between the information obtained from the first estimated area cut out from the n+1th first image and the latest element of the group of predicted existence areas of the same object, which is each element of the predicted existence area list.

[0048] ≪Specific example 2 of "Information obtained from the first estimated region"≫ The center of gravity of the first estimated area may be used as information obtained from the first estimated area, the center of gravity of the predicted existence area may be used as information of the predicted existence area, and the distance between the center of gravity may be used as the degree of match between information obtained from an object rectangular area belonging to the (n+1)th rotated input image and the latest element of each group of predicted existence areas of the same object in the predicted existence area list.

[0049] The present invention is not limited to the above-described embodiments, but includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0050] 1…Object recognition device 2. Image acquisition section 3...3D shape approximation determination section 4...Image area estimation section 5…Area selection section 6…Learning device 7…Product(group) 8…Trolley 9. Industrial robots

Claims

1. an image acquisition unit that acquires a first image composed of two-dimensional pixels; a three-dimensional shape approximation determination unit that determines whether image information of a predetermined rectangular area in the first image can approximate predetermined three-dimensional shape information; an image area estimation unit that cuts out the rectangular area as a first estimated area based on a result of the determination by the three-dimensional shape approximation determination unit; a region selection unit that selects, from among the plurality of first estimated regions, a region having a smallest area of ​​the first estimated region; An object recognition device comprising:

2. The object recognition device according to claim 1 , An object recognition device characterized in that the three-dimensional shape information is at least one of a box type, a cylinder type, a cone type, a triangular prism type, a triangular pyramid type, a square pyramid type, a sphere type, and a torus type, and also includes types that can be expressed by a combination of the three-dimensional shape information.

3. The object recognition device according to claim 1 , An object recognition device characterized in that, in the image acquisition unit, at least one rotated image obtained by rotating the first image is generated, and the rotated image is used as the first image.

4. The object recognition device according to claim 3, The object recognition device generates n rotated images (1≦n≦N) for each fixed discrete rotation angle value.

5. The object recognition device according to claim 1 , An object recognition device characterized in that, in the image acquisition unit, when acquiring the first image from the image sensor, the image sensor is rotated around an optical axis direction of the image sensor as an axis to acquire at least one of the first images.

6. The object recognition device according to claim 5 , An object recognition device, characterized in that the image sensor is rotated n times (1≦n≦N) for each constant discrete rotation angle value.

7. The object recognition device according to claim 1 , The object recognition device is characterized in that the three-dimensional shape approximation determination unit determines whether the specified three-dimensional shape information can be approximated by using a learning device that has been trained using a second image including a general object that can be approximated to the specified three-dimensional shape information, information indicating a rectangular area including the general object in the second image, and the specified three-dimensional shape information as learning data.

8. The object recognition device according to claim 1 , The object recognition device is characterized in that the three-dimensional shape approximation determination unit estimates the position of the specified rectangular area in the first image using a learning device that has been trained using a second image including a general object that can be approximated to the specified three-dimensional shape information, information indicating a rectangular area including the general object in the second image, and the specified three-dimensional shape information as learning data, and determines whether the specified three-dimensional shape information can be approximated.

9. The object recognition device according to claim 7 or 8, The object recognition device is characterized in that the learning data is composed of information indicating a rectangular area including the 3D model and the specified three-dimensional shape information, the image being obtained by placing a 3D model having a size randomly selected from a certain range based on the specified three-dimensional shape information in a random position and orientation within a virtual environment, and capturing the image with a virtual camera placed in a position and orientation that allows an image of the surface of the 3D model, and the second image being information indicating a rectangular area including the 3D model and the specified three-dimensional shape information.

10. The object recognition device according to claim 4, The object recognition device is characterized in that the area selection unit identifies rectangular areas containing the same object from all of the first estimated areas to generate at least one group of identical object rectangular areas, and selects a rectangular area with the smallest area for each of the group of identical object rectangular areas.

11. The object recognition device according to claim 10, generating an identical rectangle-containing region group having as its element information obtained from the first estimated region cut out from the n-th first image, and an identical object tracking list having as its element the identical object rectangular region group; estimating information of a predicted existence area in the n+1th first image using a rotation angle at which the n+1th first image was generated and information obtained from the latest first estimated area of ​​the same object rectangular area group that is each element of the same object tracking list; Using the information on the predicted existence area and the same object tracking list, generate a group of predicted existence areas of the same object having information obtained from the first estimated area or the information on the predicted existence area as an element, and generate a predicted existence area list having the group of predicted existence areas of the same object as an element; An object recognition device characterized by generating the same object rectangular area group by determining whether to add the information obtained from the first estimated area cut out from the n+1th first image as an element of the same object rectangular area group and update the same object tracking list, or to update the same object tracking list as a new same object rectangular area group, depending on the degree of match between information obtained from the first estimated area cut out from the n+1th first image and the latest element of the same object predicted existence area group, which is each element of the predicted existence area list.

12. The object recognition device according to claim 11, An object recognition device characterized in that an Axis Aligned Bounding Box or an Oriented Bounding Box is used as the information obtained from the first estimated area and the information of the predicted existence area, and an Intersection over Union (IOU) is used as the degree of agreement between the information obtained from the first estimated area cut out from the n+1th first image and the latest element of the group of predicted existence areas of the same object, which is each element of the predicted existence area list.

13. The object recognition device according to claim 11, Using a center of gravity of the first estimated area as information obtained from the first estimated area, The center of gravity of the predicted existence area is used as information of the predicted existence area, An object recognition device characterized in that the distance between center points is used as the degree of agreement between information obtained from an object rectangular area belonging to the (n+1)th rotated input image and the latest element of each predicted existence area group of the same object in the predicted existence area list.

14. A method for object recognition comprising the steps of: (a) acquiring a first image composed of two dimensional pixels; (b) determining whether image information of a predetermined rectangular area in the first image can approximate predetermined three-dimensional shape information; (c) extracting the rectangular area as a first estimated area based on the determination result of the (b) step; (d) selecting, from among the plurality of first estimated regions, the region having the smallest area as the first estimated region;

15. The object recognition method according to claim 14, The object recognition method is characterized in that the three-dimensional shape information is at least one of a box type, a cylinder type, a cone type, a triangular prism type, a triangular pyramid type, a square pyramid type, a sphere type, and a torus type, and also includes types that can be expressed by a combination of the three-dimensional shape information.

Images