OBJECT RECOGNITION DEVICE, ROBOT SYSTEM, AND OBJECT RECOGNITION METHOD

Abstract

To provide an object recognition device capable of recognizing a lump of conveyed objects (the same article group) as a conveyance unit on the basis of the arrangement of individual articles in the article group.SOLUTION: An object recognition device that recognizes the same article group as a conveyance unit under an environment where a plurality of articles exist, includes: an input unit that acquires images of the plurality of articles; an article detection unit that detects an article region where the articles exist, from the images; and a same article group region estimation unit that acquires inter-article region information as information on the arrangement of the article region, calculates frequency distribution for the inter-article region information, and estimates a region of the same article group on the basis of the frequency distribution.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an object recognition device, a robot system, and an object recognition method. [Background technology]

[0002] In recent years, the use of robots such as articulated arm robots for transporting goods has been increasing in logistics warehouses, etc. When this type of robot transports, for example, multiple PET bottles all wrapped in clear plastic wrap, it needs to recognize the group of PET bottles wrapped in clear plastic wrap as the transport unit, rather than each individual PET bottle, and then grasp the group of PET bottles, which is a single object to be transported, in an appropriate position.

[0003] A known prior art technology capable of recognizing a group of items wrapped entirely in transparent wrap as a single object to be transported is the cargo handling control device described in Patent Document 1. For example, the abstract of Patent Document 1 describes the problem of "appropriately moving items even when multiple items are bundled together under a sheet-like covering," and as a solution to this problem, it describes that "the cargo handling control of an embodiment includes a transceiver and a control unit. The transceiver is provided in a cargo handling device that grasps and moves items placed on a placement unit, transmits ultrasonic or radio waves as transmission waves in the direction of the placement unit, and receives reflected waves of the transmission waves. When the control unit recognizes that multiple items are present on the placement unit based on a captured image of the items placed on the placement unit, it determines whether the multiple items are bundled together under a sheet-like covering based on the recognition result based on the captured image and the reception result of the reflected waves from the transceiver, and controls the cargo handling device based on the determination result."

[0004] Furthermore, paragraph 0068 of the same document states, regarding a covering for bundling multiple items, that "Covering B is a transparent or translucent vinyl sheet or the like. Covering B is also called bundling material, wrapping sheet (sheet), or shrink film (film)." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-109257 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the cargo handling control device of Patent Document 1, as shown in Figures 1, 6, 7, 9, etc. of the document, has a sensor device 20 attached to the gripping part 113 at the tip of the robot arm, and determines whether multiple items are bundled together under a sheet-like covering by identifying the reflection point of the ultrasonic waves transmitted and received by this sensor device 20, so there were the following operational problems.

[0007] (1) Because the detection distance of the sensor device 20, which uses ultrasonic waves or radio waves, is short, when determining whether or not there is a covering object, it is necessary to bring the sensor device 20 (i.e., the gripping unit 113 at the tip of the robot arm) close to the object and identify the point where the ultrasonic waves are reflected (see Figures 3, 7, 9, etc. of Patent Document 1). For this reason, with the cargo handling control device of Patent Document 1, when it is desired to detect all covering objects in a captured image, it is necessary to scan the entire area of ​​the captured image with the sensor device 20, which poses the problem that it takes a very long time to determine whether or not there is a covering object.

[0008] (2) When the items to be transported are a group of items bound on the sides with transparent wrap, cardboard, etc., there is no covering on top of the group of items, so the cargo handling control device of Patent Document 1 cannot recognize the group of items as a single object to be transported, and there is a problem in that it mistakes the group of items for multiple items simply placed together in one place.

[0009] In consideration of the above problems, the present invention aims to provide an object recognition device and an object recognition method that can recognize a group of transport objects (same group of items) that is a transport unit based on the arrangement of individual items within the group of items. [Means for solving the problem]

[0010] In order to solve the above problem, the object recognition device of the present invention is an object recognition device that recognizes a group of identical items that is a transport unit in an environment where multiple items are present, and is equipped with an input unit that acquires images of the multiple items, an item detection unit that detects an item area where the items are present from the image, and an identical item group area estimation unit that acquires information between each item area, which is information regarding the arrangement of the item areas, calculates a frequency distribution of the item area information, and estimates the area of ​​the identical item group based on the frequency distribution. [Effects of the Invention]

[0011] According to the object recognition device and object recognition method of the present invention, it is possible to recognize a group of conveyance targets (same group of items) that is a conveyance unit, based on the arrangement of individual items within the group of items. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram illustrating a usage environment of an object recognition device according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing a hardware configuration of an object recognition device according to a first embodiment. [Figure 3] FIG. 1 is a functional block diagram of an object recognition device according to a first embodiment. [Figure 4] 5A and 5B are diagrams for explaining the processing of an item detection unit and an identical item region estimation unit. [Figure 5] FIG. 10 is a diagram for explaining the processing of the object recognition apparatus according to the second embodiment. [Figure 6] FIG. 10 is a diagram for explaining the processing of the object recognition apparatus according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the object recognition device of the present invention will be described with reference to the drawings. [Example]

[0014] First, an object recognition device 1 according to a first embodiment of the present invention and a robot system using the same will be described with reference to FIGS.

[0015] Fig. 1 is a schematic diagram showing the environment in which the object recognition device 1 is used. In the figure, reference numeral 2 denotes a multi-articulated arm robot (hereinafter simply referred to as "robot") having a multi-articulated arm 21 and a hand 22 and controlled by the object recognition device 1, 3 denotes a stereo camera that transmits stereo images captured synchronously by a left camera 3L and a right camera 3R to the object recognition device 1, 4 denotes various items (e.g., plastic bottles, toilet paper, cardboard boxes, etc.) that are transported by the robot 2, and 5 denotes a pallet on which an item 4 is placed. As is clear from Fig. 1, the robot 2 is installed in a location where it can grasp any item 4 on the pallet 5 with the hand 22 by moving the multi-articulated arm 21, and the stereo camera 3 is installed in a location where it can capture an image of the entire top surface of the pallet 5.

[0016] 2 is a block diagram showing the hardware configuration of the object recognition device 1. As shown here, the object recognition device 1 is a computer including a processor 11 such as a CPU, a storage device 12 such as a semiconductor memory, an input device 13 such as a keyboard or a mouse, an output device 14 such as an LCD display, a communication interface 15 for communicating with the robot 2 and the stereo camera 3, and a bus 16 connecting these. Note that the processor 11 executes an object recognition processing program 12a stored in the storage device 12 to realize an item detection unit 11a and an identical item group area estimation unit 11b, which will be described later.

[0017] 3 is a functional block diagram of the object recognition device 1. As shown here, the object recognition device 1 includes an input unit 15a that receives stereo images from the stereo camera 3, an item detection unit 11a that detects items 4 by processing the received stereo images, a identical item group area estimation unit 11b that estimates an identical item area corresponding to a transport unit based on the arrangement of the detected items 4, and an output unit 15b that transmits a command to the robot 2 based on the position of the estimated identical item area. The input unit 15a and the output unit 15b are functional units realized by the communication interface 15 in FIG. 2.

[0018] <Details of the Item Detection Unit 11a and the Same Item Group Region Estimation Unit 11b> Next, the details of the processing performed by the item detection unit 11a and the same item group area estimation unit 11b will be explained in order from step S1 to step S5 with reference to Fig. 4. In the following, it is assumed that the items 4 transported by the robot 2 are PET bottles, and that the transport unit (a mass of objects to be transported) is a 3x3 array of nine PET bottles that are wrapped entirely in clear plastic wrap or bound on the sides and bottom with clear plastic wrap.

[0019] <<Step S1 (Acquisition of Imaging Information)>> In step S1, the item detection unit 11a acquires image information of the object based on the stereo images captured by the stereo camera 3. Because the left camera 3L and right camera 3R of the stereo camera 3 are positioned a predetermined distance apart, when the left image captured by the left camera 3L and the right image captured by the right camera 3R are compared, a parallax corresponding to the distance to the object is observed. Therefore, the item detection unit 11a can calculate the distance to the captured object by processing the stereo images using the principle of triangulation.

[0020] When a monocular camera is used instead of the stereo camera 3, the captured image itself may be acquired as the imaging information.

[0021] <<Step S2 (Detection of the object area)>> In step S2, the item detection unit 11a detects item regions where individual PET bottles (items 4) exist based on the imaging information acquired in step S1. Since the distance to the subject is calculated in step S1, each of the flat circular regions on the same plane, as shown in the image diagram of Fig. 4(a), can be extracted as an item region where the top end of the item 4 (specifically, the cap of the PET bottle) exists.

[0022] If a monocular camera is used instead of the stereo camera 3, the object area where each individual PET bottle (object 4) exists can be extracted by pattern matching the image, which is the image capture information, with the known top view shape and color of the PET bottle (object 4).

[0023] <<Step S3 (Acquisition of information between each item area)>> In step S3, the same item group area estimation unit 11b acquires inter-item area information relating to the arrangement of the item areas detected in step S2. Specifically, as shown in the image diagram of FIG. 4(b), the same item group area estimation unit 11b calculates the distance (indicated by a solid line in FIG. 4(b)) to adjacent item areas in the vertical or horizontal direction in the image for each item area detected in step S2, and acquires this as inter-item area information. Note that the angle with adjacent item areas or the normal direction of each item area may also be used as inter-item area information. Pattern information may also be used as inter-item area information.

[0024] <<Step S4 (Calculation of frequency distribution)>> In step S4, the same item group area estimation unit 11b calculates the frequency distribution of each item area information (distance, angle, normal direction) acquired in step S3. For example, as shown in the right graph in Figure 4(c), the same item group area estimation unit 11b first arranges distance information, which is one type of item area information acquired in step S3, on a graph with the horizontal axis representing the item-to-item distance and the vertical axis representing frequency. In this example, two types of item-to-item distances are calculated as the frequency distribution of each item area information: a high-frequency distance group indicated by a solid line and a low-frequency distance group indicated by a dotted line. Note that a graph is created in the same way when the item area information is angle or normal direction information.

[0025] <<Step S5 (Estimation of the Same Item Group Area)>> In step S5, the same object group region estimation unit 11b estimates the same object group region based on the frequency distribution calculated in step S4. As illustrated in FIG. 4(c), the distances between the objects 4 are divided into a relatively short distance group indicated by the solid lines and a relatively long distance group indicated by the dotted lines. Since the latter distance group is considered to be the distance between transport units, the same object group region estimation unit 11b regards the portions indicated by the dotted lines where the distances between the objects 4 are relatively long as the boundaries of the transport units, and estimates two same object group regions as shown in FIG. 4(d). Position information of the two estimated same object group regions is transmitted to the control unit of the robot 2 via the output unit 15b and is used by the robot 2 to properly grasp the transport unit consisting of nine plastic bottles (objects 4) arranged in a 3x3 array.

[0026] Note that if the distance between the objects indicated by the dotted lines in Figure 4(c) is sufficiently long, the boundaries of the same object group region can be clearly identified. However, if the difference between the distance between objects within the same object group region and the distance between the same object group regions is small, it may be impossible to determine whether the group of objects captured by the stereo camera 3 is a single same object group region or multiple same object group regions. Therefore, in such cases, the probability that the captured group of objects is a single same object group region and the probability that it is multiple same object group regions may be calculated and output to the outside. This allows the robot 2 to determine its behavior based on both possibilities. Various methods for calculating the probability are possible. For example, the relative frequency may be output directly as the probability, or a clustering method such as K-means may be used.

[0027] <Effects of this Example> As described above, the object recognition device 1 of this embodiment can recognize a group of objects to be conveyed (a group of identical objects) as a unit of conveyance based on the arrangement of the objects imaged by a camera, without using an ultrasonic sensor or a radio wave sensor. As a result, the robot 2 can move the hand 22 to an appropriate position for grasping the group of objects to be conveyed that are bundled together in transparent wrap or the like. [Example]

[0028] Second Embodiment Next, an object recognition device 1 according to a second embodiment of the present invention will be described with reference to Fig. 5. Note that a duplicated description of points common to the first embodiment will be omitted.

[0029] When nine PET bottles (items 4) arranged in a 3x3 pattern were entirely wrapped in transparent plastic wrap as in Example 1, the stereo camera 3 was able to clearly capture the position of each PET bottle (item 4), and therefore, information between each item area (information about the distance between items) could be easily obtained, as shown in Figure 5(a).

[0030] However, many transparent wraps have patterns 6 printed on some parts, and when a PET bottle (item 4) is wrapped in such a patterned transparent wrap, the position of some of the PET bottles (item 4) may not be clearly captured.

[0031] Therefore, in step S3 of this embodiment, when the same item group area estimation unit 11b acquires information between each item area, it acquires the distance between the pattern 6 and the item 4, the angle between the pattern 6 and the item 4, or the normal direction of the pattern 6, as shown in Figure 5(b).

[0032] When a graph like the graph on the right of Figure 4(c) is created based on the information between each item region in this embodiment, high frequency groups and low frequency groups are formed, just as in Example 1.Therefore, by regarding the positions where low frequency groups exist as boundaries of transport units, it is possible to divide a large number of items 4 into an appropriate number of identical object group regions. [Example]

[0033] Next, an object recognition device 1 according to a third embodiment of the present invention will be described with reference to Fig. 6. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0034] In the above example, the characteristics of the group of items that constitute the unit of transport (the frequency distribution of information between each item area) were unknown, so it was necessary to estimate the state of the unit of transport based on the image each time it was captured.However, in this example, the characteristics of the group of items that constitute the unit of transport (the frequency distribution of information between each item area) are known, and it is possible to extract the group of items that constitute the unit of transport based on the known characteristics (the frequency distribution of information between each item area).

[0035] For example, as illustrated in Figure 6(a), if it is known in advance that four PET bottles (item 4) arranged in a 2x2 pattern are the transport unit, the frequency distribution of information between each item area, as shown in the graph on the right of Figure 6(a), is pre-registered as a criterion in the same item group area estimation unit 11b of this embodiment.

[0036] In this case, the eight plastic bottles (items 4) captured by the stereo camera 3 are arranged as shown in Figure 6(b), and even if the frequency distribution of the information between each item region calculated from that image shows a small difference between the solid line group and the dotted line group, as shown in the right graph of Figure 6(b), it can be determined that the solid line distance matches the characteristics of the reference transport unit (the frequency distribution of the information between each item region), and the dashed line distance does not match the characteristics of the reference transport unit (the frequency distribution of the information between each item region), so correction processing can be performed to emphasize the frequency of the solid line group.As a result, the object recognition device 1 of this embodiment can correctly divide the eight plastic bottles (items 4) in a 2 x 4 array into two identical item group regions consisting of four plastic bottles (items 4) in a 2 x 2 array.

[0037] In the above explanation, it is assumed that the distribution information of each inter-item area information is registered in advance, but future estimation of the same item group area may also be performed based on the frequency distribution of each inter-item area information calculated in the past. [Example]

[0038] Next, an object recognition device 1 according to a fourth embodiment of the present invention will be described. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0039] In this embodiment, the inter-item area information is a template of an area that includes two or more item areas, or a pattern feature acquired from that area. For example, if the items 4 to be conveyed are plastic bottles with patterns drawn on their caps and the orientation of the patterns is always consistent within the same item group, a distribution is calculated in which the horizontal axis represents the value of the pattern feature and the vertical axis represents frequency. Based on this calculation result, the same item group area can be estimated by detecting differences in the templates or differences in the pattern feature (differences on the horizontal axis of the frequency distribution) to detect deviations in the orientation of the patterns. [Explanation of symbols]

[0040] 1 Object recognition device 11 processors 11a Item detection unit 11b Same article group area estimator 12 Storage Devices 12a Object recognition processing program 13 Input Devices 14 Output Devices 15 Communication Interface 15a Input section 15b Output section 16 Bus 2. Robot 21 Articulated Arm 22 hands 3 Stereo Camera 3L Left camera 3R Right Camera 4 Goods 5 palettes 6 Pattern

Claims

1. An object recognition device that recognizes a group of identical items as a transport unit in an environment where multiple items exist, an input unit for acquiring images of the plurality of articles; an object detection unit that detects an object area where the object is present from the image; a same item group area estimation unit that acquires information between item areas, which is information about the arrangement of the item areas, calculates a frequency distribution of the information between item areas, and estimates an area of ​​the same item group based on the frequency distribution; The group of identical items is a transport unit in which a plurality of items are all wrapped in transparent wrap, The transparent wrap has a pattern printed on it, The object recognition device is characterized in that the information between each item area is the distance between the pattern and the adjacent item area, the angle between the pattern and the adjacent item area, or the normal direction of the pattern.

2. An object recognition device that recognizes a group of identical items as a transport unit in an environment where multiple items exist, an input unit for acquiring images of the plurality of articles; an object detection unit that detects an object area where the object is present from the image; a same item group area estimation unit that acquires information between item areas, which is information about the arrangement of the item areas, calculates a frequency distribution of the information between item areas, and estimates an area of ​​the same item group based on the frequency distribution; The object recognition device is characterized in that the same item group area estimation unit further calculates the probability that the multiple items exist in a single same item group area and the probability that the multiple items are distributed across multiple same item group areas.

3. An object recognition device that recognizes a group of identical items as a transport unit in an environment where multiple items exist, an input unit for acquiring images of the plurality of articles; an object detection unit that detects an object area where the object is present from the image; a same item group area estimation unit that acquires information between item areas, which is information about the arrangement of the item areas, calculates a frequency distribution of the information between item areas, and estimates an area of ​​the same item group based on the frequency distribution; An object recognition device characterized in that a reference frequency distribution is registered in advance in the same item group area estimation unit, and the area of ​​the same item group is estimated based on the reference.

4. In the object recognition device according to claim 2 or claim 3, An object recognition device characterized in that the group of identical articles is a transport unit in which multiple articles are bound on the sides with plastic wrap or cardboard.

5. In the object recognition device according to claim 3, An object recognition device, wherein the reference frequency distribution is a frequency distribution previously calculated by the identical object group region estimation unit.

6. A robot system comprising the object recognition device according to any one of claims 1, 2, 3, and 5, and a robot that transports the group of identical items, The object recognition device an output unit that outputs information about the estimated area of ​​the same item group; a control unit that controls the robot based on the information on the area of ​​the group of identical items output from the output unit; A robot system comprising:

7. An object recognition method for recognizing a group of identical items as a transport unit in an environment where multiple items exist, comprising: an input step of acquiring images of the plurality of articles; an article detection step of detecting an article region where the article is present from the image; a same item group area estimation step of acquiring information between item areas, which is information about the arrangement of the item areas, calculating a frequency distribution of the information between item areas, and estimating an area of ​​the same item group based on the frequency distribution, The group of identical items is a transport unit in which a plurality of items are all wrapped in transparent wrap, The transparent wrap has a pattern printed on it, The object recognition method is characterized in that the information between each item region is the distance between the pattern and the adjacent item region, the angle between the pattern and the adjacent item region, or the normal direction of the pattern.

8. An object recognition method for recognizing a group of identical items as a transport unit in an environment where multiple items exist, comprising: an input step of acquiring images of the plurality of articles; an article detection step of detecting an article region where the article is present from the image; a same item group area estimation step of acquiring information between item areas, which is information about the arrangement of the item areas, calculating a frequency distribution of the information between item areas, and estimating an area of ​​the same item group based on the frequency distribution, The object recognition method is characterized in that the same item group area estimation step further calculates the probability that the multiple items exist in a single same item group area and the probability that they are distributed across multiple same item group areas.

9. An object recognition method for recognizing a group of identical items as a transport unit in an environment where multiple items exist, comprising: an input step of acquiring images of the plurality of articles; an article detection step of detecting an article region where the article is present from the image; a same item group area estimation step of acquiring information between item areas, which is information about the arrangement of the item areas, calculating a frequency distribution of the information between item areas, and estimating an area of ​​the same item group based on the frequency distribution, An object recognition method characterized in that in the identical item group area estimation step, a reference frequency distribution is registered in advance, and the area of ​​the identical item group is estimated based on the reference.

Images