Object recognition device, robot system, and object recognition method

Abstract

To provide an object recognition device capable of recognizing a lump of conveyed objects (the same article group) as a conveyance unit on the basis of the arrangement of individual articles in the article group.SOLUTION: An object recognition device that recognizes the same article group as a conveyance unit under an environment where a plurality of articles exist, includes: an input unit that acquires images of the plurality of articles; an article detection unit that detects an article region where the articles exist, from the images; and a same article group region estimation unit that acquires inter-article region information as information on the arrangement of the article region, calculates frequency distribution for the inter-article region information, and estimates a region of the same article group on the basis of the frequency distribution.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an object recognition device, a robot system, and an object recognition method. [Background technology]

[0002] In recent years, the use of robots such as articulated arm robots for transporting goods has been increasing in logistics warehouses, etc. When this type of robot transports, for example, multiple plastic bottles all wrapped in clear plastic wrap, it is necessary for the robot to recognize the group of plastic bottles wrapped in clear plastic wrap as the transport unit, rather than each individual plastic bottle, and to grasp the appropriate position of the group of plastic bottles, which is a mass of objects to be transported.

[0003] A cargo handling control device of Patent Document 1 is known as a conventional technology capable of recognizing a group of items wrapped entirely in transparent wrap as a single transport target. For example, the abstract of Patent Document 1 describes the problem of "properly moving items even when multiple items are bundled together under a sheet-like covering material," and describes a solution to the problem as follows: "The cargo handling control of the embodiment includes a transceiver and a control unit. The transceiver is provided in a cargo handling device that grasps an item placed on a placement unit and moves the item, transmits ultrasonic waves or radio waves as transmission waves in the direction in which the placement unit is located, and receives reflected waves of the transmission waves. When the control unit recognizes that multiple items are present on the placement unit based on an image of the items placed on the placement unit, the control unit determines whether the multiple items are bundled together under a sheet-like covering material based on the recognition result based on the image and the reception result of the reflected waves of the transceiver, and controls the cargo handling device based on the result of the determination."

[0004] Furthermore, paragraph 0068 of the same document states, with regard to a covering for bundling multiple items, that "Covering B is a transparent or translucent vinyl sheet or the like. Covering B is also called a bundling material, a wrapping sheet (sheet), or a shrink film (film)." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-109257 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the cargo handling control device of Patent Document 1, as shown in Figures 1, 6, 7, 9, etc. in the document, is provided with a sensor device 20 on the gripping part 113 at the tip of the robot arm, and determines whether multiple items are bundled together under a sheet-like covering by identifying the reflection point of the ultrasonic waves transmitted and received by this sensor device 20, so that it has the following operational problems.

[0007] (1) Since the detection distance of the sensor device 20 that uses ultrasonic waves or radio waves is short, when determining whether or not there is a covering object, it is necessary to bring the sensor device 20 (i.e., the gripping unit 113 at the tip of the robot arm) close to the object and identify the point where the ultrasonic waves are reflected (see Figures 3, 7, 9, etc. of Patent Document 1). For this reason, in the cargo handling control device of Patent Document 1, when it is desired to detect all covering objects in a captured image, it is necessary to scan the entire area of ​​the captured image with the sensor device 20, which causes the problem that it takes a very long time to determine whether or not there is a covering object.

[0008] (2) When the objects to be transported are a group of items bound on the sides with transparent wrap, cardboard, or the like, there is no covering on the top of the group of objects, so the cargo handling control device of Patent Document 1 is unable to recognize the group of objects as a single mass of objects to be transported, and there is a problem in that it mistakes the group of objects for multiple objects simply placed together in one place.

[0009] In consideration of the above problems, the present invention aims to provide an object recognition device and an object recognition method that can recognize a group of transport objects (a group of identical items), which is a transport unit, based on the arrangement of individual items within the group of items. [Means for solving the problem]

[0010] In order to solve the above problems, the object recognition device of the present invention is an object recognition device that recognizes a group of identical items, which is a transport unit, in an environment where multiple items are present, and is equipped with an input unit that acquires an image of the multiple items, an item detection unit that detects an item area where the items are present from the image, and a same item group area estimation unit that acquires information between each item area, which is information regarding the arrangement of the item areas, calculates a frequency distribution of the item area information, and estimates the area of ​​the same item group based on the frequency distribution. Effect of the Invention

[0011] According to the object recognition device and object recognition method of the present invention, it is possible to recognize a mass of conveying objects (same object group) that is a conveying unit, based on the arrangement of individual objects within the object group. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram showing a usage environment of an object recognition device according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing a hardware configuration of an object recognition device according to a first embodiment. [Diagram 3] FIG. 1 is a functional block diagram of an object recognition device according to a first embodiment. [Figure 4] 5A and 5B are diagrams for explaining the processing of an item detection unit and an identical item region estimation unit. [Diagram 5] FIG. 11 is a diagram for explaining the process of the object recognition apparatus according to the second embodiment. [Figure 6] FIG. 11 is a diagram for explaining the processing of the object recognition apparatus according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of an object recognition device according to the present invention will be described with reference to the drawings. EXAMPLES

[0014] First Embodiment First, an object recognition device 1 according to a first embodiment of the present invention and a robot system using the same will be described with reference to FIGS.

[0015] Fig. 1 is a schematic diagram showing the environment in which an object recognition device 1 is used. In the figure, reference numeral 2 denotes a multi-joint arm robot (hereinafter simply referred to as "robot") having a multi-joint arm 21 and a hand 22 and controlled by the object recognition device 1, reference numeral 3 denotes a stereo camera that transmits stereo images captured synchronously by a left camera 3L and a right camera 3R to the object recognition device 1, reference numeral 4 denotes various items (e.g., plastic bottles, toilet paper, cardboard boxes, etc.) that are carried by the robot 2, and reference numeral 5 denotes a pallet on which an item 4 is placed. As is clear from Fig. 1, the robot 2 is installed at a location where it can grasp any item 4 on the pallet 5 with the hand 22 by moving the multi-joint arm 21, and the stereo camera 3 is installed at a location where it can capture an image of the entire top surface of the pallet 5.

[0016] 2 is a block diagram showing a hardware configuration of the object recognition device 1. As shown in the figure, the object recognition device 1 is a computer including a processor 11 such as a CPU, a storage device 12 such as a semiconductor memory, an input device 13 such as a keyboard or a mouse, an output device 14 such as a liquid crystal display, a communication interface 15 for communicating with the robot 2 and the stereo camera 3, and a bus 16 connecting them. Note that the processor 11 executes an object recognition processing program 12a stored in the storage device 12 to realize an item detection unit 11a and a same item group area estimation unit 11b, which will be described later.

[0017] 3 is a functional block diagram of the object recognition device 1. As shown in the figure, the object recognition device 1 includes an input unit 15a that receives stereo images from the stereo camera 3, an item detection unit 11a that detects items 4 by processing the received stereo images, a same item group area estimation unit 11b that estimates a same item area corresponding to a transport unit by focusing on the arrangement of the detected items 4, and an output unit 15b that transmits a command based on the position of the estimated same item area to the robot 2. The input unit 15a and the output unit 15b are functional units realized by the communication interface 15 in FIG. 2.

[0018] <Details of the item detection unit 11a and the same item group area estimation unit 11b> Next, the details of the processing performed by the item detection unit 11a and the same item group region estimation unit 11b will be explained in sequence from step S1 to step S5 with reference to Fig. 4. In the following, it is assumed that the items 4 transported by the robot 2 are plastic bottles, and that a transport unit (a lump of transport target object) is a set of nine plastic bottles in a 3 x 3 array, either entirely wrapped in clear plastic wrap or bound at the sides and bottom with clear plastic wrap.

[0019] <<Step S1 (Acquisition of Imaging Information)>> In step S1, the article detection unit 11a acquires imaging information of a subject based on stereo images captured by the stereo camera 3. Since the left camera 3L and the right camera 3R of the stereo camera 3 are disposed at a predetermined distance apart, a parallax corresponding to the distance to the subject is observed when a left image captured by the left camera 3L and a right image captured by the right camera 3R are compared. Therefore, the article detection unit 11a can calculate the distance to the captured subject by processing the stereo images using the principle of triangulation.

[0020] When a monocular camera is used instead of the stereo camera 3, the captured image itself may be acquired as the imaging information.

[0021] <<Step S2 (Detection of object area)>> In step S2, the item detection unit 11a detects item regions in which individual plastic bottles (items 4) exist based on the imaging information acquired in step S1. Since the distance to the subject is calculated in step S1, each of the flat circular regions on the same plane, as shown in the image diagram of FIG. 4(a), can be extracted as an item region in which the top end of the item 4 (specifically, the cap of the plastic bottle) exists.

[0022] In addition, if a monocular camera is used instead of the stereo camera 3, the object area in which each PET bottle (object 4) exists can be extracted by pattern matching the image, which is the imaging information, with the known top view shape and color of the PET bottle (object 4).

[0023] <<Step S3 (Acquisition of information between each item area)>> In step S3, the same item group region estimation unit 11b acquires inter-item region information relating to the arrangement of the item regions detected in step S2. Specifically, as shown in the image diagram of Fig. 4(b), the same item group region estimation unit 11b calculates the distance (indicated by a solid line in Fig. 4(b)) to adjacent item regions in the vertical or horizontal direction in the image for each item region detected in step S2, and acquires this as inter-item region information. Note that the angle with the adjacent item region or the normal direction of each item region may also be used as inter-item region information. Pattern information may also be used as inter-item region information.

[0024] <<Step S4 (Calculation of frequency distribution)>> In step S4, the same item group region estimation unit 11b calculates a frequency distribution of each item region inter-information (distance, angle, normal direction) acquired in step S3. For example, as shown in the right graph in Fig. 4(c), the same item group region estimation unit 11b first arranges distance information, which is one type of item region inter-information acquired in step S3, on a graph with item inter-distance on the horizontal axis and frequency on the vertical axis. In this example, two types of item inter-distances, a high frequency distance group indicated by a solid line and a low frequency distance group indicated by a dotted line, are calculated as the frequency distribution of each item region inter-information. Note that a graph is also created in the same way when the item region inter-information is angle or normal direction information.

[0025] <<Step S5 (Estimation of the Same Product Group Area)>> In step S5, the same object group region estimation unit 11b estimates the same object group region based on the frequency distribution calculated in step S4. As shown in FIG. 4(c), the distance between the objects 4 includes a relatively short distance group indicated by a solid line and a relatively long distance group indicated by a dotted line. The latter distance group is considered to be the distance between the transport units, so the same object group region estimation unit 11b regards the portion indicated by the dotted line where the distance between the objects 4 is relatively long as the boundary between the transport units, and estimates two same object group regions as shown in FIG. 4(d). The position information of the two same object group regions estimated here is transmitted to the control unit of the robot 2 via the output unit 15b, and is used for the robot 2 to appropriately grasp the transport unit consisting of nine plastic bottles (objects 4) arranged in a 3×3 array.

[0026] If the distance between the objects shown by the dotted lines in FIG. 4(c) is sufficiently long, the boundary of the same object group region can be clearly identified. However, if the difference between the distance between the objects in the same object group region and the distance between the same object group regions is small, it may be impossible to determine whether the object group captured by the stereo camera 3 is one same object group region or multiple same object group regions. In such a case, the probability that the captured object group is one same object group region and the probability that it is multiple same object group regions may be calculated and output to the outside. This allows the robot 2 to determine its operation taking into account both possibilities. Various methods can be considered for calculating the probability. For example, the relative frequency may be output as the probability as it is, or a clustering method such as K-means may be used.

[0027] <Effects of this embodiment> As described above, the object recognition device 1 of this embodiment can recognize a lump of transport objects (a group of identical objects) that is a transport unit based on the arrangement of objects captured by a camera without using an ultrasonic sensor or a radio wave sensor. As a result, the robot 2 can move the hand 22 to an appropriate position for grasping the lump of transport objects bundled in transparent wrap or the like. EXAMPLES

[0028] Second Embodiment Next, an object recognition device 1 according to a second embodiment of the present invention will be described with reference to Fig. 5. Note that a duplicated description of points common to the first embodiment will be omitted.

[0029] When nine plastic bottles (items 4) arranged in a 3 × 3 array were entirely wrapped in transparent plastic wrap as in Example 1, the stereo camera 3 could clearly capture the position of each plastic bottle (item 4), and therefore it was possible to easily obtain information between each item area (information about the distance between the items) as shown in Figure 5 (a).

[0030] However, many transparent wraps have patterns 6 printed on some parts, and when a plastic bottle (item 4) is wrapped in such a patterned transparent wrap, it may not be possible to clearly image the position of some of the plastic bottles (item 4).

[0031] Therefore, in step S3 of this embodiment, when the same item group area estimation unit 11b acquires information between each item area, the distance between the pattern 6 and the item 4, the angle between the pattern 6 and the item 4, or the normal direction of the pattern 6, as shown in Figure 5(b), is acquired as information between each item area.

[0032] When a graph like the graph on the right of Figure 4(c) is created based on the information between each item region in this embodiment, high frequency groups and low frequency groups are formed, just as in Example 1, and by regarding the positions where the low frequency groups exist as boundaries between conveying units, it is possible to divide a large number of items 4 into an appropriate number of identical object group regions. EXAMPLES

[0033] Next, an object recognition device 1 according to a third embodiment of the present invention will be described with reference to Fig. 6. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0034] In the above embodiment, the characteristics of the group of items that constitute the unit of transport (frequency distribution of information between each item area) were unknown, so it was necessary to estimate the state of the unit of transport based on the image each time it was captured.However, in this embodiment, the characteristics of the group of items that constitute the unit of transport (frequency distribution of information between each item area) are known, and it is possible to extract the group of items that constitute the unit of transport based on the known characteristics (frequency distribution of information between each item area) as a standard.

[0035] For example, when it is known in advance that the transport unit is four plastic bottles (items 4) arranged in a 2 x 2 pattern as illustrated in Figure 6(a), the frequency distribution of information between each item region, as shown in the graph on the right of Figure 6(a), is pre-registered in the same item group region estimation unit 11b of this embodiment as a criterion.

[0036] In this case, the eight plastic bottles (items 4) captured by the stereo camera 3 are arranged as shown in Fig. 6(b), and even if the frequency distribution of the information between each item region calculated from the image shows a small difference between the solid line group and the dotted line group, as shown in the right graph of Fig. 6(b), it can be determined that the solid line distance matches the characteristics of the reference transport unit (frequency distribution of information between each item region) and the dashed line distance does not match the characteristics of the reference transport unit (frequency distribution of information between each item region), so a correction process can be performed to emphasize the frequency of the solid line group. As a result, the object recognition device 1 of this embodiment can correctly divide the eight plastic bottles (items 4) in a 2 x 4 arrangement into two identical item group areas consisting of four plastic bottles (items 4) in a 2 x 2 arrangement.

[0037] In the above explanation, it is assumed that information on the distribution of information between each item region is registered in advance, but future estimations of the same item group region may also be performed based on the frequency distribution of information between each item region calculated in the past. EXAMPLES

[0038] Next, an object recognition device 1 according to a fourth embodiment of the present invention will be described. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0039] In this embodiment, the inter-item area information is a template of an area including two or more item areas, or a pattern feature acquired from the area. For example, if the item 4 to be transported is a plastic bottle, and the cap of the plastic bottle has a pattern drawn thereon, and the orientation of the pattern is always consistent within the same item group, a distribution is calculated in which the horizontal axis is the value of the pattern feature and the vertical axis is the frequency, and based on the calculation result, the difference in the template or the difference in the pattern feature (the difference on the horizontal axis of the frequency distribution) is detected as a deviation in the orientation of the pattern, thereby making it possible to estimate the same item group area. [Explanation of symbols]

[0040] 1 Object recognition device 11 Processors 11a Item detection unit 11b Same article group area estimator 12 Storage Devices 12a Object recognition processing program 13 Input Devices 14 Output Devices 15 Communication Interface 15a Input section 15b Output section 16 Bus 2. Robot 21 Articulated Arm 22 Hands 3 Stereo Camera 3L Left camera 3R Right Camera 4 Goods 5 Palette 6. Pattern

Claims

1. An object recognition device that recognizes a group of identical objects as a transport unit in an environment where a plurality of objects are present, an input unit for acquiring images of the plurality of articles; an object detection unit that detects an object region in which the object is present from the image; a same item group area estimation unit that acquires item area inter-information, which is information regarding the arrangement of the item areas, calculates a frequency distribution of the item area inter-information, and estimates an area of ​​the same item group based on the frequency distribution; The group of identical articles is a transport unit in which a plurality of articles are entirely wrapped in transparent wrap, The transparent wrap has a pattern printed on it, The object recognition device is characterized in that the information between each object region is a distance between the pattern and an adjacent object region, an angle between the pattern and an adjacent object region, or a normal direction of the pattern.

2. An object recognition device that recognizes a group of identical objects as a transport unit in an environment where a plurality of objects are present, an input unit for acquiring images of the plurality of articles; an object detection unit that detects an object region in which the object is present from the image; a same item group area estimation unit that acquires item area inter-information, which is information regarding the arrangement of the item areas, calculates a frequency distribution of the item area inter-information, and estimates an area of ​​the same item group based on the frequency distribution; The object recognition device is characterized in that the same item group area estimation unit further calculates the probability that the multiple items exist in a single same item group area and the probability that the multiple items are distributed across multiple same item group areas.

3. An object recognition device that recognizes a group of identical objects as a transport unit in an environment where a plurality of objects are present, an input unit for acquiring images of the plurality of articles; an object detection unit that detects an object region in which the object is present from the image; a same item group area estimation unit that acquires item area inter-information, which is information regarding the arrangement of the item areas, calculates a frequency distribution of the item area inter-information, and estimates an area of ​​the same item group based on the frequency distribution; An object recognition device characterized in that a reference frequency distribution is registered in the same item group area estimation unit in advance, and the area of ​​the same item group is estimated based on the reference.

4. In the object recognition device according to claim 2 or claim 3, An object recognition device characterized in that the group of identical objects is a transport unit in which multiple objects are bound on their sides with plastic wrap or cardboard.

5. In the object recognition device according to claim 3, An object recognition device characterized in that the reference frequency distribution is a frequency distribution previously calculated by the same object group area estimation unit.

6. A robot system comprising the object recognition device according to any one of claims 1, 2, 3, and 5, and a robot that transports the group of identical objects, The object recognition device includes: An output unit that outputs information about the estimated area of ​​the same article group; A control unit that controls the robot based on the information on the area of ​​the group of identical items output from the output unit; A robot system comprising:

7. An object recognition method for recognizing a group of identical objects as a transport unit in an environment where a plurality of objects are present, comprising: an input step of acquiring images of the plurality of articles; an article detection step of detecting an article region in which the article is present from the image; a same item group area estimation step of acquiring inter-item area information, which is information regarding the arrangement of the item areas, calculating a frequency distribution of the inter-item area information, and estimating an area of ​​the same item group based on the frequency distribution, The group of identical articles is a transport unit in which a plurality of articles are entirely wrapped in transparent wrap, The transparent wrap has a pattern printed on it, The object recognition method, characterized in that the information between each object region is a distance between the pattern and an adjacent object region, an angle between the pattern and an adjacent object region, or a normal direction of the pattern.

8. An object recognition method for recognizing a group of identical objects as a transport unit in an environment where a plurality of objects are present, comprising: an input step of acquiring images of the plurality of articles; an article detection step of detecting an article region in which the article is present from the image; a same item group area estimation step of acquiring inter-item area information, which is information regarding the arrangement of the item areas, calculating a frequency distribution of the inter-item area information, and estimating an area of ​​the same item group based on the frequency distribution, The object recognition method is characterized in that the same item group area estimation step further calculates the probability that the multiple items exist in a single same item group area and the probability that they are distributed across multiple same item group areas.

9. An object recognition method for recognizing a group of identical objects as a transport unit in an environment where a plurality of objects are present, comprising: an input step of acquiring images of the plurality of articles; an article detection step of detecting an article region in which the article is present from the image; a same item group area estimation step of acquiring inter-item area information, which is information regarding the arrangement of the item areas, calculating a frequency distribution of the inter-item area information, and estimating an area of ​​the same item group based on the frequency distribution, An object recognition method characterized in that in the identical item group region estimating step, a reference frequency distribution is registered in advance, and the region of the identical item group is estimated based on the reference.

Images