Image processing apparatus

Abstract

An image processing device according to one embodiment of the present disclosure is provided with: a three-dimensional information acquisition unit that acquires, from a vision sensor, three-dimensional information including distance information of a two-dimensional image and a two-dimensional position; an image recognition unit that recognizes a candidate region of an object from the two-dimensional image; an overlap determination unit that determines whether or not the candidate regions overlap; and a distance determination unit that determines whether or not two candidate regions determined to overlap by the overlap determination unit are present on the same plane on the basis of the distance information, and that determines whether or not the two candidate regions are present on the same plane when it is determined that the two candidate regions are present on the same plane. When it is determined that the candidate region having a smaller area is a part of the object indicated by the candidate region having a larger area, the candidate region having a smaller area is a part of the object indicated by the candidate region having a larger area. It is determined that the candidate region having a smaller area is an independent object located in front of the object indicated by the candidate region having a larger area.

Description

Technical Field

[0001] This invention relates to an image processing apparatus. Background Technology

[0002] To improve logistics efficiency, it is widely practiced to transport multiple goods stacked on pallets in multiple layers. To further enhance efficiency, it is desirable to automatically remove these stacked goods one by one by robots or similar devices. Therefore, the following research was conducted: acquiring distance images of the stacked goods taken from above, using pattern matching technology to analyze the planar position and vertical relationship of the goods, and then removing them sequentially starting from the top layer.

[0003] Specifically, the following scheme is proposed: Features considered to be corners or edges of goods are extracted from a distance image and extracted from the image representing the uppermost distance range. The position of the goods is estimated by matching these features with a template of pre-registered goods contour information (see, for example, Patent Document 1). In distance images, edges may sometimes be unidentifiable when adjacent goods are close together. Furthermore, information may be missing in distance images, for example, due to diffuse reflection caused by tape used to seal the goods. Therefore, the following scheme is also proposed: matching with a shape generated by combining contour information, or matching that takes into account the missing information.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 10-31742 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] The study also explored the scenario of various goods of different shapes and sizes being mixed and loaded on the same pallet. In such cases, it is difficult to pre-register the outline information of all types of goods. Therefore, an image processing technology capable of accurately identifying goods even when they are stacked in multiple layers of varying shapes and sizes is desired.

[0009] Solution for solving the problem

[0010] One aspect of the image processing apparatus disclosed herein includes: a three-dimensional information acquisition unit that acquires three-dimensional information from a vision sensor, the three-dimensional information including a two-dimensional image and distance information of a two-dimensional position; an image recognition unit that identifies candidate regions of an object from the two-dimensional image; an overlap determination unit that determines whether the candidate regions overlap; and a distance determination unit that, for two candidate regions determined by the overlap determination unit to overlap, determines whether they exist on the same plane based on the distance information. If they are determined to exist on the same plane, the candidate region with the smaller area is determined to be part of the object represented by the candidate region with the larger area. If they are determined not to exist on the same plane, the candidate region with the smaller area is determined to be an independent object located in front of the object represented by the candidate region with the larger area. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating the structure of a destacking system that includes the image processing apparatus according to the first embodiment of this disclosure.

[0012] Figure 2 It is shown Figure 1 The flowchart shows the image recognition process of the image processing device.

[0013] Figure 3 This is a block diagram illustrating the structure of a destacking system that includes the image processing apparatus according to the second embodiment of this disclosure. Detailed Implementation

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Furthermore, for the embodiments described below, the same reference numerals will be used for the same constituent elements as in the previously described embodiments, and repeated descriptions will be omitted.

[0015] (First Embodiment)

[0016] Figure 1 This is a block diagram showing the structure of the depalletizing system 1. The depalletizing system 1 is a device that removes multiple objects (goods) L one by one from a pallet P. The multiple objects L removed by the depalletizing system 1 can also be a mixture of objects with different shapes (length, width, and height ratios), sizes, and patterns (including tape for sealing boxes, labels for information display, etc.).

[0017] The depalletizing system 1 includes: a 3D sensor 10, which takes a picture of the object L on the pallet P from above and generates 3D information including a 2D image and distance information of each 2D position; a robot 20, which holds the object L and moves it; and an image processing device 30, which instructs the robot 20 to remove the object L one by one based on the 3D information generated by the 3D sensor 10.

[0018] The 3D sensor 10 only needs to generate 3D information that includes 2D images and distance information of 2D position (typically each pixel of the 2D image). It can be a combination of a stereo camera that captures multiple 2D images and generates distance information based on their parallax, or a combination of a 2D camera that captures 2D images and a laser scanner that acquires distance images.

[0019] Robot 20 can be a vertical multi-joint robot as shown in the figure, but it is not limited to this. It can also be, for example, an orthogonal coordinate robot, a horizontal multi-joint robot, or a parallel robot. Robot 20 has a head 21 at its end effector that can hold various objects L. The structure of the head 21 is not particularly limited. In addition to a structure with an adhesive pad as shown in the figure, it can also have a structure with gripping fingers for opening and closing.

[0020] The image processing apparatus 30 includes a memory, a processor, an input / output interface, etc., and can be implemented by one or more computer devices that execute appropriate control programs. In addition to the functions of the image processing apparatus disclosed herein, the image processing apparatus 30 of this embodiment also has the functions of a robot control device for controlling the robot 20; however, the functions of the robot control device can also be implemented independently by different computer devices.

[0021] The image processing apparatus 30 includes a 3D information acquisition unit 31, a machine learning unit 32, an image recognition unit 33, an overlap determination unit 34, an area threshold setting unit 35, a distance determination unit 36, a distance threshold setting unit 37, an information integration unit 38, an identifier assignment unit 39, a candidate exclusion unit 40, and a robot control unit 41. Furthermore, all of these components except the robot control unit 41 are components of the image processing apparatus of this disclosure. Additionally, the aforementioned components may be components categorized by the functions of the image processing apparatus 30, or they may not be components that can be clearly distinguished by physical structure and program structure.

[0022] The 3D information acquisition unit 31 acquires 3D information, including 2D images and distance information for each 2D position, from the 3D sensor 10. The acquisition of 3D information can be performed each time one object L is retrieved, or it can be performed only once to retrieve multiple objects L of the same layer (within a range of approximately equal height and position).

[0023] The machine learning unit 32 forms a model for recognizing objects L in a two-dimensional image using AI. Specifically, the machine learning unit 32 generates a learning model for recognizing regions in the two-dimensional image where objects L are believed to exist, i.e., candidate regions, by inputting the two-dimensional image with identified candidate regions into a neural network for supervised learning. In the image recognition unit 33, the object L can be represented as a square-shaped outline in the two-dimensional image. Ideally, the machine learning unit 32 uses appropriate input data for learning to minimize the possibility of misidentifying patterns, such as those on the packaging material of the object L, as candidate regions.

[0024] The image recognition unit 33 identifies candidate regions that are considered to be image regions of the object L from the two-dimensional image acquired by the three-dimensional information acquisition unit 31. The image recognition unit 33 of this embodiment intends to perform image recognition using the learning model provided by the machine learning unit 32, but it is not limited to this; the image recognition unit 33 may also be configured to identify candidate regions through template matching or the like. Furthermore, the candidate regions identified by the image recognition unit 33 may include labels affixed to the object L or patterns of the object L that are mistakenly identified as independent objects L.

[0025] When the overlap determination unit 34 identifies multiple candidate regions in the same two-dimensional image, it determines whether the candidate regions overlap. Specifically, the overlap determination unit 34 may be configured such that if the ratio of the overlapping area of ​​two candidate regions to the area of ​​the smaller one is above an area threshold, the two candidate regions are determined to overlap.

[0026] The area threshold setting unit 35 accepts an input from the user of a threshold value representing the overlap area ratio used by the overlap determination unit 34 to determine whether overlap occurs. The area threshold setting unit 35 can also be configured to apply a preset default value in the absence of user input. The area threshold setting unit 35 can also accept any numerical input, but it can also allow the user to select a value from a plurality of preset values ​​to be set as the area threshold.

[0027] The distance determination unit 36 ​​determines whether two candidate regions identified as overlapping by the overlap determination unit 34 exist on the same plane based on distance information. Specifically, the distance determination unit 36 ​​may be configured such that if the difference between the representative values ​​of the distances from the three-dimensional sensor 10 to each candidate region is below a distance threshold, it determines that the two candidate regions exist on the same plane. When the distance determination unit 36 ​​determines that the two candidate regions exist on the same plane, it determines that the candidate region with the smaller area is part of the object L shown by the candidate region with the larger area, for example, the area of ​​a label attached to the back of the object L rather than the area of ​​the independent object L. When the distance determination unit 36 ​​determines that the two candidate regions do not exist on the same plane, it determines that the candidate region with the smaller area is the area of ​​the independent object L located in front of the object L shown by the candidate region with the larger area.

[0028] The representative distance confirmed by the distance determination unit 36 ​​can be set to the distance of one pixel representing the candidate region, the average distance of multiple pixels, the mode of multiple pixels, etc. This reduces computational load and more accurately determines whether a candidate region is a region of an independent object L. Ideally, when the representative distance is derived as the distance of one or a few pixels representing the candidate region, the pixel representing the candidate region is selected from the central portion separated from the outer edge of the candidate region.

[0029] Alternatively, the distance determination unit 36 ​​can be configured to estimate a plane with two candidate regions based on distance information, and determine whether the two candidate regions exist on the same plane by the distance between the two planes and the angle of the normal.

[0030] The distance threshold setting unit 37 accepts input of a distance threshold used by the distance determination unit 36 ​​to determine whether a candidate area with a smaller area is an independent object. The distance threshold setting unit 37 may also be configured to apply a preset default value in the absence of user input. The distance threshold setting unit 37 may also accept any numerical input, but it may also allow the user to select a value from a plurality of preset values ​​as the area threshold.

[0031] The information integration unit 38 integrates the information of the smaller candidate area, which is determined by the distance determination unit 36 ​​to be part of the candidate area shown behind the object L, into the information of the larger candidate area. Therefore, for the object L behind the larger candidate area, information such as the location of the label considered as the smaller candidate area can be assigned. Thus, by considering the location of the target label identified as the smaller candidate area, the retrieved object L is placed at the handover position, making it easier for the operator to identify the object L that follows. Furthermore, by integrating the information of the smaller candidate area into the information of the larger candidate area, the smaller candidate area is excluded from the candidate area.

[0032] The identifier assignment unit 39 assigns an identifier indicating that a candidate region with a larger area, which is determined by the distance determination unit to be part of the candidate region with the smaller area, has been determined to be part of itself. By assigning such an identifier, it is possible to move tagged object L and untagged object L to different locations, for example, when a candidate region with a smaller area is determined to be part of the object L behind it is a tag of the object L with a larger area. Thus, it is possible to properly classify the object L taken from the tray P.

[0033] The candidate exclusion unit 40 excludes, at least temporarily, the candidate regions with larger areas that are determined by the distance determination unit 36 ​​to be independent object L from the candidate regions. This prevents the problem of other object L being damaged or falling off when attempting to remove the object L placed on the lower side of the object L.

[0034] The robot control unit 41 instructs the robot 20 to perform actions such that the robot 20 retrieves the object L that was ultimately left as a candidate region representing an independent object L. The structure of the robot control unit 41 can be the same as that of conventional robot control devices.

[0035] Next, in Figure 2 The image processing apparatus 30 is shown to perform object recognition processing. The object recognition processing of the image processing apparatus 30 includes a three-dimensional information acquisition process (step S01), a two-dimensional image recognition process (step S02), a candidate region quantity confirmation process (step S03), a candidate region combination setting process (step S04), an overlap ratio confirmation process (step S05), a distance confirmation process (step S06), an information integration process (step S07), a distance determination identifier assignment process (step S08), a candidate region exclusion process (step S09), and an end confirmation process (step S10).

[0036] In the three-dimensional information acquisition step S01, three-dimensional information is acquired from the three-dimensional sensor 10 by the three-dimensional information acquisition unit 31. In the two-dimensional image recognition step S02, the candidate region of the object L is identified from the two-dimensional image contained in the three-dimensional information by the image recognition unit 33.

[0037] In the candidate region number confirmation step S03, it is confirmed whether there are multiple candidate regions identified in the 3D image recognition step. If the number of candidate regions is one or less, the object recognition process ends. If the number of candidate regions is multiple, the process proceeds to the candidate region combination setting step S04, where a combination of candidate regions to be confirmed is set, that is, two candidate regions are selected.

[0038] If a combination of candidate regions is set, in the overlap ratio confirmation process of step S05, the overlap determination unit 34 calculates the ratio of the overlap area of ​​the two candidate regions to the area of ​​the smaller candidate region, and confirms whether the ratio of the overlap area is above the area threshold. If the ratio of the overlap area in the smaller candidate region is above the area threshold, it is determined that the two candidate regions overlap, and the process proceeds to step S06. If the ratio of the overlap area in the smaller candidate region is less than the area threshold, it is determined that the two candidate regions do not overlap, and the process proceeds to step S10.

[0039] In the distance confirmation process of step S06, the distance determination unit 36 ​​determines the representative distance of each candidate region and compares the difference of the representative distance with a distance threshold. If the difference of the representative distance is below the distance threshold, it is determined that the candidate region with the smaller area is a part of the object L shown by the candidate region with the larger area. If the difference of the representative distance exceeds the distance threshold, it is determined that the candidate region with the smaller area represents an independent object L placed on the object L shown by the candidate region with the larger area.

[0040] If in step S06 it is determined that the candidate region of the smaller area is part of the candidate region of the larger area, in step S07, the information integration unit 38 integrates the information of the candidate region of the smaller area into the information of the candidate region of the larger area. In step S08, after the identifier assignment unit 39 assigns an identifier indicating that it is determined to be part of the candidate region of the larger area, the process proceeds to step S10.

[0041] If in step S06 the candidate region with the smaller area is determined to represent an independent object L, in step S09, after the candidate region with the larger area is excluded from the candidate region by the candidate exclusion unit 40, the process proceeds to step S10.

[0042] In step S10, for all combinations of candidate regions except those integrated in step S07 and excluded in step S09, it is confirmed whether the processing after step S04 has been performed. If there are combinations of candidate regions that have not been executed, the processing after step S04 is performed for the new combinations. If the processing for all combinations of candidate regions has been completed, the object recognition process ends.

[0043] As described above, the image processing apparatus 30 of this embodiment first determines the candidate region through image recognition of a two-dimensional image with relatively low error. Therefore, it can extract the candidate region more accurately than using distance images or point cloud data with large errors. Especially when box-shaped objects are arranged closely together, although the boundaries of the objects cannot be identified in distance images, image recognition of the two-dimensional image can separate and identify the closely arranged objects with a high probability. In addition, since the image processing apparatus 30 determines whether the image of the candidate region is based on the outline of the object L or the label of the object L based on the difference in the representative distance of the candidate region, it can more easily correct misidentification caused by the two-dimensional image. Therefore, compared with pattern matching of two-dimensional images and pattern matching of distance images and point cloud data, the image processing apparatus 30 can correctly identify the object L.

[0044] (Second Implementation)

[0045] Figure 3 This is a block diagram showing the structure of the depalletizing system 1A. The depalletizing system 1A and... Figure 1 Similarly, the destacking system 1 is a device for removing multiple objects L placed on a pallet P one by one.

[0046] The depalletizing system 1 includes: a 3D sensor 10, which takes a picture of the object L on the pallet P from above and generates 3D information including a 2D image and distance information of each 2D position; a robot 20, which holds the object L and moves it; and an image processing device 30A, which instructs the robot 20 to remove the object L one by one based on the 3D information generated by the 3D sensor 10.

[0047] The image processing apparatus 30A includes a memory, a processor, an input / output interface, etc., and can be implemented by one or more computer devices that execute appropriate control programs. In addition to the functions of the image processing apparatus of this disclosure, the image processing apparatus 30A of this embodiment also has the functions of a robot control device for controlling the robot 20; however, the functions of the robot control device can also be implemented independently by different computer devices.

[0048] The image processing apparatus 30A includes a 3D information acquisition unit 31, a machine learning unit 32A, a comprehensive estimation unit 42, and a robot control unit 41. The machine learning unit 32A and the comprehensive estimation unit 42 are constituent elements of the image processing apparatus of this disclosure.

[0049] The machine learning unit 32A forms a model for identifying an object L based on 3D information using AI. Specifically, the machine learning unit 32A performs supervised learning by inputting 3D information, such as candidate regions and patterns to be excluded, from a 2D image into a neural network. As a result, the machine learning unit 32A generates a learning model for identifying regions in the 2D image where the object L is believed to exist, i.e., candidate regions, in the comprehensive estimation unit 42.

[0050] The comprehensive estimation unit 42 identifies candidate regions in the two-dimensional image that are considered to be object L, based on the three-dimensional information acquisition unit 31. Using a learning model generated by the machine learning unit 32A, the comprehensive estimation unit 42 excludes patterns and labels attached to object L, and identifies only regions corresponding to the outline of object L that should be retrieved by robot 20 as candidate regions. Ideally, the comprehensive estimation unit 42 is configured to further consider distance information to determine candidate regions in the two-dimensional image representing object L located at the very front (in reality, at the top). This allows the determination of the object L that robot 20 should retrieve next.

[0051] As described above, the image processing apparatus 30 of this embodiment uses AI to exclude patterns and labels of the object L and only determines the candidate region corresponding to the object L. Therefore, it can determine the object L that the robot 20 should take out next with fewer steps.

[0052] The following notes further disclose the above-described embodiments and variations.

[0053] (Postscript 1)

[0054] An image processing apparatus (30) includes: a three-dimensional information acquisition unit (31) that acquires three-dimensional information from a vision sensor, the three-dimensional information including a two-dimensional image and distance information of a two-dimensional position; an image recognition unit (33) that identifies candidate regions of an object from the two-dimensional image; an overlap determination unit (34) that determines whether the candidate regions overlap; and a distance determination unit (36) that, for two candidate regions determined to overlap by the overlap determination unit (34), determines whether they exist on the same plane based on the distance information. If they are determined to exist on the same plane, the candidate region with the smaller area is determined to be part of the object shown by the candidate region with the larger area. If they are determined not to exist on the same plane, the candidate region with the smaller area is determined to be an independent object located in front of the object shown by the candidate region with the larger area.

[0055] (Postscript 2)

[0056] The image processing apparatus (30) in Appendix 1 may also include an information integration unit (38), which integrates information of a candidate region with a smaller area that is determined by the distance determination unit to be part of the candidate region with a larger area into the information of the candidate region with a larger area.

[0057] (Note 3)

[0058] The image processing apparatus (30) in Appendix 1 or 2 may also include an identifier assignment unit (39) that assigns an identifier to a candidate region of the larger area that is determined by the distance determination unit (36) to be part of itself, indicating that the determination has been made.

[0059] (Postscript 4)

[0060] The image processing apparatus (30) of any of the appendices 1 to 3 may also include a candidate exclusion unit (40), which excludes candidate regions whose smaller area is determined by the distance determination unit to be independent objects from the candidate regions.

[0061] (Note 5)

[0062] The image processing apparatus (30) of any of the appendices 1 to 4 may also include an area threshold setting unit (35) that accepts input of a threshold for the overlap area ratio used by the overlap determination unit (34) to determine whether an overlap occurs.

[0063] (Note 6)

[0064] The image processing apparatus (30) of any of the appendices 1 to 5 may also include a distance threshold setting unit (37) that accepts an input of a distance threshold used by the distance determination unit (36) to determine whether an object is an independent object.

[0065] (Note 7)

[0066] An image processing apparatus (30A) includes: a three-dimensional information acquisition unit (31) that acquires three-dimensional information, the three-dimensional information including a two-dimensional image and distance information of a two-dimensional position; a machine learning unit (32A) that performs machine learning to determine a model of an object in the two-dimensional image by using the three-dimensional information as input to a neural network; and a comprehensive estimation unit (42) that estimates a candidate region of the object in the three-dimensional information using the model learned by the machine learning unit (32A).

[0067] (Note 8)

[0068] In the image processing apparatus (30A) in Appendix 7, the comprehensive estimation unit (42) may also determine the candidate region of the object that exists on the frontmost side.

[0069] The present disclosure has been described in detail above, but it is not limited to the various embodiments described above. These embodiments may include various additions, substitutions, modifications, partial deletions, etc., without departing from the spirit of the present disclosure or from the main points derived from the claims and their equivalents. For example, in the embodiments described above, the order of actions or the order of processes is illustrative and is not limited to these orders.

[0070] Explanation of reference numerals in the attached figures

[0071] 1.1A: Depalletizing system

[0072] 10: Three-dimensional sensor

[0073] 20: Robot

[0074] 21: Head

[0075] 30, 30A: Image processing device

[0076] 31: 3D Information Acquisition Department

[0077] 32, 32A: Machine Learning Department

[0078] 33: Image Recognition Department

[0079] 34: Overlap Detection Unit

[0080] 35: Area threshold setting unit

[0081] 36: Distance Judgment Unit

[0082] 37: Distance threshold setting unit

[0083] 38: Information Integration Department

[0084] 39: Identifier Assignment Section

[0085] 40: Candidate Exclusion Section

[0086] 41: Robot Control Department

[0087] 42: Comprehensive Judgment Department

[0088] L: Object

[0089] P: Tray

Claims

1. An image processing apparatus comprising: The three-dimensional information acquisition unit acquires three-dimensional information from a vision sensor, which includes two-dimensional images and distance information of two-dimensional positions; An image recognition unit identifies candidate regions of objects from the two-dimensional image; An overlap determination unit determines whether the candidate regions overlap. as well as The distance determination unit determines whether two candidate regions determined to be overlapping by the overlap determination unit exist on the same plane based on the distance information. If they are determined to exist on the same plane, the candidate region with the smaller area is determined to be part of the object shown by the candidate region with the larger area. If they are determined not to exist on the same plane, the candidate region with the smaller area is determined to be an independent object located in front of the object shown by the candidate region with the larger area.

2. The image processing apparatus according to claim 1, wherein, It also includes an information integration unit, which integrates information of the candidate region with smaller area that is determined by the distance determination unit to be part of the object shown in the candidate region with larger area into the information of the candidate region with larger area.

3. The image processing apparatus according to claim 1 or 2, wherein, It also includes an identifier assignment unit, which assigns an identifier to the candidate region of the larger area that is determined by the distance determination unit to be part of the candidate region of the smaller area, indicating that the determination has been made.

4. The image processing apparatus according to any one of claims 1 to 3, wherein, It also includes a candidate exclusion unit, which excludes candidate regions whose smaller area is determined by the distance determination unit to be independent objects, and candidate regions whose larger area is determined to be independent objects from the candidate regions.

5. The image processing apparatus according to any one of claims 1 to 4, wherein, It also includes an area threshold setting unit, which accepts input of a threshold for the overlap area ratio used by the overlap determination unit to determine whether there is overlap.

6. The image processing apparatus according to any one of claims 1 to 5, wherein, It also includes a distance threshold setting unit, which accepts input of a distance threshold used by the distance determination unit to determine whether an object is an independent object.

7. An image processing apparatus comprising: The three-dimensional information acquisition unit acquires three-dimensional information, which includes a two-dimensional image and distance information of the two-dimensional position. The machine learning unit performs machine learning, using the three-dimensional information as input to a neural network, to determine a model of an object in the two-dimensional image; and The comprehensive estimation unit uses the model learned by the machine learning unit to estimate candidate regions of objects in the three-dimensional information.

8. The image processing apparatus according to claim 7, wherein, The comprehensive estimation unit determines the candidate region that exists on the frontmost side of the object.

Images