Picking apparatus

Abstract

To provide a picking apparatus which can implement low-cost and highly accurate random bin picking.SOLUTION: A picking apparatus which performs random bin picking includes: a picking unit which picks up a target object located in a predetermined area; a camera unit equipped with 2D and 3D cameras that image the target object; and a processing unit which recognizes the target object from the image captured by the camera unit. The processing unit includes: an acquisition unit which separates an image of a region corresponding to the target object, from the image, and acquires a separated image; a determination unit which determines whether the target object in the separated image is a target object registered in advance; and a motion control unit which executes, when the target object is the registered object, motion control to cause the picking unit to pick up the target object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a picking device, and more particularly to a picking device that can achieve low-cost and high-precision bulk picking.

Background Art

[0002] Conventionally, 3D sensors have generally been used for bulk picking.

[0003] Techniques have been proposed for providing a processing system that controls a three-dimensional distance sensor to project a light pattern onto a target surface (see Patent Document 1, etc.).

[0004] Patent Document 1 includes steps of controlling a three-dimensional distance sensor to project a light pattern onto a target surface, controlling a two-dimensional camera to obtain a first image of the light pattern on the target surface, controlling a light receiving system of the three-dimensional distance sensor to obtain a second image of the light pattern on the target surface, associating a set of two-dimensional coordinates of a first light point among the plurality of light points with a distance coordinate of a set of three-dimensional coordinates of the first light point to form a single set of coordinates for the first light point, and deriving a relational expression from the single set of coordinates.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the technique of Patent Document 1, an expensive 3D sensor is used, so the cost of the entire unit with a 2D camera is too high.

[0007] The present invention has been made in view of the above points, and provides a picking device capable of realizing low-cost and high-precision bulk picking.

Means for Solving the Problems

[0008] That is, the picking device according to the embodiment is a picking device that performs bulk picking, and includes a picking unit that picks an object existing within a predetermined region, a camera unit that includes a 2D camera and a 3D camera for photographing the object, and a processing unit that recognizes the object from the image acquired by the camera unit. The processing unit includes an acquisition unit that separates an image of a region corresponding to the object from the image and acquires it as a separated image, a determination unit that determines whether the object in the separated image is a pre-registered object, and when the object is a pre-registered object, an operation control unit that executes operation control for picking the object with respect to the picking unit.

[0009] Furthermore, in the picking device, the processing unit may further include a parameter setting unit that performs stereo calibration between the 2D camera and the 3D camera and sets conversion parameters for converting the 3D point cloud generated by photographing with the 3D camera to corresponding positions within the planar image generated by photographing with the 2D camera, a coordinate conversion unit that performs coordinate conversion on the 3D point cloud using the parameters to a planar image, and a coordinate estimation unit that projects the 3D point cloud coordinate-converted to the planar image onto the planar image and estimates the spatial coordinates of the 3D point cloud projected onto the planar image generated by photographing with the 2D camera.

[0010] Furthermore, in the picking device, the object may be a bag in which objects are bulked.

[0011] Furthermore, in the picking device, the parameter setting unit may set the conversion parameters for converting the 3D point cloud generated by photographing with the 3D camera to corresponding positions within the planar image generated by photographing with the 2D camera by image processing using AI through stereo calibration between the 2D camera and the 3D camera.

[0012] Furthermore, in the picking device, it may include a base, an arm portion extending from the base, and a suction portion connected to the tip of the arm portion for sucking an object, and the suction portion may include a suction pad at its tip and a suction portion for applying negative pressure to the suction pad.

[0013] Furthermore, in the picking device, the processing unit may further include a priority assignment unit that assigns a priority for picking in the order of the area of the separation image being large, an area extraction unit that extracts an image area where the suction pad fits in the separation image and sets the center of gravity position of the image area as the position in the plane coordinates when the suction pad contacts, an angle detection unit that compares the orientation of the object with the orientation of the object in a predetermined image by feature point matching and detects an angle centered on the height direction of the object, and a suction position and posture determination unit that estimates the spatial coordinates of the suction position where the suction pad sucks the object, applies the angle of the object, and determines the position and posture where the suction pad sucks the object.

Effect of the Invention

[0014] The picking device of the present invention includes a picking unit for picking an object existing within a predetermined area, a camera unit including a 2D camera and a 3D camera for photographing the object, and a processing unit for recognizing the object from the images acquired by the camera unit. The processing unit includes an acquisition unit that separates an image of an area corresponding to the object from the image and acquires it as a separation image, a determination unit that determines whether the object in the separation image is a pre-registered object, and an operation control unit that executes operation control for picking the object with respect to the picking unit when the object is a pre-registered object. Therefore, by using an in-house camera unit with a 2D camera and an inexpensive 3D camera, the cost of the entire system can be reduced. Also, since the image of the object is separated and extracted by AI, it is possible to recognize loose stacking that is resistant to environmental changes.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0016] The picking device according to the embodiment is a picking device capable of realizing low-cost and high-precision bulk picking.

[0017] In the conventional picking device, because an expensive 3D sensor is used, the cost of the entire unit with the 2D camera is too high.

[0018] Therefore, the picking device, etc. (method) of the embodiment is positioned as a technology for enabling low-cost and high-precision bulk picking by using a camera unit composed of a 2D camera and an inexpensive 3D camera. The 2D camera captures two-dimensional information in the horizontal and vertical (X, Y) directions, and the 3D camera captures depth in addition to the horizontal and vertical information. The 3D camera of the embodiment complements the 2D camera by being able to measure the height that cannot be measured by the 2D camera. The 2D camera is highly accurate and complements the low-accuracy 3D camera. Bulk picking refers to the operation of sucking out the plurality of small bags 3 deposited in the container 2 one by one, taking them out of the container 2, and putting them into a small box.

[0019] The configuration of the picking device 1 of the embodiment is represented as a schematic diagram in FIG. 1. The picking device 1 includes a camera unit 10 that images an object. The camera unit 10 includes a 2D camera 11 and a 3D camera 12. The 2D camera 11 and the 3D camera 12 image the plurality of small bags 3 deposited in the container 2.

[0020] FIG. 2 is a perspective view showing the entirety of the picking device 1. A plurality of small bags 3 are deposited in the container 2. Inside the small bag 3, for example, snack foods, jam, sauce, screws, and electronic components are contained, and the size of the small bag 3 is, for example, 10 cm in length and 5 cm in width. The 2D camera 11 is installed at a distance of 50 cm to 1 m directly above the container 2 and images the two-dimensional images of the plurality of small bags 3 deposited in the container 2. The 3D camera 12 is installed immediately behind the 2D camera 11 and images the three-dimensional images of the plurality of small bags 3 deposited in the container 2.

[0021] The picking unit 20 is disposed at a position closer to the container 2 than the camera unit 10 and picks up the plurality of small bags 3 deposited in the container 2. The picking unit 20 has a robot structure and includes a base portion 21, an arm portion 22 extending from the base portion 21, and a suction portion 23 connected to the tip of the arm portion 22 for sucking an object. The suction portion 23 includes a suction pad 25 at its tip and a pump-type suction portion 24 that makes the suction pad 25 under negative pressure. The suction pad 25 is composed of a suction cup or rubber.

[0022] The processing unit 30 is arranged behind the picking unit 20 and is connected to the camera unit 10 and the picking unit 20. The processing unit 30 is, in terms of hardware, constituted by a computer, and inside, arithmetic elements such as a CPU and a GPU, and storage elements are implemented in a ROM, a RAM, an HDD, an SSD, etc. The processing unit 30 uses various electronic computers (computing resources) such as a personal computer (PC), a mainframe, a workstation, a cloud computing system, and further, a tablet terminal, a smartphone, etc. In the embodiment, a tablet terminal (computer) is used for the processing unit 30. The picking method is realized, in terms of software, by a picking program or the like loaded into the main memory.

[0023] FIG. 3 is a block diagram showing the functional units of the picking device 1. The picking device 1 includes a picking unit 20, a camera unit 10, and a processing unit 30. The processing unit 30 includes an acquisition unit 40, a determination unit 50, an operation control unit 60, a parameter setting unit 70, a coordinate conversion unit 80, a coordinate estimation unit 90, a priority assignment unit 100, a region extraction unit 110, an angle detection unit 120, and a suction position and posture determination unit 130. The picking unit 20 and the camera unit 10 are as described above.

[0024] The acquisition unit 40 separates an image of a region corresponding to the object from the image acquired by the camera unit 10 and acquires it as a separated image. The separated image is an image separated by segmentation. That is, the object to be picked is surrounded by a rectangular frame as one section and distinguished from other objects. Thereby, only the object to be picked can be separated.

[0025] The determination unit 50 determines whether the object in the separated image is a pre-registered object. For example, if the object to be picked is a "bag of snack snacks" and the "bag of snack snacks" is pre-registered, it is determined that the object is a pre-registered object.

[0026] When the object is a pre-registered object, the operation control unit 60 executes operation control to pick the object for the picking unit 20.

[0027] The parameter setting unit 70 performs stereo calibration of the 2D camera 11 and the 3D camera 12, and sets conversion parameters for converting the 3D point cloud generated by the shooting of the 3D camera 12 to the corresponding position in the planar image generated by the shooting of the 2D camera 11. Stereo calibration is a type of stereo matching. Stereo matching is a technique for estimating the depth of the scene shown in the image using two images taken from different viewpoints of the same static scene. The conversion parameter is a predetermined matrix, and a certain point (x 1 , y 1 , z 1 ) in space is combined with a predetermined conversion matrix described later to be moved to another position (x 2 , y 2 , z 2 ).

[0028] The coordinate conversion unit 80 performs coordinate conversion of the 3D point cloud into a planar image using the parameters. This will be described later with reference to FIG. 8.

[0029] The coordinate estimation unit 90 projects the 3D point cloud coordinate-converted into the planar image onto the planar image, and estimates the spatial coordinates of the 3D point cloud projected onto the planar image generated by the shooting of the 2D camera 11. The spatial coordinates are Cartesian coordinates represented as (x, y, z).

[0030] The priority assignment unit 100 assigns a priority for picking in the order of decreasing area of the separated image. For example, the priority assignment unit picks the object with the largest area of the separated image first, and picks the object with the second largest area of the separated image second. As a result, since the objects are picked starting from the ones with large areas, the plurality of sachets 3 deposited in the container 2 become more visible. This will be described later with reference to FIG. 4.

[0031] The area extraction unit 110 extracts the image area in the separated image where the suction pad 25 fits, and sets the center of gravity position of the image area as the position in the plane coordinates when the suction pad 25 comes into contact. This will be described later with reference to FIG. 5.

[0032] The angle detection unit 120 compares the orientation of the object with the orientation of the object in a predetermined image, and detects the angle centered on the height direction of the object. The comparison is performed by feature point matching. Feature point matching is to compare geometrically characterized positions such as the four corners of the pouch or the center of gravity position of the pouch on the three-dimensional coordinate side and the two-dimensional coordinate side, and measure the deviation of the angle from the perpendicular line.

[0033] The suction position and posture determination unit 130 estimates the spatial coordinates of the suction position where the suction pad 25 sucks the object, and applies the angle θ of the object to determine the position and posture where the suction pad 25 sucks the object. Thereby, the suction pad 25 can accurately suck the object.

[0034] FIG. 4 is a top view showing the process of detecting the area of the pouch in the container in the embodiment. The process of detecting the area of this pouch 3 is performed by the 2D camera 11 and by instance segmentation using AI (artificial intelligence). Instance segmentation is a technique for detecting the foreground mask of "instances of object classes" shown in an image or an RGB-D image, distinguishing each instance. The portion surrounded by a square in FIG. 4 is the area of one pouch 3.

[0035] FIG. 5 is a top view showing the process of extracting the suction candidate points of the pouch 3 in the container 2. This process is also performed by the 2D camera 11, and points where the suction pad 25 fits are extracted as suction candidate points P from the area of the pouch 3 detected by the process of detecting the area in FIG. 4.

[0036] FIG. 6 is a top view showing a process of detecting the orientation of the sachet 3 in the container 2. This process is also performed by the 2D camera 11, and compares the orientation of the object extracted by the region extraction unit 120 with the orientation of the object in the pre-registered image of the object, and detects the deviation in orientation as the angle θ of the deviation with respect to the perpendicular line in the height direction of the object. This comparison of orientations is performed by feature point matching. In FIG. 6, the vertical orientation of the sachet 3 is represented by arrow A, and the horizontal orientation is represented by arrow B.

[0037] FIG. 7 is a top view showing a process of determining the adsorption position and posture of the sachet 3 in the container. This process is performed by the 3D camera 12, and determines the position and posture where the adsorption pad 25 adsorbs the object using the coordinates of the 3D point cloud of the three-dimensional image. In FIG. 7 as well, the vertical orientation of the sachet 3 is represented by arrow A, and the horizontal orientation is represented by arrow B.

[0038] FIG. 8 is a diagram showing a process of converting a 3D coordinate point cloud into 2D coordinates. As shown in FIG. 8, the upper side is the three-dimensional coordinates to which the 3D point cloud is mapped. P 1 、P 2 、P 3 are each one point among the 3D point cloud. When moving a point on the three-dimensional coordinates to the two-dimensional coordinates, some distortion occurs. Therefore, the matrix is optimized, corrected, and rectified by an algorithm created by AI according to the location. The original point P is multiplied by a certain matrix A to become a new point P'. For example, it is represented by an expression of 3 rows and 3 columns as follows.

[0039]

Equation

[0040] FIG. 9 is a diagram showing the processing by the angle detection unit 120. The orientation of the object extracted by the region extraction unit 110 is compared with the orientation of the object in the pre-registered image of the object, and the deviation in orientation is detected as the deviation angle θ with respect to the perpendicular line H in the height direction of the object. 3A is a pouch to be grasped, and 3B and 3C are other objects. A perpendicular line H is drawn downward from the center of gravity position of 3A, and the rotation angle θ centered on the perpendicular line H is detected as the deviation angle between the orientation of the object extracted by the region extraction unit 110 and the deviation in the orientation of the object in the pre-registered image of the object.

[0041] Based on this, the picking method and the picking program of the embodiment will be described together with reference to the flowchart of FIG. 10. The picking method of the embodiment is executed by the computer (processing unit 30) of the picking device 1 of the embodiment based on the picking program (see FIGS. 2 and 3). The picking program of the embodiment causes the computer of the picking device 1 to realize a parameter setting function, a coordinate conversion function, a coordinate estimation function, an acquisition function, a determination function, a priority assignment function, a region extraction function, an angle detection function, a suction position and posture determination function, a suction function, and an operation control function. Since each function overlaps with the description of the picking device 1 of the above-described embodiment, the details are omitted.

[0042] The flowchart of FIG. 10 shows the flow of an information processing method of one form of the embodiment, and includes various steps such as a parameter setting step (S1), an acquisition step (S2), a coordinate conversion step (S3), a coordinate estimation step (S4), a region extraction step (S5), a determination step (S6), a priority assignment step (S7), an angle detection step (S8), a suction position and posture determination step (S9), a suction step (S10), and an operation control step (S11). In addition, the picking method also includes various appropriately necessary steps not shown in the figures.

[0043] The parameter setting function performs stereo calibration of the 2D camera 11 and the 3D camera 12, and sets conversion parameters for converting the 3D point cloud generated by the shooting of the 3D camera 12 to the corresponding position in the planar image generated by the shooting of the 2D camera 11 (S1; parameter setting step). The acquisition function separates and acquires an image of the region corresponding to the object from the images acquired by the camera unit 10 as a separated image (S2: acquisition function).

[0044] The coordinate conversion function performs coordinate conversion of the 3D point cloud into a planar image using the parameters (S3; coordinate conversion step). The coordinate estimation function projects the 3D point cloud coordinate-converted into the planar image onto the planar image, and estimates the spatial coordinates of the 3D point cloud projected onto the planar image generated by the shooting of the 2D camera 11 (S4: coordinate estimation step).

[0045] The region extraction function extracts the image region in the separated image where the suction pad 25 fits, and sets the centroid position of the image region as the position in the planar coordinates when the suction pad 25 comes into contact (S5: region extraction function). The determination function determines whether the object in the separated image is a pre-registered object (S6: determination function).

[0046] The priority assignment function assigns a priority for picking in the order of the area of the separated image being large (S7: priority assignment function). The angle detection function compares the orientation of the object extracted by the region extraction function with the orientation of the object in the pre-registered object image, and detects the deviation of the orientation as the angle of deviation with respect to the perpendicular line in the height direction of the object (S8: angle detection function). The suction position and posture determination function estimates the spatial coordinates of the suction position where the suction pad 25 sucks the object, and applies the angle of the object to determine the position and posture where the suction pad 25 sucks the object (S9: suction position and posture determination function).

[0047] The suction function sucks the object (S10: suction function). The operation control function executes operation control for picking the object with respect to the picking unit when the object is a pre-registered object (S11: operation control function).

[0048] The information processing program of the embodiment can be implemented using, for example, script languages such as ActionScript, JavaScript (registered trademark), Python, Ruby, and compiler languages such as C language, C++, C#, Objective-C, Swift, Java (registered trademark).

Explanation of Signs

[0049] 1 Picking device 2 Container 3 Pouch 10 Camera unit 11 2D camera 12 3D camera 20 Picking section 30 Processing section 40 Acquisition section 50 Judgment section 60 Operation control section 70 Parameter setting section 80 Coordinate conversion section 90 Coordinate estimation section 100 Priority assignment section 110 Region extraction section 120 Angle detection section 130 Adsorption position and posture determination section

Claims

1. A picking unit that picks an object existing within a predetermined area, A camera unit including a 2D camera and a 3D camera that photograph the object, A processing unit that recognizes the object from an image acquired by the camera unit, and includes: The processing unit: An acquisition unit that separates an image of a region corresponding to the object from the image and acquires it as a separated image; A determination unit that determines whether the object in the separated image is a pre-registered object; When the object is a pre-registered object, an operation control unit that executes operation control to pick the object for the picking unit. A picking device characterized by the above.

2. The processing unit: A parameter setting unit that performs stereo calibration between the 2D camera and the 3D camera and sets conversion parameters for converting a 3D point cloud generated by photographing with the 3D camera to corresponding positions within a planar image generated by photographing with the 2D camera; A coordinate conversion unit that performs coordinate conversion of the 3D point cloud to the planar image using the parameters; A coordinate estimation unit that projects the 3D point cloud coordinate-converted to the planar image onto the planar image and estimates the spatial coordinates of the 3D point cloud projected onto the planar image generated by photographing with the 2D camera. The picking device according to claim 1, further comprising the above.

3. The picking device according to claim 1, wherein the object is a bag in which items are bulk-packed.

4. The picking device according to claim 1, wherein the recognition unit performs image processing using AI. The picking device according to claim 1, wherein the parameter setting unit performs stereo calibration between the 2D camera and the 3D camera and sets conversion parameters for converting a 3D point cloud generated by photographing with the 3D camera to corresponding positions within a planar image generated by photographing with the 2D camera by image processing using AI.

5. The picking unit includes a base, an arm portion extending from the base, and a suction portion connected to the tip of the arm portion that sucks the object. The suction portion includes a suction pad at its tip and a suction portion that applies negative pressure to the suction pad. The picking device according to claim 1, characterized by the above.

6. The processing unit: A priority assignment unit that assigns a priority for picking in descending order of the area of the separated image. An area extraction unit that extracts an image area in the separated image where the suction pad fits, and sets the center of gravity position of the image area as the position in the plane coordinates when the suction pad comes into contact; An angle detection unit that compares the orientation of the object extracted by the area extraction unit with the orientation of the object in a pre-registered image of the object, and detects the deviation in orientation as the angle of deviation with respect to the perpendicular line in the height direction of the object; An adsorption position and posture determination unit that estimates the spatial coordinates of the adsorption position where the adsorption pad adsorbs the object, applies the angle of the object, and determines the position and posture where the adsorption pad adsorbs the object; The picking device according to claim 4, further comprising the above.

Images