System for tidily placing objects in box body

Abstract

The invention provides a system for tidily placing objects in a box body, and the system comprises a height map generation module which is used for obtaining point cloud maps of the box body and target objects stored in the inner side of the box body, and generating a height map according to the point cloud maps; the placement map generation module is used for clustering and layering the height map to form a plurality of layer spaces which are sequentially arranged from low to high, and a free placement map is constructed in each layer space; the placement position calculation module is used for searching idle positions in the free placement map from the bottom layer to the upper layer in sequence, when the idle positions exist, the placement positions are output, and the placement positions are used for the robot to place the target object; and the control module is used for controlling the height map generation module, the placement map generation module and the placement position calculation module to operate repeatedly until the box body is filled with the objects or the sorting of the task quantity is completed. According to the invention, a target object can be placed in the box body in order.

Description

Technical Field

[0001] The present invention relates to a picking robot, and in particular to a system for neatly placing objects in a box. Background Art

[0002] The picking robot is an intelligent device equipped with sensors, objective lenses and electronic optical systems that can quickly sort and move goods.

[0003] More and more visual sensors and force sensors will be used in picking robots, and picking robots will become more and more intelligent. With the advancement of sensing and recognition systems, artificial intelligence and other technologies, robots are moving from being controlled in one direction to storing and applying data by themselves, and are gradually becoming information-based.

[0004] When goods are sorted and transported by a picking robot, it is necessary to capture images of the goods through a visual sensor, identify the images of the goods to determine the posture of the goods, and then the picking robot picks or transports the goods through a loaded clamp or suction cup.

[0005] When packing incoming materials, it is necessary to store the incoming materials in multiple layers in the box. The packing method in the existing technology has a poor accuracy. Improper placement of items will reduce the number of items loaded in each layer, or affect the number of layers of packing. Too much gap between adjacent items will also affect the packing rate. Therefore, a new solution is needed. Summary of the invention

[0006] In view of the defects in the prior art, an object of the present invention is to provide a system for neatly placing objects in a box.

[0007] The system for neatly placing objects in a box provided by the present invention comprises:

[0008] A height map generation module is used to obtain a point cloud map of the box and the target object stored inside it, and generate a height map according to the point cloud map;

[0009] A placement map generation module is used to cluster and layer the height map to form a plurality of layer spaces arranged in sequence from low to high, and to construct a free placement map in each layer space;

[0010] The placement position calculation module is used to search for free positions in the free placement map from the bottom layer to the top layer. When there is a free position, the placement position is output. The placement position is used by the robot to place the target object:

[0011] The placement control module is used to control the height map generation module, the placement map generation module and the placement position calculation module to run repeatedly until the objects fill the box or the picking of the task quantity is completed.

[0012] Preferably, the height map generation module includes the following units:

[0013] A point cloud acquisition unit is used to acquire a point cloud image of the box and the target object stored inside the box, project the point cloud image in the camera coordinate system to the image coordinate system, calculate the coordinates of each point in the point cloud image in the image coordinate system, and obtain a two-dimensional point cloud in the image coordinate system;

[0014] A coordinate system conversion unit is used to convert the two-dimensional point cloud in the image coordinate system to the pixel coordinate system, calculate the coordinates of the two-dimensional point cloud in the image coordinate system in the image pixel coordinate system, and obtain the two-dimensional point cloud in the pixel coordinate system;

[0015] The height map generating unit is used to project the two-dimensional point cloud in the pixel coordinate system and the height value corresponding to each point into a preset height map model to generate the target height map.

[0016] Preferably, the point cloud acquisition unit includes the following parts:

[0017] A projector control part, used to control the projector in the depth camera to project structured light onto the box;

[0018] A sensor control part, used for controlling the light receiving sensor in the depth camera to receive the structured light reflected by the object to generate a plurality of structured light images;

[0019] The depth reconstruction part is used to perform depth reconstruction based on the multiple structured light images to generate a point cloud image of the box and the target object stored inside it.

[0020] Preferably, each pixel point in the free placement map indicates whether the point can support the target object in the layer space, and a plurality of pixel points that can stably support the target object form the idle position.

[0021] Preferably, the placement position calculation module includes the following units:

[0022] A location search unit, used to search for an idle location along a preset oblique direction in a free-placement map located at the bottom layer;

[0023] A placement control unit, used to control the robot to place the target object at the idle position when searching for an idle position, thereby triggering the height map generation module, and triggering the search control unit when there is no idle position;

[0024] The search control unit is used to search for free positions on the free placement maps on the upper layer in sequence until the free placement map on the top layer.

[0025] Preferably, the placement position calculation module further includes the following units:

[0026] The top layer construction unit is used to construct a free placement map on a higher layer after increasing a preset target height according to the free placement map on the top layer, so as to place the target object in the free placement map on the higher layer.

[0027] Preferably, when the robot is controlled to place the target object at an idle position, the following units are used:

[0028] A size determination unit, used to obtain a point cloud image of a target object, and determine a three-dimensional size of the target object according to the point cloud image of the target object;

[0029] A scale adjustment unit, used for acquiring a preset scale adjustment factor, and adjusting the three-dimensional size of the target object according to the scale adjustment factor to generate a target three-dimensional size;

[0030] A placement control unit is used to control the position search unit to search for an idle position according to the target three-dimensional size, and then control the robot unit to place the target object at the idle position.

[0031] Preferably, clustering and stratifying the height map comprises:

[0032] dilating and corroding the height map;

[0033] Evenly dividing the minimum height value and the maximum height value of the height map, and storing them to form a data array;

[0034] Performing kernel density estimation on each value in the data array to generate a KDE value, and then calculating a local maximum value based on the KDE value;

[0035] Each pixel value in the height map is replaced by the corresponding and closest local maximum value in the layer space.

[0036] Preferably, when performing the kernel density estimation, a Gaussian kernel function is used as the kernel function.

[0037] Preferably, the pixel coordinate system takes the upper left corner of the image as the origin. P , the row and column directions of pixels are respectively the two-dimensional rectangular coordinate system established in the direction of the U and V coordinate axes;

[0038] The image coordinate system is a two-dimensional rectangular coordinate system established with the intersection of the lens optical axis and the image plane as the origin O1 and the row and column directions parallel to the pixels as the X and Y coordinate axis directions;

[0039] The camera coordinate system takes the optical center of the lens as the origin. C , the row and column directions parallel to the pixels are XC Axis and Y C Axis direction, determine Z according to the right-hand coordinate system rule C A three-dimensional rectangular coordinate system established along the axis direction.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] In the present invention, a height map is first generated according to a point cloud map of the box and the target object stored inside it, and then the height map is clustered and layered, and a layer space is constructed layer by layer from low to high. A free placement map is constructed in each layer space, and free positions are searched in the free placement map from the bottom layer to the upper layer. When there is a free position, the placement position is output. The placement position is used for the robot to place the target object, and the objects are repeatedly placed in the box to fill the box or complete the task quantity of picking, thereby achieving neat placement of the target objects in the box. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0043] Figure 1 It is a schematic diagram of the modules of the system for neatly placing objects in a box according to an embodiment of the present invention;

[0044] Figure 2 Schematic diagram of a module for generating a height map in an embodiment of the present invention;

[0045] Figure 3 Schematic diagram of a module of a point cloud acquisition unit in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a module for placing a map generation module in an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of a module for placing a position calculation module in an embodiment of the present invention;

[0048] Figure 6 It is a schematic diagram of a module for placing a position calculation module in a modified example of the present invention;

[0049] Figure 7 A flowchart of the steps of finding an idle position in the prior art; and

[0050] Figure 8 The following is a flowchart of the steps of searching for free locations in various embodiments of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0052] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0054] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0055] Figure 1 FIG. 1 is a schematic diagram of a module of a system for neatly placing objects in a box according to an embodiment of the present invention. Figure 1 As shown, the system for neatly placing objects in a box provided by the present invention comprises:

[0056] A height map generation module is used to obtain a point cloud map of the box and the target object stored inside it, and generate a height map according to the point cloud map;

[0057] A placement map generation module is used to cluster and layer the height map to form a plurality of layer spaces arranged in sequence from low to high, and to construct a free placement map in each layer space;

[0058] The placement position calculation module is used to search for free positions in the free placement map from the bottom layer to the top layer. When there is a free position, the placement position is output. The placement position is used by the robot to place the target object:

[0059] The placement control module is used to control the height map generation module, the placement map generation module and the placement position calculation module to run repeatedly until the objects fill the box or the picking of the task quantity is completed.

[0060] In an embodiment of the present invention, the point cloud is a data set, each point in the data set represents a set of X, Y, Z geometric coordinates and an intensity value, which records the intensity of the return signal according to the reflectivity of the object surface. When these points are combined together, a point cloud is formed, that is, a collection of data points representing a 3D shape or object in space. The point cloud can also be automatically colored to achieve more realistic visualization.

[0061] Figure 2 Schematic diagram of a module for generating a height map in an embodiment of the present invention. Figure 2 As shown, the height map generation module includes the following units:

[0062] A point cloud acquisition unit is used to acquire a point cloud image of the box and the target object stored inside the box, project the point cloud image in the camera coordinate system to the image coordinate system, calculate the coordinates of each point in the point cloud image in the image coordinate system, and obtain a two-dimensional point cloud in the image coordinate system;

[0063] A coordinate system conversion unit is used to convert the two-dimensional point cloud in the image coordinate system to the pixel coordinate system, calculate the coordinates of the two-dimensional point cloud in the image coordinate system in the image pixel coordinate system, and obtain the two-dimensional point cloud in the pixel coordinate system;

[0064] The height map generating unit is used to project the two-dimensional point cloud in the pixel coordinate system and the height value corresponding to each point into a preset height map model to generate the target height map.

[0065] In the embodiment of the present invention, the pixel coordinate system takes the upper left corner of the image as the origin. P , a two-dimensional rectangular coordinate system is established with the row and column directions of the pixels being the U and V coordinate axis directions respectively.

[0066] The image coordinate system is a two-dimensional rectangular coordinate system established with the intersection of the lens optical axis and the image plane as the origin O1 and the row and column directions parallel to the pixels as the X and Y coordinate axis directions.

[0067] The camera coordinate system takes the optical center of the lens as the origin. C , the row and column directions parallel to the pixels are X C Axis and Y CAxis direction, determine Z according to the right-hand coordinate system rule C A three-dimensional rectangular coordinate system established along the axis direction.

[0068] In an embodiment of the present invention, the height map model is divided into squares to obtain a plurality of grids, and the grids are numbered to obtain grid numbers. Similarly, the XOY plane in the camera coordinate system and the image coordinate system is also divided into squares of the same size to obtain a plurality of grids, and the grids are numbered. According to the grid number and the size of each grid, the x-axis and y-axis coordinates corresponding to the grid can be obtained. According to the conversion relationship between the camera coordinate system, the image coordinate system and the pixel coordinate system, the x-axis and y-axis coordinates are converted to the camera coordinate system to obtain the x-axis and y-axis coordinates in the camera coordinate system, and then according to the x-axis and y-axis coordinates in the camera coordinate system, the grid corresponding to a certain grid in the pixel coordinate system in the camera coordinate system can be obtained. Then, the height value in the height map model corresponding to the multi-frame two-dimensional point cloud in the pixel coordinate system is calculated by the conversion relationship.

[0069] Figure 3 Schematic diagram of the module of the point cloud acquisition unit in the embodiment of the present invention. Figure 3 As shown, the point cloud acquisition unit includes the following parts:

[0070] A projector control part, used to control the projector in the depth camera to project structured light onto the box;

[0071] A sensor control part, used for controlling the light receiving sensor in the depth camera to receive the structured light reflected by the object to generate a plurality of structured light images;

[0072] The depth reconstruction part is used to perform depth reconstruction based on the multiple structured light images to generate a point cloud image of the box and the target object stored inside it.

[0073] In an embodiment of the present invention, the projector is a DLP projector, the light receiving sensor is a camera unit, which is used to collect the structured light pattern formed by the structured light on the box, and the processor module is used to control the synchronous switching of the projector and the camera unit, and generate a point cloud image or depth image of the box and the target object stored inside it according to the structured light pattern.

[0074] Figure 4 FIG. 1 is a schematic diagram of a module of a system for neatly placing objects in a box according to an embodiment of the present invention. Figure 4 As shown, the layout map generation module includes the following units:

[0075] A clustering and stratification unit, used for clustering and stratifying the height map to form a plurality of layer spaces arranged in sequence from low to high;

[0076] A map construction unit, used to construct a free-placement map in each layer space and determine the layer height of the free-placement map, wherein the layer height of the free-placement map is greater than or equal to the height of the target object;

[0077] The height determination unit is used to determine the height of the freely placed map according to the layer height of the freely placed map, and the height of the freely placed map is the lowest point of the freely placed maps on this layer.

[0078] Each pixel point in the free placement map indicates whether the point can support the target object in the layer space, and a plurality of pixel points that can stably support the target object form the idle position.

[0079] In an embodiment of the present invention, a kernel density estimation (KDE) method is used, and a Gaussian kernel function is selected as the kernel function to cluster and stratify the height map, including the following steps:

[0080] dilating and corroding the height map;

[0081] Evenly separate the minimum height value and the maximum height value of the height map and store them into a data array. For example, if the minimum height value is 10 and the maximum height value is 50, separated by 10, the data array is [10, 20, 30, 40, 50].

[0082] Performing kernel density estimation on each value in the data array to generate a KDE value, and then calculating a local maxima based on the KDE value;

[0083] Each pixel value in the height map is replaced by the corresponding and closest local maximum value in the layer space.

[0084] Figure 5 FIG. 1 is a schematic diagram of a module for placing a position calculation module in an embodiment of the present invention. Figure 5 As shown, the placement position calculation module includes the following units:

[0085] A location search unit, used to search for an idle location along a preset oblique direction in a free-placement map located at the bottom layer;

[0086] A placement control unit, used to control the robot to place the target object at the idle position when searching for an idle position, and search for another idle position row by row until there is no idle position and the construction control unit is triggered;

[0087] The search control unit is used to search for free positions on the free placement maps on the upper layer in sequence until the free placement map on the top layer.

[0088] In the embodiment of the present invention, an idle position is searched along a 45° oblique angle direction, such as Figure 8 As shown, it is possible to avoid Figure 7 In the case of missing free positions when searching for free positions row by row or column by column.

[0089] Set the length of the target object to l, the width to w, and the height to h. When searching for free positions, check whether the area is free by placing l along the long side of the material box and w along the short side of the material box; then check whether the area is free by placing l along the short side of the material box and w along the long side of the material box.

[0090] In a variation of the present invention, the map construction unit is controlled to construct a free placement map of the upper layer after increasing a preset target height according to the height of the free placement map, so that the position search unit can search for an idle position on the free placement map of the upper layer.

[0091] The preset target height is determined according to the height tolerance error of the objects placed in the box, such as 1 cm and 0.5 cm.

[0092] Figure 6 FIG. 1 is a schematic diagram of a module for placing a position calculation module in a modified example of the present invention, such as Figure 6 As shown, when the robot is controlled to place the target object in an idle position, it is realized by the following units:

[0093] A size determination unit, used to obtain a point cloud image of a target object, and determine a three-dimensional size of the target object according to the point cloud image of the target object;

[0094] A scale adjustment unit, used for acquiring a preset scale adjustment factor, and adjusting the three-dimensional size of the target object according to the scale adjustment factor to generate a target three-dimensional size;

[0095] A placement control unit is used to control the position search unit to search for an idle position according to the target three-dimensional size, and then control the robot unit to place the target object at the idle position.

[0096] In a variation of the present invention, when the gap between adjacent target objects is greater than a preset gap threshold, the proportional adjustment factor is less than 1. For example, when the gap threshold along the length direction of the target object is 10% of the length of the target object, the proportional adjustment factor can be set to between 0.92 and 0.98.

[0097] When adjacent target objects collide and overlap, the proportional adjustment factor is greater than 1, and the proportional adjustment factor can be set between 1.02 and 1.1.

[0098] In an embodiment of the present invention, a height map is first generated based on a point cloud map of the box and the target object stored inside it, and then the height map is clustered and layered to construct a layer space from low to high. A free placement map is constructed in each layer space, and free positions are searched in the free placement map from the bottom layer to the top layer. When there is a free position, the placement position is output. The placement position is used for the robot to place the target object, and the process is repeated many times to fill the box with objects or complete the picking of the task quantity, thereby achieving neat placement of the target objects in the box.

[0099] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0100] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A system for neatly placing objects in a box, characterized in that: include: A height map generation module is used to obtain a point cloud map of the box and the target object stored inside it, and generate a height map according to the point cloud map; A placement map generation module is used to cluster and layer the height map to form a plurality of layer spaces arranged in sequence from low to high, and to construct a free placement map in each layer space; The placement position calculation module is used to search for free positions in the free placement map from the bottom layer to the top layer. When there is a free position, the placement position is output. The placement position is used by the robot to place the target object: The placement control module is used to control the height map generation module, the placement map generation module and the placement position calculation module to run repeatedly until the objects fill the box or the picking of the task quantity is completed.

2. The system for neatly placing objects in a box according to claim 1, characterized in that: The height map generation module includes the following units: A point cloud acquisition unit is used to acquire a point cloud image of the box and the target object stored inside the box, project the point cloud image in the camera coordinate system to the image coordinate system, calculate the coordinates of each point in the point cloud image in the image coordinate system, and obtain a two-dimensional point cloud in the image coordinate system; A coordinate system conversion unit is used to convert the two-dimensional point cloud in the image coordinate system to the pixel coordinate system, calculate the coordinates of the two-dimensional point cloud in the image coordinate system in the image pixel coordinate system, and obtain the two-dimensional point cloud in the pixel coordinate system; The height map generating unit is used to project the two-dimensional point cloud in the pixel coordinate system and the height value corresponding to each point into a preset height map model to generate the target height map.

3. The system for neatly placing objects in a box according to claim 2, characterized in that: The point cloud acquisition unit includes the following parts: A projector control part, used to control the projector in the depth camera to project structured light onto the box; A sensor control part, used for controlling the light receiving sensor in the depth camera to receive the structured light reflected by the object to generate a plurality of structured light images; The depth reconstruction part is used to perform depth reconstruction based on the multiple structured light images to generate a point cloud image of the box and the target object stored inside it.

4. The system for neatly placing objects in a box according to claim 1, characterized in that: Each pixel point in the free placement map indicates whether the point can support the target object in the layer space, and a plurality of pixel points that can stably support the target object form the idle position.

5. The system for neatly placing objects in a box according to claim 4, characterized in that: The placement position calculation module includes the following units: A location search unit, used to search for an idle location along a preset oblique direction in a free-placement map located at the bottom layer; A placement control unit, used to control the robot to place the target object at the idle position when searching for an idle position, thereby triggering the height map generation module, and triggering the search control unit when there is no idle position; The search control unit is used to search for free positions on the free placement maps on the upper layer in sequence until the free placement map on the top layer.

6. The system for neatly placing objects in a box according to claim 5, characterized in that: The placement position calculation module also includes the following units: The top layer construction unit is used to construct a free placement map on a higher layer after increasing a preset target height according to the free placement map on the top layer, so as to place the target object in the free placement map on the higher layer.

7. The system for neatly placing objects in a box according to claim 1, characterized in that: When controlling the robot to place the target object in an idle position, it is achieved through the following units: A size determination unit, used to obtain a point cloud image of a target object, and determine a three-dimensional size of the target object according to the point cloud image of the target object; A scale adjustment unit, used for acquiring a preset scale adjustment factor, and adjusting the three-dimensional size of the target object according to the scale adjustment factor to generate a target three-dimensional size; A placement control unit is used to control the position search unit to search for an idle position according to the target three-dimensional size, and then control the robot unit to place the target object at the idle position.

8. The system for neatly placing objects in a box according to claim 1, characterized in that: The height map is clustered and layered, including: dilating and corroding the height map; Evenly dividing the minimum height value and the maximum height value of the height map, and storing them to form a data array; Performing kernel density estimation on each value in the data array to generate a KDE value, and then calculating a local maximum value based on the KDE value; Each pixel value in the height map is replaced by the corresponding and closest local maximum value in the layer space.

9. The system for neatly placing objects in a box according to claim 8, characterized in that: When performing the kernel density estimation, a Gaussian kernel function is used as the kernel function.

10. The system for neatly placing objects in a box according to claim 2, characterized in that: The pixel coordinate system takes the upper left corner of the image as the origin. P , the row and column directions of pixels are respectively the two-dimensional rectangular coordinate system established in the direction of the U and V coordinate axes; The image coordinate system is a two-dimensional rectangular coordinate system established with the intersection of the lens optical axis and the image plane as the origin O1 and the row and column directions parallel to the pixels as the X and Y coordinate axis directions; The camera coordinate system takes the optical center of the lens as the origin. C , the row and column directions parallel to the pixels are X C Axis and Y C Axis direction, determine Z according to the right-hand coordinate system rule C A three-dimensional rectangular coordinate system established along the axis direction.

Images