System for arranging objects in a box

Abstract

The application provides a system for neatly arranging objects in a box, comprising: a height map generation module, configured to obtain a point cloud map of the box and objects to be stored in the box, and generate a height map according to the point cloud map; an arrangement map generation module, configured to cluster and layer the height map to form a plurality of layer spaces arranged in order from low to high, and construct a free arrangement map in each layer space; a placement position calculation module, configured to sequentially search for idle positions in the free arrangement map from the bottom layer to the top layer, and output a placement position when an idle position exists, the placement position being used for a robot to place the objects; and an arrangement control module, configured to control the height map generation module, the arrangement map generation module and the placement position calculation module to repeatedly run until the objects are arranged in the box or the number of tasks is completed. In the application, the objects can be neatly arranged in the box.

Description

Technical Field

[0001] This invention relates to picking robots, and more specifically, to a system for neatly arranging objects inside a box. Background Technology

[0002] Picking robots are intelligent devices equipped with sensors, lenses, and electro-optical systems that can quickly sort and transport goods.

[0003] More and more visual sensors and force sensors will be used in picking robots, making them increasingly intelligent. With advancements in sensing and recognition systems, artificial intelligence, and other technologies, robots are evolving from being controlled unidirectionally to storing and applying their own data, gradually becoming information-based.

[0004] When picking robots sort and move goods, they need to acquire images of the goods through vision sensors, identify the posture of the goods by recognizing the images, and then the picking robot picks or moves the goods by using the loaded grippers or suction cups.

[0005] When packing incoming materials, multiple layers of items need to be stored inside the box. Existing packing methods lack precision; improper placement of items will reduce the quantity of items per layer or affect the number of layers. Excessive gaps between adjacent items will also affect the packing efficiency. Therefore, a new solution is needed. Summary of the Invention

[0006] In view of the deficiencies in the prior art, the purpose of this invention is to provide a system for neatly arranging objects inside a box.

[0007] The box-shaped object neat arrangement system provided by the present invention includes:

[0008] The height map generation module is used to obtain the point cloud map of the box and the target object stored inside it, and generate a height map based on the point cloud map;

[0009] The placement map generation module is used to cluster and layer the height map to form multiple layers arranged from low to high, and to construct a free placement map in each layer.

[0010] The placement location calculation module searches for available locations on the free placement map from the bottom layer to the top layer. When an available location exists, it outputs the placement location, which the robot uses to place the target object.

[0011] The placement control module controls the repeated operation of the height map generation module, the placement map generation module, and the placement position calculation module until the box is filled with objects or the required number of picking tasks is completed.

[0012] Preferably, the height map generation module includes the following units:

[0013] The point cloud acquisition unit is used to acquire the point cloud image of the box and the target object stored inside it, project the point cloud image in the camera coordinate system to the image coordinate system, calculate the coordinates of each point in the point cloud image in the image coordinate system, and obtain a two-dimensional point cloud in the image coordinate system.

[0014] The coordinate system transformation unit is used to transform the two-dimensional point cloud in the image coordinate system to the pixel coordinate system, calculate the coordinates of the two-dimensional point cloud in the image coordinate system in the image pixel coordinate system, and obtain the two-dimensional point cloud in the pixel coordinate system.

[0015] The height map generation unit is used to project the two-dimensional point cloud in the pixel coordinate system and the height value corresponding to each point onto a preset height map model to generate the target height map.

[0016] Preferably, the point cloud acquisition unit includes the following parts:

[0017] The projector control section is used to control the projector in the depth camera to project structured light onto the enclosure.

[0018] The sensor control section is used to control the light receiving sensor in the depth camera to receive the structured light reflected from the object and generate multiple structured light images;

[0019] The depth reconstruction section is used to generate a point cloud map of the box and the target object stored inside it by performing depth reconstruction based on multiple structured light images.

[0020] Preferably, each pixel in the free-placement map indicates whether the point can support the target object in that layer of space, and multiple pixels that can stably support the target object form the free position.

[0021] Preferably, the placement location calculation module includes the following units:

[0022] The location search unit is used to find available locations along a preset diagonal direction within the freely placed map at the bottom layer.

[0023] The placement control unit is used to control the robot to place the target object into the available location when a free location is found. This triggers the height map generation module. If no free location is found, the search control unit is triggered.

[0024] The search control unit is used to sequentially search for available positions on the free-placed maps located on the upper layer until the free-placed map is located on the top layer.

[0025] Preferably, the placement location calculation module further includes the following unit:

[0026] The top-level construction unit is used to construct a higher-level free placement map by adding a preset target height to the top-level free placement map, so as to place the target object in the higher-level free placement map.

[0027] Preferably, the robot places the target object into an empty position using the following unit:

[0028] A size determination unit is used to acquire a point cloud image of a target object and determine the three-dimensional size of the target object based on the point cloud image of the target object.

[0029] A scaling unit is used to acquire a preset scaling factor and adjust the three-dimensional dimensions of the target object according to the scaling factor to generate the target three-dimensional dimensions.

[0030] The placement control unit is used to control the position finding unit to find an empty position based on the three-dimensional dimensions of the target, and then control the robot unit to place the target object into the empty position.

[0031] Preferably, the height map is clustered and hierarchically structured, including:

[0032] The height map is then subjected to dilation and erosion.

[0033] The minimum and maximum height values ​​of the height map are evenly divided and stored to form a data array;

[0034] Kernel density estimation is performed on each value in the data array to generate a KDE value, and then local maxima are calculated based on the KDE values;

[0035] Each pixel value in the height map is replaced with the corresponding and closest local maximum value to generate the layer space.

[0036] Preferably, a Gaussian kernel function is used as the kernel function when performing the kernel density estimation.

[0037] Preferably, the pixel coordinate system is based on the top-left corner of the image as the origin O. P A two-dimensional rectangular coordinate system is established with the row and column directions of the pixels being the U and V coordinate axes, respectively;

[0038] The image coordinate system is a two-dimensional rectangular coordinate system with the intersection of the lens optical axis and the image plane as the origin O1, and the X and Y coordinate axes parallel to the row and column directions of the pixels.

[0039] The camera coordinate system is based on the lens optical center as its origin O. C The row and column directions parallel to the pixel are X and X, respectively.C axis and Y C The Z-axis direction is determined according to the right-hand coordinate system rule. C A three-dimensional rectangular coordinate system is established along the axial direction.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] In this invention, a height map is first generated based on the point cloud map of the box and the target objects stored inside it. Then, the height map is clustered and layered, and a layer space is constructed layer by layer from low to high. A free placement map is constructed in each layer space. From the bottom layer to the top layer, an empty position is searched in the free placement map. When an empty position exists, the placement position is output. This placement position is used by the robot to place the target object. The robot repeats this process multiple times to fill the box with objects or complete the required number of picking tasks, thereby achieving the neat placement of the target objects in the box. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0043] Figure 1 This is a schematic diagram of the module of the system for neatly arranging objects inside the box in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the height map generation module in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the point cloud acquisition unit in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the map generation module placement in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the placement position calculation module in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the placement position calculation module in a modified embodiment of the present invention;

[0049] Figure 7 A flowchart illustrating the steps for finding a free location in existing technologies; and

[0050] Figure 8 This is a flowchart illustrating the steps for finding a free location in various embodiments of the present invention. Detailed Implementation

[0051] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0052] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0053] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0054] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0055] Figure 1 This is a schematic diagram of the module of the system for neatly arranging objects inside a box in an embodiment of the present invention, as shown below. Figure 1 As shown, the present invention provides a system for neatly arranging objects inside a box, comprising:

[0056] The height map generation module is used to obtain the point cloud map of the box and the target object stored inside it, and generate a height map based on the point cloud map;

[0057] The placement map generation module is used to cluster and layer the height map to form multiple layers arranged from low to high, and to construct a free placement map in each layer.

[0058] The placement location calculation module searches for available locations on the free placement map from the bottom layer to the top layer. When an available location exists, it outputs the placement location, which the robot uses to place the target object.

[0059] The placement control module controls the repeated operation of the height map generation module, the placement map generation module, and the placement position calculation module until the box is filled with objects or the required number of picking tasks is completed.

[0060] In this embodiment of the invention, the point cloud is a dataset, where each point represents a set of X, Y, and Z geometric coordinates and an intensity value. This intensity value records the intensity of the returned signal based on the reflectivity of the object's surface. When these points are combined, they form a point cloud, which is a collection of data points representing 3D shapes or objects in space. The point cloud can also be automatically colored for more realistic visualization.

[0061] Figure 2 This is a schematic diagram of the height map generation module in an embodiment of the present invention, as shown below. Figure 2 As shown, the height map generation module includes the following units:

[0062] The point cloud acquisition unit is used to acquire the point cloud image of the box and the target object stored inside it, project the point cloud image in the camera coordinate system to the image coordinate system, calculate the coordinates of each point in the point cloud image in the image coordinate system, and obtain a two-dimensional point cloud in the image coordinate system.

[0063] The coordinate system transformation unit is used to transform the two-dimensional point cloud in the image coordinate system to the pixel coordinate system, calculate the coordinates of the two-dimensional point cloud in the image coordinate system in the image pixel coordinate system, and obtain the two-dimensional point cloud in the pixel coordinate system.

[0064] The height map generation unit is used to project the two-dimensional point cloud in the pixel coordinate system and the height value corresponding to each point onto a preset height map model to generate the target height map.

[0065] In this embodiment of the invention, the pixel coordinate system is based on the top left corner of the image as the origin O. P A two-dimensional rectangular coordinate system is established with the row and column directions of the pixels being the U and V coordinate axes, respectively.

[0066] The image coordinate system is a two-dimensional rectangular coordinate system with the intersection of the lens optical axis and the image plane as the origin O1, and the X and Y coordinate axes being parallel to the row and column directions of the pixels.

[0067] The camera coordinate system is based on the lens optical center as its origin O. C The row and column directions parallel to the pixel are X and X, respectively. C axis and Y CThe Z-axis direction is determined according to the right-hand coordinate system rule. C A three-dimensional rectangular coordinate system is established along the axial direction.

[0068] In this embodiment of the invention, the height map model is divided into multiple grids according to squares, and these grids are numbered to obtain grid numbers. Similarly, the XOY plane in both the camera coordinate system and the image coordinate system is also divided into multiple grids according to squares of the same size, and these grids are numbered. Based on the grid number and the size of each grid, the x-axis and y-axis coordinates corresponding to that grid can be obtained. The x-axis and y-axis coordinates are transformed to the camera coordinate system according to the transformation relationship between the camera coordinate system, the image coordinate system, and the pixel coordinate system, resulting in the x-axis and y-axis coordinates in the camera coordinate system. Then, based on the x-axis and y-axis coordinates in the camera coordinate system, the corresponding grid in the camera coordinate system for a given grid in the pixel coordinate system can be obtained. Finally, the height values ​​in the height map model corresponding to the multi-frame two-dimensional point cloud in the pixel coordinate system are calculated using the transformation relationship.

[0069] Figure 3 This is a schematic diagram of the point cloud acquisition unit in an embodiment of the present invention, as shown below. Figure 3 As shown, the point cloud acquisition unit includes the following parts:

[0070] The projector control section is used to control the projector in the depth camera to project structured light onto the enclosure.

[0071] The sensor control section is used to control the light receiving sensor in the depth camera to receive the structured light reflected from the object and generate multiple structured light images;

[0072] The depth reconstruction section is used to generate a point cloud map of the box and the target object stored inside it by performing depth reconstruction based on multiple structured light images.

[0073] In this embodiment of the invention, the projector is a DLP projector, the light receiving sensor is a camera unit, used to acquire the structured light pattern formed on the cabinet by the structured light, and the processor module is used to control the synchronous switching of the projector and the camera unit, and to generate point cloud images or depth images of the cabinet and the target object stored inside it according to the structured light pattern.

[0074] Figure 4 This is a schematic diagram of the module of the system for neatly arranging objects inside a box in an embodiment of the present invention, as shown below. Figure 4 As shown, the map generation module includes the following units:

[0075] Clustering hierarchical unit, used to cluster and hierarchically divide the height map into multiple layers arranged in ascending order;

[0076] A map building unit is used to build a freely oriented map in each of the layer spaces and determine the layer height of the freely oriented map, wherein the layer height of the freely oriented map is greater than or equal to the height of the target object.

[0077] The height determination unit is used to determine the height of the freely placed map based on the layer height of the freely placed map, wherein the height of the freely placed map is the lowest point of the freely placed map on that layer.

[0078] Each pixel in the free-placement map indicates whether the point can support the target object in that layer of space, and multiple pixels that can stably support the target object form the free position.

[0079] In this embodiment of the invention, the height map is clustered and hierarchically structured using the kernel density estimation (KDE) method, with a Gaussian kernel function selected as the kernel function. This includes the following steps:

[0080] The height map is then subjected to dilation and erosion.

[0081] The minimum and maximum height values ​​of the height map are evenly divided and stored to form a data array. For example, if the minimum height value is 10 and the maximum height value is 50, and the data array is divided by 10, then the data array is [10,20,30,40,50].

[0082] Kernel density estimation is performed on each value in the data array to generate a KDE value, and then a local maxima is calculated based on the KDE value;

[0083] Each pixel value in the height map is replaced with the corresponding and closest local maximum value to generate the layer space.

[0084] Figure 5 This is a schematic diagram of the placement position calculation module in an embodiment of the present invention, as shown below. Figure 5 As shown, the placement location calculation module includes the following units:

[0085] The location search unit is used to find available locations along a preset diagonal direction within the freely placed map at the bottom layer.

[0086] The placement control unit is used to control the robot to place the target object into the available position when searching for an available position, and then search for another available position row by row until there are no available positions left and trigger the construction control unit.

[0087] The search control unit is used to sequentially search for available positions on the free-placed maps located on the upper layer until the free-placed map is located on the top layer.

[0088] In this embodiment of the invention, an empty position is found along a 45° oblique angle, such as... Figure 8 As shown, it can avoid Figure 7 In this context, it refers to situations where empty positions are missed when searching for empty positions row by row or column by column.

[0089] The target object is set to have a length of l, a width of w, and a height of h. When searching for an empty space, the object is placed with l along the long side of the bin and w along the short side of the bin to check if the area is empty. Then, the object is placed with l along the short side of the bin and w along the long side of the bin to check if the area is empty.

[0090] In a variation of the present invention, the map building unit is controlled to build a free-placed map of the upper layer after increasing a preset target height according to the height of the free-placed map, so that the location search unit can search for vacant locations on the free-placed map of the upper layer.

[0091] The preset target height is determined based on the height tolerance error of the objects placed inside the box, such as 1 cm and 0.5 cm.

[0092] Figure 6 This is a schematic diagram of the placement position calculation module in a modified embodiment of the present invention, as shown below. Figure 6 As shown, the robot places the target object into an empty position using the following unit:

[0093] A size determination unit is used to acquire a point cloud image of a target object and determine the three-dimensional size of the target object based on the point cloud image of the target object.

[0094] A scaling unit is used to acquire a preset scaling factor and adjust the three-dimensional dimensions of the target object according to the scaling factor to generate the target three-dimensional dimensions.

[0095] The placement control unit is used to control the position finding unit to find an empty position based on the three-dimensional dimensions of the target, and then control the robot unit to place the target object into the empty position.

[0096] In a variation of the present invention, when the gap between adjacent target objects is greater than a preset gap threshold, the proportional adjustment factor is less than 1. For example, when the gap threshold along the length direction of the target object is 10% of the length of the target object, the proportional adjustment factor can be set between 0.92 and 0.98.

[0097] When adjacent target objects collide and overlap, the scaling factor is greater than 1, and the scaling factor can be set between 1.02 and 1.1.

[0098] In this embodiment of the invention, a height map is first generated based on the point cloud map of the box and the target object stored inside it. Then, the height map is clustered and layered, and a layer space is constructed layer by layer from low to high. A free placement map is constructed in each layer space. From the bottom layer to the top layer, an empty position is searched in the free placement map. When an empty position exists, the placement position is output. This placement position is used by the robot to place the target object. The process is repeated multiple times to fill the box with objects or complete the picking task, thereby achieving the neat placement of the target object in the box.

[0099] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0100] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A system for neatly arranging objects inside a box, characterized in that, include: The height map generation module is used to obtain the point cloud map of the box and the target object stored inside it, and generate a height map based on the point cloud map; The placement map generation module is used to cluster and layer the height map to form multiple layers arranged from low to high, and to construct a free placement map in each layer. The placement location calculation module is used to search for free locations in the free placement map from the bottom layer to the top layer. When a free location exists, it outputs the placement location, which is used by the robot to place the target object. The placement control module is used to control the height map generation module, the placement map generation module, and the placement position calculation module to run repeatedly until the box is filled with objects or the picking task is completed. The height map generation module includes the following units: The point cloud acquisition unit is used to acquire the point cloud image of the box and the target object stored inside it, project the point cloud image in the camera coordinate system to the image coordinate system, calculate the coordinates of each point in the point cloud image in the image coordinate system, and obtain a two-dimensional point cloud in the image coordinate system. The coordinate system transformation unit is used to transform the two-dimensional point cloud in the image coordinate system to the pixel coordinate system, calculate the coordinates of the two-dimensional point cloud in the image coordinate system in the image pixel coordinate system, and obtain the two-dimensional point cloud in the pixel coordinate system. The height map generation unit is used to project the two-dimensional point cloud in the pixel coordinate system and the height value corresponding to each point onto a preset height map model to generate the height map. The point cloud acquisition unit includes the following parts: a projector control section, used to control the projector in the depth camera to project structured light onto the box.

2. The system for neatly arranging objects inside a box according to claim 1, characterized in that, The point cloud acquisition unit also includes the following components: The sensor control section is used to control the light receiving sensor in the depth camera to receive the structured light reflected by the target object stored inside the box and generate multiple structured light images; The depth reconstruction section is used to generate a point cloud map of the box and the target object stored inside it by performing depth reconstruction based on multiple structured light images.

3. The system for neatly arranging objects inside a box according to claim 1, characterized in that, Each pixel in the free-placement map represents a pixel that can support a target object in that layer of space, and multiple pixels that can stably support the target object form the free space.

4. The system for neatly arranging objects inside a box according to claim 3, characterized in that, The placement location calculation module includes the following units: The location search unit is used to find available locations along a preset diagonal direction within the freely placed map at the bottom layer. The placement control unit is used to control the robot to place the target object in an empty space when an empty space is found. This triggers the height map generation module. If no empty space is found, the search control unit is triggered. The search control unit is used to sequentially search for available positions on the free-placed maps located on the upper layer until the free-placed map is located on the top layer.

5. The system for neatly arranging objects inside a box according to claim 4, characterized in that, The placement location calculation module also includes the following units: The top-level construction unit is used to construct a higher-level free placement map by adding a preset target height to the top-level free placement map, so as to place the target object in the higher-level free placement map.

6. The system for neatly arranging objects inside a box according to claim 4, characterized in that, The robot places the target object into an empty position using the following unit: A size determination unit is used to acquire a point cloud image of a target object and determine the three-dimensional size of the target object based on the point cloud image of the target object. A scaling unit is used to acquire a preset scaling factor and adjust the three-dimensional dimensions of the target object according to the scaling factor to generate the target three-dimensional dimensions. The placement control unit is used to control the position finding unit to find an empty position based on the three-dimensional dimensions of the target, and then control the robot to place the target object into the empty position.

7. The system for neatly arranging objects inside a box according to claim 1, characterized in that, Clustering and hierarchizing the height map includes: The height map is then subjected to dilation and erosion. The minimum and maximum height values ​​of the height map are evenly divided and stored to form a data array; Kernel density estimation is performed on each value in the data array to generate a KDE value, and then local maxima are calculated based on the KDE values; Each pixel value in the height map is replaced with the corresponding and closest local maximum value to generate the layer space.

8. The system for neatly arranging objects inside a box according to claim 7, characterized in that, When performing the kernel density estimation, a Gaussian kernel function is used as the kernel function.

9. The system for neatly arranging objects inside a box according to claim 1, characterized in that, The pixel coordinate system is based on the top left corner of the image as the origin O. P A two-dimensional rectangular coordinate system is established with the row and column directions of the pixels being the U and V coordinate axes, respectively; The image coordinate system is a two-dimensional rectangular coordinate system with the intersection of the lens optical axis and the image plane as the origin O1, and the X and Y coordinate axes being parallel to the row and column directions of the pixels. The camera coordinate system is based on the lens optical center as its origin O. C The row and column directions parallel to the pixel are X and X, respectively. C axis and Y C The Z-axis direction is determined according to the right-hand coordinate system rule. C A three-dimensional rectangular coordinate system is established along the axial direction.

Images