Robot automatic boxing method, system and device based on multi-mode perception feedback control

Abstract

The invention relates to the technical field of automatic boxing, in particular to a robot automatic boxing method, system and device based on multi-mode perception feedback control, and the method comprises the steps that the geometric dimension of goods to be loaded is obtained; a boxing space coordinate system fixedly associated with the container body is established above the boxing area, surface appearance data of goods placed in the container area are collected in real time, a three-dimensional height distribution diagram is constructed, and a local protruding area is identified based on the three-dimensional height distribution diagram; and on the basis of the local convex area, in combination with the geometric dimension of the to-be-filled goods, planning an optimal box falling position, and recording the optimal box falling position as a box falling area. According to the method, the packing space coordinate system fixedly associated with the box body is established, the stacking state in the box and the goods to be loaded are sensed in real time in combination with three-dimensional depth imaging, the three-dimensional height distribution diagram reflecting the actual morphology in the box is constructed, and on the basis, the physical dimension of the goods and the height distribution information in the box are fused, so that the packing space coordinate system is obtained. And planning a box falling area with the optimal stability.

Description

Technical Field

[0001] This invention relates to the field of automated packing technology, specifically to a robot automated packing method, system, and apparatus based on multimodal perception feedback control. Background Technology

[0002] With the continuous improvement of logistics automation and intelligent manufacturing, robotic packing equipment has been gradually applied to scenarios such as warehousing, sorting, and end-of-line production. However, in the actual packing process, the stacking state of goods inside the box is often highly uncertain. Goods of different sizes, shapes, and placement orders will form irregular height distributions and local bulges inside the box. Existing automated packing technologies mostly rely on preset rules or ideal planes for placement planning, which makes it difficult to reflect the actual spatial shape of the goods already placed inside the box in a timely and accurate manner. This can easily lead to problems such as goods being suspended, tilted, or subjected to excessive local stress, affecting packing stability and transportation safety.

[0003] On the other hand, most existing container loading path planning is based on simplified two-dimensional projection or fixed height assumptions, which do not adequately consider changes in cargo posture and spatial interference. Especially when the internal space of the container is limited and the stacking height varies greatly, collisions during the grabbing process or sudden changes in posture at the moment of dropping the container are likely to occur.

[0004] Therefore, there is an urgent need for a robotic automatic packing method that can integrate multi-source perception information, acquire the stacking status and spatial pose of goods in the box in real time, and realize stable box drop area recognition and constrained path planning on this basis, so as to solve the problems of insufficient packing stability and high collision risk of existing technologies in complex stacking environments. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a robot automatic packing method, system, and apparatus based on multimodal perception feedback control.

[0006] This invention employs the following technical solution: an automatic packing method for robots based on multimodal perception feedback control, comprising:

[0007] Obtain the geometric dimensions of the cargo to be loaded. , These represent the length, width, and height of the goods to be loaded, respectively.

[0008] A loading space coordinate system is established above the loading area and is fixedly associated with the container. The surface topography data of the goods placed in the container area are collected in real time to construct a three-dimensional height distribution map. Based on the three-dimensional height distribution map, local protruding areas are identified.

[0009] Based on the local protruding areas and the geometric dimensions of the goods to be loaded, the optimal drop position is planned and denoted as the drop area.

[0010] Obtain a 3D height distribution map of the box-dropping area and calculate the corresponding box-dropping height benchmark;

[0011] By using a 3D depth imaging unit deployed in the transport area, the goods to be packed are scanned frame by frame to obtain a discrete point cloud set, and the point cloud set is uniformly mapped to the packing space coordinate system to obtain a discrete point cloud set of goods.

[0012] The three-dimensional height distribution map of the box-dropping area, the discrete point cloud of the goods, and the box-dropping height benchmark are used to generate a box-dropping path movement command. Based on the box-dropping path movement command, the automatic gripping mechanism is continuously controlled to achieve stable and non-crushing placement of goods within the target box-dropping area.

[0013] As a further description of the above technical solution: the method for identifying local protrusion regions is as follows:

[0014] By using a 3D depth imaging unit positioned above the container, the upper surface of the placed goods is scanned frame by frame to obtain a discrete point cloud set. This point cloud set is then mapped uniformly to the container's spatial coordinate system to obtain a discrete point cloud set inside the container opening. , These are points within the discrete point cloud set inside the box opening;

[0015] In the space coordinate system plane of the packing box Build regular grid cells on top , Indicates the first OK The grid cells are arranged in columns, and the maximum height value is calculated within each grid cell, thereby forming a three-dimensional height distribution map that reflects the real-time stacking status inside the box. , Represents the x and y coordinates in the packing space coordinate system;

[0016] Based on the aforementioned three-dimensional height distribution map, calculate the height change rate between adjacent grid cells. The region whose height value simultaneously meets the preset conditions is identified as a local protrusion region.

[0017] As a further description of the above technical solution: the preset conditions include: ; ,in This represents the average stacking height inside the boxes. and Threshold parameters set based on historical stability data; For the first OK Column grid.

[0018] As a further description of the above technical solution: the method for planning the optimal box placement location includes:

[0019] Project the cargo to be loaded onto the container space coordinate system. On a three-dimensional height distribution map, for a plane with a continuous area greater than [missing information] × Furthermore, regions with a height change rate less than a preset height change rate threshold are used to construct a set of candidate placement regions;

[0020] In the set of candidate placement regions, the stability evaluation function of each candidate region is calculated by combining the spatial distribution relationship of the local protrusion regions;

[0021] The candidate region with the optimal stability evaluation function is selected as the final box drop location and marked as the box drop region;

[0022] Obtain a 3D height distribution map of the box-dropping area and calculate the corresponding box-dropping height benchmark.

[0023] As a further description of the above technical solution: the method for generating packing path movement instructions includes:

[0024] Based on the discrete point cloud set of the cargo, the current spatial pose of the cargo to be filled is determined in the container space coordinate system, and a cargo geometric model representing the spatial occupancy range of the cargo to be filled is constructed in combination with the geometric dimensions of the cargo to be filled.

[0025] Based on the spatial occupancy range of the box-dropping area and the box-dropping height reference, the target box-dropping posture of the goods to be loaded is determined in the box-loading space coordinate system. The target box-dropping posture includes the target planar position, the target height position, and the cargo posture angle.

[0026] Establish path planning constraints between the current spatial pose and the target box landing pose;

[0027] Under the premise of satisfying the path planning constraints, based on the cargo geometry model and the three-dimensional height distribution map of the dropping area, the movement reach space of the cargo to be loaded is analyzed, and a continuous dropping path is generated by connecting the transition approach pose and the final dropping pose in sequence.

[0028] As a further description of the above technical solution: the path planning constraints include minimum safe clearance constraints with the placed goods, box drop height constraints, and attitude change constraints.

[0029] As a further description of the above technical solution: the method for analyzing the reachable space of the cargo to be loaded and generating a continuous dropping path composed of the transition approach pose and the final dropping pose connected in sequence includes:

[0030] Step S01: Within the container loading space coordinate system, based on the three-dimensional height distribution map of the container dropping area, determine the transition approach pose of the goods to be loaded. ;

[0031] in, These are the x-coordinate, y-coordinate, and vertical coordinate of the cargo to be loaded, as well as the cargo attitude angle;

[0032] Step S02: Determine the final container placement pose based on the cargo geometry model and the 3D height distribution map of the container placement area. , ;

[0033] in, These are the x-coordinate, y-coordinate, and vertical coordinates of the final landing position of the cargo, as well as the cargo attitude angle;

[0034] Step S03: Based on the cargo geometry model and the occupancy information of the cargo already placed in the drop box area, construct a set of reachable spaces for the cargo to be loaded. In the set of reachable spaces, exclude all spatial areas that do not meet the path planning constraints to ensure that the generated path will not collide or violate the attitude constraints during the movement.

[0035] Step S04: Within the reachable space set, sequentially transfer the approach pose. and final landing position Connect them to form a continuous drop path for the boxes.

[0036] As a further description of the above technical solution: the method for constructing the reachable space set of goods to be loaded based on the cargo geometry model and the occupancy information of goods already placed within the container dropping area includes:

[0037] The container landing area and its safe height range above it are discretized into three-dimensional grid cells in the container space coordinate system. Each grid cell represents the possible spatial location of the cargo's core.

[0038] For each mesh cell, based on the outer contour of the cargo geometry model, the volume occupied by the cargo at its corresponding position and orientation in that mesh cell is mapped to form a set of voxels at that position.

[0039] The intersection of the occupied space formed by the goods placed in the drop box area and the occupied voxel set of each grid cell is detected. If any grid cell overlaps with the occupied space of the goods placed, the grid cell is marked as an unreachable grid cell.

[0040] Further path planning constraints are imposed on reachable grid cells, including minimum safety clearance constraints, drop height constraints, and attitude change constraints;

[0041] The minimum safety clearance constraint is as follows: for each candidate grid cell, calculate the minimum distance between the outer contour of the cargo and the obstacle. If the minimum distance is less than the preset distance threshold, then exclude the grid cell.

[0042] The drop height constraint is: if the height of a grid cell is less than the drop height reference plus the safety height threshold, then the grid cell is excluded.

[0043] The attitude change constraint is: if the attitude angle of the cargo in the corresponding grid cell is... Exceeding the allowed range, i.e. If so, then exclude that grid cell, where, Let be the rotation angle of the cargo about the X-axis. Let be the rotation angle of the cargo about the Y-axis. Let be the rotation angle of the cargo around the Z-axis. This represents the magnitude of the attitude angle vector of the cargo within that grid cell. This indicates the maximum permissible attitude deflection angle threshold for the cargo;

[0044] All grid cells that satisfy the path planning constraints are combined to form the final reachable space set.

[0045] An automated robot packing system based on multimodal perception feedback control, comprising the following steps:

[0046] The size acquisition module acquires the geometric dimensions of the goods to be loaded.

[0047] The container space establishment module establishes a container space coordinate system fixedly associated with the container body above the container area, collects surface topography data of goods placed in the container area in real time, constructs a three-dimensional height distribution map, and identifies local protruding areas based on the three-dimensional height distribution map;

[0048] The container dropping area planning module, based on local protruding areas and combined with the geometric dimensions of the goods to be loaded, plans the optimal container dropping position, which is denoted as the container dropping area;

[0049] The height reference acquisition module acquires a three-dimensional height distribution map of the box-dropping area and calculates the corresponding box-dropping height reference.

[0050] The point cloud acquisition and mapping module uses a 3D depth imaging unit arranged in the transport area to scan the goods to be packed frame by frame to obtain a discrete point cloud set, and then maps the point cloud set to the packing space coordinate system to obtain a discrete point cloud set of the goods.

[0051] The packing path generation module generates packing path movement instructions from the three-dimensional height distribution map of the packing area, the discrete point cloud set of the goods, and the packing height benchmark. Based on the packing path movement instructions, it continuously controls the automatic gripping mechanism to achieve stable and non-crushing placement of goods within the target packing area.

[0052] A robotic automated box-packing device based on multimodal perception feedback control includes a device and a robotic automated box-packing system based on multimodal perception feedback control. The device includes:

[0053] device body;

[0054] The conveying mechanism, located within the device body, is used to convey goods to be packed.

[0055] Three-dimensional depth imaging units are installed inside the device body, at least two of them, one of which is arranged above the conveying area to collect point cloud data of the goods to be packed, and the other is arranged above the container to collect point cloud data of the goods inside the container.

[0056] An automatic gripping mechanism, located within the device body, is used to grip goods to be packed.

[0057] Delivery mechanism, used to drive the automatic gripping mechanism to move within the packing space to realize the delivery and packing of goods;

[0058] The device body is provided with a support platform, on which cargo boxes of goods to be stacked are placed;

[0059] The robotic automatic packing system based on multimodal perception feedback control is connected to a three-dimensional depth imaging unit, an automatic grasping mechanism, and a delivery mechanism to control the coordinated operation of the above mechanisms, thereby realizing real-time perception, path planning, and packing of goods.

[0060] Beneficial effects:

[0061] In the above technical solution, by establishing a loading space coordinate system fixedly associated with the container, and combining three-dimensional depth imaging to perceive the stacking state inside the container and the goods to be loaded in real time, a three-dimensional height distribution map reflecting the actual shape inside the container is constructed, so as to accurately identify local protruding areas and placement space. On this basis, the geometric dimensions of the goods are integrated with the height distribution information inside the container to plan the optimal dropping area, effectively avoiding the problem of unstable loading caused by uneven stacking, local suspension or compression, and improving the reliability and consistency of the automatic loading process of multi-specification goods.

[0062] Furthermore, a realistic geometric model is constructed based on the discrete point cloud of the cargo, and multiple constraints such as minimum safety clearance, drop height, and attitude change are introduced during the path planning process. Through reachability space analysis, a continuous packing path including transitional approach pose and final drop pose is generated, enabling the cargo to be placed smoothly and in a controlled manner into the target area. This method significantly reduces the collision risk and attitude change during the packing process, ensures that the cargo is dropped without being squeezed, and improves the safety, intelligence level, and engineering applicability of the automated packing system in complex stacking environments. Attached Figure Description

[0063] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0064] Figure 1 This is a flowchart illustrating an automated robot packing method based on multimodal perception feedback control provided in Embodiment 1 of the present invention.

[0065] Figure 2 This is a flowchart of the method for identifying local protrusion regions provided in Embodiment 1 of the present invention;

[0066] Figure 3 This is a flowchart of a method for generating packing path movement instructions provided in Embodiment 1 of the present invention;

[0067] Figure 4 This is a module connection diagram of a robot automatic packing system based on multimodal perception feedback control provided in Embodiment 2 of the present invention;

[0068] Figure 5 This is a schematic diagram of the structure of a robotic automated packing device based on multimodal perception feedback control provided in Embodiment 3 of the present invention. Figure 1 ;

[0069] Figure 6 This is a schematic diagram of the structure of a robotic automated packing device based on multimodal perception feedback control provided in Embodiment 3 of the present invention. Figure 2 ;

[0070] Reference numerals in the attached drawings: 1. Device body; 2. Conveying mechanism; 3. Three-dimensional depth imaging unit; 4. Automatic gripping mechanism; 5. Delivery mechanism; 6. Cargo box; 7. Carrier platform. Detailed Implementation

[0071] To make the technical means, creative features, objectives, and effects of this invention readily understandable, the invention is further described below with reference to specific illustrations. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0072] Example 1

[0073] Please see Figures 1-3 This invention provides a technical solution: an automatic packing method for robots based on multimodal perception feedback control, comprising:

[0074] Obtain the geometric dimensions of the cargo to be loaded. , These represent the length, width, and height of the goods to be loaded, respectively.

[0075] It should be noted that before automatically grabbing the goods, the geometric dimensions of the goods to be filled are directly obtained based on the verified goods specification information in the historical packing database.

[0076] Establish a spatial coordinate system for packing that is fixedly associated with the container above the packing area. Real-time surface topography data of goods placed in the container area is collected to construct a three-dimensional height distribution map, and local protruding areas are identified based on the three-dimensional height distribution map;

[0077] The method for identifying localized raised areas is as follows:

[0078] By using a 3D depth imaging unit positioned above the container, the upper surface of the placed goods is scanned frame by frame to obtain a discrete point cloud set. This point cloud set is then mapped uniformly to the container's spatial coordinate system to obtain a discrete point cloud set inside the container opening. , This represents a point in the discrete point cloud set within the box opening;

[0079] It should be noted that the above steps also include preprocessing the point cloud data, including noise filtering of the collected point cloud data, removal of isolated points and outliers, and reduction of the number of point clouds by downsampling or voxel gridding while maintaining geometric features; the above preprocessing methods are all existing technologies and will not be described in detail here.

[0080] The method for mapping the point cloud set to the container space coordinate system to obtain the discrete point cloud set inside the container opening is as follows: Camera calibration is performed on the collected point cloud data to obtain the intrinsic parameters of the depth imaging unit and its extrinsic parameters relative to the container space reference coordinate system, including rotation and translation. Then, the point cloud data is mapped from the depth imaging unit coordinate system to the unified container space coordinate system through rotation and translation transformations, thereby ensuring that all point clouds are aligned with the geometric information of the container and cargo under the same spatial reference. The coordinate transformation method based on calibration is existing technology and will not be elaborated here.

[0081] In the space coordinate system plane of the packing box Build regular grid cells on top , Indicates the first OK The grid cells are arranged in columns, and the maximum height value is calculated within each grid cell, thereby forming a three-dimensional height distribution map that reflects the real-time stacking status inside the box. ; This represents the x-coordinate and y-coordinate in the packing space coordinate system.

[0082] Based on the aforementioned three-dimensional height distribution map, calculate the height change rate between adjacent grid cells. And areas whose height values ​​simultaneously meet preset conditions are identified as local protrusion areas;

[0083] It should be noted that: ,in, Represents a three-dimensional height distribution map )along The partial derivative in the direction is used to characterize the rate of change of height of the stacked surface between adjacent grid cells in that direction. Represents a three-dimensional height distribution map along The partial derivative in the direction is used to characterize the rate of change of height of the stacked surface between adjacent grid cells in that direction; Represents a three-dimensional height distribution map The height gradient vector in the packing space coordinate system is used to describe the trend and intensity of the change in the height of the stacked surface in the horizontal direction.

[0084] The preset conditions include: ; ,in This represents the average stacking height inside the boxes. and Threshold parameters set based on historical stability data; For the first OK Column grid;

[0085] It should be noted that, among them This indicates that the height of this point is significantly higher than the current overall stacking level of the boxes. This indicates that the point has a significant slope change relative to its neighborhood, rather than being a flat plateau. Combining these two characteristics can effectively exclude areas that are generally elevated but flat, and only identify local protrusions that are prone to compression.

[0086] Based on the local protruding areas and the geometric dimensions of the goods to be loaded, the optimal placement position of the container is planned.

[0087] Methods for planning the optimal drop-off location include:

[0088] Project the cargo to be loaded onto the container space coordinate system. On a three-dimensional height distribution map, for areas that satisfy planar continuity × A set of candidate placement regions is constructed for regions whose height is greater than and whose height change rate is less than a preset height change rate threshold.

[0089] In the set of candidate placement regions, the stability evaluation function of each candidate region is calculated by combining the spatial distribution relationship of the local protrusion regions;

[0090] The stability evaluation function The expression is:

[0091] ;

[0092] In the formula, This represents the height variance within the candidate region. Indicates the distance between the candidate region and the nearest convex region;

[0093] The candidate region with the optimal stability evaluation function is selected as the final box drop location and marked as the box drop region;

[0094] Obtain the 3D height distribution map of the box dropping area. , These are the spatial coordinates along the X-axis. The corresponding spatial coordinate value in the Y-axis direction, Given the height value, calculate the corresponding drop height reference Hjz;

[0095] The method for calculating the reference height for box dropping includes:

[0096] Extract the height values ​​of all grid cells within the box-dropping area to form a local height set. Then, obtain the highest point within this local height set as the height reference and the height of the box-dropping area.

[0097] Using a 3D depth imaging unit deployed in the transport area, the goods to be packed are scanned frame by frame to obtain a discrete point cloud set. This point cloud set is then mapped uniformly to the packing space coordinate system to obtain the discrete point cloud set of the goods. , Represents a point in a discrete point cloud set of goods;

[0098] It should be noted that the above steps also include preprocessing the power data within the discrete point cloud set, including noise filtering of the collected point cloud data, removal of isolated points and outliers, and reduction of the number of point clouds by downsampling or voxel meshing while maintaining geometric features; the above preprocessing methods are all existing technologies and will not be described in detail here.

[0099] The cargo attitude angle is obtained by fitting a discrete point cloud set of cargo. The specific methods for obtaining the angle include:

[0100] Three-dimensional depth cameras deployed in the transport area are used to acquire point cloud data of the cargo surface;

[0101] Perform 3D fitting on point cloud data, for example, fitting it into a rectangular parallelepiped or other geometric model;

[0102] The attitude angle is calculated using the rotation matrix R between the fitted model coordinate system and the packing space coordinate system. ;

[0103] ;

[0104] This function represents the conversion of the rotation matrix R into Euler angles. The rotation matrix represents the local coordinate system of the cargo relative to the reference coordinate system of the packing space; Let be the rotation angle of the cargo about the X-axis. The rotation angle of the cargo around the Y-axis, The rotation angle of the cargo around the Z-axis.

[0105] The three-dimensional height distribution map of the box-dropping area, the discrete point cloud of the goods, and the box-dropping height benchmark are used to generate a box-dropping path movement command. Based on the box-dropping path movement command, the automatic gripping mechanism is continuously controlled to achieve stable and non-crushing placement of goods within the target box-dropping area.

[0106] Methods for generating packing path movement instructions include:

[0107] Based on the discrete point cloud set of the cargo, the current spatial pose of the cargo to be filled is determined in the container space coordinate system, and a cargo geometric model representing the spatial occupancy range of the cargo to be filled is constructed in combination with the geometric dimensions of the cargo to be filled.

[0108] Based on the spatial occupancy range of the box-dropping area and the box-dropping height reference, the target box-dropping posture of the goods to be loaded is determined in the box-loading space coordinate system. The target box-dropping posture includes the target planar position, the target height position, and the cargo posture angle.

[0109] Establish path planning constraints between the current spatial pose and the target box-dropping pose. The path planning constraints include minimum safe clearance constraints with the already placed goods, box-dropping height constraints, and attitude change constraints.

[0110] Under the premise of satisfying the path planning constraints, based on the cargo geometry model and the three-dimensional height distribution map of the dropping area, the movement reach space of the cargo to be loaded is analyzed, and a continuous dropping path is generated by connecting the transition approach pose and the final dropping pose in sequence.

[0111] The method for analyzing the reachable space of the cargo to be loaded and generating a continuous drop path consisting of a transitional approach pose and a final drop pose is as follows:

[0112] Step S01: Within the container loading space coordinate system, based on the three-dimensional height distribution map of the container dropping area, determine the transition approach pose of the goods to be loaded. ;

[0113] in, These are the x-coordinate, y-coordinate, and vertical coordinate of the cargo to be loaded, as well as the cargo attitude angle;

[0114] Step S02: Determine the final container placement pose based on the cargo geometry model and the 3D height distribution map of the container placement area. , ;

[0115] Among them, the horizontal, vertical, and axial coordinates of the final landing position of the cargo, as well as the cargo attitude angle;

[0116] Step S03: Based on the cargo geometry model and the occupancy information of the cargo already placed in the drop box area, construct a set of reachable spaces for the cargo to be loaded. In the set of reachable spaces, exclude all spatial areas that do not meet the path planning constraints to ensure that the generated path will not collide or violate the attitude constraints during the movement.

[0117] Step S04: Within the reachable space set, sequentially transfer the approach pose. and final landing position Connect them to form a continuous drop path for the boxes.

[0118] Based on the cargo geometry model and the occupancy information of the cargo already placed within the container dropping area, methods for constructing the reachable space set of the cargo to be loaded include:

[0119] The container landing area and its safe height range above it are discretized into three-dimensional grid cells in the container space coordinate system. Each grid cell represents the possible spatial location of the cargo's core.

[0120] For each mesh cell, based on the outer contour of the cargo geometry model, the volume occupied by the cargo at its corresponding position and orientation in that mesh cell is mapped to form a set of voxels at that position.

[0121] The intersection of the occupied space formed by the goods placed in the drop box area and the occupied voxel set of each grid cell is detected. If any grid cell overlaps with the occupied space of the goods placed, the grid cell is marked as an unreachable grid cell.

[0122] Further path planning constraints are imposed on reachable grid cells, including minimum safety clearance constraints, drop height constraints, and attitude change constraints;

[0123] The minimum safety clearance constraint is as follows: for each candidate grid cell, calculate the minimum distance between the outer contour of the cargo and the obstacle. If the minimum distance is less than the preset distance threshold, then exclude the grid cell.

[0124] The drop height constraint is: if the height of a grid cell is less than the drop height reference plus the safety height threshold, then the grid cell is excluded.

[0125] The attitude change constraint is: if the attitude angle of the cargo in the corresponding grid cell is... Exceeding the allowed range, i.e. If so, then exclude that grid cell;

[0126] in, Let be the rotation angle of the cargo about the X-axis. Let be the rotation angle of the cargo about the Y-axis. Let be the rotation angle of the cargo around the Z-axis. This represents the magnitude of the attitude angle vector of the cargo within that grid cell;

[0127] Used to measure the overall deflection of goods. This indicates the maximum allowable attitude deflection angle threshold for the cargo. If this threshold is exceeded, the cargo may tilt, flip, or become unstable.

[0128] All grid cells that satisfy the path planning constraints are combined to form the final reachable space set.

[0129] It should be noted that the method of sequentially connecting the transition approach pose and the final box landing pose to form a continuous box landing path includes: discretizing the pose state space in the reachable space set with a preset spatial resolution to obtain multiple candidate pose nodes, wherein each candidate pose node satisfies the minimum safety gap constraint, the box landing height constraint, and the attitude change constraint.

[0130] Secondly, based on the spatial adjacency relationship between candidate pose nodes, node connectivity is established. For any two adjacent candidate pose nodes, it is determined whether the geometric model of the cargo to be loaded is always within the reachable space set during the process of the cargo moving from the previous pose node to the next pose node, and whether it does not spatially interfere with the box or the cargo already placed. If the above conditions are met, it is considered that there is a feasible connection between the two candidate pose nodes.

[0131] Based on this, the candidate pose nodes are searched step by step from the initial pose to generate multiple candidate paths composed of adjacent feasible connections. During the search process, path branches with collision risks, pose abrupt changes, or high violations are continuously eliminated.

[0132] When a candidate path successfully connects to the target pose, the path is determined as a valid drop-box path; if there are multiple valid paths, the valid paths are evaluated by weighted summation based on path length and smoothness, and the path with the best evaluation result is selected as the final continuous drop-box path.

[0133] Example 2

[0134] Please see Figure 4 This invention provides a technical solution: an automated robot packing system based on multimodal perception feedback control, comprising the following steps:

[0135] The size acquisition module acquires the geometric dimensions of the goods to be loaded.

[0136] The container space establishment module establishes a container space coordinate system fixedly associated with the container body above the container area, collects surface topography data of goods placed in the container area in real time, constructs a three-dimensional height distribution map, and identifies local protruding areas based on the three-dimensional height distribution map;

[0137] The container dropping area planning module, based on local protruding areas and combined with the geometric dimensions of the goods to be loaded, plans the optimal container dropping position, which is denoted as the container dropping area;

[0138] The height reference acquisition module acquires a three-dimensional height distribution map of the box-dropping area and calculates the corresponding box-dropping height reference.

[0139] The point cloud acquisition and mapping module uses a 3D depth imaging unit arranged in the transport area to scan the goods to be packed frame by frame to obtain a discrete point cloud set, and then maps the point cloud set to the packing space coordinate system to obtain a discrete point cloud set of the goods.

[0140] The packing path generation module generates packing path movement instructions from the three-dimensional height distribution map of the packing area, the discrete point cloud set of the goods, and the packing height reference. Based on the packing path movement instructions, it continuously controls the automatic gripping mechanism 4 to achieve stable and non-crushing placement of goods within the target packing area.

[0141] Example 3

[0142] Please see Figures 5-6 This invention provides a technical solution: a robot automatic packing device based on multimodal perception feedback control, comprising a device and a robot automatic packing system based on multimodal perception feedback control. The device includes:

[0143] Device body 1;

[0144] Conveying mechanism 2, located inside device body 1, is used to convey goods to be packed.

[0145] Three-dimensional depth imaging unit 3 is set inside the device body 1, at least two of them, one of which is arranged above the conveying area to collect point cloud data of the goods to be packed, and the other is arranged above the container to collect point cloud data of the goods inside the container.

[0146] An automatic gripping mechanism 4 is installed inside the device body 1 and is used to grip goods to be packed.

[0147] Delivery mechanism 5 is used to drive 4 to move within the packing space to realize the delivery and packing of goods;

[0148] The device body 1 is equipped with a support platform 7, on which the cargo box 6 of the goods to be stacked is placed;

[0149] A robotic automatic packing system based on multimodal perception feedback control is connected to a three-dimensional depth imaging unit 3, an automatic gripping mechanism 4, and a delivery mechanism 5. It is used to control the coordinated work of the above-mentioned mechanisms to realize real-time perception, path planning, and packing of goods.

[0150] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A robot automated bin-packing method based on multi-modal sensory feedback control, characterized by, The application relates to a method for realizing stable and non-extrusion placement of goods in a target falling box area. acquiring the geometrical dimensions of the goods to be loaded , respectively indicate the length, width and height of the goods to be loaded The method comprises the following steps: establishing a falling box space coordinate system associated with a box body above a boxing area; collecting surface topography data of goods placed in the boxing area in real time; constructing a three-dimensional height distribution map; identifying a local convex area based on the three-dimensional height distribution map; planning an optimal falling box position based on the local convex area and the geometric size of the goods to be filled, and marking the position as a falling box area; obtaining a three-dimensional height distribution map of the falling box area; calculating a corresponding falling box height reference; scanning the goods to be boxed frame by frame through a three-dimensional depth imaging unit arranged in a conveying area; obtaining a discrete point cloud set; and mapping the point cloud set to the falling box space coordinate system to obtain a discrete point cloud set of the goods; generating a boxing path movement instruction based on the three-dimensional height distribution map of the falling box area, the discrete point cloud set of the goods and the falling box height reference; and continuously controlling the automatic grabbing mechanism based on the boxing path movement instruction to realize stable and non-extrusion placement of the goods in the target falling box area. The method for identifying the local convex area comprises the following steps: obtaining a three-dimensional height distribution map of the boxing area; calculating a three-dimensional height distribution map of the falling box area; and calculating a falling box height reference corresponding to the three-dimensional height distribution map of the falling box area. The method for planning the optimal falling box position comprises the following steps: calculating a stability evaluation function of each candidate area in the candidate placement area set in combination with the spatial distribution relationship of the local convex area; selecting a candidate area with the optimal stability evaluation function as the final falling box position; and marking the position as the falling box area. The method for generating the boxing path movement instruction comprises the following steps: determining a current spatial pose of the goods to be filled in the falling box space coordinate system based on the discrete point cloud set of the goods; constructing a goods geometric model representing the spatial occupation range of the goods to be filled in combination with the geometric size of the goods to be filled; determining a target falling box pose of the goods to be filled in the falling box space coordinate system based on the spatial occupation range of the falling box area and the falling box height reference, wherein the target falling box pose comprises a target plane position, a target height position and a goods attitude angle; establishing a path planning constraint condition between the current spatial pose and the target falling box pose; analyzing a motion reachable space of the goods to be filled based on the goods geometric model and the three-dimensional height distribution map of the falling box area to generate a continuous falling box path connected by a transition approaching pose and a final falling box pose in sequence under the premise of meeting the path planning constraint condition. The path planning constraint condition comprises a minimum safety clearance constraint of the placed goods, a falling box height constraint and an attitude change constraint.

2. The robotic automated bin-packing method based on multi-modal perception feedback control according to claim 1, wherein, The method for analyzing the motion reachable space of the goods to be filled to generate the continuous falling box path connected by the transition approaching pose and the final falling box pose in sequence comprises the following steps: constructing a reachable space set of the goods to be filled based on the goods geometric model and the occupation information of the placed goods in the falling box area; and excluding all space areas not meeting the path planning constraint condition in the reachable space set to ensure that the generated path does not collide or violate the attitude limitation in the motion process. The upper surface of the placed goods is scanned frame by frame by a three-dimensional depth imaging unit arranged above the box, a discrete point cloud set is obtained, and the point cloud set is uniformly mapped to the boxing space coordinate system to obtain a discrete point cloud set in the box opening , is a point in the discrete point cloud set in the box opening In the container loading space coordinate system plane The upper building rule grid unit , The first Row Column grid unit, and calculate the maximum height value in each grid unit, thereby forming a three-dimensional height distribution map reflecting the real-time stacking state inside the container , Indicates the horizontal coordinate and vertical coordinate of the container loading space coordinate system; Based on the three-dimensional height distribution map, a height change rate between adjacent grid cells is calculated and the region whose height value satisfies the preset condition at the same time is determined as a local convex region.

3. The robotic automated bin-packing method based on multi-modal perception feedback control according to claim 2, wherein, The preset condition comprises: ; , wherein is the average stacking height in the bin, and is a threshold parameter set based on historical stability data; is the first row column grid.

4. The robotic automated bin-packing method based on multi-modal perception feedback control of claim 1, wherein, ​ Projecting a cargo to be loaded into a container space coordinate system On a three-dimensional height distribution map, a candidate placement region set is constructed for a region satisfying a plane continuous area greater than × and a height variation rate less than a preset height variation rate threshold. ​ ​ ​ 5. The robotic automated bin-packing method based on multi-modal perception feedback control according to claim 1, wherein, ​ ​ ​ ​ ​ 6. The robotic automated bin-packing method based on multi-modal perception feedback control according to claim 5, wherein, ​ 7. The robotic automated bin-packing method based on multi-modal sensory feedback control according to claim 6, wherein, ​ Step S01, in the binning space coordinate system, based on the three-dimensional height distribution map of the falling bin area, the transition approaching pose of the to-be-filled goods is determined ; wherein, respectively, the horizontal, vertical and vertical coordinates of the cargo to be loaded and the cargo attitude angle; Step S02, determining the final box landing pose according to the cargo geometric model and the three-dimensional height distribution map of the box landing area , ; wherein, respectively the horizontal coordinate, the vertical coordinate and the vertical coordinate of the final container drop position of the cargo, and the cargo attitude angle; ​ Step S04, in the reachable space set, sequentially connecting the transition approaching poses and the final drop-off pose to form a continuous drop-off path.

8. The robotic automated bin-packing method based on multi-modal sensory feedback control according to claim 7, wherein, ​ discretize the falling-in-box area and its above safety height interval into three-dimensional grid cells in the packing space coordinate system, each grid cell representing a spatial position where the cargo centroid can be located; for each grid cell, map the occupied volume of the cargo in the position and attitude of the grid cell according to the outer contour of the cargo geometric model, forming an occupied voxel set of the position; carry out intersection detection between the occupied space formed by the placed cargo in the falling-in-box area and the occupied voxel set of each grid cell, if any grid cell has overlap with the occupied space of the placed cargo, mark the grid cell as an unreachable grid cell; further impose path planning constraints on the reachable grid cells, including minimum safety gap constraint, falling-in-box height constraint and attitude change constraint; wherein the minimum safety gap constraint is that for each candidate grid cell, calculate the minimum distance between the cargo outer contour and the obstacle, if the minimum distance is less than the preset distance threshold, exclude the grid cell; the falling-in-box height constraint is that if the grid cell height is less than the falling-in-box height reference plus the safety height threshold, exclude the grid cell; The posture change constraint is: if the posture angle of the goods corresponding to the grid unit is out of the allowed range, that is, , the grid unit is excluded, wherein, is the rotation angle of the goods around the X axis, represents the rotation angle of the goods around the Y axis, represents the rotation angle of the goods around the Z axis, represents the modulus of the posture angle vector of the goods in the grid unit, represents the maximum posture deflection angle threshold allowed by the goods; form a final reachable space set from the set of all grid cells satisfying the path planning constraints.

9. A multi-modal perception feedback control based robotic case packing system implementing a multi-modal perception feedback control based robotic case packing method as claimed in any one of claims 1 to 8, characterized in that, comprise: a size acquisition module that acquires the geometric size of the to-be-packed cargo; a packing space establishment module that establishes a packing space coordinate system fixedly associated with the box above the packing area, acquires surface topography data of the placed cargo in the packing area in real time, constructs a three-dimensional height distribution map, and identifies a local protruding area based on the three-dimensional height distribution map; a falling-in-box area planning module that plans an optimal falling-in-box position based on the local protruding area and in combination with the geometric size of the to-be-packed cargo, and records the position as the falling-in-box area; a height reference acquisition module that acquires the three-dimensional height distribution map of the falling-in-box area and calculates a corresponding falling-in-box height reference; a point cloud acquisition and mapping module that acquires a discrete point cloud set by sequentially scanning the to-be-packed cargo through a three-dimensional depth imaging unit arranged in the conveying area, and maps the point cloud set to the packing space coordinate system to obtain a cargo discrete point cloud set; a packing path generation module that generates a packing path movement instruction based on the three-dimensional height distribution map of the falling-in-box area, the cargo discrete point cloud set and the falling-in-box height reference, and continuously controls the automatic grabbing mechanism based on the packing path movement instruction to realize stable and non-squeezing placement of the cargo in the target falling-in-box area.

10. A robot automated binning apparatus based on multi-modal sensory feedback control, characterized in that, comprise a device and a robot automatic packing system based on multi-modal perception feedback control, the device comprising: a device body (1); a conveying mechanism (2) arranged in the device body (1) and used for conveying the to-be-packed cargo; a three-dimensional depth imaging unit (3) arranged in the device body (1) and comprising at least two units, one of which is arranged above the conveying area and used for acquiring point cloud data of the to-be-packed cargo, and the other of which is arranged above the box and used for acquiring point cloud data of the cargo in the box; an automatic grabbing mechanism (4) arranged in the device body (1) and used for grabbing the to-be-packed cargo; a delivery mechanism (5) used for driving the automatic grabbing mechanism (4) to move in the packing space to realize delivery and packing of the cargo; A bearing table (7) is arranged in the device body (1), and a goods box (6) to be stacked is placed on the bearing table (7); The robot automatic boxing system based on the multi-modal sensing feedback control is connected with a three-dimensional depth imaging unit (3), an automatic grabbing mechanism (4) and a delivery mechanism (5), is used for controlling the cooperative work of each mechanism, and realizes real-time sensing, path planning and boxing of goods.

Images