Target grabbing collision-free planning method and system

Abstract

The invention provides a target capturing collision-free planning method and system, and relates to a data processing technology, the method comprises the steps: carrying out the segmentation processing of original point cloud data of a target scene, generating a plurality of target point cloud clusters, generating a target envelope based on the target point cloud clusters, and building an environment geometric model according to the target envelope; according to a grabbing task strategy, target goods are determined from the environmental geometric model, and candidate grabbing poses of the target goods are generated; converting the environmental geometric model into voxel grids, inputting the voxel grids and the candidate grabbing poses into a pre-trained prediction neural network, and outputting a path thermodynamic diagram; path searching is carried out in the environmental geometric model based on a path thermodynamic diagram, a target motion path from the initial state of the robot to the candidate grabbing poses is generated, blindness of a traditional algorithm can be overcome, the sampling direction is guided according to environmental information, and rapid planning is achieved on the premise that safety is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a target collision-free grasping planning method and system. BACKGROUND

[0002] Planning of target grasping is mainly applied to intelligent warehouse logistics automation, and its main task is to plan a motion trajectory for a robot arm from a starting state to a target grasping pose in a complex three-dimensional space full of obstacles, so as to reduce the collision between the goods and the robot arm. In the logistics unloading scene, the working space usually presents a narrow and closed structure, such as the inside of a container or the cargo compartment of a truck, and the goods are densely and disorderly stacked, and the robot arm needs to move in a limited space and grasp the target goods, which poses a high challenge to the grasping motion path and adaptability.

[0003] In the prior art, the path planning mainly adopts an algorithm based on random sampling. However, such a method has significant defects when facing a narrow channel scene such as container unloading, because the random sampling strategy adopted by the algorithm lacks understanding and guidance of the geometric features of the environment, resulting in that most of the sampling points fall in the obstacle region or invalid space, so that the planning time is long, and the efficiency of logistics operation is restricted.

[0004] Therefore, how to overcome the blindness of the traditional algorithm, guide the sampling direction according to the environmental information, and realize fast planning under the premise of safety has become a problem to be solved. SUMMARY

[0005] The present application provides a target collision-free grasping planning method and system, which can overcome the blindness of the traditional algorithm, guide the sampling direction according to the environmental information, and realize fast planning under the premise of safety.

[0006] In a first aspect, the present application provides a target collision-free grasping planning method, comprising: segmenting the original point cloud data of a target scene to generate a plurality of target point cloud clusters, generating a target envelope based on the target point cloud clusters, and constructing an environmental geometric model according to the target envelope; determining a target goods from the environmental geometric model according to a grasping task strategy, and generating a candidate grasping pose of the target goods; converting the environmental geometric model into a voxel grid, and inputting the voxel grid and the candidate grasping pose into a pre-trained prediction neural network to output a path heat map; performing path search in the environmental geometric model based on the path heat map to generate a target motion path from a starting state of a robot to the candidate grasping pose.

[0007] Optionally, in a possible implementation manner of the first aspect, the segmentation processing on the original point cloud data of the target scene to generate a plurality of target point cloud clusters comprises: plane fitting is performed on the original point cloud data, background plane points in the original point cloud data are removed, and remaining point cloud data is obtained; the remaining point cloud data is clustered and segmented by using a Euclidean clustering algorithm to obtain the plurality of target point cloud clusters.

[0008] Optionally, in a possible implementation manner of the first aspect, the target envelope is generated based on the target point cloud cluster, and the environment geometric model is constructed according to the target envelope, which comprises: principal component analysis is performed on each target point cloud cluster to determine a main direction of the point cloud cluster; a directional bounding box tightly surrounding the target point cloud cluster is generated as the target envelope based on the main direction; the pose of the target envelope is converted from a camera coordinate system to a first coordinate system of the robot based on calibration parameters of the robot; the converted target envelope is fused with a job boundary model in the first coordinate system to generate the environment geometric model.

[0009] Optionally, in a possible implementation manner of the first aspect, the target goods are determined from the environment geometric model according to a grasping task strategy, and a candidate grasping pose of the target goods is generated, which comprises: geometric center coordinates of each target envelope in the first coordinate system in the environment geometric model are obtained; Z-axis values of the geometric center coordinates are compared, and a target envelope corresponding to goods with the largest Z-axis value is selected as the target goods; a surface normal of a target point cloud cluster corresponding to the target goods is calculated; an initial grasping pose set is generated by sampling in a deflection space defined by the preset angle range with the surface normal as a reference direction; collision detection is performed on each pose in the initial grasping pose set by using the environment geometric model; poses in the initial grasping pose set that interfere with the environment geometric model are eliminated to obtain the candidate grasping pose.

[0010] Optionally, in a possible implementation manner of the first aspect, the environment geometric model is converted into a voxel grid, and the voxel grid and the candidate grasping pose are input into a pre-trained prediction neural network to output a path heat map, which comprises: discretize a three-dimensional space where the environment geometric model is located according to a preset resolution to obtain a three-dimensional grid space, and map spatial occupancy information of the environment geometric model into the three-dimensional grid space to generate the voxel grid; Encode spatial coordinates of the candidate grasping pose into pose feature data consistent with dimensions of the voxel grid based on spatial dimensions of the voxel grid; Splice the voxel grid and the pose feature data in a channel dimension to form multi-channel input data, input the multi-channel input data into the pre-trained prediction neural network for processing to obtain a three-dimensional probability distribution field, and take the three-dimensional probability distribution field as the path heat map.

[0011] Optionally, in a possible implementation manner of the first aspect, the path searching in the environment geometric model based on the path heat map to generate a target motion path from a robot starting state to the candidate grasping pose comprises: Map a voxel numerical value distribution in the path heat map to a non-uniform sampling probability field; Determine a non-occupied region in a three-dimensional space range where the environment geometric model is located; Sample in the non-occupied region according to the non-uniform sampling probability field to obtain a sampling point; Take the robot starting state as a root node, use the sampling point to construct a search tree extending to the candidate grasping pose, and generate the target motion path when the search tree is connected to the candidate grasping pose.

[0012] Optionally, in a possible implementation manner of the first aspect, the method further comprises: Control the robot to grasp the target goods based on the target motion path and place the target goods at a unloading area.

[0013] Optionally, in a possible implementation manner of the first aspect, before the step of placing the target goods at the unloading area, the method comprises: Retrieve a standard goods model corresponding to the target goods and a feature identifier; Identify an actual goods model of the target goods, and align the standard goods model with the actual goods model based on the feature identifier; Obtain an intersection of each standard external surface of the standard goods model and a corresponding actual external surface in the actual goods model to obtain a placement area of each external surface corresponding to the target goods; Determine a placement surface of the target goods based on the placement area.

[0014] Optionally, in a possible implementation manner of the first aspect, the placing surface of the target cargo is determined based on the placing area, and the determining comprises: sequentially sorting the corresponding actual outer surfaces in the actual cargo model in descending order based on the placing area to obtain an outer surface sequence; selecting a first actual outer surface in the outer surface sequence as the placing surface of the target cargo.

[0015] In a second aspect of the present application, a collision-free planning system for a target is provided, comprising: a segmentation module configured to perform segmentation processing on original point cloud data of a target scene to generate a plurality of target point cloud clusters, generate a target envelope based on the target point cloud clusters, and construct an environmental geometric model according to the target envelope; a determination module configured to determine a target cargo from the environmental geometric model according to a grasping task strategy, and generate a candidate grasping pose of the target cargo; a conversion module configured to convert the environmental geometric model into a voxel grid, and input the voxel grid and the candidate grasping pose into a pre-trained prediction neural network to output a path heat map; a generation module configured to perform path search in the environmental geometric model based on the path heat map to generate a target motion path from a starting state of a robot to the candidate grasping pose.

[0016] In a third aspect of the present application, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect and various possible aspects related to the method.

[0017] The present application has the following beneficial effects: 1. The present application realizes fast collision-free path planning for a grasping target in a complex stacked scene by fusing point cloud segmentation and environmental modeling. The original point cloud is segmented and a target envelope is generated, and then an environmental geometric model composed of regular geometric bodies is constructed. This not only accurately restores the complex spatial structure of cargo stacking in logistics unloading and other scenes, but also effectively reduces the complexity of subsequent calculations. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of the collision-free planning method for a grasping target provided by the present application; Figure 2 A schematic diagram of the system architecture provided by the present application; Figure 3 A structural schematic diagram of the collision-free planning system for a grasping target provided by the present application. DETAILED DESCRIPTION

[0019] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0020] The terms "first", "second", "third", "fourth" and the like in the description, claims, and drawings of the present application, and the foregoing, if any, are used to distinguish similar objects, and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the present application described herein can be carried out in other than the order shown or described herein.

[0021] It should be understood that, in various embodiments of the present application, the magnitude of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0022] It should be understood that, in the present application, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0023] It should be understood that, in the present application, "a plurality of" means two or more. "And / or" is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it. "Including A, B and C", "including A, B, C" means that A, B and C are all included, "including A, B or C" means that one of A, B and C is included, and "including A, B and / or C" means that any one or any two or three of A, B and C is included.

[0024] It should be understood that, in the present application, "B corresponding to A", "B corresponding to A", "A corresponding to B" or "B corresponding to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean that B is determined only according to A, but also can be determined according to A and / or other information. The matching of A and B means that the similarity of A and B is greater than or equal to a preset threshold.

[0025] Depending on the context, "if' as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting."

[0026] The technical solutions of the present application are described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in some examples.

[0027] The present application provides a collision-free planning method for grabbing targets, such as Figure 1 As shown, comprising steps S1-S4: S1, the original point cloud data of the target scene is segmented and processed to generate a plurality of target point cloud clusters, based on the target point cloud cluster, a target envelope is generated, and an environmental geometric model is constructed according to the target envelope.

[0028] It should be noted that the original point cloud data collected by the 3D camera is very large and disordered, usually containing tens of thousands or even millions of discrete coordinate points. If these original point cloud data is directly used for collision detection with the robot arm model, the calculation amount is large, which cannot meet the requirements of real-time operation. Moreover, the point cloud contains a lot of noise, which will mislead the planning.

[0029] Among them, the target scene refers to the working area of the robot arm for grabbing work, such as the inside of a container; the original point cloud data refers to the set of three-dimensional coordinate points collected by the vision sensor; the target point cloud cluster refers to the point cloud set after segmentation, which belongs to the same independent goods, such as a carton; the target envelope refers to a simplified geometric shape that can wrap the target point cloud cluster, usually refers to a directional cuboid; the environmental geometric model refers to a three-dimensional space model composed of all recognized goods envelopes and scene boundaries, such as the walls of the car.

[0030] It can be understood that the processing device first controls the high-resolution 3D camera installed above the container door or the mechanical arm to perform a global scan on the target scene, obtaining raw point cloud data containing tens of thousands of coordinate points. Subsequently, the device preprocesses these data: removes background points such as the ground and the carriage wall that are outside the scope of the operation using a pass-through filter, and removes outlier points generated by sensor noise using statistical filtering. For the remaining cargo point cloud, the device uses the Euclidean clustering algorithm to set a reasonable distance threshold and merge points that are close in space into a class, thereby successfully segmenting the chaotic overall point cloud into dozens of independent target point cloud clusters, i.e., identifying independent cartons. Then, the device performs geometric analysis on each target point cloud cluster, specifically by calculating three eigenvectors of the point cloud distribution using principal component analysis, and fitting a minimum-volume oriented bounding box as the target envelope based on this direction. Finally, the device integrates all generated OBB envelope bodies and the geometric models of pre-modeled fixed obstacles such as the carriage wall and the base into the same coordinate system to construct an environment geometric model composed of regular geometric bodies, which will serve as the basis map for subsequent neural network input and collision detection.

[0031] Referring to Figure 2 For the overall architecture of our system, subsequent processing of point cloud data is based on this system architecture, which includes a hardware layer, a data layer, a core algorithm layer, and an application control layer from bottom to top. The hardware layer is the foundation support, and the hardware layer is the physical carrier of the system, mainly responsible for data acquisition and physical execution of instructions. The camera / radar as the acquisition device of the system is usually configured as a high-resolution 3D depth camera or a laser radar, responsible for collecting the original 3D point cloud data of the working scene, such as the inside of a container. The industrial computer / controller as the processor of the system carries all software algorithm running and logic judgment, responsible for processing sensor data and issuing control instructions, the robot arm and end effector as the execution device are responsible for receiving trajectory instructions from the upper layer and driving the motor to perform specific grabbing, carrying, and placing actions.

[0032] The data layer is the information hub, responsible for storing dynamic environment information and static knowledge models required for system operation, providing data support for the algorithm layer. Environment model: dynamically stores the scene representation generated by the perception module, including raw point cloud data, abstracted and simplified OBB envelope models, and voxel maps for path planning. Path existence prediction neural network model: stores a pre-off-line trained deep learning model that is used for real-time inference and prediction of the probability distribution of collision-free paths, i.e., a heat map.

[0033] The core algorithm layer is the decision core, which contains a series of closely coordinated algorithm modules. The perception and abstraction module is responsible for processing the original point cloud from the hardware layer, filtering, segmentation, and converting the unstructured point cloud into a simplified geometric envelope OBB, providing input for the planning algorithm. The grasping planning module calculates the feasible candidate grasping pose of the robot end for the determined target goods. The collision-free path planning module integrates a neural network guided sampling planning algorithm. It calls the neural network model of the data layer to generate a heat map, and quickly generates a safe path from the starting point to the grasping point under this guidance, completely solving the problem of blind search of traditional algorithms. The trajectory optimization and control module smoothes and time regularizes the rough path planned, and converts it into a time-optimal trajectory instruction executable by the robot. The application control layer is located at the topmost layer, directly facing the user, and is responsible for the macroscopic scheduling of the task. The task scheduling and human-computer interaction module provides a visual operation interface, allows the user to start or stop the system, specifies a specific grasping target strategy, such as prioritizing the upper goods, and monitors the running state and health status of the system in real time In some embodiments, the (original point cloud data of the target scene in step S1 is segmented to generate a plurality of target point cloud clusters) includes: The original point cloud data is fitted with a plane, and the background plane point cloud in the original point cloud data is removed to obtain residual point cloud data.

[0034] The residual point cloud data is clustered and segmented using a Euclidean clustering algorithm to obtain the plurality of target point cloud clusters.

[0035] Among them, the plane fitting refers to finding a set of points that meet the mathematical model of the plane in the point cloud through a mathematical algorithm, such as the RANSAC random sample consensus algorithm; the background plane point cloud refers to the point cloud data identified as a fixed environmental structure, such as the ground, walls, or ceilings, etc.; the Euclidean clustering algorithm refers to a point cloud segmentation algorithm based on Euclidean distance neighbor search, which is not described here as prior art; the target point cloud cluster refers to the point cloud set representing a single independent goods after clustering, such as a carton.

[0036] It can be understood that the processing device first performs a plane model segmentation on the pre-processed original point cloud. Specifically, the device calls a point cloud processing library, such as the RANSAC random sample consensus algorithm in the PCL library, and sets a distance threshold. The algorithm randomly selects several points to construct a plane model, calculates the distance of other points to the plane, and iterates repeatedly until the plane with the largest number of points is found. The points conforming to the plane model are marked as background plane points and removed from the data set, and the process is repeated until all walls and floors are removed, and the remaining point cloud data containing only goods is obtained. Then, the device uses the Euclidean clustering extraction algorithm on the remaining suspended point cloud. The device constructs a KD-Tree to speed up the search and sets a clustering tolerance as a distance threshold for determining whether two points belong to the same object. The algorithm traverses each point, finds all points in its neighborhood, and merges them into the same cluster until all connected points are processed. Finally, the device filters out noise with too few points or clusters that may be connected, and outputs a series of independent target point cloud clusters, each corresponding to an actual box in the physical space. This is prior art and will not be described in detail here.

[0037] In some embodiments, the step S1 (generating a target envelope based on the target point cloud cluster, and constructing an environment geometric model according to the target envelope) comprises: Performing principal component analysis on each target point cloud cluster to determine the principal direction of the point cloud cluster.

[0038] Generating a directional bounding box tightly surrounding the target point cloud cluster as the target envelope based on the principal direction.

[0039] Converting the pose of the target envelope from the camera coordinate system to the first coordinate system of the robot based on the calibration parameters of the robot.

[0040] Retrieving a job boundary model, and performing data fusion on the converted target envelope and the job boundary model in the first coordinate system to generate an environment geometric model.

[0041] It should be noted that although the point cloud after clustering and segmentation is divided into independent points, they are still composed of hundreds or thousands of scattered points, and the shape is irregular. If these scattered points are directly used to calculate the collision, the calculation amount is still very large. In addition, goods are usually regular geometric bodies, but they may be placed obliquely in the carriage. If a box parallel to the coordinate axis is simply used to frame them, a large empty area will be generated, for example, a large box is placed obliquely, and the four corners of the large box are actually empty, which will cause the mechanical arm to pass by, but it is misjudged as a collision.

[0042] The principal component analysis refers to a statistical analysis method for converting a set of variables that may have correlations into a set of linearly uncorrelated variables through orthogonal transformation. For example, the principal axis direction of an object is determined by calculating the eigenvectors of the point cloud covariance matrix. The principal direction refers to the direction in which the point cloud data is most divergent and has the largest variance. For example, the length direction or width direction of the cargo. The oriented bounding box refers to a minimum volume cuboid that can rotate with the object and tightly wrap the object. For example, a cuboid geometry model corresponding to a paper box placed at an angle. The calibration parameters of the robot refer to a mathematical transformation matrix that describes the relative position and attitude relationship between the camera and the robot base. For example, a hand-eye calibration matrix. The first coordinate system refers to the reference coordinate system for motion planning and control of the robot. For example, the robot base coordinate system. The work boundary model refers to a pre-constructed geometric model that describes the boundary of the fixed work environment and static obstacles. For example, the three side walls, ceiling of the container, and the base model of the robot itself. The environment geometric model refers to a full-element three-dimensional space model that integrates dynamic cargo and static environment information. For example, a voxel map or geometric set that contains all cargo OBBs and compartment wall models.

[0043] It can be understood that the processing device traverses each segmented target point cloud cluster. For each cluster, the device calculates the coordinate covariance matrix of all points thereof, and performs eigenvalue decomposition on the matrix. The three mutually orthogonal eigenvectors solved represent the three principal directions of the point cloud cluster in spatial distribution, which usually correspond to the length, width, and height directions of the cargo. Then, the device projects all points in the point cloud cluster onto the three principal directions to find the maximum and minimum values in each direction, thereby determining the center point coordinates and length, width, and height dimensions of the bounding box. Based on these parameters, the device generates an oriented bounding box (OBB) that can tightly fit the actual pose of the cargo, and defines the OBB as a target envelope. Compared with the axis-aligned bounding box, the OBB can more truly reflect the spatial occupation of the cargo. The processing device retrieves a pre-stored hand-eye calibration matrix, and performs rigid transformation operation on the pose matrix of each target envelope. For example, by matrix multiplication, the center coordinates and rotation angle are converted from the camera coordinate system to the robot base coordinate system in the first coordinate system. Subsequently, the device retrieves a pre-set work boundary model from the database. The model is usually composed of simple geometric bodies and describes the compartment walls, ground, and impassable boundaries. The device performs data fusion of all dynamic cargo OBBs converted in the first coordinate system and the static work boundary model, i.e., adds them into the same object list or spatial indexing structure. The fused collection is constructed as the final environment geometric model.

[0044] S2, according to the grasping task strategy, determining a target cargo from the environment geometric model, and generating a candidate grasping pose of the target cargo.

[0045] In some embodiments, the step S2 (determining a target cargo from the environment geometry model according to a grasping task strategy, and generating a candidate grasping pose of the target cargo) comprises: obtaining geometric center coordinates of each target envelope in the environment geometry model in the first coordinate system.

[0046] comparing the Z-axis values of each geometric center coordinate; selecting the cargo corresponding to the target envelope with the largest Z-axis value as the target cargo.

[0047] calculating the surface normal of the target point cloud cluster corresponding to the target cargo.

[0048] sampling in the deflection space defined by the preset angle range with the surface normal as the reference direction to generate an initial grasping pose set.

[0049] performing collision detection on each pose in the initial grasping pose set using the environment geometry model.

[0050] eliminating the poses in the initial grasping pose set that interfere with the environment geometry model to obtain a candidate grasping pose.

[0051] It should be noted that in the unloading scenario, the cargos are usually stacked and placed. If the robot arm first grasps the bottom cargo, the cargos above will collapse, causing danger. Therefore, the safest and most logical strategy is to grasp from top to bottom, and after determining which box to grasp, the ideal grasping angle for a suction cup type robot arm is perpendicular to the box surface, i.e., along the normal direction. However, in actual operation, a perfect vertical angle may not be reached due to joint limitations of the robot arm, or it may be blocked by the surrounding boxes. Therefore, only one angle cannot be recognized, and several alternative solutions should be prepared.

[0052] wherein the first coordinate system refers to the robot base coordinate system; the geometric center coordinates refer to the spatial positions of the diagonal intersection points of the target envelope; the target cargo refers to the cargo selected as the priority operation object in the current grasping cycle; the surface normal refers to the vector arrow perpendicular to the top surface of the target cargo and pointing outward; the preset angle range refers to the maximum tolerance for the deviation of the suction cup or gripper from the vertical direction, such as 15 degrees or 30 degrees, which can be artificially pre-set; the initial grasping pose set refers to a list of feasible grasping positions and angles generated without collision detection; collision detection refers to the process of calculating whether two geometric bodies overlap in volume in a computer virtual environment; interference refers to the contact or overlap between the end effector of the robot arm and non-target objects, such as other boxes or walls, at the grasping pose It can be understood that the constructed environment geometry model is traversed to read the center point coordinates of each target envelope body in the robot base coordinate system. Assuming that there are three boxes A, B and C on the site, and the center point height Z values of the three boxes are 0.5 meters, 1.2 meters and 2.0 meters respectively. The processing device compares the Z values and determines that 2.0 meters is the maximum value. Therefore, the processing device locks the box C represented by the envelope body with a height of 2.0 meters, marks it as the current target cargo, and prepares to plan the grabbing of the target cargo, obtains the original point cloud cluster corresponding to the target cargo, calculates the surface normal of the top surface thereof by using principal component analysis or plane fitting algorithm, which is usually a vector perpendicular upward. Then, the device sets a deflection space, for example, a conical space with a top angle of 30 degrees, with the normal as the central axis. The device performs discrete sampling in this space to generate dozens of different approaching vectors, i.e., the direction of the mechanical arm end towards the box. Combined with the position of the center point of the box, the device generates a series of initial grabbing poses, such as vertically downward, 5 degrees left tilt, 10 degrees right tilt, and stores them in a list. Then, the environment geometry model constructed by S1 is called, which contains the OBBs of all obstacles and the walls. The device traverses each initial grabbing pose generated just now, and virtually places the end effector model of the mechanical arm, such as a suction cup, at these poses. The device uses a geometric intersection test algorithm to quickly check whether the end effector model overlaps with any non-target obstacle in the environment. If it is detected that a certain pose will cause the end effector to collide with the surrounding cargo or the vehicle wall, the device will exclude that pose from the set. Only those poses that pass the detection and are in the free space will be retained to form the final candidate grabbing pose set for use by the subsequent path planning module.

[0053] S3, converting the environment geometry model into a voxel grid, and inputting the voxel grid and the candidate grabbing poses into a pre-trained prediction neural network to output a path heat map.

[0054] In some embodiments, the step S3 (converting the environment geometry model into a voxel grid, and inputting the voxel grid and the candidate grabbing poses into a pre-trained prediction neural network to output a path heat map) comprises: Discretizing the three-dimensional space where the environment geometry model is located according to a preset resolution to obtain a three-dimensional grid space, and mapping the spatial occupancy information of the environment geometry model into the three-dimensional grid space to generate the voxel grid.

[0055] Encoding the spatial coordinates of the candidate grabbing poses into pose feature data consistent with the dimensions of the voxel grid based on the spatial dimensions of the voxel grid.

[0056] The voxel grid and the pose feature data are spliced in the channel dimension to form multi-channel input data, and the multi-channel input data is input into the pre-trained prediction neural network for processing to obtain a three-dimensional probability distribution field, and the three-dimensional probability distribution field is taken as the path heat map.

[0057] wherein, the preset resolution refers to the actual physical size represented by each small block in the voxel grid, for example, 5 cm x 5 cm x 5 cm; the three-dimensional grid space refers to the matrix container divided by the working area; the voxel grid refers to the binary three-dimensional array finally generated carrying the environmental obstacle distribution information; the spatial dimension refers to the length, width and height size of the voxel grid; the pose feature data refers to the matrix generated by mapping the target point coordinates into the grid, which is exactly the same size as the obstacle grid; the channel dimension refers to the dimension in the deep learning data tensor for distinguishing different feature types, for example, channel 1 is obstacle and channel 2 is target; the multi-channel input data refers to the composite data tensor fused with environmental and target information; the pre-trained prediction neural network refers to the deep learning model trained in advance using a large amount of path planning data, for example, 3D U-Net, etc., which is not described here; the path heat map, i.e., the three-dimensional probability distribution field, refers to the three-dimensional grid output by the network, which represents the probability value of each point in the space as a collision-free path.

[0058] It can be understood that the processing device first sets a preset resolution, for example, each voxel represents 2 centimeters in reality. The device constructs an empty three-dimensional grid space according to the working space range of the robot. Then, the device traverses each OBB envelope body in the environmental geometric model to calculate which positions they occupy in the space. The grid cells that overlap with the OBB are set to 1 by the device, representing obstacle occupation; the remaining unoccupied cells remain 0, representing free space. The three-dimensional matrix finally generated full of 0 and 1 is the voxel grid, which obtains the coordinates of the candidate grasping poses generated by S2. The device creates a full-zero grid consistent with the dimension of the voxel grid. Then, the device converts the physical coordinates into grid indexes and fills the values, for example, 1, or generates a Gaussian heat distribution centered at the point, at the corresponding positions of the new grid. The three-dimensional grid carrying the target position information is the pose feature data. The voxel grid and the pose feature data are stacked and spliced in the channel dimension to form a multi-channel input data. Then, the device inputs the data into the pre-trained prediction neural network loaded in the memory, for example, a model based on the V-Net architecture. The network performs forward inference calculation and outputs a three-dimensional probability distribution field with the same size as the input space. In this field, the value of each voxel represents the possibility of the position belonging to the collision-free path from the starting point to the ending point, between 0 and 1. The device defines the output result as the path heat map for subsequent sampling algorithms.

[0059] S4, performing path search in the environment geometry model based on the path heat map, and generating a target motion path from a robot initial state to the candidate grasping pose.

[0060] In some embodiments, the path search in the environment geometry model based on the path heat map in step S4 includes: mapping a numerical value distribution of voxels in the path heat map to a non-uniform sampling probability field.

[0061] determining a non-occupied region in a three-dimensional space range of the environment geometry model.

[0062] sampling in the non-occupied region according to the non-uniform sampling probability field to obtain a sampling point.

[0063] taking the robot initial state as a root node, constructing a search tree extending to the candidate grasping pose by using the sampling point, and generating the target motion path when the search tree is connected to the candidate grasping pose.

[0064] It should be noted that the traditional path planning algorithm, such as RRT, can only grab everywhere in the room, that is, uniform random sampling, which is low in efficiency and wastes most of the time in dead ends or areas far from the target.

[0065] The non-occupied region refers to a free space in the environment geometry model that is not occupied by any object.

[0066] It can be understood that first, the path heat map is read, and the numerical value of each voxel in the heat map, the probability value from 0 to 1, is normalized to convert into a sampling probability. The higher the numerical value of the voxel, the greater the weight of the corresponding space region in subsequent sampling, thereby constructing a non-uniform sampling probability field. Then, in combination with the constructed environment geometry model, it is identified which coordinates in the space are inside the obstacle, the occupied region, and which are in the air, the non-occupied region. When generating the sampling point, the device executes the sampling strategy: generating a random number. If the random number is less than a predetermined threshold, for example, 0.9, which can be pre-set, the device generates coordinates in the high-heat region according to the probability field; if the random number is greater than the threshold, full-image random sampling is performed to prevent missing the path. Regardless of which way, the generated coordinates must be subjected to validity check to ensure that they are located in the non-occupied region, that is, they do not overlap with any OBB, and finally an effective sampling point is obtained.

[0067] On the basis of the above-mentioned embodiments, the method further includes: controlling the robot to grasp the target goods based on the target motion path and place the target goods at the unloading area.

[0068] Before the step of placing the target cargo at the unloading area, comprising: Accessing a standard cargo model corresponding to the target cargo, and a feature identifier.

[0069] Identifying an actual cargo model of the target cargo, and aligning the standard cargo model with the actual cargo model based on the feature identifier.

[0070] Obtaining an intersection of each standard external surface of the standard cargo model and a corresponding actual external surface of the actual cargo model, to obtain a placement area of each external surface corresponding to the target cargo.

[0071] Determining a placement surface of the target cargo based on the placement area.

[0072] It should be noted that during the logistics transportation process, the cargo, especially the carton, is often squeezed and deformed. If the bottom of the box is pressed and deformed, it may not be stable when placed directly on the conveyor belt, or even overturned. In order to ensure the stability after placement, we need to choose the flattest surface as the bottom surface.

[0073] Wherein, the unloading area refers to the place outside the container for temporarily storing or circulating the cargo, such as the entrance of the conveyor belt or the AGV pallet; the standard cargo model refers to the pre-stored ideal cargo 3D model with perfect shape, such as a standard cube; the feature identifier refers to the identifier set on the cargo, such as the logo, barcode or fragile icon printed on the side of the box; the actual cargo model refers to the point cloud model of the cargo with squeezing and deformation obtained by real-time scanning and grabbing through the sensor, and the two models can be superimposed through the identifiers at multiple positions on the carton.

[0074] It can be understood that the standard cargo model and the actual cargo model are aligned in the virtual space, so that their identifiers are superimposed, and the intersection of the standard external surface and the corresponding actual external surface of the actual cargo model is the placement area.

[0075] In some embodiments, in the step of determining the placement surface of the target cargo based on the placement area, comprising: Based on the placement area, the corresponding actual external surface of the actual cargo model is sorted in descending order to obtain an external surface sequence.

[0076] The first actual external surface in the external surface sequence is selected as the placement surface of the target cargo.

[0077] Specifically, the device sorts the placement area values. The device selects the surface with the largest placement area, which means that the surface is the flattest and most intact as the placement surface.

[0078] Referring toFigure 3 is a structural schematic diagram of a target collision-free planning system provided by an embodiment of the present application, and the target collision-free planning system comprises: The segmentation module is configured to perform segmentation processing on original point cloud data of a target scene to generate a plurality of target point cloud clusters, generate a target envelope body based on the target point cloud clusters, and construct an environment geometric model according to the target envelope body. The determination module is configured to determine a target cargo from the environment geometric model according to a target task strategy and generate a candidate target grabbing pose of the target cargo. The conversion module is configured to convert the environment geometric model into a voxel grid and input the voxel grid and the candidate target grabbing pose into a pre-trained prediction neural network to output a path heat map. The generation module is configured to perform path searching in the environment geometric model based on the path heat map to generate a target motion path from a starting state of a robot to the candidate target grabbing pose.

[0079] The present application also provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method provided by the various embodiments.

[0080] The readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transfer of computer programs from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special purpose computer. For example, the readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). In addition, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in the communication device. The readable storage medium can be a read only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0081] The present application also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to enable the device to implement the method provided by the various embodiments.

[0082] In the embodiments of the above apparatus, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or can also be any conventional processor, etc. The steps of the method disclosed in the present application can be directly embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0083] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A collision-free planning method for grasping a target, characterized in that, The method comprises the following steps: segmenting original point cloud data of a target scene to generate a plurality of target point cloud clusters, generating a target envelope based on the target point cloud clusters, and constructing an environment geometric model according to the target envelope; determining a target cargo from the environment geometric model according to a grabbing task strategy, and generating a candidate grabbing pose of the target cargo; converting the environment geometric model into a voxel grid, inputting the voxel grid and the candidate grabbing pose into a pre-trained prediction neural network, and outputting a path heat map; performing path searching in the environment geometric model based on the path heat map to generate a target motion path from a starting state of a robot to the candidate grabbing pose.

2. The method of claim 1, wherein the segmenting of the original point cloud data of the target scene to generate a plurality of target point cloud clusters comprises: plane fitting of the original point cloud data to remove background plane points in the original point cloud data and obtain residual point cloud data; clustering segmentation of the residual point cloud data using an Euclidean clustering algorithm to obtain the plurality of target point cloud clusters.

3. The method of claim 1, wherein the generating of the target envelope based on the target point cloud clusters and the constructing of the environment geometric model according to the target envelope comprise: principal component analysis of each of the target point cloud clusters to determine a main direction of the point cloud cluster; generating a directional bounding box tightly surrounding the target point cloud cluster as the target envelope based on the main direction; conversion of a pose of the target envelope from a camera coordinate system to a first coordinate system of the robot based on calibration parameters of the robot; data fusion of the converted target envelope and a job boundary model in the first coordinate system to generate the environment geometric model.

4. The method of claim 1, wherein the determining of the target cargo from the environment geometric model according to the grabbing task strategy and the generating of the candidate grabbing pose of the target cargo comprise: obtaining geometric center coordinates of each target envelope in the first coordinate system in the environment geometric model; comparing Z-axis values of the geometric center coordinates; and selecting a cargo corresponding to a target envelope with the largest Z-axis value as the target cargo; calculating a surface normal of a target point cloud cluster corresponding to the target cargo; sampling within a deflection space defined by the preset angle range with the surface normal as a reference direction to generate an initial grabbing pose set; performing collision detection on each pose in the initial grabbing pose set using the environment geometric model; eliminating poses in the initial grabbing pose set that interfere with the environment geometric model to obtain the candidate grabbing pose.

5. The method of claim 1, wherein the converting of the environment geometric model into a voxel grid and the inputting of the voxel grid and the candidate grabbing pose into a pre-trained prediction neural network to output a path heat map comprise: ​ ​ ​ ​ discretize the three-dimensional space where the environment geometric model is located according to a preset resolution to obtain a three-dimensional grid space, and map the spatial occupancy information of the environment geometric model into the three-dimensional grid space to generate the voxel grid; encode the spatial coordinates of the candidate grasping pose into pose feature data consistent with the dimensions of the voxel grid based on the spatial dimensions of the voxel grid; concatenate the voxel grid and the pose feature data in the channel dimension to form multi-channel input data, input the multi-channel input data into the pre-trained prediction neural network for processing to obtain a three-dimensional probability distribution field, and use the three-dimensional probability distribution field as the path heat map.

6. The method of claim 1, wherein the path search in the environment geometric model based on the path heat map to generate a target motion path from a starting state of the robot to the candidate grasping pose comprises: mapping the voxel value distribution in the path heat map to a non-uniform sampling probability field; determining a non-occupied region in the three-dimensional space range of the environment geometric model; sampling in the non-occupied region according to the non-uniform sampling probability field to obtain sampling points; using the sampling points to construct a search tree extending to the candidate grasping pose with the starting state of the robot as a root node, and generating the target motion path when the search tree is connected to the candidate grasping pose. Further comprising:

7. The method of claim 1, wherein, controlling the robot to grasp the target goods based on the target motion path and placing the target goods at the unloading area.

8. The method of claim 7, wherein before the step of placing the target goods at the unloading area, comprising: calling a standard goods model corresponding to the target goods and a feature identifier; aligning the standard goods model with an actual goods model of the target goods based on the feature identifier; obtaining the intersection of each standard external surface of the standard goods model and the corresponding actual external surface of the actual goods model to obtain the placement area of each external surface of the target goods; determining the placement surface of the target goods based on the placement area.

9. The method of claim 8, wherein the determination of the placement surface of the target goods based on the placement area comprises: sorting the corresponding actual external surfaces of the actual goods model in descending order based on the placement area to obtain an external surface sequence; selecting the first actual external surface in the external surface sequence as the placement surface of the target goods.

10. A collision-free planning system for grasping a target, comprising: a segmentation module configured to segment original point cloud data of a target scene to generate a plurality of target point cloud clusters, generate a target envelope based on the target point cloud clusters, and construct an environment geometric model based on the target envelope; a determination module configured to determine a target goods from the environment geometric model according to a grasping task strategy, and generate a candidate grasping pose of the target goods. ​ ​ ​ A conversion module is configured to convert the environment geometry model into a voxel grid, and input the voxel grid and the candidate grasping pose into a pre-trained prediction neural network to output a path heat map; A generation module is configured to perform path search in the environment geometry model based on the path heat map to generate a target motion path from a starting state of the robot to the candidate grasping pose.

Images