An object grasping method based on a denoising diffusion model
By using a denoising diffusion model to complete point cloud data and calculate pose, the problem of inaccurate grasping pose under low-quality RGBD cameras is solved, and a more efficient object grasping effect is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NINGBO ARTIFICIAL INTELLIGENCE RES INST OF SHANGHAI JIAOTONG UNIV
- Filing Date
- 2024-03-14
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies lack accuracy and stability in generating six-DOF grasping poses based on point clouds in low-quality RGBD camera scenarios, especially in scenarios with unknown objects where the grasping effect is poor.
A denoising diffusion model is used to complete incomplete and noisy point clouds. A complete point cloud is generated through forward denoising and reverse denoising processes. Combined with robotic arm simulation, the grasping posture is calculated, and a stable posture is selected using geometric methods.
It improves the accuracy and stability of the grasping posture, and enhances the success rate and versatility of the robotic arm in grasping objects in complex scenarios.
Smart Images
Figure CN118024253B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robotics, and in particular to an object grasping method based on a denoising diffusion model. Background Technology
[0002] In recent years, with the advancement of technology and social development, robotics has rapidly developed in autonomous perception and human-computer interaction, and has been widely applied in various industrial scenarios. For example, robotics is often used to achieve parts grasping in automated assembly lines, improving the efficiency of factory assembly line operations. Specifically, this involves using a robotic arm combined with an RGBD camera to achieve autonomous perception in different environments using point cloud data of the scene, enabling the grasping of objects and parts in various scenarios. RGBD stands for RGB (Red, Green, Blue) image, a color mode combined with a depth map. RGBD combines color and depth information to provide three-dimensional data. Using an RGBD camera (depth camera), point cloud data of the scene can be acquired in real time, thus determining the grasping depth at that location, which is very convenient for robotic grasping tasks. However, due to the accuracy issues of non-high-end RGBD cameras in imaging, and the presence of a small number of outliers or random holes in the point cloud due to noise, the use of such defective point clouds for processing has a significant impact on the subsequent generation of six-DOF grasping poses. Therefore, it is necessary to design an efficient method that can overcome the defects of point cloud data in this regard and generate suitable grasping poses based on the defective point clouds so that the robotic arm can successfully grasp objects in the scene.
[0003] The paper "A Deep Learning-Based Robotic Arm Grasping Method" (application number CN202311215718.5) uses RGB and depth images as input. First, it designs an algorithm for target object detection with a rotation angle using the RGB image. The rotation angle of the object to be grasped relative to the current camera pose and its position from the camera's viewpoint are obtained from the RGB image. Then, the grasping depth of the object is obtained from the depth image, thus determining the current grasping posture. However, this method restricts the grasping posture to a vertical, top-down movement of the robotic arm gripper. When there are stacked objects in the scene, the grasping effect becomes poor, leading to collisions with other objects.
[0004] The patent "A Six-DOF Grasping Pose Estimation Method and Device for a Robotic Arm" (application number CN202210513187.X) directly utilizes point clouds with incomplete manual annotations and designs an encoder-decoder structure. First, a neural network encoder extracts features from the point cloud. In the decoder, a fully connected neural network maps the features extracted by the encoder to a suitable six-DOF grasping posture. This method effectively alleviates the problem of object grasping in stacked scenarios present in the patent "A Deep Learning-Based Robotic Arm Grasping Method." However, this method heavily relies on the quality of the point cloud generated by the sensor. When the point cloud contains significant noise and voids, this method becomes highly unstable. The grasping accuracy is greatly affected when dealing with unknown objects.
[0005] It is evident that existing technologies, based on six degrees of freedom grasping with defective point clouds and the diversity of grasping postures for unknown objects, result in poor quality of grasping posture generation due to the incompleteness of the point cloud in application scenarios with low-quality RGBD cameras.
[0006] Therefore, those skilled in the art are dedicated to developing an item grasping method based on a denoising diffusion model. Summary of the Invention
[0007] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to improve the accuracy and stability of the grasping posture in scenarios with missing point cloud data and unstable factors such as noise.
[0008] The inventors developed a grasping algorithm based on a denoising diffusion model. This algorithm completes incomplete and noisy point clouds to obtain more comprehensive point clouds. Based on these comprehensive point clouds, it generates six-degree-of-freedom diverse grasping postures for parallel grippers. For ease of description, the "scene" in this specification refers to an object grasping scenario.
[0009] In one embodiment of the present invention, an item grasping method based on a denoising diffusion model is provided, comprising:
[0010] S100 Scene point cloud data processing: Acquire scene point cloud data and process it to obtain scene feature vectors;
[0011] S200, the training of the noisy network in the forward process, performs multi-step Gaussian noise addition on the scene feature vector in the forward process of the denoising diffusion model to obtain the noisy feature distribution;
[0012] S300, point cloud data completion, using the inverse process of the denoising diffusion model to gradually denoise Gaussian noise, to obtain the completed point cloud data;
[0013] S400, object grasping simulation, simulates grasping scenarios, calculates the location and probability of successful grasping, and calculates the grasping heatmap;
[0014] S500, Feature Aggregation: Extract features from the captured heatmap, and aggregate the obtained captured heatmap vector and location features.
[0015] S600, Aggregate feature noise addition: Input the aggregated features into the denoising diffusion model, add noise to the aggregated features in the forward process of the denoising diffusion model, and obtain the noisy feature vector of the item grabbing features.
[0016] S700, Feature Heatmap Extraction: The feature heatmap corresponding to the completed point cloud data is obtained by using the reverse process of the denoising diffusion model.
[0017] S800: Final grasping posture matrix calculation. Based on the grasping feature heatmap, the grasping posture matrix is calculated using geometric methods, and the posture is filtered.
[0018] Optionally, in the item grasping method based on the denoising diffusion model in the above embodiments, step S100 includes:
[0019] S110. Scene point cloud data acquisition: Acquire RGB and depth images and process them to obtain scene point cloud data;
[0020] S120. Scene point cloud data preprocessing: The acquired scene point cloud data is preprocessed to obtain the input data for the denoising diffusion model.
[0021] S130. Point cloud data feature extraction: Extract features from the above input data to obtain the feature vector of the intermediate layer.
[0022] Optionally, in the item grasping method based on the denoising diffusion model in the above embodiments, step S110 includes:
[0023] S111. Acquire RGB and depth images. Use an RGBD camera to take pictures of the calibration board in a specified scene to acquire RGB and depth images.
[0024] S112. Extract the location of corner points. Use the ConerHarris function in the OpenCV library to extract the location of corner points in the RGB image and depth image.
[0025] S113. Align the RGB image and the depth image. Use the SURF algorithm to map the corner points in the RGB image to the corner points in the depth image, thereby aligning the RGB image and the depth image.
[0026] S114, Scene point cloud data: The aligned image is converted into point cloud xyz format using the intrinsic coefficients of the RGBD camera to obtain scene point cloud data.
[0027] Furthermore, in the object grasping method based on the denoising diffusion model in the above embodiments, the calculation method used in step S114 to convert the aligned image into point cloud xyz format is as follows:
[0028]
[0029] in These are the coordinates of the scene in the point cloud. These are the internal parameters of an RGBD camera. This represents the focal length of the RGBD camera along the x-axis. This represents the focal length of the RGBD camera along the y-axis. This represents the offset distance of the image along the center of the x-axis. This represents the offset distance of the image along the center of the y-axis. These are the corresponding coordinate values in the RGB image, where D is the depth conversion ratio, which is a constant and can be obtained from the product manual of the RGBD camera.
[0030] Optionally, in the object grasping method based on the denoising diffusion model in any of the above embodiments, the RGBD camera can be replaced with a RealSense camera or a Kinect camera.
[0031] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S120 includes:
[0032] S121, Point cloud data normalization, including point cloud data displacement and point cloud data rotation;
[0033] S122. Obtain the point cloud data of the scene. The point cloud data obtained after normalization is ,in Indicates the number of items in the scene. This represents the number of individual points contained in a complete object point cloud. To indicate that a point is composed of It consists of three coordinates.
[0034] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S130 uses the PointNet++ network to extract features from the point cloud data, and the resulting feature vector is: ,in, The number of items encoded in one feature. Dimensions of the encoded features.
[0035] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S200 includes:
[0036] S210, Positional attention embedding: Input the scene feature vector extracted in step S100 into the attention fusion network with cosine positional embedding to obtain the feature vector with positional attention embedding.
[0037] S220, Input the denoising diffusion model, input the feature vector with positional attention embedding into the denoising diffusion model;
[0038] S230. Training the noise-adding network: The feature vector with positional attention embedding is subjected to multi-step Gaussian noise addition using the forward process of the denoising diffusion model to obtain the noise-adding feature distribution, and the noise-adding network of the forward process is trained.
[0039] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S230 includes:
[0040] S231. Add Gaussian noise: Gradually add Gaussian noise to the input feature vector with positional attention embedding according to the number of noise addition steps, forming a Markov chain of the noisy features. N represents the number of steps for adding noise;
[0041] S232. Calculate the eigenvector after adding noise, and then calculate the eigenvector at step t-1. After adding noise, the noisy feature vector in the next step is represented as follows: ,in It is Gaussian noise. Let be the noise amplitude added in step t;
[0042] S233. Predict the noise amplitude. The noise amplitude is predicted by a neural network. It is the amplitude of noise at different steps during the multi-step Gaussian noise addition process, and it gradually decreases as the number of noise addition steps increases;
[0043] S234. The noisy network during the training process: the input feature vector with positional attention embedding becomes random noise through continuous noise addition. The currently added random noise is then utilized... With the noise vector The mean squared error loss is used to train the forward network.
[0044] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S300 includes:
[0045] S310. Noise removal: Use the reverse process of the noise reduction diffusion model to remove noise. Input random Gaussian noise, and in the... Through the first The noise values in the first step predict the denoised feature vector, and the mean and variance of the denoised feature vector satisfy the following:
[0046]
[0047]
[0048] in For the first Feature encoding in the denoising process. The original feature vector is obtained by encoding the initially completed point cloud data using PointNet++.
[0049] S320. Loss function design: The loss function for the denoising and matching process is the chamfer distance loss function.
[0050]
[0051]
[0052] in In the reverse denoising process, when there is a priori feature vector, the first... The feature vector after denoising step The feature vector is obtained by directly denoising randomly generated Gaussian noise. for The set of points formed for The set of points formed, where Let be the feature vector obtained after denoising in step t-1. Let be the denoised feature vector at step t. The initial random Gaussian noise is used.
[0053] Furthermore, in the item grasping method based on the denoising diffusion model in the above embodiments, the point cloud shape obtained after t steps of denoising in step S310 is: That is, the sampled point cloud contains 2048 points, and each point contains Three location information.
[0054] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S400 includes:
[0055] S410, Object Simulation: Import CAD models of different objects into the pybullet simulation environment. For complex object models, use an approximate convex decomposition algorithm to convert the CAD model into a convex model and export the CAD model as a .obj format.
[0056] S420, robotic arm simulation: Import the robotic arm model from aubo-i10, build a 3D scene for actual grasping, randomly generate a rotation matrix centered on the object to be grasped, and form a 6D grasping posture with the current position of the object in space. Set the parallel gripper of the robotic arm to the generated 6D grasping posture, and then turn off the parallel gripper.
[0057] S430. Calculate the gripping posture. Calculate the posture of the parallel gripper relative to the current object in the pybullet simulation environment.
[0058] S440. Obtain point cloud data, control the robotic arm to move freely a short distance, and recalculate the attitude of the parallel gripper relative to the current object. If the Euclidean distance between the two relative attitudes before and after the movement is less than... If the grasp is collision-free and stable, then the posture is marked as positive sample data, the current item is sampled as point cloud data, the point cloud count information at the contact position with the parallel gripper is calculated, and the count at the corresponding position is incremented by 1.
[0059] S450, normalization calculation, repeat S410-S440 five thousand to ten thousand times to obtain the number of successful grabs at different positions in the grab object point cloud, perform normalization calculation for the success frequency at different positions, and obtain the point cloud position contact frequency table.
[0060] S460. Generate a grabbing heatmap. Using the point cloud position contact frequency table, generate a grabbing heatmap of the grabbed item. The higher the frequency, the higher the R value in the RGB value of the point cloud color at that position.
[0061] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S500 includes:
[0062] S510: Feature extraction from the captured heatmap. A three-layer fully connected layer is used to extract features from the captured heatmap.
[0063] S520. Location information feature vector extraction: The location information of the point cloud is extracted using the farthest point sampling algorithm to obtain feature vectors at different sampling ratios.
[0064] S530, Feature Vector Aggregation: The k-nearest neighbor algorithm is used to aggregate feature vectors under different sampling ratios. The input is cosine and sine position embedding to obtain position features. The obtained heatmap vector and position features are superimposed to perform feature aggregation.
[0065] Furthermore, in the item grasping method based on the denoising diffusion model in the above embodiments, the farthest point sampling algorithm includes:
[0066] S521. Select a set of sampling points. For the 2048 points in the input point cloud, arbitrarily select one point from these points. As the initial point, we obtain the set of sampling points. ;
[0067] S522. Update the sampling point set and calculate the distance from all points to the sampling point set. The Euclidean distances form a 2048-dimensional array. From array Select the point corresponding to the maximum value as Update the set of sampling points ;
[0068] S523. Update the distance to the point corresponding to the maximum value, and calculate the distance to all points. The Euclidean distance for each point If its distance The distance is less than Then update For this point to The distance;
[0069] S524. Update the sampling point set again, selecting the array. The point corresponding to the maximum value in the middle is taken as Update the set of sampling points ;
[0070] S525. Calculate the distance to other points in the point cloud, iterating through S522-S524 to calculate the distance from other points in the point cloud to the set of sampling points. The distance is calculated until the number of points in the sampling point set meets the required number of sampling points under different sampling ratios.
[0071] Furthermore, in the item grasping method based on the denoising diffusion model in the above embodiments, the required number of sampling points is 4-6 times at a downsampling rate of 50%, which downsamples the point cloud to 1024 points, 512 points, 256 points and 128 points respectively, to obtain features under different sampling rates.
[0072] Furthermore, in the item grasping method based on the denoising diffusion model in the above embodiments, the k-nearest neighbor algorithm includes:
[0073] S531. Calculate the Euclidean distance: calculate the Euclidean distance between the feature vector at each sampling rate and the feature vectors at all different sampling rates.
[0074] S532. Output aggregated features: Select the k feature vectors with the largest distance among all feature vectors as aggregated features and output them.
[0075] Furthermore, in the item grabbing method based on the denoising diffusion model in the above embodiments, step S600 further includes calculating the KL divergence error between the denoised feature vector and the added random Gaussian noise, and training the denoising diffusion model.
[0076] Furthermore, in the item grasping method based on the denoising diffusion model in the above embodiments, the expression for the KL divergence error is as follows:
[0077]
[0078] in In the reverse denoising process, when there is a priori feature vector, the first... The feature vector after denoising step Let i be the i-th dimension of the eigenvector. The feature vector is obtained by directly denoising randomly generated Gaussian noise, where Let be the feature vector obtained after denoising in step t-1. Let be the denoised feature vector at step t. The initial random Gaussian noise is used.
[0079] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S700 includes:
[0080] S710, Feature Extraction: The completed point cloud data is used for feature extraction via PointNet++.
[0081] S720, Noise superposition, input features after superimposing randomly generated Gaussian noise;
[0082] S730: Capture feature heatmap acquisition, and obtain the capture feature heatmap corresponding to the completed point cloud data through the reverse process of the denoising diffusion model.
[0083] Optionally, in the item grasping method based on the denoising diffusion model in any of the above embodiments, step S800 includes:
[0084] S810, Normal Vector Calculation: Calculate the normal vector of the surface formed by the cross product of the vectors formed by the coordinate differences between any two points.
[0085] S820, the center position coordinates of the parallel gripper and the gripping plane of the parallel gripper are calculated. Using the completed point cloud data, the symbolic distance field (SDF) of the scene is calculated. The intersection of the calculated normal vector and the symbolic distance field is used as another contact point between the normal vector and the object surface to calculate the center position coordinates of the parallel gripper and the gripping plane of the parallel gripper.
[0086] S830, candidate grasping posture sampling, in the obtained grasping plane, according to the given precision °, sampled One candidate crawling posture, The value is between 15° and 30° and is divisible by 360;
[0087] S840. Grasping pose calculation: Using the FCL (the flexible collision library, an open-source collision detection library that can detect whether two objects collide given the parameters of two object models) collision detection library, we can calculate in parallel whether the candidate grasping poses in the current scene collide with the point cloud. If a collision occurs, it is filtered out to obtain the final collision-free and stable grasping pose.
[0088] This invention proposes a grasping method based on a denoising diffusion model, which completes incomplete point clouds and point clouds with noise to obtain complete point clouds. Based on the complete point clouds, it realizes the generation of six-degree-of-freedom diverse grasping postures for parallel grippers, thereby improving the success rate, accuracy and diversity of grasping.
[0089] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description
[0090] Figure 1 This is a flowchart illustrating an item grasping method based on a denoising diffusion model according to an exemplary embodiment;
[0091] Figure 2 This is the forward noise addition process for point cloud completion in the item grasping method based on the denoising diffusion model in the exemplary embodiment;
[0092] Figure 3 This is a positive noise addition process for heatmap features in an item grabbing method based on a denoising diffusion model, as described in an exemplary embodiment. Detailed Implementation
[0093] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.
[0094] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of components is schematically exaggerated in some places in the drawings.
[0095] The inventors designed an item grasping method based on a denoising diffusion model, such as... Figure 1 As shown, it includes:
[0096] S100. Process scene point cloud data: acquire scene point cloud data and process it to obtain scene feature vectors; including:
[0097] S110. Scene point cloud data acquisition: Acquire RGB and depth images and process them to obtain scene point cloud data; specifically including:
[0098] S111. Acquire RGB and depth images. Use an RGBD camera to take pictures of the calibration board in a specified scene to acquire RGB and depth images.
[0099] S112. Extract the location of corner points. Use the ConerHarris function in the OpenCV library to extract the location of corner points in the RGB image and depth image.
[0100] S113. Align the RGB image and the depth image. Use the SURF algorithm to map the corner points in the RGB image to the corner points in the depth image, thereby aligning the RGB image and the depth image.
[0101] S114. Scene point cloud data: The aligned image is converted into point cloud xyz format using the intrinsic parameters of the RGBD camera to obtain scene point cloud data. The calculation method is as follows:
[0102]
[0103] in These are the coordinates of the scene in the point cloud. These are the internal parameters of an RGBD camera. This represents the focal length of the RGBD camera along the x-axis. This represents the focal length of the RGBD camera along the y-axis. This represents the offset distance of the image along the center of the x-axis. This represents the offset distance of the image along the center of the y-axis. These are the corresponding coordinate values in the RGB image, where D is the depth conversion ratio, which is a constant and can be obtained from the product manual of the RGBD camera.
[0104] S120. Scene point cloud data preprocessing: The acquired scene point cloud data is preprocessed to obtain the input data for the denoising diffusion model; specifically including:
[0105] include:
[0106] S121, Point cloud data normalization, including point cloud data displacement and point cloud data rotation;
[0107] S122. Obtain the point cloud data of the scene. The point cloud data obtained after normalization is ,in Indicates the number of items in the scene. This represents the number of individual points contained in a complete object point cloud. To indicate that a point is composed of It consists of three coordinates.
[0108] S130. Point cloud data feature extraction: Feature extraction is performed on the above input data. The PointNet++ network is used to extract the features of the point cloud data, and the resulting feature vector is: ,in, The number of items encoded in one feature. Dimensions of the encoded features.
[0109] S200, the forward process of training the noisy network involves multi-step Gaussian noise addition to the scene feature vectors during the forward process of the denoising diffusion model, resulting in the noisy feature distribution; such as Figure 2 As shown, it includes:
[0110] S210, Positional attention embedding: Input the scene feature vector extracted in step S100 into the attention fusion network with cosine positional embedding to obtain the feature vector with positional attention embedding.
[0111] S220, Input the denoising diffusion model, input the feature vector with positional attention embedding into the denoising diffusion model;
[0112] S230. Training the noisy network: The noisy network is trained by performing multi-step Gaussian noise addition on the feature vector with positional attention embedding using the forward process of the denoising diffusion model, resulting in a noisy feature distribution. The training of the noisy network using the forward process includes:
[0113] S231. Add Gaussian noise: Gradually add Gaussian noise to the input feature vector with positional attention embedding according to the number of noise addition steps, forming a Markov chain of the noisy features. N represents the number of steps for adding noise;
[0114] S232. Calculate the eigenvector after adding noise, and then calculate the eigenvector at step t-1. After adding noise, the noisy feature vector in the next step is represented as follows: ,in It is Gaussian noise. Let be the noise amplitude added in step t;
[0115] S233. Predict the noise amplitude. The noise amplitude is predicted by a neural network. It is the amplitude of noise at different steps during the multi-step Gaussian noise addition process, and it gradually decreases as the number of noise addition steps increases;
[0116] S234. The noisy network during the training process: the input feature vector with positional attention embedding becomes random noise through continuous noise addition. The currently added random noise is then utilized... With the noise vector The mean squared error loss is used to train the forward network.
[0117] S300, Point cloud data completion: Gaussian noise is progressively denoised using the inverse process of a denoising diffusion model to obtain the completed point cloud data; including:
[0118] S310. Noise removal: Use the reverse process of the noise reduction diffusion model to remove noise. Input random Gaussian noise, and in the... Through the first The noise values in the first step predict the denoised feature vector, and the mean and variance of the denoised feature vector satisfy the following:
[0119]
[0120]
[0121] in For the first Feature encoding in the denoising process. The original feature vector is obtained by encoding the initially padded point cloud data using PointNet++; the shape of the point cloud after t-step denoising is as follows: That is, the sampled point cloud contains 2048 points, and each point contains Three location information;
[0122] S320. Loss function design: The loss function for the denoising and matching process is the chamfer distance loss function.
[0123]
[0124]
[0125] in In the reverse denoising process, when there is a priori feature vector, the first... The feature vector after denoising step The feature vector is obtained by directly denoising randomly generated Gaussian noise. for The set of points formed for The set of points formed, where Let be the feature vector obtained after denoising in step t-1. Let be the denoised feature vector at step t. The initial random Gaussian noise is used.
[0126] S400, object grasping simulation, simulates grasping scenarios, calculates the location and probability of successful grasping, and generates a grasping heatmap; including:
[0127] S410, Object Simulation: Import CAD models of different objects into the pybullet simulation environment. For complex object models, use an approximate convex decomposition algorithm to convert the CAD model into a convex model and export the CAD model as a .obj format.
[0128] S420, robotic arm simulation: Import the robotic arm model from aubo-i10, build a 3D scene for actual grasping, randomly generate a rotation matrix centered on the object to be grasped, and form a 6D grasping posture with the current position of the object in space. Set the parallel gripper of the robotic arm to the generated 6D grasping posture, and then turn off the parallel gripper.
[0129] S430. Calculate the gripping posture. Calculate the posture of the parallel gripper relative to the current object in the pybullet simulation environment.
[0130] S440. Obtain point cloud data, control the robotic arm to move freely a short distance, and recalculate the attitude of the parallel gripper relative to the current object. If the Euclidean distance between the two relative attitudes before and after the movement is less than... If the grasp is collision-free and stable, then the posture is marked as positive sample data, the current item is sampled as point cloud data, the point cloud count information at the contact position with the parallel gripper is calculated, and the count at the corresponding position is incremented by 1.
[0131] S450, normalization calculation, repeat S410-S440 five thousand to ten thousand times to obtain the number of successful grabs at different positions in the grab object point cloud, perform normalization calculation for the success frequency at different positions, and obtain the point cloud position contact frequency table.
[0132] S460. Generate a grasping heatmap. Using the point cloud position contact frequency table, generate a grasping heatmap of the grasped item. The higher the frequency, the higher the R value in the RGB value of the point cloud color at that position.
[0133] S500, Feature Aggregation: Features are extracted from the captured heatmap. The resulting heatmap vector and location features are then superimposed to perform feature aggregation, such as... Figure 3 As shown, it includes:
[0134] S510: Feature extraction from the captured heatmap. A three-layer fully connected layer is used to extract features from the captured heatmap.
[0135] S520. Location information feature vector extraction: The location information of the point cloud is extracted using the farthest point sampling algorithm to obtain feature vectors at different sampling ratios. The farthest point sampling algorithm includes:
[0136] S521. Select a set of sampling points. For the 2048 points in the input point cloud, arbitrarily select one point from these points. As the initial point, we obtain the set of sampling points. ;
[0137] S522. Update the sampling point set and calculate the distance from all points to the sampling point set. The Euclidean distances form a 2048-dimensional array. From array Select the point corresponding to the maximum value as Update the set of sampling points ;
[0138] S523. Update the distance to the point corresponding to the maximum value, and calculate the distance to all points. The Euclidean distance for each point If its distance The distance is less than Then update For this point to The distance;
[0139] S524. Update the sampling point set again, selecting the array. The point corresponding to the maximum value in the middle is taken as Update the set of sampling points ;
[0140] S525. Calculate the distance to other points in the point cloud, iterating through S522-S524 to calculate the distance from other points in the point cloud to the set of sampling points. The distance is calculated until the number of points in the sampling point set meets the required number of sampling points under different sampling ratios. That is, sampling is performed 4-6 times at a downsampling rate of 50%, and the point cloud is downsampled to 1024 points, 512 points, 256 points and 128 points respectively to obtain features under different sampling ratios.
[0141] S530. Feature vector aggregation: The k-nearest neighbor algorithm is used to aggregate feature vectors at different sampling ratios. The vectors are then input into cosine and sine position embeddings to obtain positional features. The resulting heatmap vectors and positional features are superimposed for feature aggregation. The k-nearest neighbor algorithm includes:
[0142] S531. Calculate the Euclidean distance: calculate the Euclidean distance between the feature vector at each sampling rate and the feature vectors at all different sampling rates.
[0143] S532. Output aggregated features: Select the k feature vectors with the largest distance among all feature vectors as aggregated features and output them.
[0144] S600. Adding noise to aggregated features: Input the aggregated features into the denoising diffusion model. During the forward pass of the denoising diffusion model, noise is added to the aggregated features to obtain the noisy feature vector of the item grabbing features. Calculate the KL divergence error between the noisy feature vector and the added random Gaussian noise. Train the denoising diffusion model. The expression for the KL divergence error is as follows:
[0145]
[0146] in In the reverse denoising process, when there is a priori feature vector, the first... The feature vector after denoising step Let i be the i-th dimension of the eigenvector. The feature vector is obtained by directly denoising randomly generated Gaussian noise, where Let be the feature vector obtained after denoising in step t-1. Let be the denoised feature vector at step t. The initial random Gaussian noise;
[0147] S700, Feature Heatmap Extraction: The feature heatmap corresponding to the completed point cloud data is obtained through the reverse process of a denoising diffusion model; including:
[0148] S710, Feature Extraction: The completed point cloud data is used for feature extraction via PointNet++.
[0149] S720, Noise superposition, input features after superimposing randomly generated Gaussian noise;
[0150] S730: Capture feature heatmap acquisition, and obtain the capture feature heatmap corresponding to the completed point cloud data through the reverse process of the denoising diffusion model.
[0151] S800, Final Grasping Pose Matrix Calculation: Based on the grasping feature heatmap, the grasping pose matrix is calculated using geometric methods, and pose selection is performed, including:
[0152] S810, Normal Vector Calculation: Calculate the normal vector of the surface formed by the cross product of the vectors formed by the coordinate differences between any two points.
[0153] S820, the center position coordinates of the parallel gripper and the gripping plane of the parallel gripper are calculated. Using the completed point cloud data, the symbolic distance field (SDF) of the scene is calculated. The intersection of the calculated normal vector and the symbolic distance field is used as another contact point between the normal vector and the object surface to calculate the center position coordinates of the parallel gripper and the gripping plane of the parallel gripper.
[0154] S830, candidate grasping posture sampling, in the obtained grasping plane, according to the given precision °, sampled One candidate crawling posture, The value is between 15° and 30° and is divisible by 360;
[0155] S840. Grasping pose calculation: Using the FCL (the flexible collision library, an open-source collision detection library that can detect whether two objects collide given the parameters of two object models) collision detection library, we can calculate in parallel whether the candidate grasping poses in the current scene collide with the point cloud. If a collision occurs, it is filtered out to obtain the final collision-free and stable grasping pose.
[0156] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. An article grasping method based on a denoising diffusion model, characterized by, include: S100 Scene point cloud data processing: Acquire scene point cloud data and process it to obtain scene feature vectors; S200, the training of the noisy network in the forward process, in which the scene feature vector is subjected to multi-step Gaussian noise addition in the forward process of the denoising diffusion model, to obtain the feature distribution after noise addition; S300. Point cloud data completion: Gaussian noise is gradually denoised using the inverse process of the denoising diffusion model to obtain the completed point cloud data. S400, object grasping simulation, simulates grasping scenarios, calculates the location and probability of successful grasping, and calculates the grasping heatmap; S500, Feature aggregation: Extract features from the captured heatmap, and aggregate the obtained captured heatmap vector and position features by superimposing them. S600. Add noise to aggregated features: Input the aggregated features into the denoising diffusion model, and add noise to the aggregated features in the forward process of the denoising diffusion model to obtain the feature vector of the item grabbing features after adding noise. S700, Feature heatmap capture: The feature heatmap corresponding to the completed point cloud data is obtained by using the reverse process of the denoising diffusion model. S800. Final grasping posture matrix calculation: Based on the grasping feature heatmap, the grasping posture matrix is calculated using geometric methods, and postures are filtered.
2. The object grasping method based on a denoising diffusion model according to claim 1, wherein, Step S100 includes: S110. Scene point cloud data acquisition: Acquire RGB and depth images and process them to obtain scene point cloud data; S120. Scene point cloud data preprocessing: The acquired scene point cloud data is preprocessed to obtain the input data of the denoising diffusion model. S130. Point cloud data feature extraction: Extract features from the input data to obtain the feature vector of the intermediate layer.
3. The object grasping method based on a de-noising diffusion model according to claim 2, wherein, Step S120 includes: S121, Point cloud data normalization, including point cloud data displacement and point cloud data rotation; S122, obtain the point cloud data of the scene , the point cloud data obtained after normalization is , wherein represents the number of items in the scene, represents the number of single points contained in a complete item point cloud, represents that a point is composed of three coordinates.
4. The object grasping method based on a de-noising diffusion model according to claim 2, wherein, Step S200 includes: S210, Position attention embedding: The scene feature vector is input into an attention fusion network with cosine position embedding to obtain a feature vector with position attention embedding. S220. Input the denoising diffusion model by inputting the feature vector with positional attention embedding into the denoising diffusion model. S230. Training the noise-adding network: The feature vector with positional attention embedding is subjected to multi-step Gaussian noise addition using the forward process of the denoising diffusion model to obtain the noise-adding feature distribution, and the noise-adding network of the forward process is trained.
5. The item grasping method based on a denoising diffusion model as described in claim 4, characterized in that, Step S300 includes: S310. Noise removal: Remove noise using the reverse process of the denoising diffusion model. Input random Gaussian noise, and in the... Step, through the first The noise values in the first step predict the denoised feature vector, and the mean and variance of the denoised feature vector satisfy the following: ； ； in For the first Feature encoding in the denoising process. The original feature vector is obtained by encoding the initially completed point cloud data using PointNet++. Let be the noise amplitude added in step t; S320. Loss function design: The loss function for the denoising and matching process is the chamfer distance loss function. ； in In the reverse denoising process, when there is a priori feature vector, the first... The feature vector after denoising step The feature vector is obtained by directly denoising randomly generated Gaussian noise. for The set of points formed for The set of points formed, where Let be the feature vector obtained after denoising in step t-1. Let be the denoised feature vector at step t. The initial random Gaussian noise is used.
6. The item grasping method based on a denoising diffusion model as described in claim 5, characterized in that, Step S400 includes: S410, Object Simulation: Import CAD models of different objects into the pybullet simulation environment. For complex object models, use an approximate convex decomposition algorithm to convert the CAD model into a convex model and export the CAD model as a .obj format. S420, robotic arm simulation: import the robotic arm model from aubo-i10, build a 3D scene for actual grasping, randomly generate a rotation matrix centered on the grasped object, and form a 6D grasping posture with the current position of the object in space. Set the parallel gripper of the robotic arm to the generated 6D grasping posture and turn off the parallel gripper. S430. Calculate the gripping posture, and calculate the posture of the parallel gripper relative to the current item in the pybullet simulation environment; S440. Obtain point cloud data, control the robotic arm to move arbitrarily a short distance, and recalculate the posture of the parallel gripper relative to the current object. If the Euclidean distance between the two relative postures before and after the movement is less than... If the grasping is collision-free and stable, the posture is marked as positive sample data, the current item is sampled as point cloud data, the point cloud count information at the contact position with the parallel gripper is calculated, and the count at the corresponding position is incremented by 1. S450, normalization calculation, repeat S410-S440 five thousand to ten thousand times to obtain the number of successful grabs at different positions in the grab object point cloud, perform normalization calculation for the success frequency at different positions, and obtain the point cloud position contact frequency table. S460. Generate a grasping heatmap. Using the point cloud position contact frequency table, generate a grasping heatmap of the grasped item. The higher the frequency, the higher the R value in the RGB value of the point cloud color at that position.
7. The item grasping method based on a denoising diffusion model as described in claim 6, characterized in that, Step S500 includes: S510: Feature extraction from the captured heatmap. A three-layer fully connected layer is used to extract features from the captured heatmap. S520. Location information feature vector extraction: The location information of the point cloud is extracted using the farthest point sampling algorithm to obtain feature vectors at different sampling ratios. S530. Feature vector aggregation: The k-nearest neighbor algorithm is used to aggregate the feature vectors under different sampling ratios. The input is cosine and sine position embedding to obtain position features. The obtained heat map vector and position features are superimposed to perform feature aggregation.
8. The item grasping method based on the denoising diffusion model as described in claim 7, characterized in that, Step S600 further includes calculating the KL divergence error between the denoised feature vector and the added random Gaussian noise, and training the denoising diffusion model.
9. The item grasping method based on a denoising diffusion model as described in claim 8, characterized in that, Step S700 includes: S710, Feature Extraction: The completed point cloud data is used for feature extraction via PointNet++. S720, Noise superposition, input features after superimposing randomly generated Gaussian noise; S730. Capture feature heatmap acquisition: The capture feature heatmap corresponding to the completed point cloud data is obtained through the reverse process of the denoising diffusion model.
10. The item grasping method based on a denoising diffusion model as described in claim 5, characterized in that, Step S800 includes: S810, Normal Vector Calculation: Calculate the normal vector of the surface formed by the cross product of the vectors formed by the coordinate differences between any two points. S820, the center position coordinates of the parallel gripper and the gripping plane of the parallel gripper are calculated. Using the completed point cloud data, the symbolic distance field (SDF) of the scene is calculated. The intersection of the calculated normal vector and the symbolic distance field is used as another contact point between the normal vector and the object surface. The center position coordinates of the parallel gripper and the gripping plane of the parallel gripper are calculated. S830, Candidate grasping posture sampling, in the obtained grasping plane, according to the given precision °, sampled One candidate crawling posture, The value is between 15° and 30° and is divisible by 360; S840 Grasping posture calculation: The FCL collision detection library is used to calculate in parallel whether the candidate grasping postures in the current scene collide with the point cloud. If a collision occurs, the postures are filtered out to obtain the final collision-free and stable grasping postures.