Robot seven-degree-of-freedom grabbing method based on generative adversarial learning

Through the seven-degree of freedom crawling method based on generative adversarial learning, combined with enhanced crawling evaluation and multi-stream channel correlation attention network, the traditional six-degree of freedom crawling method is solved. The lack of crawling stability and robustness in complex environments is solved. Problem, achieving higher crawl success rate and accuracy.

CN118990500AActive Publication Date: 2024-11-22DONGHUA UNIV

Patent Information

Application Number
CN202411291302.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-11-22
Estimated Expiration
2044-09-14

AI Technical Summary

Technical Problem

When the traditional six-degree of freedom grasping method deals with objects of complex shapes and diverse materials, the force closure indicator cannot fully reflect the grasping stability, resulting in the grab failure, and the generalization ability and robustness of the model are insufficient, so the grab is successful. Low rate and robustness.

Method used

Using the seven-degree-of-freedom crawling method based on generative adversarial learning, new scoring indicators are introduced by enhancing the crawling evaluation method and generative adversarial learning, point cloud feature extraction is performed in combination with Hough voting and PnP3D network, and multi-stream channel correlation attention is used. The network obtains seven degrees of freedom information and optimizes the feature extraction results through the discriminator to form a more robust capture and prediction network.

Benefits of technology

It significantly improves the grab stability, contact point distribution balance and grab prediction accuracy, improves the grab success rate and robustness, and overcomes the problems of insufficient precision and robustness of traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118990500A_ABST
    Figure CN118990500A_ABST
Patent Text Reader

Abstract

The invention relates to a robot seven-degree-of-freedom grabbing method based on generative adversarial learning, which comprehensively considers multiple scoring indexes such as grabbing stability, contact point distribution balance, centroid distance and contact point non-coplanarity by introducing an enhanced grabbing evaluation method and generative adversarial learning, and comprehensively evaluates a grabbing pose. A set abstraction layer network based on Hough voting and a PnP3D network feature extractor are combined with a multi-flow channel association attention network, point cloud features are finely extracted, and the capture prediction precision is improved; by generating an adversarial optimization feature extraction result, the model has higher robustness and precision under different environments and object changes, the grabbing success rate is remarkably increased, the problems that a traditional six-degree-of-freedom grabbing method is low in success rate and robustness of grabbing prediction and low in precision and stability are solved, and the grabbing accuracy is improved. Diversified objects can be accurately grabbed in a complex environment, it is ensured that the grabbing posture is closer to a real value, and powerful support is provided for development of the intelligent robot grabbing technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of robot visual grasping, and in particular to a seven-degree-of-freedom grasping method based on generative adversarial learning. Background Art

[0002] In recent years, with the rapid development of robotics technology, robotic grasping technology has been widely used in many fields such as industrial automation, logistics sorting, and home services. The core of robotic grasping technology lies in how to achieve stable and accurate grasping of objects of different shapes, materials, and sizes, which places high demands on the robot's perception, planning, and control capabilities. In the robotic grasping task, the selection of grasping posture and the evaluation of grasping stability are two key issues that directly affect the robot's grasping success rate and task execution effect.

[0003] Traditional six-degree-of-freedom (6-DoF) grasping methods have been widely used in many applications. However, traditional 6-DoF grasping evaluation methods mainly rely on the force closure index, which judges the feasibility of grasping by evaluating whether the contact force between the robot gripper and the object to be grasped can form a stable balance. However, the force closure index may not fully reflect the stability of grasping when dealing with objects of complex shapes and diverse materials, resulting in grasping failure. When evaluating the grasping posture, traditional methods often ignore the distribution balance of contact points, which makes it easy for torque imbalance to occur during the grasping process, affecting the stability and safety of the grasping. Most existing methods only consider the local information of the grasping point, without comprehensively considering the force conditions and posture changes of the object during the entire grasping process, resulting in low grasping success rate and robustness.

[0004] Traditional grasp prediction methods lack generalization and robustness when dealing with different environments and diverse objects, and are easily affected by data noise and sample distribution differences. Common methods rely on static feature extraction and scoring mechanisms and cannot dynamically adapt to changes in data sets, resulting in low accuracy and stability in grasp prediction. Summary of the invention

[0005] In view of the problems of low success rate and robustness of grasping prediction, low accuracy and stability of traditional six-degree-of-freedom grasping methods, a seven-degree-of-freedom grasping method based on generative adversarial learning is proposed. Taking the robot's seven-degree-of-freedom grasping as the research object, enhanced grasping evaluation method and generative adversarial learning method are introduced, and an efficient and stable robot grasping technology is proposed, which can achieve accurate grasping of diverse objects in complex environments.

[0006] The technical solution of the present invention is:

[0007] A robot seven-degree-of-freedom grasping method based on generative adversarial learning comprises the following steps:

[0008] S1. Obtain and input the original data set containing point cloud data, perform data enhancement on the original data set through the enhanced grasping evaluation method, use the enhanced grasping evaluation method EGE, introduce a new scoring indicator, comprehensively consider the distribution balance of contact points and the stability of grasping, and improve the existing grasping scoring mechanism as a learning tendency;

[0009] S2, using the backbone network Backbone improved based on Hough voting Vonet and a plug-and-play enhanced feature PnP3D for point cloud downsampling and feature extraction;

[0010] S3, the grasping feature extraction network is implemented through a multi-layer perceptron to extract the features of the sampled point cloud, grasp the required position and the candidate set of points to be grasped and the set of feature vectors;

[0011] S4, obtain the six-degree-of-freedom posture and seven-degree-of-freedom information of the gripper width through the multi-stream channel association attention network;

[0012] S5. A discriminator network is added between the feature extraction network and the grasping optimization network. By identifying the intermediate features of the feature extraction network and the real labels of the dataset optimized by EGE, a more robust seven-DOF grasping prediction network is obtained.

[0013] S6, inputting the RGB-D image of the object to be grasped into the seven-degree-of-freedom grasping prediction network obtained in step S5, to obtain the seven-degree-of-freedom pose information of the object to be grasped;

[0014] S7. Perform camera intrinsic parameter calibration, and use the Tsai-Lenz algorithm to calibrate the camera and the robot arm hand-eye to obtain the hand-eye relationship matrix between the camera coordinate system and the robot base coordinate system. Map the seven-degree-of-freedom posture information obtained in step 6 from the camera coordinate system to the robot base coordinate system, and use Moveit to deploy the robot arm to move to the position to be grasped to perform object grasping.

[0015] Further, the method specifically includes the following steps:

[0016] S11. Initialize the grasping posture and force closure index in the original data set, and obtain the grasping posture and force closure index score from the original data set. fc , as initial data;

[0017] S12. Calculate the normal vector and the connecting line vector of the contact point, use the K-nearest neighbor (KNN) method to find the contact points c1 and c2 that are closest to the object to be grasped and the robot gripper in a grasping posture, and select 15 nearby points to fit the surface. Calculate the first-order partial derivative of the surface at the contact points c1 and c2 through the fitted surface equation to obtain the normal vector of the surface at the two contact points. and the line vector connecting the two contact points By formula

[0018]

[0019] Calculate the cosine similarity between the two normal vectors and the connecting vector as the torque balance index sore s , θ1, θ2 represent and The angle of

[0020] S13, calculate the centroid distance index, calculate the contact point center coordinates C by the KNN method of S12, and obtain the centroid coordinates O of the object point cloud by calculating the average coordinates of the point cloud, using the formula

[0021] score d =||OC||2

[0022] Calculate the distance between the object to be grasped and the center of the gripper, and use this distance as the object's center of mass distance index score d ;

[0023] S14, calculate the contact point distribution balance index, and use the contact point normal vector obtained by S12 Calculate the contact point distribution balance index; the formula is as follows

[0024]

[0025] score b It is an index of contact point distribution balance, which is used to indicate the distribution balance of the contact points between the gripper and the object.

[0026] S15, calculate the contact point non-coplanarity index, using the contact point center coordinates C and the object mass center coordinates O obtained in step S13, using the formula

[0027]

[0028] Calculate the contact point non-coplanarity index score v ;

[0029] S16, comprehensively evaluate the grasping posture, use the comprehensive evaluation formula for the original grasping posture, and update the score of each grasping posture. The formula is as follows:

[0030] score=α×score s +β×score d +γ×score fc +μ×(score b ×score v )

[0031] In the formula, α, β, γ, μ are weight coefficients, and score represents the comprehensive score of the crawl;

[0032] S21, split the input point cloud data into Cartesian coordinates and feature descriptors, use multi-layer Hough voting-based point set abstraction and feature enhancement, sample and extract features of the point cloud data through multi-layer Hough voting point set abstraction layer and PnP3D module to enhance feature representation; then perform feature fusion and upsampling: fuse and upsample features at different levels to obtain the final feature representation;

[0033] S3, the grasping feature extraction network is implemented through a multi-layer perceptron to extract the features of the sampled point cloud, grasp the required position and the candidate set of points to be grasped and the set of feature vectors;

[0034] S41, select four captured feature vectors X1, X2, X3, X4 of different scales as the input of the multi-stream channel association attention network, and obtain the merged feature map X through add feature fusion;

[0035] S42, perform convolution operations on X through the query convolution layer, key convolution layer and value convolution layer respectively to obtain the projected query feature p q , projection key feature p k and the value matrix p v ;

[0036] S43. Calculate the similarity matrix mat between the projection query feature and the projection key feature s

[0037] mat s =p k ·P q T

[0038] S44, mat obtained by S43 s Perform the maximum pooling operation and use the Softmax activation function to obtain the channel correlation matrix mat c ;

[0039] S45, place the mat c Matrix and projection value features p v Perform a dot product operation and weight the comprehensive feature map X to calculate the output transition feature map. The formula is:

[0040] xm =mat c ·p v +ρX

[0041] x m is the transition feature map, mat c is the channel correlation matrix calculated by S44, and ρ is a learnable weight;

[0042] S46, the four different scales of the captured feature vectors X1, X2, X3, X4 of S41 are used by formula

[0043]

[0044] X o Represents the output feature map, X k represents X1, X2, X3, X4;

[0045] Get the output feature vector X o , through the multi-layer perceptron, nonlinear mapping is performed, and the feature vector X is gradually transformed through the hierarchical structure of the network. o Convert to the required seven-degree-of-freedom grasping posture parameter group (x, y, z, ψ, θ, φ, w); where (x, y, z) represents the Cartesian coordinates of the gripper in the camera coordinate system, (φ, θ, φ) represents the rotation around the x, y, z coordinate axes, and w represents the width of the gripper;

[0046] S5, adding a discriminator network between step S3 and step S4, using the feature vector set to be captured obtained in step S3 and the real label to perform generative adversarial training through the Wasserstein loss function to optimize the overall neural network parameters; specifically including:

[0047] S51. Build a discriminator network through a multi-layer perceptron;

[0048] S52, input the intermediate feature vector output by S3 and the real grasping posture label in the data set into the discriminator network, and the discriminator outputs the judgment value of the false grasping perspective and the judgment value of the real grasping perspective;

[0049] S53. Define the discriminator loss function:

[0050]

[0051] L D is the loss function of the discriminator D, Indicates that when the input x is from the real data distribution P r When sampling in, the expected value of the discriminator output is, Represents input Generate data from distribution P gThe expected value of the discriminator output, λ represents the parameter that controls the strength of the penalty term in the loss function, Represents the discriminator about the input The L2 norm of the gradient of ;

[0052] S54. Define the seven-degree-of-freedom grasp prediction network loss function:

[0053]

[0054] L G represents the loss function of the generator G, N cls represents the number of classification task categories, represents the classification loss function, c i represents the model prediction category, represents the true category, t represents the training round, λ1(t) represents the time weight coefficient used to adjust the contribution of each item in the loss function, N reg represents the number of items related to the regression task, represents the weight of the sample selected for regression, 1(·) is the indicator function, indicating It takes 1 when it is, otherwise it takes 0. represents the regression loss function, s ij is the predicted regression value, is the true regression value, λ2 is the regularization weight, represents the generator weight;

[0055] S55. During model training, the losses of the discriminator and the seven-degree-of-freedom grasp prediction network are calculated respectively, and the parameters of each part of the model are calculated by back propagation;

[0056] S6, inputting the RGB-D image of the object to be grasped into the seven-degree-of-freedom grasping prediction network obtained in step 5 to obtain the seven-degree-of-freedom pose information of the object to be grasped;

[0057] S7. Perform camera intrinsic parameter calibration, and use the Tsai-Lenz algorithm to calibrate the camera and the robot arm hand-eye to obtain the hand-eye relationship matrix between the camera coordinate system and the robot base coordinate system. Map the seven-degree-of-freedom posture information obtained in step 6 from the camera coordinate system to the robot base coordinate system, and use Moveit to deploy the robot arm to move to the position to be grasped to perform object grasping.

[0058] Furthermore, in step S11, the original data set used is the public data set GraspNet-1Billion.

[0059] Furthermore, step S55 is specifically as follows:

[0060] In each training cycle, the learning rate and batch normalization momentum are adjusted, the model is set to training mode, forward and backward propagation is performed through batch data, the loss is calculated and the network parameters are updated, the discriminator is trained multiple times to ensure the balance between the generator and the discriminator, and during the generator training process, the total loss, including the generator loss and the grasping loss, is calculated, and the generator parameters are updated.

[0061] The beneficial effects of the present invention are:

[0062] Compared with traditional multi-degree-of-freedom grasping methods, the present invention has significant advantages in grasping stability, contact point distribution balance and grasping prediction accuracy. It can effectively overcome the defects of existing methods and has broad application prospects and important technical significance.

[0063] Traditional six-degree-of-freedom grasping methods mainly rely on force closure indicators, with a single evaluation standard and easily ignoring the distribution balance of contact points, resulting in insufficient grasping stability and accuracy. In addition, the feature extraction process is simple, the generalization ability and robustness of the model are weak, and it is difficult to adapt to complex environments and diverse objects. The present invention proposes an enhanced robot grasping method, which comprehensively evaluates the grasping posture by introducing an enhanced grasping evaluation method and generative adversarial learning, taking into account a variety of scoring indicators such as grasping stability, contact point distribution balance, centroid distance and contact point non-coplanarity; based on the Hough voting set abstract layer network and PnP3D network feature extractor, combined with a multi-stream channel association attention network, finely extracts point cloud features and improves grasping prediction accuracy; by generating adversarial optimization feature extraction results, the model has higher robustness and accuracy under different environments and object changes, significantly improves the grasping success rate, overcomes the limitations of traditional methods such as lack of precision and grasping robustness, and has broad application prospects and important technical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0065] Figure 2 It is the actual deployment diagram of the present invention;

[0066] Figure 3 This is a structural diagram of the multi-stream channel associated attention network model of the present invention. DETAILED DESCRIPTION

[0067] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0068] This paper proposes a robot seven-degree-of-freedom grasping method based on generative adversarial learning. By combining the enhanced grasping evaluation method (EGE), the feature extraction network and the multi-stream channel association attention network, and the introduction of the generative adversarial method, the accuracy and stability of grasping prediction are significantly improved. This method comprehensively considers the contact point distribution balance and grasping stability, improves the existing grasping scoring mechanism, and ensures that the grasping posture is closer to the true value. In addition, through camera intrinsic parameter calibration and hand-eye calibration, the grasping parameters output by the network are deployed in the actual environment, realizing the efficient grasping operation of the robot in a complex environment. This method not only improves the grasping success rate, but also enhances the robustness of the robot in actual grasping, providing strong support for the development of intelligent robot grasping technology.

[0069] like Figure 1 As shown, this embodiment provides a robot seven-degree-of-freedom grasping method based on generative adversarial learning. The method process is referenced Figure 1 , the actual deployment is as follows Figure 2 As shown, the following steps are included:

[0070] S1. Obtain and input the original data set containing point cloud data through the Internet, actual measurement, etc., enhance the original data set through the enhanced grasping evaluation method, use the enhanced grasping evaluation method (Enhanced GraspEvaluation, EGE), introduce a new scoring indicator, the EGE method comprehensively considers the distribution balance of contact points and the stability of grasping, and improves the existing grasping scoring mechanism as a learning tendency; EGE specifically includes:

[0071] S11. Initialize the grasping posture and force closure index in the original data set, and obtain the grasping posture and force closure index score from the original data set. fc , as the initial data, the original dataset used is the public dataset GraspNet-1Billion.

[0072] S12. Calculate the normal vector and line vector of the contact point, use the K-nearest neighbor (KNN) method to find the contact points c1 and c2 that are closest to the object to be grasped and the robot gripper in a grasping posture, and select 15 nearby points to fit the surface. Calculate the first-order partial derivative of the surface at the contact points c1 and c2 through the fitted surface equation to obtain the normal vector of the surface at the two contact points. and the line vector connecting the two contact points By formula

[0073]

[0074] Calculate the cosine similarity between the two normal vectors and the connecting vector as the torque balance index score s , θ1, θ2 represent and Angle.

[0075] S13, calculate the centroid distance index, calculate the contact point center coordinates C by the KNN method of S12, and obtain the centroid coordinates O of the object point cloud by calculating the average coordinates of the point cloud, using the formula

[0076] score d =||OC||2

[0077] Calculate the distance between the object to be grasped and the center of the gripper, and use this distance as the object's center of mass distance index score d .

[0078] S14, calculate the contact point distribution balance index, and use the contact point normal vector obtained by S12 Calculate the contact point distribution balance index. The formula is as follows

[0079]

[0080] score b It is an index of contact point distribution balance, which is used to indicate the distribution balance of the contact points between the gripper and the object.

[0081] S15, calculate the contact point non-coplanarity index, using the contact point center coordinates C and the object mass center coordinates O obtained in step 13, using the formula

[0082]

[0083] Calculate the contact point non-coplanarity index score v .

[0084] S16, comprehensively evaluate the grasping posture, use the comprehensive evaluation formula for the original grasping posture, and update the score of each grasping posture. The formula is as follows:

[0085] score=α×score s +β×score d +γ×score fc +μ×(score b ×score v )

[0086] In the formula, α, β, γ, and μ are weight coefficients, and score represents the comprehensive score of the crawl.

[0087] S2. Use the backbone network (Backbone) improved by Hough voting (Votenet) and a plug-and-play enhanced feature (PnP3D) for point cloud downsampling and feature extraction, including:

[0088] S21. Split the input point cloud data into Cartesian coordinates and feature descriptors, use multi-layer Hough voting-based point set abstraction and feature enhancement, sample and extract features of the point cloud data through multi-layer Hough voting point set abstraction layer and PnP3D module to enhance feature representation; then perform feature fusion and upsampling: fuse and upsample features at different levels to obtain the final feature representation.

[0089] S3, the grasping feature extraction network is implemented through a multi-layer perceptron to extract the features of the sampled point cloud, grasp the required position and the candidate set of points to be grasped and the set of feature vectors;

[0090] S4, through the multi-stream channel association attention network, the six-degree-of-freedom posture and the seven-degree-of-freedom information of the gripper width degree of freedom are obtained, such as Figure 3 As shown, specifically including:

[0091] S41, select four captured feature vectors X1, X2, X3, X4 of different scales as the input of the multi-stream channel association attention network, and obtain the merged feature map X through add feature fusion;

[0092] S42, perform convolution operations on X through the query convolution layer, key convolution layer and value convolution layer respectively to obtain the projected query feature p q , the projection key feature p k and the value matrix p v ;

[0093] S43. Calculate the similarity matrix mat between the projection query feature and the projection key feature s

[0094] mat s =p k ·p q T

[0095] S44, mat obtained by S43 s Perform the maximum pooling operation and use the Softmax activation function to obtain the channel correlation matrix mat c ;

[0096] S45, place the mat c Matrix and projection value features p v Perform a dot product operation and weight the comprehensive feature map X to calculate the output transition feature map. The formula is:

[0097] x m =mat c ·p v +ρX

[0098] x mis the transition feature map, mat c is the channel correlation matrix calculated by S44, and ρ is a learnable weight;

[0099] S46, the four different scales of the captured feature vectors X1, X2, X3, X4 of S41 are used by formula

[0100]

[0101] X o Represents the output feature map, X k Represents X1, X2, X3, X4.

[0102] Get the output feature vector X o , through the multi-layer perceptron, nonlinear mapping is performed, and the feature vector X is gradually transformed through the hierarchical structure of the network. o Converted into the required seven-degree-of-freedom grasping posture parameter group (x, y, z, ψ, θ, φ, w) (the grasping position posture parameters are not explicitly provided at the input, but they are generated in the output part of the neural network / multi-layer perceptron and continuously updated to the true value through the optimization algorithm); where (x, y,) represents the Cartesian coordinates of the gripper in the camera coordinate system, (ψ, θ, φ) represents the rotation around the x, y, z coordinate axes, and w represents the width of the gripper.

[0103] S5. In order to further improve the accuracy of grasp prediction, the present invention adds a discriminator network between the feature extraction network and the grasp optimization network. By identifying the intermediate features of the feature extraction network and the real labels of the data set optimized by EGE, a more robust seven-degree-of-freedom grasp prediction network is obtained.

[0104] A discriminator network is added between step 3 and step 4, and the feature vector set to be captured obtained in step 3 and the real label are used to generate adversarial training through the Wasserstein loss function to optimize the overall neural network parameters. Specifically, it includes:

[0105] S51. Build a discriminator network through a multi-layer perceptron;

[0106] S52, input the intermediate feature vector output by S3 and the real grasping posture label in the data set into the discriminator network, and the discriminator outputs the judgment value of the false grasping perspective and the judgment value of the real grasping perspective;

[0107] S53. Define the discriminator loss function:

[0108]

[0109] L D is the loss function of the discriminator D, Indicates that when the input x is from the real data distribution P r When sampling in, the expected value of the discriminator output is, Represents input Generate data from distribution P g The expected value of the discriminator output, λ represents the parameter that controls the strength of the penalty term in the loss function, Represents the discriminator about the input The L2 norm of the gradient of ;

[0110] S54. Define the seven-degree-of-freedom grasp prediction network loss function:

[0111]

[0112] L G represents the loss function of the generator G, B cls represents the number of classification task categories, represents the classification loss function, c i represents the model prediction category, represents the true category, t represents the training round, λ1(t) represents the time weight coefficient used to adjust the contribution of each item in the loss function, N reg represents the number of items related to the regression task, represents the weight of the sample selected for regression, 1(·) is the indicator function, indicating It takes 1 when it is, otherwise it takes 0. represents the regression loss function, s ij is the predicted regression value, is the true regression value, λ2 is the regularization weight, represents the generator weight.

[0113] S55. During model training, the losses of the discriminator and the seven-degree-of-freedom grasping prediction network are calculated separately, and the parameters of each part of the model are calculated through back propagation.

[0114] In each training cycle, the learning rate and batch normalization momentum are adjusted, the model is set to training mode, forward and backward propagation is performed through batch data, the loss is calculated and the network parameters are updated, the discriminator is trained multiple times to ensure the balance between the generator and the discriminator, and during the generator training process, the total loss, including the generator loss and the grasping loss, is calculated, and the generator parameters are updated.

[0115] S6. Input the RGB-D image of the object to be grasped into the seven-degree-of-freedom grasping prediction network obtained in step 5 to obtain the seven-degree-of-freedom pose information of the object to be grasped.

[0116] S7. Perform camera intrinsic parameter calibration, and use the Tsai-Lenz algorithm (an algorithm for hand-eye calibration) to perform camera and robotic arm hand-eye calibration to obtain the hand-eye relationship matrix between the camera coordinate system and the robot base coordinate system. Map the seven-degree-of-freedom posture information obtained in step 6 from the camera coordinate system to the robot base coordinate system, and use Moveit (an open source, general robot operating system motion framework) to deploy the robotic arm to move to the position to be grasped to perform object grasping.

[0117] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A robot seven-degree-of-freedom grasping method based on generative adversarial learning, characterized in that: The following steps are involved: S1. Obtain and input the original data set containing point cloud data, perform data enhancement on the original data set through the enhanced grasping evaluation method, use the enhanced grasping evaluation method EGE, introduce a new scoring indicator, comprehensively consider the distribution balance of contact points and the stability of grasping, and improve the existing grasping scoring mechanism as a learning tendency; S2, using the backbone network Backbone improved based on Hough voting Vonet and a plug-and-play enhanced feature PnP3D for point cloud downsampling and feature extraction; S3, the grasping feature extraction network is implemented through a multi-layer perceptron to extract the features of the sampled point cloud, grasp the required position and the candidate set of points to be grasped and the set of feature vectors; S4, obtain the six-degree-of-freedom posture and seven-degree-of-freedom information of the gripper width through the multi-stream channel association attention network; S5. A discriminator network is added between the feature extraction network and the grasping optimization network. By identifying the intermediate features of the feature extraction network and the real labels of the dataset optimized by EGE, a more robust seven-DOF grasping prediction network is obtained. S6, inputting the RGB-D image of the object to be grasped into the seven-degree-of-freedom grasping prediction network obtained in step S5, to obtain the seven-degree-of-freedom pose information of the object to be grasped; S7. Perform camera intrinsic parameter calibration, and use the Tsai-Lenz algorithm to perform camera and robot hand-eye calibration to obtain the hand-eye relationship matrix between the camera coordinate system and the robot base coordinate system. Map the seven-degree-of-freedom posture information obtained in step S6 from the camera coordinate system to the robot base coordinate system, and use Moveit to deploy the robot arm to move to the position to be grasped to perform object grasping.

2. The robot seven-degree-of-freedom grasping method based on generative adversarial learning according to claim 1 is characterized in that: The specific steps include: S11. Initialize the grasping posture and force closure index in the original data set, and obtain the grasping posture and force closure index score from the original data set. fe , as initial data; S12. Calculate the normal vector and the connecting line vector of the contact point, use the K-nearest neighbor (KNN) method to find the contact points c1 and c2 that are closest to the object to be grasped and the robot gripper in a grasping posture, and select 15 nearby points to fit the surface. Calculate the first-order partial derivative of the surface at the contact points c1 and c2 through the fitted surface equation to obtain the normal vector of the surface at the two contact points. and the line vector connecting the two contact points By formula Calculate the cosine similarity between the two normal vectors and the connecting vector as the torque balance index score s , θ1, θ2 represent and The angle of S13, calculate the centroid distance index, calculate the contact point center coordinates C by the KNN method of S12, and obtain the centroid coordinates O of the object point cloud by calculating the average coordinates of the point cloud, using the formula score d =||OC||2 Calculate the distance between the object to be grasped and the center of the gripper, and use this distance as the object's center of mass distance index score d ; S14, calculate the contact point distribution balance index, and use the contact point normal vector obtained by S12 Calculate the contact point distribution balance index; the formula is as follows score b It is an index of contact point distribution balance, which is used to indicate the distribution balance of the contact points between the gripper and the object. S15, calculate the contact point non-coplanarity index, using the contact point center coordinates C and the object mass center coordinates O obtained in step S13, using the formula Calculate the contact point non-coplanarity index score v ; S16, comprehensively evaluate the grasping posture, use the comprehensive evaluation formula for the original grasping posture, and update the score of each grasping posture. The formula is as follows: score=α×score s +β×score d +γ×score fc +μ×(score b ×score v ) In the formula, α, β, γ, μ are weight coefficients, and score represents the comprehensive score of the crawl; S21, split the input point cloud data into Cartesian coordinates and feature descriptors, use multi-layer Hough voting-based point set abstraction and feature enhancement, sample and extract features of the point cloud data through multi-layer Hough voting point set abstraction layer and PnP3D module to enhance feature representation; then perform feature fusion and upsampling: fuse and upsample features at different levels to obtain the final feature representation; S3, the grasping feature extraction network is implemented through a multi-layer perceptron to extract the features of the sampled point cloud, grasp the required position and the candidate set of points to be grasped and the set of feature vectors; S41, select four captured feature vectors X1, X2, X3, X4 of different scales as the input of the multi-stream channel association attention network, and obtain the merged feature map X through add feature fusion; S42, perform convolution operations on X through the query convolution layer, key convolution layer and value convolution layer respectively to obtain the projected query feature p q , the projection key feature p k and the value matrix p v ; S43. Calculate the similarity matrix mat between the projection query feature and the projection key feature s mat s =p k ·p q T S44, mat obtained by S43 s Perform the maximum pooling operation and use the Softmax activation function to obtain the channel correlation matrix mat c ; S45, place the mat c Matrix and projection value features p v Perform a dot product operation and weight the comprehensive feature map X to calculate the output transition feature map. The formula is: x m =mat c ·p v +ρX x m is the transition feature map, mat c is the channel correlation matrix calculated by S44, and ρ is a learnable weight; S46, the four different scales of the captured feature vectors X1, X2, X3, X4 of S41 are used by formula X o Represents the output feature map, X k represents X1, X2, X3, X4; Get the output feature vector X o , through the multi-layer perceptron, nonlinear mapping is performed, and the feature vector X is gradually transformed through the hierarchical structure of the network. o Convert to the required seven-degree-of-freedom grasping posture parameter group (x, y, z, ψ, θ, φ, w); where (x, y, z) represents the Cartesian coordinates of the gripper in the camera coordinate system, (ψ, θ, φ) represents the rotation around the x, y, z coordinate axes, and w represents the width of the gripper; S5, adding a discriminator network between step S3 and step S4, using the feature vector set to be captured obtained in step S3 and the real label to perform generative adversarial training through the Wasserstein loss function to optimize the overall neural network parameters; specifically including: S51. Build a discriminator network through a multi-layer perceptron; S52, input the intermediate feature vector output by S3 and the real grasping posture label in the data set into the discriminator network, and the discriminator outputs the judgment value of the false grasping perspective and the judgment value of the real grasping perspective; S53. Define the discriminator loss function: L D is the loss function of the discriminator D, Indicates that when the input x is from the real data distribution P r When sampling in, the expected value of the discriminator output is, Represents input Generate data from distribution P g The expected value of the discriminator output, λ represents the parameter that controls the strength of the penalty term in the loss function, Represents the discriminator about the input The L2 norm of the gradient of ; S54. Define the seven-degree-of-freedom grasp prediction network loss function: L G represents the loss function of the generator G, N cls represents the number of classification task categories, represents the classification loss function, c i represents the model prediction category, represents the true category, t represents the training round, λ1(t) represents the time weight coefficient used to adjust the contribution of each item in the loss function, N reg represents the number of items related to the regression task, represents the weight of the sample selected for regression, 1(·) is the indicator function, indicating It takes 1 when it is, otherwise it takes 0. represents the regression loss function, s ij is the predicted regression value, is the true regression value, λ2 is the regularization weight, represents the generator weight; S55. During model training, the losses of the discriminator and the seven-degree-of-freedom grasp prediction network are calculated respectively, and the parameters of each part of the model are calculated by back propagation; S6, inputting the RGB-D image of the object to be grasped into the seven-degree-of-freedom grasping prediction network obtained in step 5 to obtain the seven-degree-of-freedom pose information of the object to be grasped; S7. Perform camera intrinsic parameter calibration, and use the Tsai-Lenz algorithm to calibrate the camera and the robot arm hand-eye to obtain the hand-eye relationship matrix between the camera coordinate system and the robot base coordinate system. Map the seven-degree-of-freedom posture information obtained in step 6 from the camera coordinate system to the robot base coordinate system, and use Moveit to deploy the robot arm to move to the position to be grasped to perform object grasping.

3. The robot seven-degree-of-freedom grasping method based on generative adversarial learning according to claim 2 is characterized in that: In step S11, the original data set used is the public data set GraspNet-1Billion.

4. The robot seven-degree-of-freedom grasping method based on generative adversarial learning according to claim 2 is characterized in that: Step S55 is specifically as follows: In each training cycle, the learning rate and batch normalization momentum are adjusted, the model is set to training mode, forward and backward propagation is performed through batch data, the loss is calculated and the network parameters are updated, the discriminator is trained multiple times to ensure the balance between the generator and the discriminator, and during the generator training process, the total loss, including the generator loss and the grasping loss, is calculated, and the generator parameters are updated.

Citation Information

Patent Citations

  • Multispecific binding molecules having specificity to dystroglycan and laminin-2

    CN110997714A

  • Point disturbance adversarial attack method for three-dimensional target tracking model

    CN113808165A

  • Mesoscale eddy identification method and system based on multi-task learning

    CN116895023A

  • Robot digital twin grabbing method and device based on virtual-real collaborative learning strategy

    CN117392663A

  • Multidimensional robot movement control - by dividing each movement into component steps and storing movement information which is later reproduced

    DE2750302A1

Cited By

  • Auxiliary walking device based on multi-mode perception and interaction and control method thereof

    CN120516665A