Point cloud-based 6-DOF human-robot collaborative posture planning and humanoid interactive motion generation method
Through the 6-DOF item handover attitude planning and improved dynamic motion primitive model based on point cloud, the problem of uncertainty in attitude planning and motion trajectory in item handover tasks in human-machine collaboration is solved, and safe, reliable, and Humanized item handover enhances the application value of the collaboration system.
Patent Information
- Application Number
- CN202410003701.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-02
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-01-02
AI Technical Summary
In the prior art, in the object handover task in human-machine collaboration, the degree of freedom of robot grasping posture planning is limited, and the generated motion trajectory uncertainty leads to an increase in security risks, making it difficult to achieve safe, reliable, and human-like interactive collaboration.
Using a point cloud-based 6-DOF item handover attitude planning method, combined with FasterR-CNN and GraspNet, the handover point cloud is extracted and a stable and secure 6-degree of freedom grabbing posture is generated; using an improved dynamic motion primitive (DMP) model Imitate human teaching trajectories and generate highly adaptable interactive motion trajectories.
It realizes safe, stable and human-like handover of items on diverse items and new objects, reduces the risk of contact between robotic arms and human bodies, and improves the reliability and generalization ability of handover tasks.
Smart Images

Figure CN117726654B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the research field of human-machine interactive collaboration and robot operation, and specifically relates to a point cloud-based 6-DOF human-machine collaborative posture planning and humanoid interactive motion generation system design method, including two subtasks: a point cloud-based 6-DOF object handover posture planning method in human-machine collaboration and a humanoid interactive motion generation model design based on an improved dynamic motion primitive (DMP). Background Art
[0002] In human-robot collaboration, humans are at the center of the entire collaborative process, and robots are equivalent to human assistants, which can assist humans in doing repetitive and tiring work. Due to its important application value in assisting labor, helping the elderly and the disabled, it has become an important research direction in the field of robotics. Object handover is one of the core and key tasks in the interactive collaboration between humans and robots. How robots can safely and reliably take over various types of objects delivered in different postures in a human-like manner has always been a hot and difficult research issue in the field of human-robot collaboration.
[0003] The implementation of the object handover task in human-machine collaboration specifically includes two subtasks, (1) planning the robot's handover posture and (2) generating the motion trajectory for taking over the object. For subtask (1), it is necessary to consider the geometric shape and diverse postures of the target object and the occlusion of the human hand. Previous studies used methods such as GG-CNN for handover planning, but the grasping posture generated by the 2D image-based method can only grasp from a single direction, limiting the freedom of the grasping posture. In contrast, point cloud representation can provide richer 3D information, provide better support for 6D grasping, and have better generalization ability on new objects. In the present invention, a 6-DOF object handover posture planning method based on point cloud is provided, which realizes accurate, stable and safe handover posture planning. For subtask (2), the smoothness, naturalness and human-likeness of the robot's interactive movement are crucial, while the existing methods using RRT have some defects, such as the generated path may be too long, inconsistent in shape and uncertain. In actual handover tasks, the uncertainty of the trajectory path can easily lead to contact between the robot arm and the human body, increasing safety risks. In the present invention, a humanoid interactive motion generation model based on an improved dynamic motion primitive (DMP) is proposed, which can imitate human-taught trajectories and generate humanoid trajectories with shorter paths and predictable shapes. Combining the two models, the present invention provides a safe, reliable, and humanoid human-machine interactive collaboration system. This has important application value in human-machine collaborative tasks. Summary of the invention
[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a 6-DOF human-machine collaborative posture planning and humanoid interactive motion generation system design method based on point cloud, which is specifically implemented through the following technologies:
[0005] In view of the diversity of types and postures of object handover in human-machine collaboration, the present invention proposes a point cloud-based 6-degree-of-freedom (6-DOF) object handover posture planning method, which includes two subtasks: handover point cloud extraction and 6-DOF grasping posture generation.
[0006] In the process of human-machine handover, in order to prevent the robot from touching the human hand, it is necessary to accurately predict the graspable area of the object. The FasterR-CNN model is used in the present invention to accurately identify the position and size of the human hand and the object in the handover scene from the 2D image. The human hand position area is defined as an untouchable area. Considering the safety of the interaction, the end of the robot can only take over the object in the remaining area on the object. Therefore, combined with the scene depth (Depth map) and camera calibration information, the detected human hand and object images are transformed into 3D space. Through the point cloud segmentation algorithm and statistical filtering technology, the local point cloud on the object that is not blocked by the hand is extracted as the handover point cloud, and it is used as the input of the grasping planning module.
[0007] The 6-DOF grasping posture generation subtask needs to predict a set of 6-DOF grasping candidates based on the input handover point cloud, and output the most feasible 6-DOF grasping posture that can stably take over the object. The present invention constructs a 6-DOF grasping planning model based on GraspNet, where GraspNet is an end-to-end deep neural network that can predict candidate 6-DOF grasping postures on a 3D point cloud. However, not all candidate grasps are suitable for the robot to stably take over the object. The present invention proposes a grasping selection strategy that takes into account both safety and stability to ensure the generation of a more reliable and safe 6-DOF grasping plan.
[0008] The above point cloud-based 6-DOF object handover posture planning method proposed in the present invention has two key advantages. On the one hand, it helps to achieve safe grasping for human-machine handover tasks. On the other hand, it can generate feasible 6-DOF grasping postures from single-view point clouds and has good generalization ability for new objects of different shapes and sizes.
[0009] Aiming at the problem of generating humanoid interactive motion, the present invention uses imitation learning method to learn teaching interactive motion and generate interactive motion trajectory that can adapt to new grasping posture. A humanoid interactive motion generation model based on improved dynamic motion primitive (DMP) method is proposed. Different from the position learning in Cartesian space, the learning of motion posture needs to consider additional constraints, so the position and posture need to be learned separately. The present invention combines the DMP for learning position and the Orientation-DMP for learning posture to form a humanoid interactive motion generation model, which is used to learn the posture trajectory characteristics in Cartesian space.
[0010] The original DMP and Orientation-DMP models can only learn actions from a single teaching trajectory. Since a single teaching data is often affected by noise and errors, the original DMP and Orientation-DMP models may not be able to generate the required motion trajectory. Therefore, the present invention proposes to introduce Gaussian mixture model (GMM) and Gaussian mixture regression (GMR) into the DMP and Orientation-DMP models. Through GMM and GMR, probabilistic reference trajectories can be generated from multiple trajectories, and DMP and Orientation-DMP learn motion features from the probabilistic reference trajectory, which effectively improves the adaptability of the DMP and Orientation-DMP models and overcomes the limitation that the original DMP and Orientation-DMP can only learn from a single trajectory.
[0011] Based on the above method, the human-machine collaboration system constructed by the present invention can safely transfer objects with people. Since it has good generalization and adaptability and can generate human-like interactive movements, it has high practical application value in human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 It is a flow chart of the present invention.
[0013] Figure 2 This is the effect of FasterR-CNN recognizing hands and objects.
[0014] Figure 3 It is a point cloud-based 6-DOF human-machine collaborative posture planning flow chart designed by the present invention.
[0015] Figure 4 It is a process diagram for generating intersection point cloud.
[0016] Figure 5 This is an application effect diagram of the point cloud-based 6-DOF human-machine collaborative posture planning designed by the present invention.
[0017] Figure 6 It is a humanoid interactive motion generation graph based on the improved Dynamic Motion Primitive (DMP).
[0018] Figure 7 It is a three-dimensional picture of humanoid interactive motion.
[0019] Figure 8 The present invention realizes the handover object diagram. DETAILED DESCRIPTION
[0020] The present invention is further described below in conjunction with the accompanying drawings.
[0021] Figure 1This is the specific process of the human-machine collaboration system designed by the present invention, which includes two stages: (1) 6-DOF human-machine collaboration posture planning based on point cloud and (2) human-like interactive motion generation system design. The implementation details of stage (1) are as follows: Figure 2 As shown, the point cloud-based 6-DOF object handover posture planning method in human-machine collaboration proposed by the present invention includes two parts: extraction of handover point cloud and generation of 6-DOF grasping posture.
[0022] In the junction point cloud extraction stage: Faster R-CNN is used to detect the position of objects and hands in the 2D scene, and the identified bounding boxes are used to segment the hands and objects. Figure 3 The recognition effect diagram is shown, where the red box is the recognized hand position and the yellow box is the object to be handed over. The area within the object bounding box is considered graspable, but when it overlaps with the hand bounding box, the overlapping area is considered ungraspable. In addition, the area within the hand bounding box is also considered ungraspable. Based on the above segmentation conditions, a mask image of the grasping area can be generated on the object image, which is simple and computationally efficient. Combining the RGB image, depth image and mask image, the intersection point cloud in the scene can be extracted. Figure 4 The detailed process of intersection point cloud extraction is demonstrated.
[0023] Since the intersection point cloud obtained from the mask of the graspable area may contain background points and outliers, this will affect the accuracy of the grasp prediction. The present invention adopts a two-step approach to solve this problem. First, the point cloud is plane segmented using the Random Sample Consensus (RANSAC) algorithm to effectively filter out background points. Subsequently, a statistical filtering technique is applied to remove obvious outliers (see Figure 4 ). This method helps to obtain a more accurate intersection point cloud, thus improving the overall accuracy of grasp prediction.
[0024] 6-DOF grasping posture generation stage: This paper proposes a grasping planning model based on point cloud data. The model also consists of two parts: 6-DOF grasping planning generation and optimal grasping posture selection (see Figure 2 ).
[0025] The intersection point cloud is input, and the end-to-end network GraspNet is used in the present invention to generate a variety of 6-DOF candidate grasping postures. The GraspNet network can predict the 6-DOF pose of the parallel gripper relative to the camera coordinate system on the 3D point cloud, where the 6-DOF pose is represented by the translation along the X, Y, and Z axes in the camera coordinate system and the rotation around the X, Y, and Z axes. However, due to the explicit constraint problem, accurately predicting the rotation matrix is still challenging for neural networks. Therefore, the network adopts a clever method to decouple the prediction of the rotation matrix representing the direction into two independent parts: viewpoint classification and in-plane rotation angle prediction. This decomposition strategy enables neural networks to effectively learn and predict 6-DOF grasping postures.
[0026] GraspNet uses PointNet++ as the backbone network, which includes three networks: Approach network, Operation network and Tolerance network. Each network consists of a multi-layer perceptron. The Approach network is used to predict feasible grasping points and their viewpoints. The loss function L of the Approach network is A The definition is as follows:
[0027]
[0028] Among them, c i Indicates the binary prediction of whether point i is graspable. If point i is positive, then Assign a value of 1, if point i is negative, assign a value of 0. cls is the total number of predicted points; λ1 is a constant representing the learning rate; s ij represents the prediction confidence score of the j-th viewpoint of point i, is the corresponding truth value; N reg is the total number of viewpoints; v ij represents the approximate vector prediction of the jth virtual view of graspable point i, is the corresponding truth value; represents the angle difference, which is the loss term in the first indicator function 1(), which limits the loss of the approximate vector; L cls represents the softmax loss of binary classification, L reg represents the smoothed L1 loss function.
[0029] The Operation network is responsible for predicting the rotation angle in the plane, the distance the gripper moves forward, the gripper width, and the confidence.
[0030] Loss function L R Defined as:
[0031]
[0032] In the formula, R ij represents the degree of the angle grouping of the j-th viewpoint of point i, is the corresponding truth value; S ij , W ij are the grasp confidence score and grasp width of the j-th viewpoint of point i, respectively. is the corresponding truth value;
[0033] is the sigmoid cross entropy loss function of the d-th group proximity distance, is the smoothed L1 loss function of the proximity distance of the dth group; λ2 and λ3 are both constants, representing the learning rate; K is the total number of angle groups; N cls is the total number of predicted points, N reg is the total number of viewpoints.
[0034] The Tolerance Network introduces a representation called Grasp Affinity Fields (GAFs) to predict the tolerance of each grasp pose to perturbations. When the ground truth of the grasp pose is provided, the tolerance network searches its neighborhood in the ball space and sets the grasp pose with a grasp score s>0.5 and the farthest distance from the ground truth as the target of GAFs.
[0035] The loss function L of the Tolerance network F The definition is as follows:
[0036]
[0037] Where T ij represents the maximum disturbance that the grasping posture of the j-th viewpoint of point i can resist, is the corresponding true value; K is the total number of angle groups; N reg is the total number of viewpoints; is the smoothed L1 loss function of the proximity distance of the d-th group.
[0038] During training, the formula L for minimizing the GraspNet parameters is as follows:
[0039] L=L A ({c i},{s ij})+αL R (R ij ,S ij ,W ij )+βL F (T ij ) (4)
[0040] L A is the loss function of the Approach network, ci Indicates the binary prediction of whether point i is graspable, s ij represents the prediction confidence score of the j-th viewpoint of point i; L R is the loss function of the Operation network, Rij represents the degree of the angle grouping of the j-th viewpoint of point i, S ij , W ij are the grasp confidence score and grasp width of the j-th viewpoint of point i respectively; L F is the loss function of the Tolerance network. ij It represents the maximum disturbance that the grasping posture of the j-th viewpoint of point i can resist; α and β are both constants, representing the learning rate.
[0041] The original GraspNet evaluates the reliability of grasping by predicting the stability score. However, in the scenario where a person and a machine hand over an item, a grasping posture with high stability may not be feasible. Stability and safety need to be considered at the same time. The grasping posture cannot be too close to the human hand or the edge of the object. To ensure that the generated grasping posture is located in the central area of the intersection point cloud, the present invention calculates the length, width and height of the three-dimensional bounding box of the intersection point cloud, and screens out the candidate grasping postures located in the central area of the intersection point cloud. In addition, if the predicted grasping direction is close to the direction in which a person delivers an object, it is considered an unsafe grasping posture. The postures in each direction are evaluated through the above filtering, and the candidate grasping posture with the highest score is selected as the final 6-DOF grasping posture to ensure that the grasping process is safer and more reliable. Figure 5 The effect diagram of the method for generating 6-DOF grasping posture in the human-machine handover task proposed by the present invention is shown.
[0042] In the humanoid interactive motion generation stage: the present invention proposes a humanoid interactive motion generation model based on an improved dynamic motion primitive (DMP). Figure 1 Task (2) describes the specific process of generating interactive motion trajectories.
[0043] The present invention is used represents the set of teaching data, where t n,h ∈R I is the input, ξ n,h ∈R O It is important to note that the probabilistic encoding of the teaching allows the input t and the output ξ to represent different types of variables.
[0044] The present invention uses the GMM method to obtain the joint probability from the teaching data distributed:
[0045]
[0046] Where C represents the number of Gaussian components, and πc ,μ c and∑ c They represent the prior probability, mean and covariance of the Cth Gaussian component respectively; t is the input time series, ξ is the output trajectory point; is the normal distribution function. Then, a probability reference trajectory is retrieved from formula (5) through GMR Each point associated with it Mean and covariance Description. The probabilistic reference trajectory is used as the representative of the trajectory set, and the formula is as follows:
[0047]
[0048] In the formula, represents the probabilistic reference trajectory within time period t, are the mean and covariance of the probability reference trajectory; is the normal distribution function.
[0049] For the position trajectory and the quaternion trajectory describing the posture in the teaching trajectory, the present invention uses the GMM and GMR methods to generate the position probability reference trajectory and the quaternion probability reference trajectory respectively. Subsequently, these two probability reference trajectories are respectively input into the DMP model and the Orientation-DMP model to learn the trajectory features.
[0050] The basic concept of DMP is to mimic the motion pattern through a second-order dynamic system with self-stabilizing properties. The system has two main goals: first, to ensure convergence to the desired attractor point, thereby allowing the modification of the trajectory target position; second, to control the convergence process to facilitate the learning of the trajectory shape. To achieve these goals, the DMP model consists of a spring-damper system and a nonlinear forcing function.
[0051] The vectorized form of the DMP equation encoding the position components of the trajectory is as follows.
[0052]
[0053]
[0054]
[0055] In formulas (7)-(9), g p is the target position of the trajectory; p represents the motion trajectory; z represents the velocity of the trajectory; represents the acceleration of the trajectory; f p (x) is the nonlinear forcing term, τ is the time constant, and α z and β z is a positive constant; irepresents the Gaussian basis function; N is the number of Gaussian basis functions; w i p represents the weight, p0 is the starting position; diag() represents a diagonal matrix; x is the state quantity.
[0056] Probabilistic reference trajectory given the center point position of the robot end effector T is the maximum length of the time series. The forcing term can be calculated
[0057]
[0058] where p t , They represent the position, velocity and acceleration at sampling time t respectively; g p The final position point of the trajectory; p t is the starting point of the generalization trajectory; τ represents the time constant, α z and β z is a normal number.
[0059] Weight coefficient w i p This can be obtained by solving the following equation using Locally Weighted Regression (LWR):
[0060]
[0061] In the formula is the learned forcing function; f p (x) is the forcing term function obtained by weighting the basis functions.
[0062] Directly modeling quaternion trajectories becomes complicated due to the unit norm constraint. Therefore, the quaternion trajectories are converted to Euclidean space to make it easier to build a probabilistic model of the converted trajectories. The vectorized form of the DMP equation encoding the quaternion trajectory is as follows:
[0063]
[0064]
[0065]
[0066] In formulas (12)-(14), g o represents the target quaternion, represents the conjugate value of the initial quaternion; q is the quaternion trajectory, represents the trajectory of its quaternion conjugate value; η is a quaternion with a scalar part of 0; f o(x) is the forcing function; Ψ i represents the Gaussian basis function; w i o represents the weight; N is the number of Gaussian basis functions; diag() represents a diagonal matrix; τ represents the time constant, α z and β z is a normal number; x is a state quantity.
[0067] The quaternion is projected into Euclidean space through a logarithmic transformation, defined as follows:
[0068]
[0069] Where q is the quaternion; v is the real part of the quaternion; and u is the imaginary part of the quaternion.
[0070] Probabilistic reference trajectory given the robot end-effector orientation where q t , Respectively represent the quaternion, angular velocity and angular acceleration at the sampling time t. Angular velocity ω and quaternion derivative The relationship between is given by the following formula: Similarly, the weight parameter w can be obtained i o .
[0071] Figure 6 The figure shows the humanoid interactive motion generation based on the improved dynamic motion primitive (DMP). The "Teaching trajectory" column shows the position (x, y, z) trajectory and quaternion (qs, qx, qy, qz) trajectory of the four teaching trajectories collected, and the "Adapted pose point trajectory" column shows the position trajectory and quaternion trajectory generated by the DMPs model to adapt to the specific grasping posture. The horizontal axis represents the time corresponding to each point on the trajectory, and the end point appears at 1.48 seconds. Figure 7 It is a three-dimensional diagram of human-like interactive motion. The curves represent the probabilistic reference trajectory and the trajectory reproduced by DMP after learning from the reference trajectory. The complete overlap of the curves indicates that DMP has a strong ability to reproduce the trajectory; the dotted curve represents the trajectory generated by DMP when adapting to the expected point.
[0072] Figure 8 The effectiveness of the proposed human-computer interactive collaboration system is verified through physical experiments.
[0073] In order to demonstrate the system's ability in handing over diverse objects, the experiment selected items in the YCB object set and previously unseen objects for testing. Each object was handed over by the experimenter 10 times in a row, allowing the experimenter to adjust the handover posture as needed. The experiment uses the number of successes as an evaluation metric. When the robot safely and stably grasps the object from the giver, the handover is considered successful. On the contrary, failure to meet the above conditions is considered a failure. In addition, the number of failures caused by grasping planning prediction and trajectory execution is also recorded. Detailed data is provided in Table 1, and it can be seen that the system proposed in the present invention is able to hand over objects of different shapes and sizes with humans and performs well in collaborative tasks.
[0074] Table 1: Experimental results (number of times), each object was handed over 10 times
[0075] Banana Eraser Marker Bottle Pencase Controller Knife Number of successes 8 6 8 6 5 7 9 Crawl prediction failed 1 3 2 2 5 2 1 Trajectory planning failed 1 1 0 2 0 1 0
[0076] In order to verify the rationality of the system settings, an ablation study was conducted to remove the grasp selection strategy to evaluate its impact on system performance. While maintaining the consistency of experimental methods and evaluation criteria, Table 2 compares the number of successful handovers for each object with and without the grasp selection strategy.
[0077] Table 2: Comparison of experiments with and without grab selection strategies
[0078]
[0079] Experimental results show that in the manipulation of most objects, the number of successes is significantly reduced when the grasp selection strategy is not adopted, confirming the effectiveness of the selection strategy. The grasp selection strategy effectively filters out unsafe and unfeasible grasp plans and increases the probability of successful manipulation of objects.
[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that modifying the technical solutions described in the aforementioned embodiments, or replacing some or all of the technical features therein by equivalents, does not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A point cloud-based 6-DOF human-machine collaborative posture planning and humanoid interactive motion generation method, characterized by: The method includes a junction point cloud extraction stage and a 6-DOF grasping posture generation stage; Junction point cloud extraction stage: Faster R-CNN is used to detect the position of objects and hands in the 2D scene, and the identified bounding box is used to segment the hand and the object; the area within the object bounding box is considered to be graspable, but when it overlaps with the hand bounding box, the overlapping area is considered to be an ungraspable area; the area within the hand bounding box is also considered to be ungraspable; therefore, a mask image of the grasping area is generated on the object image; the RGB image, depth image and mask image are combined to extract the junction point cloud in the scene; the random sampling consistency RANSAC algorithm is used to perform plane segmentation on the point cloud to filter background points; 6-DOF grasping posture generation stage: A grasping planning model based on point cloud data is proposed; the grasping planning model consists of two parts: 6-DOF grasping plan generation and optimal grasping posture selection; In the generation of 6-DOF grasping plan: the intersection point cloud is input, and the end-to-end network GraspNet is used to generate a variety of 6-DOF candidate grasping postures; the GraspNet network can predict the 6-DOF posture of the parallel gripper relative to the camera coordinate system on the 3D point cloud, where the 6-DOF posture is represented by the translation along the X, Y, and Z axes in the camera coordinate system and the rotation around the X, Y, and Z axes; the prediction of the rotation matrix representing the direction is decoupled into two independent parts: viewpoint classification and in-plane rotation angle prediction; A decomposition strategy is adopted to enable the neural network to learn and predict the 6-DOF grasping posture; In the selection of the optimal grasping posture: by calculating the length, width and height of the three-dimensional bounding box of the intersection point cloud, the candidate grasping posture located in the center area of the intersection point cloud is screened out; if the predicted grasping direction is close to the direction in which the person delivers the object, it is considered an unsafe grasping posture, and the postures in each direction are evaluated, and the candidate grasping posture with the highest score is selected as the 6-DOF grasping posture; The GraspNet network uses PointNet++ as the backbone network, which includes three networks: Approach network, Operation network and Tolerance network; Approach network, Operation network and Tolerance network are all composed of multi-layer perceptrons; Approach network is used to predict feasible grasp points and their viewpoints; The loss function L of the Approach network A The definition is as follows: Among them, c i Indicates the binary prediction of whether point i is graspable. If point i is positive, then Assign a value of 1, if point i is negative, assign a value of 0; N cls is the total number of predicted points; λ1 is a constant representing the learning rate; s ij represents the prediction confidence score of the j-th viewpoint of point i, is the corresponding truth value; N reg is the total number of viewpoints; v ij represents the approximate vector prediction of the jth virtual view of graspable point i, is the corresponding truth value; Represents the angle difference, which is the loss term in the first indicator function 1(), limiting the loss of the approximate vector; L cls represents the softmax loss of binary classification, L reg represents the smoothed L1 loss function; The Operation network is responsible for predicting the rotation angle in the plane, the distance the gripper moves forward, the gripper width, and the confidence level; the loss function L R Defined as: In the formula, R ij represents the degree of the angle grouping of the j-th viewpoint of point i, is the corresponding truth value; S ij , W ij are the grasp confidence score and grasp width of the j-th viewpoint of point i, respectively. is the corresponding truth value; is the sigmoid cross entropy loss function of the d-th group proximity distance, is the smoothed L1 loss function of the proximity distance of the dth group; λ2 and λ3 are both constants, representing the learning rate; K is the total number of angle groups; N cls is the total number of predicted points, N reg is the total number of viewpoints; The Tolerance network introduces a grasp affinity field (GAFs) to predict the tolerance of each grasping posture to disturbances. When the true value of the grasping posture is provided, the tolerance network searches its neighborhood in the ball space and sets the grasping posture with a grasp score s>0.5 and the farthest distance from the true value as the target of GAFs. The loss function L of the Tolerance network F The definition is as follows: Where T ij represents the maximum disturbance that the grasping posture of the j-th viewpoint at point i resists, is the corresponding true value; K is the total number of angle groups; N reg is the total number of viewpoints; is the smoothed L1 loss function of the proximity distance of the dth group; During training, the formula L for minimizing the GraspNet parameters is as follows: L=L A ({c i },{s ij })+αL R (R ij ,S ij ,W ij )+βL F (T ij ) (4) L A is the loss function of the Approach network, c i Indicates the binary prediction of whether point i is graspable, s ij represents the prediction confidence score of the j-th viewpoint of point i; L R is the loss function of the Operation network, R ij represents the degree of the angle grouping of the jth viewpoint of point i, S ij , W ij are the grasp confidence score and grasp width of the j-th viewpoint of point i respectively; L F is the loss function of the Tolerance network; T ij represents the maximum disturbance that the grasping posture of the j-th viewpoint of point i resists; α and β are both constants, representing the learning rate; This method adopts a humanoid interactive motion generation model based on an improved dynamic motion primitive DMP in the humanoid interactive motion generation stage; use represents the set of teaching data, where t n,h ∈R I is the input, ξ n,h ∈R O Represents the output; the probabilistic encoding of the teaching allows the input t and the output ξ to represent different types of variables; Joint probability derived from the teaching data using the GMM method distributed: Where C represents the number of Gaussian components, and π c , μ c and∑ c They represent the prior probability, mean and covariance of the Cth Gaussian component respectively; t is the input time series, ξ is the output trajectory point; is the normal distribution function; then a probability reference trajectory is retrieved from formula (5) through GMR Each point associated with it Mean and covariance Description: The probabilistic reference trajectory is used as the representative of the trajectory set, and the formula is as follows: In the formula, represents the probabilistic reference trajectory within time period t, are the mean and covariance of the probability reference trajectory; is the normal distribution function; For the position trajectory and the quaternion trajectory describing the posture in the teaching trajectory, the GMM and GMR methods are used to generate the position probability reference trajectory and the quaternion probability reference trajectory respectively; these two probability reference trajectories are input into the DMP model and the Orientation-DMP model to learn the trajectory features respectively; The DMP model consists of a spring-damper system and a nonlinear forcing term function; the vectorized form of the DMP equation encoding the trajectory position component is as follows; In formulas (7)-(9), gp is the target position of the trajectory; p represents the motion trajectory; z represents the velocity of the trajectory; represents the acceleration of the trajectory; f p (x) is the nonlinear forcing term, τ is the time constant, and α z and β z is a positive constant; i represents the Gaussian basis function; N is the number of Gaussian basis functions; w i p represents the weight, p0 is the starting position; diag() represents a diagonal matrix; x is the state quantity; Probabilistic reference trajectory given the center point position of the robot end effector T is the maximum length of the time series; the forcing term is calculated where p t , They represent the position, velocity and acceleration at sampling time t respectively; g p The final position point of the trajectory; p t is the starting point of the generalization trajectory; τ represents the time constant, α z and β z is a positive constant; Weight coefficient w i p The following equation is obtained by solving the local weighted regression LWR: In the formula is the learned forcing function; f p (x) is the forcing term function obtained by weighting the basis functions; The vectorized form of the DMP equation encoding the quaternion trajectory is as follows: In formulas (12)-(14), g o represents the target quaternion, represents the conjugate value of the initial quaternion; q is the quaternion trajectory, represents the trajectory of its quaternion conjugate value; η is a quaternion with a scalar part of 0; f o (x) is the forcing function; Ψ i represents the Gaussian basis function; w i o represents the weight; N is the number of Gaussian basis functions; diag() represents a diagonal matrix; τ represents the time constant, α z and β z is a normal number; x is a state quantity; The quaternion is projected into Euclidean space through a logarithmic transformation, defined as follows: Where q is the quaternion; v is the real part of the quaternion; u is the imaginary part of the quaternion; Probabilistic reference trajectory given the robot end-effector orientation where q t ,ω t , Respectively represent the quaternion, angular velocity and angular acceleration at the sampling time t; angular velocity ω and quaternion derivative The relationship between is given by the following formula:
Citation Information
Patent Citations
Seven-degree-of-freedom grabbing posture detection method based on RGB image and depth image
CN114140418A
Double-stage mechanical arm grabbing planning method and system based on multiplexing structure
CN114800511A
Multi-modal feature fusion method for simultaneously segmenting and detecting grabbing poses
CN116385546A
A method for controlling a grasping robot through a learning phase and a grasping phase
EP4155036A1
Human-Robot Collaborative Flexible Manufacturing System and Method
US20230086122A1
