Robot double-dexterous-hand cooperative grasping method based on demonstration data

Abstract

The invention discloses a robot double-dexterous-hand cooperative grasping method based on demonstration data, and belongs to the technical field of robot control, and the method comprises the steps: driving and obtaining multi-mode demonstration data of a robot through a man-machine interaction mode, marking a key frame and a point cloud of a target object, and constructing a structured data set; constructing and training to obtain a grasping posture generation model; inputting the point cloud feature vector of the object to be grasped into a grasping posture generation model to generate a candidate grasping posture vector; and the candidate grabbing posture vector is converted into a grabbing action executed by the robot, the grabbing posture is dynamically adjusted in real time, and cooperative grabbing of the double dexterous hands of the robot is achieved. According to the method, stable and executable double-dexterous-hand grabbing actions are generated by the robot in a real scene through multi-mode demonstration data learning, force feedback adjustment is conducted in combination with touch perception, the operation feasibility and grabbing stability of large-size objects are effectively improved, and wide application of the double dexterous hands in practical application is promoted.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot control, and particularly relates to a robot double dexterous hand collaborative grasping method based on demonstration data. BACKGROUND

[0002] With the continuous progress of robot technology, its application demand in industrial production, logistics and warehousing, home service and medical rehabilitation and other scenes is rapidly growing. Grasping, as the core ability of robot operation, directly determines the efficiency and adaptability of its task execution. Traditional researches mostly rely on parallel grippers or vacuum suction devices. Such end effectors have simple structure and are easy to control, and can perform high stability in regular objects and standardized environments. However, when facing large-size, complex-shaped or handle-lacking objects, the traditional method often has the problem of insufficient adaptability, which makes it difficult for robots to perform diversified actual operation tasks. Therefore, dexterous hands gradually enter the research field. Dexterous hands have multiple degrees of freedom and tactile perception ability, and can complete fine operation and complex contact tasks with a single hand, and are considered as the key direction to promote the development of robot grasping to a higher level.

[0003] In human daily life, stable grasping of large objects usually needs to be completed by two hands, such as lifting a carton, moving a barrel-shaped container or holding a ball. These objects are often difficult to achieve effective clamping and stable support by a single hand due to their large size or simple structure. In contrast, double-arm dexterous hands can expand the operation range and enhance the stability and flexibility of grasping through collaborative action, so as to better meet the operation needs of large objects or irregular objects. For robots, this double-arm cooperation capability can not only significantly improve the versatility of the system, but also further narrow the gap between the system and human operation level. Therefore, how to enable robots to achieve stable and reliable double-arm dexterous grasping based on limited perception information in real task scenarios has become an important challenge in robot operation tasks.

[0004] COMBO-Grasp adopts a constraint-guided reinforcement learning strategy, enabling the double arms to complete the grasping operation of large objects under partial occlusion conditions, embodying the idea of improving success rate through constraints in complex visual scenes. BimanGrasp-DDPM generates cooperative grasping poses of double dexterous hands based on a diffusion model, which can synthesize stable grasping configurations in a simulation environment, demonstrating the potential of generative methods in double-arm grasping. ManipTrans realizes action transfer of double dexterous hands by using residual learning, enabling robots to imitate and generalize human double-hand operation skills. AsymDex improves the coordination of double dexterous hands in operation tasks by means of the functional asymmetry and relative motion relationship of the two hands.

[0005] However, the existing method still has the following three deficiencies: first, most studies mainly face two-fingered gripper dual-arm operation, while the work for dual-dexterous hand is mostly focused on folding, assembly and other sequential tasks, lacking a systematic method for cooperative grasping of a single large-size object; second, the few explorations involving dual-dexterous hand grasping large objects mostly remain in simulation environment, relying on complete three-dimensional object models as input, which cannot be met in real scenes, and the position accessibility and action executability of the robot in actual operation are not fully considered; third, the existing method generally does not effectively integrate tactile or force feedback mechanism into the grasping process, making it difficult for the robot to perceive contact deviation and make corrections during execution, thereby affecting the stability and reliability of grasping. SUMMARY

[0006] In view of the above deficiencies in the prior art, the robot dual-dexterous hand cooperative grasping method based on demonstration data provided by the present application solves the problems of existing methods, such as being limited to two-fingered gripper, focusing on sequential tasks rather than cooperative grasping, relying on simulation environment and complete three-dimensional object models, ignoring real accessibility and operability, lacking tactile perception and force feedback mechanism, etc., realizing that the robot learns to generate stable and executable dual-dexterous hand grasping actions through multi-modal demonstration data in real scenes, and adjusts the force feedback combined with tactile perception, effectively improving the operation feasibility and grasping stability of large-size objects, and promoting the widespread landing of dual-dexterous hand in practical applications.

[0007] To achieve the above-mentioned invention purposes, the technical scheme adopted by the present application is as follows: a robot dual-dexterous hand cooperative grasping method based on demonstration data, comprising the following steps: S100, acquiring multi-modal demonstration data of the robot through human-computer interaction; S200, labeling the key frames and point clouds of the target object in the multi-modal demonstration data, and then constructing a structured data set; S300, constructing and training a grasping pose generation model based on a denoising diffusion probability model using the structured data set; S400, inputting the point cloud feature vector of the object to be grasped into the grasping pose generation model to generate a candidate grasping pose vector; S500, converting the candidate grasping pose vector into grasping actions executed by the robot, and dynamically adjusting the grasping pose in real time to realize cooperative grasping of the dual-dexterous hand of the robot.

[0008] Further, the step S100 comprises: A wearable exoskeleton device is used as a human-computer interaction interface to collect the natural actions of the operator and drive the robot dual arms and dexterous hands to demonstrate object grasping operation; During the demonstration of the object grasping operation, a joint angle change demonstration sequence of the robot arm and the fingers is obtained through the wearable exoskeleton device; During the demonstration of the object grasping operation, an action image demonstration sequence in a unified time sequence format is collected; the action image includes a depth image and a color image; The joint angle change demonstration sequence and the action image demonstration sequence are taken as the obtained multi-modal demonstration data.

[0009] Further, the step S200 includes: annotating the multi-modal demonstration data, including annotating key frames in the joint angle change demonstration sequence, and annotating point clouds of target objects in the action image demonstration sequence; encoding the point clouds of each target object to obtain corresponding point cloud feature vectors ; combining the annotated key frames and the corresponding point cloud feature vectors of different target objects, different positions and different postures to construct a binary tuple , and further construct a structured data set; wherein, represents a robot grasping posture vector corresponding to the th key frame, represents a joint angle vector of the robot arm at the th key frame, represents a joint angle vector of the robot fingers at the th key frame.

[0010] Further, the key frames in the joint angle change demonstration sequence are corresponding frames when the robot end effector completes each grasping action; The method for annotating the point clouds of target objects in the action image demonstration sequence includes: selecting a rectangular box of the region where the target object is located in the first frame color image in the action image demonstration sequence, and inputting it as a prompt to an object segmentation model, and then obtaining a mask of the target object; applying the mask to the first frame depth image, and converting the points in the depth image into three-dimensional point cloud coordinates through a perspective projection relationship, and then obtaining a point cloud set of each target object.

[0011] Further, in the step S300, the grasping posture generation model is a four-layer perception machine structure, each layer of the perception machine adopts a SiLU activation function for nonlinear transformation, and the time step and the point cloud feature vector of the target object are introduced through a conditional embedding method.

[0012] Further, the training process of the grasping posture generation model includes a forward diffusion process and a backward generation process; In the forward diffusion process, the robot grasp pose vector is perturbed step by step by noise, so that after a set time step, the distribution of the robot grasp pose vector approaches an isotropic Gaussian distribution. In the backward generation process, the grasp pose generation model gradually restores the robot grasp pose vector under the condition of the point cloud feature vector by minimizing a set model loss function.

[0013] Further, the step S400 includes the following steps: S401, inputting the point cloud feature vector of the object to be grasped into the reverse sampling process of the grasp pose generation model; S402, in the reverse sampling process, predicting the conditional probability distribution of the grasp pose vector corresponding to the current time step; S403, sampling the grasp pose vector according to the conditional probability distribution of the grasp pose vector; S404, repeating steps S402-S403 to recursively sample the state sequence of the initial grasp pose vector in turn from the initial grasp pose vector, and further to obtain the candidate grasp pose vector and to perform joint limit constraint thereon.

[0014] Further, in the step S404, the candidate grasp pose vector generated by the grasp pose generation model is represented as: ; In the formula, represents the candidate angle of the j-th joint in the robot, and the grasp pose vector of the candidate angle under the joint limit constraint is represented as: ; In the formula, and respectively represent the minimum allowable angle and the maximum allowable angle of the j-th joint of the robot, and the subscript represents the total number of joints on the arm and fingers of the robot.

[0015] Further, the step S500 includes: S501, dividing the candidate grasp pose vector into an arm joint angle vector and a finger joint angle vector ; S502, constructing an intermediate path point sequence of the robot arm grasping process according to the arm joint angle vector ; wherein, represents the initial posture of the arm, represents the intermediate posture of the robot double arm lifting, is the initial posture of the arm, and is the intermediate posture of the robot double arm lifting.​ a pose obtained by translating along the palm normal direction of the robot; S503, respectively setting a first speed of the robot arm in the path during the gripping process and a second speed of the path ; the second speed is less than the first speed; S504, when the robot arm reaches the target position, driving the robot hand with the finger joint angle in the finger joint angle vector , and then controlling the robot double arms to cooperatively press the object to be gripped; S505, setting a finger joint stable contact determination criterion, and continuously iterating the tactile sensing and driving adjustment of the robot finger joint until all fingers achieve stable contact with the object to be gripped, realizing the cooperative gripping of the robot double hands.

[0016] Further, in the step S505, the finger joint stable contact determination criterion is: ; In the formula, represents the torque sensing value of the first finger base joint in the robot finger, represents the feedback of the first tactile unit in the first finger base joint, and respectively represent the preset threshold values of the joint torque and the surface contact force; The tactile sensing based on the finger joint stable contact determination criterion determines the non-contact finger set , which is iteratively updated by a step to realize driving adjustment, and the update formula is: .

[0017] The beneficial effects of the present application are: 1) The multi-modal demonstration data is collected by the exoskeleton device, which effectively avoids the deviation caused by simulation data, can more accurately reflect the actual operation characteristics, and improves the feasibility of the method in the real platform.

[0018] 2) The grasping pose is generated by combining the diffusion model with single-view RGB-D input, which effectively overcomes the dependence on complete three-dimensional models, can output executable actions in the case of insufficient geometric information, and improves the adaptability of the method in an open environment.

[0019] 3) The execution adjustment mechanism combining motion planning and force guidance effectively avoids collision and misplacement in the grasping process, can ensure stable closure of the fingers, and improves the robustness of the double-arm cooperative grasping.

[0020] 4) The application supports the robot to complete the dual dexterous hand cooperative operation of large-size objects on a real platform, can adapt to diversified task scenarios, and provides a universal and expandable technical basis for intelligent manufacturing and service robots. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of the robot dual dexterous hand cooperative grasping method based on demonstration data is provided.

[0022] Figure 2 A teleoperation system structure and data flow diagram are provided.

[0023] Figure 3 A grasping posture adjustment schematic diagram is provided. DETAILED DESCRIPTION

[0024] The specific embodiments of the application are described below to facilitate those skilled in the art to understand the application, but it should be clear that the application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the application defined and determined by the appended claims, and all applications utilizing the concept of the application are within the scope of protection.

[0025] The embodiment of the application provides a robot dual dexterous hand cooperative grasping method based on demonstration data, multi-modal demonstration data containing vision and joint state are collected on a real platform through human-computer interaction mode, and the contents are labeled to extract key vision information and grasping posture information; a mapping relationship from visual observation to dual-arm cooperative grasping posture is constructed by combining point cloud feature extraction and diffusion generation model; the generated grasping action is executed through path planning in a real robot platform, and is adjusted combined with tactile signals to ensure the stability of the grasping process.

[0026] Referring to Figure 1 , the robot dual dexterous hand cooperative grasping method is as shown in Figure 1 S100, multi-modal demonstration data of the robot is driven and acquired through human-computer interaction mode; S200, key frames and point clouds of target objects in the multi-modal demonstration data are labeled, and then a structured data set is constructed; S300, a grasping posture generation model based on a denoising diffusion probability model is constructed and trained by using the structured data set; S400, the point cloud feature vector of the object to be grasped is input into the grasping posture generation model to generate a candidate grasping posture vector; ​S500, the candidate grasping posture vector is converted into a grasping action performed by the robot, and the grasping posture is dynamically adjusted in real time to realize cooperative grasping of the robot.

[0027] The step S100 of the embodiment of the application comprises: The wearable exoskeleton device is used as a human-computer interaction interface to collect natural actions of the operator and drive the robot arms and dexterous hands to perform object grasping operation demonstration. During the object grasping operation demonstration, the exoskeleton device is used to obtain a joint angle change demonstration sequence of the robot arms and fingers. During the object grasping operation demonstration, an action image demonstration sequence in a unified time sequence format is collected; the action image includes a depth image and a color image. The joint angle change demonstration sequence and the action image demonstration sequence are used as the obtained multi-modal demonstration data.

[0028] Specifically, the operator controls the robot to perform object grasping operation in the form of teleoperation, and collects demonstration data, as shown in FIG. 1. Figure 2 The wearable exoskeleton device is used as a human-computer interaction interface to collect natural actions of the operator and drive the robot arms and dexterous hands to perform object grasping operation demonstration.

[0029] For the arm part, the exoskeleton can record the joint angle sequences of the shoulder, elbow, wrist and the like in real time. Let the joint angle vector be: ; Wherein, represents the angle of the j-th joint, is the number of degrees of freedom of the exoskeleton arm part. Through the forward kinematics function , the pose of the operator's hand end in space can be obtained: ; Wherein, represents the homogeneous transformation matrix of the hand in three-dimensional space, which contains position and attitude information. The pose is converted into the joint solution corresponding to the robot end pose through the inverse kinematics function , so as to drive the robot arms to complete the same movement.

[0030] For the finger part, the exoskeleton can measure the bending angles of multiple degrees of freedom joints. In order to reduce the mapping complexity, the application selects several key joint angles as representative inputs, which are: ; Wherein, ​Indicates the first finger The angle of each joint This represents the number of key joints selected. This vector is directly linearly mapped to the corresponding joint angles of the robot's dexterous hand, forming the control input for the robot's fingers.

[0031] Based on the joint angles of the robot arm and fingers obtained above, a sequence of demonstrations of joint angle changes of the arm and fingers is constructed.

[0032] During the demonstration, color images and depth maps from an external camera were simultaneously acquired, denoted as follows: and This is used to characterize target objects and operational processes within the environment. The resulting demonstration data consists of a motion picture demonstration sequence in a unified time-series format. ; in, Indicates the first The color image of the frame, This represents the corresponding depth map. This represents the vector of the robot's arm joint angles at that moment. This represents the vector of the robot's finger joint angles at that moment. This indicates the length of the entire demonstration sequence.

[0033] Based on the above process, the operator grasps multiple objects, each object being grasped multiple times in different placement poses, thereby collecting rich demonstration data and providing high-quality input for subsequent data annotation and model building.

[0034] After obtaining the raw demonstration data, further data annotation is required to extract structured information that can be used for model training. Step S200 of this embodiment includes: The multimodal demonstration data is labeled, including labeling keyframes in the joint angle change demonstration sequence and labeling point clouds of target objects in the motion image demonstration sequence. Encode the point cloud of each target object to obtain the corresponding point cloud feature vector. ; By combining keyframes labeled with different target objects, positions, and poses with the corresponding point cloud feature vectors of the target objects, a binary tuple is constructed. This leads to the construction of a structured dataset; in, Indicates the first The robot's grasping posture vector corresponding to each keyframe Indicates the robotic arm in the first Joint angle vectors for each keyframe Indicates the robot finger in the first Joint angle vector of a key frame.

[0035] In this embodiment, the key frames in the joint angle change demonstration sequence are the corresponding frames when the robot end effector completes each grasping action; specifically, the action process in the joint angle change demonstration sequence is manually labeled. In each grasping process, a key frame needs to be determined, which corresponds to the moment when the robot end effector has completed the grasping action but has not yet lifted the object.

[0036] In this embodiment, the method for labeling the point cloud of the target object in the action image demonstration sequence includes: Selecting a rectangular box of the region where the target object is located in the first frame color image in the action image demonstration sequence as a prompt input into the object segmentation model, and then obtaining the mask of the target object; Applying the mask to the first frame depth map, and converting the points in the depth map into three-dimensional point cloud coordinates through the perspective projection relationship, and then obtaining the point cloud set of each target object.

[0037] Specifically, the operator only selects the region of the target object by manually drawing a box in the first frame color image of the demonstration sequence. The rectangular box is input as a prompt into the object segmentation model (such as Segment Anything Model, SAM), and the corresponding object mask is obtained. Then, the mask is applied to the first frame depth map to retain the depth information of the target object. Let the pixel coordinates be , the depth value be , and the camera intrinsic matrix be: ; wherein, focal length in the horizontal and vertical directions, respectively, and the principal point coordinates. Through the perspective projection relationship, the points in the depth map can be converted into three-dimensional point cloud coordinates: ; wherein, represents the three-dimensional coordinates corresponding to the pixel point .

[0038] After mask screening and projection conversion, the point cloud set of the target object is obtained, which provides input for the subsequent grasping pose generation model.

[0039] In the process of constructing the structured data set, PointNet++ is used to encode the point cloud of each object to obtain the point cloud feature vector , wherein This represents the feature dimension. Simultaneously, keyframe annotations for different objects, positions, and poses are collected from a large amount of demonstration data. Let the first... The robot joint angle vectors corresponding to each keyframe are: This leads to the construction of a large number of binary tuples. The structured dataset consists of grasping samples covering multiple objects and multiple poses.

[0040] In step S300 of this embodiment of the invention, the Diffusion Probabilistic Model (DPM) is a type of generative model based on probabilistic reasoning. Its core idea is to gradually add noise to the data through a forward diffusion process, so that it eventually approaches an isotropic Gaussian distribution; and then, by learning a reverse denoising process, the original data distribution is gradually restored from the noisy samples, thereby realizing the modeling of complex distributions.

[0041] The grip posture generation model in this embodiment of the invention is a four-layer perceptron structure. The SiLU activation function is used to perform nonlinear transformation between each layer of the perceptron, and the time step and the point cloud feature vector of the target object are introduced through conditional embedding. This grip posture generation model can effectively capture the complex mapping relationship between point cloud and grip posture while ensuring computational efficiency.

[0042] The training process of the grip posture generation model in this embodiment of the invention includes a forward diffusion process and a backward generation process; During the forward diffusion process, the robot's grasping posture vector is gradually perturbed into noise, so that after a set time step, the distribution of the robot's grasping posture vector is close to an isotropic Gaussian distribution. During the backward generation process, by minimizing the set model loss function, the grasping posture generation model gradually restores the robot's grasping posture vector under the point cloud feature vector conditions.

[0043] Among them, let Indicates the first step in the diffusion process The state of the step, If we represent the state in the previous step, then we have: ; in, Indicates the first The noise intensity coefficient added in the step, Represents the identity matrix.

[0044] go through After stepping, The distribution is close to an isotropic Gaussian distribution.

[0045] During the reverse generation process, a conditional denoising model is trained. ,in denotes a model parameter, denotes a time step, denotes a point cloud feature. Wherein, the model loss function is set as: ; wherein, denotes the Gaussian noise sampled in the diffusion process.

[0046] In the embodiment, by minimizing the loss function, the model can learn to gradually restore the real grasping pose under the condition .

[0047] In the embodiment of the application, in the process of generating a candidate grasping pose vector by using the grasping pose generation model, a frame of scene image and the name of the object to be grasped are input, first, a detection box is generated by a target detection model (such as GroundingDino), and an object mask is obtained by a segmentation model , combined with a depth map and camera intrinsic parameters , an object point cloud is generated , and then a point cloud feature vector is extracted by PointNet++ .

[0048] Based on this, the step S400 of the embodiment of the application includes the following steps: S401, input the point cloud feature vector of the object to be grasped into the reverse sampling process of the grasping pose generation model; S402, in the reverse sampling process, predict the conditional probability distribution of the grasping pose vector corresponding to the current time step; S403, sample the grasping pose vector according to the conditional probability distribution of the grasping pose vector; S404, repeat steps S402-S403, recursively sample from the initial grasping pose vector to obtain the state sequence of the initial grasping pose vector, and then obtain the candidate grasping pose vector, and perform joint limiting constraint on it.

[0049] In the above process, at each time step , a conditional probability distribution is predicted, which is represented as: ; wherein, denotes the grasping pose state vector at time step , denotes the state at the previous moment, is the point cloud feature vector obtained by feature extraction from the object point cloud, denotes the conditional probability distribution of the grasping pose state vector at time step given the current state Time step and point cloud features The mean vector predicted by the model under the given conditions. This represents the corresponding prediction covariance matrix. These are the model parameters.

[0050] When sampling based on the above conditional probability distribution, it can be denoted as: ; In the formula, This represents random noise sampled from a standard Gaussian distribution. It is an identity matrix.

[0051] In this way, the model introduces a noise term near the mean to ensure the diversity of the generated results. The entire generation process starts from the initial state. Initially, the state sequence is obtained through recursive iteration. Finally, the candidate gripping pose vector is obtained in step 0. .

[0052] Candidate grip posture vectors generated by the grip posture generation model Represented as: ; In the formula, Indicates the first in the robot Candidate angles of each joint, and their grasping posture vectors under joint constraint. Represented as: ; In the formula, and They represent the robot's number 1 and 2. Minimum and maximum permissible angles of each joint, subscript This represents the total number of joints in the robot's arms and fingers.

[0053] Through the above process, it is ensured that each joint angle in the final output attitude vector is limited to its corresponding allowable rotation range.

[0054] Step S500 of this embodiment of the invention includes: S501. Divide the candidate gripping posture vector into arm joint angle vectors. With finger joint angle vector ; S502. Based on the joint angle vectors of the robot arm, construct the sequence of intermediate path points during the robot arm's grasping process. ;in, Indicates the initial arm posture. This indicates the robot's intermediate pose with both arms raised to the sides, used to avoid collisions with the table caused by the arms being raised directly forward. For the reason The pose obtained by translating along the normal direction of the robot's hand is used as a pre-grasping bias. The purpose of this bias is to ensure that the two hands can slowly approach the object from the opposite side along the direction of the grasping force, thereby avoiding the object being pushed or misaligned due to premature contact. Based on the above intermediate path point sequence, dense trajectories can be generated by linear interpolation and streamed to the robot actuator at a fixed frequency. S503, Set the path during the robot arm's grasping process respectively. First speed and path The second velocity; wherein the second velocity is less than the first velocity; In a specific example, the first velocity is 0.5 m / s and the second velocity is 0.15 m / s; S504. When the robot arm reaches the target position, use the finger joint angle vector. The finger joints drive the robot's dexterous hand, thereby controlling the robot's arms to collaboratively press against the object to be grasped. Specifically, during the process of controlling the robot's two arms to perform coordinated pressing, the center positions of the left and right palms are set as follows: Its center point is: ; Control both palms to move at a low speed along the vector and The robot continues to advance synchronously until the distance moved per second at the end of its execution is less than 0.01 m, indicating that the two arms are blocked by an object and have established stable support. In a specific example, the speed of movement of both palms is 0.05 m / s; S505. Set the stable contact judgment criteria for finger joints, and continuously iterate the tactile perception drive adjustment of the robot's finger joints until all fingers achieve stable contact with the object to be grasped, so as to realize the robot's dual dexterous hand cooperative grasping. Specifically, let the first Each finger in the iteration step The joint angle is The criteria for determining stable contact of finger joints are as follows: ; In the formula, Indicates the first [unclear] in the robot's finger The torque sensing value of the base joint. Indicates the first The first finger joint Feedback from each tactile unit with respectively represent preset threshold values of joint torque and surface contact force; determining the set of fingers that are not in contact based on the criteria of stable contact of the finger joints , and the step size is The iteration is updated to achieve the driving adjustment, and the update formula is: ; In the above finger control process of the robot dexterous hand, the thumb is bent and swings laterally towards the palm at the same time, and the remaining four fingers are bent until all the fingers reach stable contact. Thus, the two arms and two hands achieve comprehensive contact coverage, thereby ensuring stable grasping of large-size objects.

[0055] The principles and implementation manners of the present application are described by using specific embodiments in the present application, and the above embodiment descriptions are only used to help understand the method of the present application and its core idea; meanwhile, for the ordinary skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above descriptions should not be understood as the limitation of the present application.

[0056] The person skilled in the art will understand that the embodiments described herein are used to help the reader understand the principles of the present application and should be understood as the protection scope of the present application not being limited to such specific descriptions and embodiments. The person skilled in the art can make various other specific modifications and combinations according to the technical inspirations disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the protection scope of the present application.

Claims

1. A method for collaborative grasping of a robot's dual dexterous hands based on demonstration data, characterized in that, Includes the following steps: S100: Drives and acquires multimodal demonstration data of the robot through human-computer interaction; S200, annotation of keyframes and point clouds of target objects in multimodal demonstration data, and then construction of structured dataset; S300. Using a structured dataset, construct and train a grasping posture generation model based on a denoised diffusion probability model. S400. Input the point cloud feature vector of the object to be grasped into the grasping posture generation model to generate candidate grasping posture vectors. S500 converts candidate grasping posture vectors into grasping actions to be performed by the robot and dynamically adjusts the grasping posture in real time to achieve collaborative grasping by the robot's two dexterous hands.

2. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 1, characterized in that, Step S100 includes: The demonstration uses a wearable exoskeleton as the human-computer interaction interface to collect the operator's natural movements and drive the robot's arms and dexterous hands to perform object grasping operations. During the demonstration of object grasping, a sequence of joint angle changes in the robot arm and fingers was obtained through a wearable exoskeleton device. During the demonstration of the object grasping operation, a sequence of motion images with a uniform time-series format was collected; the motion images included depth maps and color images. The joint angle change demonstration sequence and motion image demonstration sequence were used as the acquired multimodal demonstration data.

3. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 1, characterized in that, Step S200 includes: The multimodal demonstration data is labeled, including labeling keyframes in the joint angle change demonstration sequence and labeling point clouds of target objects in the motion image demonstration sequence. Encode the point cloud of each target object to obtain the corresponding point cloud feature vector. ; By combining keyframes labeled with different target objects, positions, and poses with the corresponding point cloud feature vectors of the target objects, a binary tuple is constructed. This leads to the construction of a structured dataset; in, Indicates the first The robot's grasping posture vector corresponding to each keyframe Indicates the robotic arm in the first Joint angle vectors for each keyframe Indicates the robot finger in the first The joint angle vector of each keyframe.

4. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 3, characterized in that, The keyframes in the joint angle change demonstration sequence are the corresponding frames when the robot's end effector completes each grasping action. The method for annotating the point cloud of the target object in the action image demonstration sequence includes: In the first frame of the color image in the motion image demonstration sequence, a rectangular box is selected in the region where the target object is located. This box is then used as a cue input into the object segmentation model to obtain the mask of the target object. The mask is applied to the first frame depth map, and the points in the depth map are converted into three-dimensional point cloud coordinates through perspective projection, thereby obtaining the point cloud set of each target object.

5. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 3, characterized in that, In step S300, the grip posture generation model is a four-layer perceptron structure. The SiLU activation function is used to perform nonlinear transformation between each layer of the perceptron, and the time step and the point cloud feature vector of the target object are introduced through conditional embedding.

6. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 5, characterized in that, The training process of the grip posture generation model includes a forward diffusion process and a backward generation process; During the forward diffusion process, the robot's grasping posture vector is gradually perturbed into noise, so that after a set time step, the distribution of the robot's grasping posture vector is close to an isotropic Gaussian distribution. In the backward generation process, by minimizing the set model loss function, the grasping posture generation model gradually restores the robot's grasping posture vector under the point cloud feature vector conditions.

7. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 1, characterized in that, Step S400 includes the following steps: S401. Input the point cloud feature vector of the object to be grasped into the backsampling process of the grasping posture generation model; S402. During the backsampling process, predict the conditional probability distribution of the grip posture vector corresponding to the current time step; S403. Sample the grip posture vector according to the conditional probability distribution of the grip posture vector; S404. Repeat steps S402 to S403, starting from the initial gripping posture vector, to recursively sample the state sequence of the initial gripping posture vector, thereby obtaining candidate gripping posture vectors, and subjecting them to joint limit constraints.

8. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 7, characterized in that, In step S404, the grip posture generation model generates candidate grip posture vectors. Represented as: ； In the formula, Indicates the first in the robot Candidate angles of each joint, and their grasping posture vectors under joint constraint. Represented as: ； In the formula, and They represent the robot's number 1 and 2. Minimum and maximum permissible angles of each joint, subscript This represents the total number of joints in the robot's arms and fingers.

9. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 1, characterized in that, Step S500 includes: S501. Divide the candidate gripping posture vector into arm joint angle vectors. With finger joint angle vector ; S502. Based on the joint angle vectors of the robot arm, construct the sequence of intermediate path points during the robot arm's grasping process. ;in, Indicates the initial arm posture. This indicates the intermediate pose of the robot's arms as they are raised to the sides. For the reason The pose obtained by translating along the normal direction of the robot's hand; S503, Set the path during the robot arm's grasping process respectively. First speed and path The second speed; the second speed is less than the first speed; S504. When the robot arm reaches the target position, use the finger joint angle vector. The finger joints drive the robot's dexterous hand, thereby controlling the robot's arms to collaboratively press against the object to be grasped. S505. Set the stable contact judgment criteria for finger joints. Through continuous iteration, adjust the tactile perception of the robot's finger joints until all fingers achieve stable contact with the object to be grasped, thus realizing the robot's dual dexterous hand cooperative grasping.

10. The robot dual-dexterity hand cooperative grasping method based on demonstration data according to claim 9, characterized in that, In step S505, the criterion for determining stable contact of finger joints is as follows: ； In the formula, Indicates the first of the robot's fingers The torque sensing value of the base joint. Indicates the first The first finger joint Feedback from each tactile unit and These represent the preset threshold values ​​for joint torque and surface contact force, respectively. The set of fingers identified as not in contact by tactile perception based on the stable contact criterion of finger joints. According to its step size Iterative updates are used to drive adjustments, and the update formula is: 。

Images