A general robot dexterous hand grasping demonstration device and grasping strategy learning method

Abstract

The application discloses a general robot dexterous hand grasping demonstration device and a grasping strategy learning method. The method comprises the following steps: acquiring multi-modal data; establishing a SLAM model, performing real-time pose calculation on the multi-modal data, and combining the action of the dexterous hand to return in real time in the SLAM model; constructing a learning network comprising a visual encoder ViT, a multi-head cross attention mechanism, a force perception encoder MLP, a splicing module and a Diffusion Policy; training the learning model; generating an action conforming to a target; obtaining delay compensation data; obtaining an updated robot action generation strategy; and predicting a subsequent action sequence in real time. The device comprises an IMU camera, a paper mirror, a fixing device, a tactile sensor and a computer program product. The demonstration device and the grasping strategy learning method can be applied to various robot platforms, and can realize the execution of various complex tasks through imitation learning, such as operation tasks in the fields of industrial automation, home service and medical care.

Description

Technical Field

[0001] This invention belongs to the field of dexterous hand control design technology, and relates to a general robot dexterous hand grasping teaching device and grasping strategy learning method. Background Technology

[0002] With the development of robotics technology, robots are increasingly being used in industrial automation, home services, and healthcare. Advances in dexterous hand technology, in particular, have provided new possibilities for robots to operate in multi-degree-of-freedom and complex environments. However, existing robot grasping and teaching systems typically rely on expensive sensor equipment, have limited operating environments, and struggle to flexibly handle diverse task scenarios. Currently, the fusion of multimodal data and precise motion control are key technologies for achieving efficient teaching and task execution.

[0003] Against this backdrop, traditional methods often employ vision- or force-feedback-based approaches for robot task learning. However, these methods are frequently limited by hardware constraints such as sensor accuracy and execution latency, making it difficult to efficiently learn dynamic tasks or complex movements. Furthermore, the time synchronization problem between different observation and action flows in existing systems has not been effectively resolved, leading to errors during multimodal data fusion and affecting the final task execution performance.

[0004] Therefore, there is an urgent need for a low-barrier-to-entry method for learning grasping strategies that enables efficient and precise control of robotic dexterity hands. Summary of the Invention This application addresses the shortcomings of existing technologies by providing a universal robot dexterous hand grasping teaching device and a grasping strategy learning method. This method aims to solve problems in existing robot teaching systems, such as reliance on expensive sensors, inaccurate data synchronization, and complex task learning. The method provided in this application enables flexible and portable data acquisition through a combination of a handheld dexterous hand and an IMU camera. It also uses Diffusion Policy to model and learn complex actions for the task, ensuring efficient and accurate robot operation without relying on expensive hardware.

[0005] Firstly, a general method for learning grasping strategies for robotic dexterity hands is provided, including: Acquire multimodal data; the multimodal data includes image data of dexterous hand movements, IMU data, tactile data, and operating environment data; Establish a SLAM model, perform real-time pose calculation on multimodal data, and combine the movements of the dexterous hand to transmit them back to the SLAM model in real time. We construct a learning model that includes a visual encoder (ViT), a multi-head cross-attention mechanism, a force-aware encoder (MLP), a stitching module, and a Diffusion Policy. Train the learning model; The trained learning model generates a random initial action distribution based on the current environmental observations. Through a multi-step process of reverse diffusion, noise is gradually removed from the noise to generate an action that meets the target. Synchronize the observation flow and multimodal data of the robotic arm and dexterous hand to obtain delay-compensated data; The delay compensation data is used to perform delay compensation on the action sequence that meets the target, so as to obtain an updated robot action generation strategy; The returned multimodal data is input into the updated robot motion generation strategy to predict subsequent motion sequences in real time.

[0006] In one possible implementation, the method for real-time pose calculation of multimodal data includes: performing real-time pose calculation using the ORB-SLAM3 algorithm to generate a 6-DOF trajectory for the camera.

[0007] In one possible implementation, the method of combining the movements of a dexterous hand for real-time backhaul in a SLAM model includes: The camera pose is obtained using image data and IMU data, thereby acquiring the pose information of the data acquisition device; the pose information is then fed back into the SLAM model.

[0008] In one possible implementation, the learning model includes a visual encoder ViT (VisionTransformer) for encoding image data into visual feature representations. Force-sensing encoders (MLPs) are used to encode tactile data into force-sensing feature representations. A multi-head cross-attention mechanism is used to fuse visual feature representations and force perception feature representations to obtain bimodal fusion data; The stitching module is used to stitch together bimodal fusion data and robot state data to obtain multimodal fusion data; Diffusion Policy is used to progressively add random noise to multimodal fusion data through a forward diffusion model, forming a gradual process from real data to a high-noise distribution; through a reverse diffusion process, noise is gradually removed over multiple time steps until the action sequence that meets the target is recovered.

[0009] In one possible implementation, the training learning model is trained using a training set; the features in the training set include image data and IMU data of the dexterous hand, and the labels include dexterous hand action data, visual input data, object position data, and force feedback data.

[0010] In one possible implementation, the method for synchronizing the observation streams of the robotic arm and dexterous hand with the delays of multimodal data to obtain delay-compensated data includes: Alignment of observation streams: Time synchronization is performed based on the stream with the highest latency by measuring the latency of each observation stream; Motion flow alignment: Motion flow alignment is achieved by synchronizing the control frequencies of the dexterous hand and the robotic arm.

[0011] Secondly, a universal robotic dexterous hand grasping strategy learning device is provided, comprising: The acquisition module is used to acquire multimodal data, which includes image data of dexterous hand movements, IMU data, and operating environment data. The model building module is used to build SLAM models, perform real-time pose calculations on multimodal data, and combine the movements of the dexterous hand to transmit them back to the SLAM model in real time. The learning model building module is used to build learning models including the visual encoder ViT, multi-head cross-attention mechanism, force-aware encoder MLP, stitching module and Diffusion Policy; The training module is used to train the learning model; The generation module is used to generate a random initial action distribution based on the current environmental observations using the trained learning model. Through a multi-step process of reverse diffusion, it gradually removes noise from the noise to generate an action that meets the target. The compensation module is used to synchronize the delay of the observation flow and multimodal data of the robotic arm and dexterous hand to obtain delay compensation data; The update module is used to perform delay compensation on the target-compliant action sequence using delay compensation data to obtain an updated robot action generation strategy. The output module is used to input the returned multimodal data into the updated robot motion generation strategy to predict subsequent motion sequences in real time.

[0012] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the learning method as described in the first aspect.

[0013] Fourthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the learning method as described in the first aspect.

[0014] Fifthly, a universal robotic dexterous hand grasping teaching device is provided, comprising: An IMU camera is used to acquire image data and IMU data of dexterous hand movements; Paper mirrors, placed on both sides of the IMU camera, are used to expand the IMU camera's field of view. A fixing device, attached to a dexterous hand, is used to secure the IMU camera and the paper mirror; A tactile sensor, installed in the dexterous hand, is used to acquire pressure sensing data from the fingers of the dexterous hand; A computer program product for performing the method described in the first aspect.

[0015] The beneficial effects of this application are: (1) The grasping strategy learning method provided in this application can achieve efficient data acquisition and accurate positioning. Through the IMU camera and ORB-SLAM3 algorithm, it can acquire multimodal data during complex operations, accurately locate the camera pose, and ensure the accuracy of robot operation.

[0016] (2) The grasping strategy learning method provided in this application utilizes the Diffusion Policy to train a neural network model using multimodal data, thereby learning and generating the action sequences required for the robot to perform complex tasks. The progressive denoising process of the diffusion model ensures that the generated actions are smooth and meet the task requirements.

[0017] (3) The grasping strategy learning method provided in this application ensures that there will be no delay error during the data fusion process by synchronously processing the observation stream and the action stream. In particular, by delay compensation of the action sequence output by the strategy, the precise synchronization of the dexterous hand and the robotic arm operation is ensured.

[0018] (4) This application also provides a universal robot dexterous hand grasping teaching device, which includes an IMU camera, a paper mirror and a fixing device, without relying on expensive sensor equipment, and can flexibly and portablely carry out data collection and teaching tasks.

[0019] (5) The teaching device and grasping strategy learning method described in this application can be applied to various robot platforms and can perform a variety of complex tasks through imitation learning, such as operational tasks in fields such as industrial automation, home services and medical care. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the general robot dexterity hand grasping strategy learning method provided in this application embodiment; Figure 2 A flowchart illustrating the Diffusion Policy learning, training, and inference process provided in this application embodiment; Figure 3 A schematic diagram of the structure of the learning model provided in the embodiments of this application; Figure 4This application provides a general robot dexterous hand grasping strategy learning device in its embodiments. Figure 5 A schematic diagram of the structure of the universal robot dexterous hand grasping teaching device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the layout structure of the universal robot dexterous hand reflector provided in the embodiments of this application; Figure 7 An application example of the universal robotic dexterous hand grasping teaching device provided in the embodiments of this application.

[0021] Figure 5 , Figure 7 In the middle: 1-IMU camera, 2-paper mirror, 3-fixation device, 4-dexterous hand, 5-tactile sensor, 6-robotic arm. Detailed Implementation

[0022] To make the technical problems and solutions to be solved by the present invention clearer, the following detailed description of a general robot dexterous hand grasping teaching system and learning strategy provided by the present invention is given in conjunction with the accompanying drawings and embodiments. The content described herein is an explanation of the present invention, not a limitation thereof.

[0023] See Figure 1 This describes a general method for learning grasping strategies for robotic dexterity hands, comprising the following steps: S100. Acquire multimodal data; the multimodal data includes image data of dexterous hand movements, IMU data, tactile data, and operating environment data.

[0024] In one possible implementation, the operating environment data refers to the specific parameters and conditions of the direct working environment in which the robot performs its tasks. These data directly affect the robot's operational accuracy, safety, and efficiency. Examples include temperature and humidity.

[0025] S200. Establish a SLAM model, perform real-time pose calculation on multimodal data, and combine the movements of the dexterous hand to transmit the data back to the SLAM model in real time.

[0026] In one possible implementation, the method for real-time pose calculation in S200 includes: performing real-time pose calculation using the ORB-SLAM3 algorithm to generate a 6-DOF trajectory for the camera.

[0027] In one possible implementation, the S200 incorporates the movements of a dexterous hand to transmit data back to the SLAM model in real time. This includes: using image data and IMU data, combining them with the ORB-SLAM3 open-source model to obtain the camera pose, thereby acquiring the pose information of the data acquisition device; and transmitting the pose information back to the SLAM model.

[0028] S300, construct a learning model including a visual encoder ViT, a multi-head cross-attention mechanism, a force-aware encoder MLP, a stitching module, and a Diffusion Policy; Among them, the visual encoder ViT (Vision Transformer) is used to encode image data into visual feature representations; Force-sensing encoders (MLPs) are used to encode tactile data into force-sensing feature representations. A multi-head cross-attention mechanism is used to fuse visual feature representations and force perception feature representations to obtain bimodal fusion data; The stitching module is used to stitch together bimodal fusion data and robot state data to obtain multimodal fusion data; Diffusion Policy is used to progressively add random noise to multimodal fusion data through a forward diffusion model, forming a gradual process from real data to a high-noise distribution; through a reverse diffusion process, noise is gradually removed over multiple time steps until the action sequence that meets the target is recovered.

[0029] Figure 3 A schematic diagram of the learning model is shown.

[0030] It's important to note that Diffusion Policy can be understood as the application of diffusion models in robot control. It combines imitation learning with the ability to learn policies by observing demonstrations from human experts. In each iteration, the noise prediction network (εθ) predicts the noise gradient of the current noise sample, which is then used to update the sample, progressively denoising it. Through stochastic Langevin dynamics steps, Diffusion Policy iteratively optimizes the action sequence until the final action is generated.

[0031] In one possible implementation, the target-compliant action sequence includes robotic arm pose information and dexterous hand grasping state.

[0032] In one possible implementation, the structure of the diffusion model includes: the network of the diffusion model includes the 1DUnet model.

[0033] In one possible implementation, the robot state data includes pose information and dexterous hand gripping state, totaling 7 dimensions.

[0034] Understandable, see Figure 3 The data flows through the learning model as follows: In order to capture pressure information at the first moment of contact between the dexterous hand and the object, and thus determine the grasping state of the dexterous hand, this embodiment adds a pressure sensor to the dexterous hand. An array of low-latency tactile sensors is used to acquire tactile data such as pressure sensing data.

[0035] In the design of the policy learning model structure, the visual image observation input is mapped to a low-dimensional space z0 after passing through a visual encoder ViT (VisionTransformer). The pressure perception data obtained from the tactile sensor is mapped into the low-dimensional space through an MLP. Then, the pressure sensor data and visual image data are fused through cross-attention to output a joint embedding vector z1. Next, z1 is concatenated with the robot state vector by a concatenation module to obtain the complete network input vector z2. Finally, z2 is input into the Diffusion Policy to obtain the output action sequence At.

[0036] S400, training and learning model.

[0037] In one possible implementation, S400 includes the following sub-steps: S410, Obtain the training set.

[0038] Furthermore, the features of the training set include image data and IMU data of the dexterous hand, and the labels include dexterous hand action data, visual input data, object position data, and force feedback data.

[0039] It should be noted that the dexterous hand action data refers to the collection of various motion parameters, sensory feedback, and control commands when simulating the fine operation of the human hand.

[0040] The visual input data refers to the image data obtained by the camera.

[0041] The object position data refers to the endpoint position when the dexterous hand approaches the target object.

[0042] The force feedback data refers to Data transmitted back from the array of low-latency tactile sensors.

[0043] S420. Train the learning model using the training set.

[0044] S430. Using the trained learning model, a random initial action distribution is generated based on the current environmental observations. Through a multi-step process of reverse diffusion, noise is gradually removed from the noise to generate an action that meets the target.

[0045] S500 synchronizes the observation stream and multimodal data delays of the robotic arm and dexterous hand to obtain delay-compensated data.

[0046] In one possible implementation, S500 includes the following sub-steps: S510, Alignment of Observation Streams: Time synchronization is performed based on the stream with the highest latency by measuring the latency of each observation stream.

[0047] Specifically, a timestamp is used to unify the sampling of vision, touch, and robot state (i.e., the observation stream), and the various sensor devices synchronize time through a shared clock.

[0048] S520, Motion Flow Alignment: Motion flow alignment is achieved by synchronizing the control frequency of the dexterous hand and the control frequency of the robotic arm.

[0049] For example, the control frequency of the Bluetooth-controlled dexterous hand is set to 10 Hz, and the control frequency of the robotic arm is also set to 10 Hz. The control frequency information of the two is synchronized through programming. That is, the inference frequency is 10 Hz, and each inference goes through a policy learning model once to obtain a 7D motion sequence (6D robotic arm end pose + dexterous hand grasping state).

[0050] S600. The delay compensation data is used to perform delay compensation on the action sequence that meets the target to obtain an updated robot action generation strategy.

[0051] The S700 inputs the returned multimodal data into the updated robot motion generation strategy to predict subsequent motion sequences in real time.

[0052] See Figure 4 A general-purpose robotic dexterous hand grasping strategy learning device, comprising: The acquisition module 41 is used to acquire multimodal data; the multimodal data includes image data of dexterous hand movements, IMU data, and operating environment data; The model building module 42 is used to build a SLAM model, perform real-time pose calculation on multimodal data, and combine the movements of the dexterous hand to transmit back in real time in the SLAM model. The learning model building module 43 is used to build a learning model that includes a visual encoder ViT, a multi-head cross-attention mechanism, a force-aware encoder MLP, a stitching module, and a Diffusion Policy. Training module 44 is used to train the learning model; The generation module 45 is used to generate a random initial action distribution based on the current environmental observation using the trained learning model, and to generate an action that meets the target by gradually removing noise from the noise through a multi-step process of reverse diffusion. Compensation module 46 is used to synchronize the delay of the observation flow and multimodal data of the robotic arm and dexterous hand to obtain delay compensation data; Update module 47 is used to perform delay compensation on the action sequence that meets the target using delay compensation data to obtain an updated robot action generation strategy; Output module 48 is used to input the returned multimodal data into the updated robot motion generation strategy to predict subsequent motion sequences in real time.

[0053] On the other hand, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the grasping strategy learning method provided by the methods described above.

[0054] In another aspect, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the grasping strategy learning method provided by the above methods.

[0055] See Figure 5 A general-purpose robotic dexterous hand grasping teaching device, comprising: IMU camera 1 is used to acquire image data and IMU data of dexterous hand movements; Paper mirrors 2 are placed on both sides of IMU camera 1 to expand the field of view of IMU camera 1; Fixing device 3, connected to dexterous hand 4, is used to fix IMU camera 1 and paper mirror 2; A tactile sensor 5 is installed on the dexterous hand 4 to acquire pressure sensing data of the dexterous hand's fingers; A computer program product for executing the crawling strategy learning methods provided by the above methods.

[0056] In one possible implementation, the reflective surface of the paper mirror 2 is disposed on both sides of the IMU camera lens, and the reflective surface extends outward relative to the camera. Figure 6 A layout structure is shown.

[0057] In one possible implementation, the fixing device is obtained by 3D printing.

[0058] Understandably, the structure of the fixing device is designed to be adapted to the specific structure of the dexterous hand, so as to meet the fixing requirements.

[0059] See Figure 7 This is a schematic diagram of the teaching device described in this embodiment used for a robotic arm.

[0060] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the invention.

Claims

1. A general method for learning grasping strategies for a robot dexterous hand, characterized in that, include: Acquire multimodal data; the multimodal data includes image data of dexterous hand movements, IMU data, tactile data, and operating environment data; A SLAM model is established, and multimodal data is used for real-time pose calculation. The movements of a dexterous hand are then combined with the movements of the dexterous hand for real-time transmission back into the SLAM model. The method of combining the movements of the dexterous hand with the movements of the dexterous hand for real-time transmission back into the SLAM model includes: obtaining the camera pose using image data and IMU data, thereby obtaining the pose information of the data acquisition device; and transmitting the pose information back into the SLAM model. We construct a learning model that includes a visual encoder (ViT), a multi-head cross-attention mechanism, a force-aware encoder (MLP), a stitching module, and a Diffusion Policy. The training and learning model is trained using a training set; the features in the training set include image data and IMU data of the dexterous hand, and the labels include dexterous hand action data, visual input data, object position data, and force feedback data. The trained learning model generates a random initial action distribution based on the current environmental observations. Through a multi-step process of reverse diffusion, noise is gradually removed from the noise to generate an action that meets the target. The method for synchronizing the observation streams and multimodal data of the robotic arm and dexterous hand to obtain delay compensation data includes: alignment of observation streams by measuring the delay of each observation stream and performing time synchronization based on the stream with the highest delay; and alignment of motion streams by synchronizing the control frequency of the dexterous hand and the control frequency of the robotic arm. The delay compensation data is used to perform delay compensation on the action sequence that meets the target, so as to obtain an updated robot action generation strategy; The returned multimodal data is input into the updated robot motion generation strategy to predict subsequent motion sequences in real time.

2. The learning method according to claim 1, characterized in that, The method for real-time pose calculation of multimodal data includes: performing real-time pose calculation using the ORB-SLAM3 algorithm to generate a 6-DOF trajectory for the camera.

3. The learning method according to claim 1, characterized in that, In the learning model, the visual encoder ViT is used to encode image data into visual feature representations; Force-sensing encoders (MLPs) are used to encode tactile data into force-sensing feature representations. A multi-head cross-attention mechanism is used to fuse visual feature representations and force perception feature representations to obtain bimodal fusion data; The stitching module is used to stitch together bimodal fusion data and robot state data to obtain multimodal fusion data; Diffusion Policy is used to progressively add random noise to multimodal fusion data through a forward diffusion model, forming a gradual process from real data to a high-noise distribution; through a reverse diffusion process, noise is gradually removed over multiple time steps until the action sequence that meets the target is recovered.

4. A universal robot dexterous hand grasping strategy learning device, used to implement the method according to any one of claims 1-3, characterized in that, include: The acquisition module is used to acquire multimodal data, which includes image data of dexterous hand movements, IMU data, and operating environment data. The model building module is used to build SLAM models, perform real-time pose calculations on multimodal data, and combine the movements of the dexterous hand to transmit them back to the SLAM model in real time. The learning model building module is used to build learning models including the visual encoder ViT, multi-head cross-attention mechanism, force-aware encoder MLP, stitching module and Diffusion Policy; The training module is used to train the learning model; The generation module is used to generate a random initial action distribution based on the current environmental observations using the trained learning model. Through a multi-step process of reverse diffusion, a target action is generated by gradually removing noise from the noise. The compensation module is used to synchronize the delay of the observation flow and multimodal data of the robotic arm and dexterous hand to obtain delay compensation data; The update module is used to perform delay compensation on the target-compliant action sequence using delay compensation data to obtain an updated robot action generation strategy. The output module is used to input the returned multimodal data into the updated robot motion generation strategy to predict subsequent motion sequences in real time.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the learning method as described in any one of claims 1 to 3.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the learning method as described in any one of claims 1 to 3.

7. A universal robot dexterous hand grasping teaching device, characterized in that, include: An IMU camera is used to acquire image data and IMU data of dexterous hand movements; Paper mirrors, placed on both sides of the IMU camera, are used to expand the IMU camera's field of view. A fixing device, attached to a dexterous hand, is used to secure the IMU camera and the paper mirror; A tactile sensor, installed in the dexterous hand, is used to acquire pressure sensing data from the fingers of the dexterous hand; A computer program product for performing the method according to any one of claims 1-3.

Images