Universal robot dexterous hand grabbing teaching device and grabbing strategy learning method

Abstract

The invention discloses a universal robot dexterous hand grabbing teaching device and a grabbing strategy learning method. The method comprises the following steps: acquiring multi-modal data; establishing an SLAM model, carrying out real-time pose calculation on the multi-modal data, and returning the multi-modal data in the SLAM model in real time in combination with the action of the dexterous hand; the method comprises the following steps: constructing a learning network comprising a visual encoder ViT, a multi-head cross attention mechanism, a force perception encoder MLP, a splicing module and a Diffusion Policy; training a learning model; generating an action conforming to a target; obtaining delay compensation data; obtaining an updated robot action generation strategy; and predicting a subsequent action sequence in real time. The device comprises an IMU camera, a paper mirror, a fixing device, a touch sensor and a computer program product. According to the teaching device and the grabbing strategy learning method, the teaching device and the grabbing strategy learning method can be suitable for various different robot platforms, and execution of various complex tasks such as operation tasks in the fields of industrial automation, family service, medical care and the like is achieved through imitation learning.

Description

Technical Field

[0001] The present invention belongs to the technical field of dexterous hand control design, and relates to a general robot dexterous hand grasping teaching device and a grasping strategy learning method. Background Art

[0002] With the development of robot technology, robots are increasingly widely used in industrial automation, home service and medical fields. Especially the progress of dexterous hand technology provides new possibilities for robots to operate in multi-degree-of-freedom and complex environments. However, existing robot grasping teaching systems usually rely on expensive sensor devices, have limited operating environments, and are difficult to flexibly handle diverse task scenarios. Currently, the fusion of multi-modal data and precise motion control are the key technologies for achieving efficient teaching and task execution.

[0003] In this context, traditional methods mostly use vision-based or force-feedback methods to achieve robot task learning, but these methods are often limited by hardware such as sensor accuracy and execution delay, and it is difficult to complete efficient learning of dynamic tasks or complex actions. In addition, the time synchronization problem of different observation streams and action streams in existing systems has not been effectively solved, resulting in errors easily occurring during multi-modal data fusion and affecting the final task execution effect.

[0004] Therefore, there is an urgent need for a grasping strategy learning method with low equipment threshold that can achieve efficient and precise control of a robot dexterous hand. Summary of the Invention In view of the deficiencies of the prior art, the present application provides a general robot dexterous hand grasping teaching device and a grasping strategy learning method. The method aims to solve problems such as dependence on expensive sensors, inaccurate data synchronization, and complex task learning in existing robot teaching systems. The method provided by the present application can achieve flexible and portable data acquisition through the combination of a handheld dexterous hand and an IMU camera, and model and learn complex actions for tasks through Diffusion Policy to ensure the efficiency and precision of robot operations, without relying on expensive hardware devices.

[0005] In a first aspect, a general robot dexterous hand grasping strategy learning method is provided, including: Obtaining multi-modal data; the multi-modal data includes image data of dexterous hand movement, IMU data, tactile data, and operating environment data; Establishing a SLAM model, performing real-time pose calculation on the multi-modal data, and combining the actions of the dexterous hand to be transmitted back in real time in the SLAM model; Constructing a learning network including a visual encoder ViT, a multi-head cross-attention mechanism, a force perception encoder MLP, a splicing module, and a Diffusion Policy; Train a learning model; Use the trained learning model to generate a random initial action distribution based on the current environmental observation, and through a multi-step process of reverse diffusion, gradually denoise from the noise to generate an action that meets the target; Synchronize the observation streams of the robotic arm and the dexterous hand and the latency of the multi-modal data to obtain latency-compensated data; Use the latency-compensated data to perform latency compensation on the action sequence that meets the target to obtain an updated robot action generation strategy; Input the back-transmitted multi-modal data into the updated robot action generation strategy to predict the subsequent action sequence in real time.

[0006] In a possible implementation manner, the method for performing real-time pose calculation on multi-modal data includes: performing real-time pose calculation through the ORB-SLAM3 algorithm to generate a 6-degree-of-freedom trajectory of the camera.

[0007] In a possible implementation manner, the method for combining the actions of the dexterous hand to perform real-time back-transmission in the SLAM model includes: Use the image data and IMU data to obtain the pose of the camera, and then obtain the pose information of the data acquisition device; transmit the pose information back to the SLAM model.

[0008] In a possible implementation manner, in the learning network, a visual encoder ViT (Vision Transformer) is used to encode image data into a visual feature representation; A force perception encoder MLP is used to encode tactile data into a force perception feature representation; A multi-head cross-attention mechanism is used to fuse the visual feature representation and the force perception feature representation to obtain bimodal fusion data; A splicing module is used to splice the bimodal fusion data and the robot state data to obtain multi-modal fusion data; Diffusion Policy is used to gradually add random noise to the multi-modal fusion data through the forward process of the diffusion model to form a progressive process from real data to a high-noise distribution; through the reverse diffusion process, gradually remove the noise through multiple time steps until an action sequence that meets the target is restored.

[0009] In a possible implementation manner, the training of the learning model is performed using a training set; the features in the training set include the image data and IMU data of the dexterous hand, and the labels include the action data executed by the dexterous hand, visual input data, object position data, and force feedback data.

[0010] In a possible implementation manner, the method for synchronizing the observation streams of the robotic arm and the dexterous hand and the delay of the multi-modal data to obtain delay compensation data includes: Alignment of the observation streams: By measuring the delay of each observation stream and performing time synchronization based on the stream with the highest delay; Alignment of the action streams: By synchronizing the control frequencies of the dexterous hand and the robotic arm to align the action streams.

[0011] In a second aspect, a general robot dexterous hand grasping strategy learning device is provided, including: An acquisition module, configured to acquire multi-modal data; the multi-modal data includes image data, IMU data, and operating environment data of the dexterous hand movement; A model construction module, configured to establish a SLAM model, perform real-time pose calculation on the multi-modal data, and combine the actions of the dexterous hand to perform real-time feedback in the SLAM model; A learning model construction module, configured to construct a learning network including a visual encoder ViT, a multi-head cross-attention mechanism, a force perception encoder MLP, a splicing module, and a Diffusion Policy; A training module, configured to train the learning model; A generation module, configured to use the trained learning model to generate a random initial action distribution according to the current environmental observation, and gradually denoise from the noise through a multi-step process of reverse diffusion to generate an action that meets the target; A compensation module, configured to synchronize the observation streams of the robotic arm and the dexterous hand and the delay of the multi-modal data to obtain delay compensation data; An update module, configured to perform delay compensation on the action sequence that meets the target by using the delay compensation data to obtain an updated robot action generation strategy; An output module, configured to input the feedback multi-modal data into the updated robot action generation strategy to predict the subsequent action sequence in real time.

[0012] In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, the learning method described in the first aspect is implemented.

[0013] In a fourth aspect, a computer program product is provided, including a computer program, where when the computer program is executed by a processor, the learning method described in the first aspect is implemented.

[0014] In a fifth aspect, a general robot dexterous hand grasping teaching device is provided, including: An IMU camera, configured to collect image data and IMU data of the dexterous hand movement; Paper mirror, arranged on both sides of the IMU camera, for expanding the viewing angle range of the IMU camera; Fixing device, connected to the dexterous hand, for fixing the IMU camera and the paper mirror; Tactile sensor, arranged on the dexterous hand, for obtaining pressure perception data of the fingers of the dexterous hand; Computer program product, which is used to execute the method according to any one of claims 1-6.

[0015] The beneficial effects of this application are as follows: (1) The grasping strategy learning method provided by this application can achieve efficient data collection and precise positioning, and through the IMU camera and the ORB-SLAM3 algorithm, it can collect multi-modal data during complex operations, accurately position the camera pose, and ensure the accuracy of robot operations.

[0016] (2) The grasping strategy learning method provided by this application uses the Diffusion Policy strategy to train the neural network model through multi-modal data, so as to learn and generate the action sequence required for the robot to perform complex tasks. The gradual denoising process of the diffusion model ensures that the generated actions are stable and meet the task requirements.

[0017] (3) The grasping strategy learning method provided by this application ensures that there will be no delay error in the data fusion process through the synchronous processing of the observation stream and the action stream, especially by compensating for the delay of the action sequence output by the strategy, ensuring the precise synchronization of the dexterous hand and the manipulator operation.

[0018] (4) This application also provides a general robot dexterous hand grasping teaching device, which includes an IMU camera, a paper mirror and a fixing device, and does not need to rely on expensive sensor equipment, and can flexibly and portably perform data collection and teaching tasks.

[0019] (5) The teaching device and the grasping strategy learning method described in this application can be applied to various different robot platforms, and can execute a variety of complex tasks through imitation learning, such as operation tasks in the fields of industrial automation, home service and medical care. Description of the Drawings

[0020] Figure 1 It is a schematic flow chart of the general robot dexterous hand grasping strategy learning method provided by the embodiment of this application; Figure 2 It is a schematic flow chart of the Diffusion Policy strategy learning training and inference process provided by the embodiment of this application; Figure 3 It is a schematic structural diagram of the learning network provided by the embodiment of this application; Figure 4A general robot dexterous hand grasping strategy learning device provided by an embodiment of the present application; Figure 5 A schematic structural diagram of a general robot dexterous hand grasping teaching device provided by an embodiment of the present application; Figure 6 A schematic layout structure diagram of a general robot dexterous hand mirror provided by an embodiment of the present application; Figure 7 An application example of a general robot dexterous hand grasping teaching device provided by an embodiment of the present application.

[0021] Figure 5 、 Figure 7 In and : 1 - IMU camera, 2 - paper mirror, 3 - fixing device, 4 - dexterous hand, 5 - tactile sensor, 6 - robotic arm. Specific implementation manners

[0022] In order to make the technical problems and technical solutions to be solved by the present invention clearer, the following will combine the accompanying drawings and embodiments to detail a general robot dexterous hand grasping teaching system and learning strategy provided by the present invention. The content is an explanation of the present invention, rather than a limitation of the present invention.

[0023] See Figure 1 , which is a general robot dexterous hand grasping strategy learning method, including the following steps: S100. Obtain multi-modal data; the multi-modal data includes image data, IMU data, tactile data, and operating environment data of the dexterous hand movement.

[0024] In a possible implementation manner, the operating environment data refers to the specific parameters and conditions of the direct working scene when it performs tasks, and these data directly affect the operation accuracy, safety, and efficiency of the robotic arm. For example, temperature and humidity, etc.

[0025] S200. Establish a SLAM model, perform real-time pose calculation on the multi-modal data, and combine the actions of the dexterous hand to perform real-time feedback in the SLAM model.

[0026] In a possible implementation manner, in S200, the method for real-time pose calculation includes: performing real-time pose calculation through the ORB-SLAM3 algorithm to generate a 6-degree-of-freedom trajectory of the camera.

[0027] In a possible implementation manner, in S200, the method for combining the actions of the dexterous hand to perform real-time feedback in the SLAM model includes: using the image data and IMU data, combining the ORB-SLAM3 open-source model to obtain the pose of the camera, and then obtaining the pose information of the data acquisition device; transmitting the pose information to the SLAM model.

[0028] S300. Construct a learning network including a vision encoder ViT, a multi-head cross-attention mechanism, a force perception encoder MLP, a splicing module, and a Diffusion Policy; Among them, the vision encoder ViT (Vision Transformer) is used to encode image data into visual feature representations; The force perception encoder MLP is used to encode tactile data into force perception feature representations; The multi-head cross-attention mechanism is used to fuse visual feature representations and force perception feature representations to obtain bimodal fusion data; The splicing module is used to splice the bimodal fusion data and robot state data to obtain multi-modal fusion data; The Diffusion Policy is used to gradually add random noise to the multi-modal fusion data through the forward process of the diffusion model, forming a progressive process from real data to a high-noise distribution; through the reverse diffusion process, the noise is gradually removed through multiple time steps until an action sequence that meets the target is restored.

[0029] Figure 3 The structural schematic diagram of the learning model is shown.

[0030] It should be noted that the Diffusion Policy can be understood as the application of the diffusion model in robot control and can combine imitation learning to learn policies by observing the demonstrations of human experts. In each iteration of this method, the noise prediction network (εθ) predicts the noise gradient of the current noise sample, which is then used to update the sample and gradually denoise. Through random Langevin dynamics steps, the Diffusion Policy iteratively optimizes the action sequence until the final action is generated.

[0031] In a possible implementation manner, the action sequence that meets the target includes robotic arm pose information and dexterous hand grasping states.

[0032] In a possible implementation manner, the structure of the diffusion model includes: the network of the diffusion model includes a 1D U-net model.

[0033] In a possible implementation manner, the robot state data includes pose information and dexterous hand grasping states, with a total of 7 dimensions.

[0034] It can be understood that referring to Figure 3 , the data flows in the learning model as follows: In order to capture the pressure information at the first moment when the dexterous hand contacts the object to determine the grasping state of the dexterous hand, in this embodiment, Array low-latency tactile sensor, which is used to obtain tactile data such as pressure perception data.

[0035] In the design of the policy learning network structure, the visual image observation input is mapped to a low-dimensional space z0 after passing through a visual encoder ViT (Vision Transformer). The pressure perception data obtained by the tactile sensor is mapped into the low-dimensional space through an MLP. Then, the pressure sensor data and the visual image data are fused through cross-attention to output a joint embedding vector z1. Then, through a splicing module, z1 is spliced with the robot state vector to obtain a complete network input vector z2. Finally, z2 is input into the Diffusion Policy to obtain the output action sequence At.

[0036] S400. Train the learning model.

[0037] In a possible implementation, S400 includes the following sub-steps: S410. Obtain the training set.

[0038] Furthermore, the features of the training set include the image data and IMU data of the dexterous hand, and the labels include the action data of the dexterous hand execution, visual input data, object position data, and force feedback data.

[0039] It should be noted that the action data of the dexterous hand execution refers to the set of various motion parameters, perception feedback, and control instructions when it simulates the fine operations of the human hand.

[0040] The visual input data refers to the image data obtained by camera shooting.

[0041] The object position data refers to the end position where the dexterous hand approaches the target object.

[0042] The force feedback data refers to The data returned by the array low-latency tactile sensor.

[0043] S420. Use the training set to train the learning model.

[0044] S400. Use the trained learning model to generate a random initial action distribution according to the current environmental observation, and through a multi-step process of reverse diffusion, gradually denoise from the noise to generate an action that meets the target.

[0045] S500. Synchronize the delays of the observation streams of the robotic arm and the dexterous hand and the multi-modal data to obtain delay compensation data.

[0046] In a possible implementation, S500 includes the following sub-steps: S510. Alignment of the observation stream: By measuring the latency of each observation stream and synchronizing time based on the stream with the highest latency.

[0047] Specifically, use timestamps to unify the sampling of vision, touch, and robot state (i.e., the observation stream), and each sensor device performs time synchronization through a shared clock.

[0048] S520. Alignment of the action stream: Align the action stream by synchronizing the control frequencies of the dexterous hand and the robotic arm.

[0049] Exemplarily, set the control frequency of the Bluetooth-controlled dexterous hand to 10 hz, and also set the control frequency of the robotic arm to 10 hz, and synchronize the control frequency information of the two through programming. That is, the inference frequency is 10 hz, and each time an inference is made, after passing through a policy learning network once, a 7D action sequence (6D end-effector pose of the robotic arm + grasping state of the dexterous hand) is obtained.

[0050] S600. Use the latency compensation data to perform latency compensation on the action sequence that meets the target, and obtain an updated robot action generation policy.

[0051] S700. Input the back-transmitted multi-modal data into the updated robot action generation policy to predict the subsequent action sequence in real time.

[0052] See Figure 4 , which is a general robot dexterous hand grasping strategy learning device, including: An acquisition module 41, configured to acquire multi-modal data; the multi-modal data includes image data of the dexterous hand movement, IMU data, and operation environment data; A model construction module 42, configured to establish a SLAM model, perform real-time pose calculation on the multi-modal data, and combine the actions of the dexterous hand to transmit back in the SLAM model in real time; A learning model construction module 43, configured to construct a learning network including a vision encoder ViT, a multi-head cross-attention mechanism, a force perception encoder MLP, a splicing module, and a Diffusion Policy; A training module 44, configured to train the learning model; A generation module 45, configured to use the trained learning model to generate a random initial action distribution according to the current environmental observation, and gradually denoise from the noise through a multi-step process of reverse diffusion to generate an action that meets the target; A compensation module 46, configured to synchronize the latencies of the observation streams of the robotic arm and the dexterous hand and the multi-modal data to obtain latency compensation data; An update module 47, configured to perform delay compensation on the action sequence meeting the target by using delay compensation data, so as to obtain an updated robot action generation strategy; An output module 48, configured to input the back-transmitted multimodal data into the updated robot action generation strategy to predict the subsequent action sequence in real time.

[0053] On the other hand, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the grasping strategy learning method provided by the above-mentioned various methods is implemented.

[0054] On another aspect, the present invention further provides a computer program product. The computer program product includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the grasping strategy learning method provided by the above-mentioned various methods.

[0055] See Figure 5 , which is a general robot dexterous hand grasping teaching device, including: An IMU camera 1, configured to collect image data and IMU data of the dexterous hand movement; Paper mirrors 2, arranged on both sides of the IMU camera 1, configured to expand the viewing angle range of the IMU camera 1; A fixing device 3, connected to the dexterous hand 4, configured to fix the IMU camera 1 and the paper mirrors 2; A tactile sensor 5, arranged on the dexterous hand 4, configured to obtain pressure perception data of the dexterous hand fingers; A computer program product, which is configured to execute the grasping strategy learning method provided by the above-mentioned various methods.

[0056] In a possible implementation manner, the reflecting surfaces of the paper mirrors 2 are arranged on both sides of the IMU camera lens, and the reflecting surfaces expand outward relative to the camera. Figure 6 A layout structure is shown.

[0057] In a possible implementation manner, the fixing device is obtained by 3D printing.

[0058] It can be understood that the structure of the fixing device is adaptively designed according to the structure of the specific dexterous hand, as long as the fixing requirements can be met.

[0059] See Figure 7 This is a schematic diagram of the teaching device described in this embodiment for a robotic arm.

[0060] As described above, it is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, improvements, etc. made by those skilled in the art within the technical scope disclosed by the present invention shall be included within the scope of protection of the invention.

Claims

1. A learning method for the grasping strategy of a general robot dexterous hand, characterized in that, Including: Obtain multimodal data; the multimodal data includes image data of dexterous hand movement, IMU data, tactile data, and operation environment data; Establish a SLAM model, perform real-time pose calculation on the multimodal data, and combine the actions of the dexterous hand to perform real-time feedback in the SLAM model; Construct a learning network including a visual encoder ViT, a multi-head cross-attention mechanism, a force perception encoder MLP, a splicing module, and a Diffusion Policy; Train the learning model; Use the trained learning model to generate a random initial action distribution based on the current environmental observation, and through a multi-step process of reverse diffusion, gradually denoise from the noise to generate an action that meets the target; Synchronize the observation streams of the robotic arm and dexterous hand and the latency of the multimodal data to obtain latency compensation data; Use the latency compensation data to perform latency compensation on the action sequence that meets the target to obtain an updated robot action generation strategy; Input the feedback multimodal data into the updated robot action generation strategy to predict the subsequent action sequence in real time.

2. The learning method according to claim 1, characterized in that, The method for performing real-time pose calculation on the multimodal data includes: performing real-time pose calculation through the ORB-SLAM3 algorithm to generate a 6-degree-of-freedom trajectory of the camera.

3. The learning method according to claim 1, wherein The method for combining the actions of the dexterous hand to perform real-time feedback in the SLAM model includes: Use the image data and IMU data to obtain the pose of the camera, and then obtain the pose information of the data acquisition device; transmit the pose information back to the SLAM model.

4. The learning method according to claim 1, characterized in that, In the learning network, the visual encoder ViT (Vision Transformer) is used to encode the image data into a visual feature representation; The force perception encoder MLP is used to encode the tactile data into a force perception feature representation; The multi-head cross-attention mechanism is used to fuse the visual feature representation and the force perception feature representation to obtain bimodal fusion data; The splicing module is used to splice the bimodal fusion data and the robot state data to obtain multimodal fusion data; Diffusion Policy is used to gradually add random noise to the multimodal fusion data through the forward process of the diffusion model to form a progressive process from real data to a high-noise distribution; through the reverse diffusion process, gradually remove the noise through multiple time steps until an action sequence that meets the target is restored.

5. The learning method according to claim 1, characterized in that The training of the learning model is performed using a training set; the features in the training set include image data and IMU data of the dexterous hand, and the labels include dexterous hand execution action data, visual input data, object position data, and force feedback data.

6. The learning method according to claim 1, wherein The method for synchronizing the observation streams of the robotic arm and dexterous hand and the latency of the multimodal data to obtain latency compensation data includes: Alignment of the observation stream: Measure the latency of each observation stream and perform time synchronization based on the stream with the highest latency; Alignment of the action stream: Align the action stream by synchronizing the control frequencies of the dexterous hand and the robotic arm.

7. A learning device for the grasping strategy of a general robot dexterous hand, characterized in that, Including: An acquisition module for acquiring multimodal data; the multimodal data includes image data of dexterous hand movement, IMU data, and operation environment data; The model construction module is used to establish a SLAM model, perform real-time pose calculation on multi-modal data, and combine the actions of the dexterous hand to transmit back in real time in the SLAM model; The learning model construction module is used to construct a learning network including a visual encoder ViT, a multi-head cross-attention mechanism, a force perception encoder MLP, a splicing module, and a Diffusion Policy; The training module is used to train the learning model; The generation module is used to use the trained learning model to generate a random initial action distribution according to the current environmental observation, and through a multi-step process of reverse diffusion, gradually denoise from the noise to generate an action that meets the target; The compensation module is used to synchronize the observation streams of the robotic arm and the dexterous hand and the delay of the multi-modal data to obtain delay compensation data; The update module is used to perform delay compensation on the action sequence that meets the target by using the delay compensation data to obtain an updated robot action generation strategy; The output module is used to input the transmitted multi-modal data into the updated robot action generation strategy to predict the subsequent action sequence in real time.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the learning method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the learning method according to any one of claims 1 to 6.

10. A teaching device for a general robot dexterous hand grasping, characterized in that, Including: An IMU camera is used to collect image data and IMU data of the dexterous hand movement; Paper mirrors are arranged on both sides of the IMU camera to expand the viewing angle range of the IMU camera; A fixing device is connected to the dexterous hand to fix the IMU camera and the paper mirrors; Tactile sensors are arranged on the dexterous hand to obtain pressure perception data of the fingers of the dexterous hand; A computer program product, and the computer program product is used to execute the method according to any one of claims 1-6.

Images