Robot control method based on tactile prediction pre-training

Abstract

A robot control method based on tactile prediction pre-training comprises the following steps: acquiring and generating a human playing data set consisting of three-channel image tensors in an offline stage, and training a constructed conditional diffusion model comprising a tactile encoder, a tactile decoder and an action and visual encoder; in the online stage, the trained conditional diffusion model is integrated into a standard imitation learning strategy network, and an action instruction of the robot is generated according to the state of the robot, the current visual features and the tactile feature vectors extracted by the imitation learning strategy network. According to the method, a specific agent task is completed by training a deep neural network model, that is, a future tactile signal sequence is predicted according to historical information and future action intentions; the model is enabled to characterize generic haptic features contacting physical dynamic laws for further migration into downstream robot control tasks.

Description

Technical Field

[0001] This invention relates to a technology in the field of robot control, specifically a robot control method based on tactile prediction and pre-training. Background Technology

[0002] In the field of robot imitation learning, enabling robots to master the ability to physically interact with their environment, especially fine manipulation relying on tactile perception, is a core challenge. Existing tactile learning algorithms mainly face the following fundamental shortcomings:

[0003] a) Data sparsity problem in direct imitation learning: In typical robot operations, effective tactile contact signals occur only in a very small number of keyframes. When training multimodal imitation learning strategies end-to-end, the model struggles to effectively learn general tactile skills from these sparse samples, and the learned features are highly coupled to specific tasks, resulting in poor generalization ability.

[0004] b) The problem of incorrect inductive bias in self-supervised learning: Existing self-supervised tactile learning methods, such as contrastive learning or masked learning, often incorrectly borrow inductive biases from the field of computer vision. For example, these methods may assume the existence of translation invariance or geometric consistency in tactile signals, but this contradicts the physical nature of tactile perception (e.g., pressure at different locations on the fingertips has completely different physical meanings), resulting in learned representations that cannot accurately describe physical interactions.

[0005] c) The fundamental fallacy of visual-tactile alignment: Such methods attempt to learn joint representations by maximizing the similarity between visual and tactile information, but this ignores the fact that the two are typically in a one-to-many relationship in physical interaction (the same visual scene can correspond to multiple different tactile signals) and complementary. Touch provides crucial physical information about force and sliding that is imperceptible to vision, while alignment methods attempt to eliminate this difference, thus losing the most critical interactive information.

[0006] d) Generating motion sequences based on abstract text instructions can only achieve generative mapping from semantics to actions, rather than modeling physical laws. It does not have the ability to process and learn multimodal physical sensing data (tactile image sequences and six-degree-of-freedom pose data) that are precisely synchronized in time. Summary of the Invention

[0007] This invention addresses the shortcomings of existing technologies that generally treat tactile signals as isolated, static observation data, neglecting their essential nature as a dynamic process of motion perception and failing to effectively learn the inherent causal relationships of contact physics. It proposes a robot control method based on tactile prediction pre-training. This method trains a deep neural network model to complete a specific agent task: predicting future tactile signal sequences based on historical information and future action intentions. After optimizing this prediction task, the model can represent the general tactile characteristics of the dynamic laws of contact physics, which can then be further transferred to downstream robot control tasks.

[0008] This invention is achieved through the following technical solution:

[0009] This invention relates to a robot control method based on tactile prediction pre-training. In the offline stage, a human play dataset consisting of three-channel image tensors is collected and generated to train a conditional diffusion model containing tactile encoders, decoders, and action and visual encoders. In the online stage, the trained conditional diffusion model is integrated into a standard imitation learning policy network. Based on the robot's own state, current visual features, and tactile feature vectors extracted by the imitation learning policy network, the robot's action commands are generated.

[0010] Technical effect

[0011] This invention fuses tactile and robot state representations using a lightweight Transformer. Predicting future tactile signals is treated as a proxy task, and a diffusion-based tactile prediction model is pre-trained on a large-scale tactile database to obtain a generalized tactile representation vector. This vector can be used for downstream robot policy learning. Compared to existing technologies, this invention accurately predicts the dynamic changes of tactile signals and learns generalized, task-independent tactile representations during the prediction process, enabling application in any task. For example, it can correctly predict the impending disappearance of tactile signals based on future action sequences (such as ceasing force application), demonstrating a deep understanding of physical causality. In real-world robot tasks with extremely high force perception requirements, this method significantly outperforms baseline algorithms. For instance, in the task of pulling a drawer containing heavy objects, the success rate is increased to 95% (baseline method: 50%); in the precise insertion stage of a block into a hole, the success rate is increased to 80% (baseline method: 60%). In other contact-sensitive tasks, it also generally delivers performance gains exceeding 15%. Attached Figure Description

[0012] Figure 1 This is a flowchart of the present invention;

[0013] Figure 2 This is a schematic diagram of a portable handheld teaching device as an example.

[0014] Figure 3 Schematic diagram of a pre-trained network for tactile prediction;

[0015] Figure 4 This is a schematic diagram of the tactile signal prediction results;

[0016] Figure 5 This is a schematic diagram illustrating the robot's performance during a real-machine test. Detailed Implementation

[0017] like Figure 1 As shown, this embodiment relates to a robot control method based on tactile prediction pre-training, including:

[0018] Step 1: Collect data using a portable handheld teaching device, and then construct and preprocess the dataset after synchronizing the multimodal data timestamps.

[0019] like Figure 2 As shown, the portable handheld teaching device includes: a six-degree-of-freedom pose tracking module, a gripper opening / closing degree measurement module, a tactile sensing module, and a visual sensing module. The six-degree-of-freedom pose tracking module uses an augmented reality (AR) controller (such as a MetaQuest3 controller) to perform 6D pose tracking of the teaching pendant. The gripper opening / closing degree measurement module uses a non-contact AS5600 magnetic rotary encoder to accurately calculate the real-time opening / closing width of the gripper by measuring the rotation angle of the gripper joints. This module provides high-resolution (12-bit) readings and is unaffected by visual occlusion. The tactile sensing module installs vision-based tactile sensors on each of the two fingertips of the teaching pendant. Each sensor contains a camera and a set of LED light sources, sensing pressure and shear force by capturing the deformation of the silicone contact layer and outputting a sequence of tactile images. The visual sensing module uses a wide-angle camera (such as a GoPro) as a first-person perspective (FPV) visual input source to record external environmental information during operation.

[0020] The aforementioned multimodal data timestamp synchronization (LatencyCalibration) is performed once before the start of each data acquisition task, and specifically includes:

[0021] 1.1) Calibration action execution: The operator holds the teaching device and performs several horizontal periodic sweeps above the stationary ArUco visual marker.

[0022] 1.2) Trajectory data extraction: The system simultaneously records two one-dimensional motion trajectories: one is the X-axis displacement trajectory f(t) from the AR pose tracking module; the other is the X-axis displacement trajectory g(t) of the ArUco visual marker in the image coordinate system, which is detected in real time by the video stream of the visual perception module.

[0023] 1.3) Delay optimization calculation: based on the time offset, i.e., delay The objective is to minimize the two trajectories, which are then solved using a binary search algorithm. The algorithm aims to find the closest trajectories after temporal alignment, specifically by considering the delay. ,in: The sampling time point is used to perform iterative search within a preset delay range. In each iteration, the current search interval is divided into M points, the MSE loss of each point is calculated, and then the point with the smallest loss and its neighborhood are selected as the new search interval until the interval range is less than the preset precision threshold (e.g., 0.0001 seconds).

[0024] 1.4) Data alignment application: Apply the delay calculated in step 1.3 It is applied to the alignment of all subsequent data streams and generates multimodal data frames that are precisely synchronized in time.

[0025] For example, the timestamps of AR pose data will be uniformly corrected and then aligned with the frame timestamps of other sensors (such as cameras and haptic sensors) through linear interpolation, thereby stabilizing the synchronization error between different data streams within 5 milliseconds.

[0026] The aforementioned dataset construction and preprocessing specifically include:

[0027] A) Dataset Construction: By having the instructor hold the above-mentioned device and interact with more than 300 different objects (covering rigid, deformable, and granular) by grasping, holding, pushing, and pulling, a human play dataset containing more than 1 million frames of data was constructed, of which the proportion of effective tactile contact frames exceeded 60%, providing rich samples for the model to learn contact dynamics.

[0028] B) Tactile Image Processing: To provide the model with richer tactile information, the original tactile images at each time step in the dataset obtained in step A are processed as follows: First, they are converted into grayscale images; then, by comparing them pixel-level with a pre-recorded reference image in a non-contact state, two binary images are generated: one is a concave map, where a pixel value of 1 indicates that a pressure indentation has occurred at that location; the other is a convex map, where a pixel value of 1 indicates that a deformation convexity has occurred at that location due to lateral force. Finally, the original grayscale image, concave map, and convex map are stacked in the channel dimension to form a 3-channel image tensor, which serves as the standard input for subsequent models.

[0029] Step 2: Construct a conditional diffusion model, specifically including:

[0030] 2.1 Constructing a probabilistic model to predict the future The tactile signals from each step are used to learn a general tactile encoder, specifically: Where: T is the tactile image sequence, and A is the action sequence. V represents the visual image at the current moment, where t is the current time tick and V is the visual image. As a historical time window, For the future time window.

[0031] In this embodiment, N=8. =4、 =4.

[0032] 2.2 Data parameterization and input representation, specifically: For the input tactile image sequence T, the resolution of the 3-channel tactile image at each time step is unified to 224×224 pixels; For the input motion sequence A, the motion at each time step is parameterized into a 7-dimensional vector, including the relative pose changes of the robot's end effector in the world coordinate system (Δx, Δy, Δz, Δroll, Δpitch, Δyaw) and the opening and closing state of the gripper (scalar); For the input visual image V, the resolution is unified to 224×224 pixels.

[0033] 2.3 such as Figure 3 As shown, a tactile prediction pre-training network is built, specifically including:

[0034] A) The multimodal coding component includes: a tactile encoder and decoder for converting complex image information into easily processed low-dimensional manifolds, and an action and visual encoder for generating feature embedding sequences and feature embedding vectors, wherein: the tactile encoder High-dimensional tactile images at each time step Compressed into a low-dimensional latent space vector ; Haptic decoder Reconstructing tactile images from latent vectors, motion encoder The sequence of actions for the entire window Encoding as a sequence of feature embeddings; visual encoder Current visual image Encode it as a single feature embedding vector.

[0035] The tactile encoder and decoder are implemented using a pre-trained KL-F16 variational autoencoder.

[0036] The motion and visual encoders described use a standard Transformer or Visual Transformer (ViT) network structure.

[0037] B) The core prediction component, namely the Conditional Latent Space Diffusion Model (LDM), includes: a historical information fusion unit, a conditional information construction unit, and a conditional denoising prediction unit. The historical information fusion unit converts the tactile latent vector sequence of historical time steps... The embedding sequence of future actions is fused with the corresponding historical action embedding sequence; the conditional information construction unit fuses the embedding sequence of future actions with the current visual embedding to construct a conditional vector containing a description of the future intent and the current scene. The conditional denoising prediction unit obtains the predicted tactile representation vector for future moments, i.e., the latent vector sequence, through iterative processing of adding noise, denoising, and conditional injection. .

[0038] The aforementioned fusion is achieved through the Transformer module using the cross-attention mechanism, generating a fused feature vector representing the historical context. .

[0039] like Figure 3 As shown, the conditional denoising prediction unit includes: a noise-adding scheduler unit and a U-Net-based Transformer noise prediction unit, wherein: the noise-adding scheduler unit receives the fused feature vector. At the current time step s, noise e is added to the feature vector to obtain a noisy feature vector; the Transformer noise prediction unit receives the noisy features at any time step, and performs cross-attention calculation with the image features and robot state features as conditions and the noisy multimodal features. The Transformer module obtains the predicted value of the noise added at the current time step s, and calculates the tactile features with noise removed based on the predicted value of the noise.

[0040] The iterative process specifically includes:

[0041] i. Adding noise: During the training phase, the real future tactile latent vector sequence is processed by a noisy scheduler unit. Gaussian noise is gradually added within a preset discrete time step S until its distribution is close to pure Gaussian noise.

[0042] ii. Denoising: During the generation phase, noise prediction is performed using the Transformer noise prediction unit. Starting with a random Gaussian noise tensor, noise is iteratively removed in S steps: in each denoising iteration s, The system receives three inputs: the noisy latent variable of the current step, the encoding of the current denoising step number s, and the conditional vector constructed above. .

[0043] iii. Conditional Injection: This involves injecting information about future actions and the current visual context. As a condition, noise prediction units of the Transformer are injected via a cross-attention mechanism. In each layer of U-Net, the model's prediction process is explicitly guided to predict tactile changes that are more in line with physical laws.

[0044] Step 3, offline pre-training and downstream task application stage, specifically includes:

[0045] 3.1 In the offline pre-training stage, the above model is pre-trained on a large scale using a human play dataset to minimize the mixture loss function. The training objective is denoted as , where: is used to optimize the Transformer noise prediction unit. diffusion loss , The vector represents the true future tactile latent vector, ϵ represents random noise, and s represents the number of denoising steps. These are preset noise scheduling coefficients; used to optimize the reconstruction loss of the VAE. = λ is the weighting coefficient that balances the two losses.

[0046] After offline pre-training, a general, task-independent haptic encoder is obtained. That is, the encoding part of VAE.

[0047] 3.2 In the downstream task application stage, the tactile encoder in the pre-trained conditional diffusion model is frozen. After obtaining the network parameters, the pre-trained visual encoder will be... As a plug-and-play feature extraction module, it is integrated into a standard imitation learning policy network, which takes at each time step the robot's own state (such as joint angles), the current visual features extracted by a separate visual backbone network, and the tactile encoder as inputs. High-quality tactile feature vectors extracted It outputs the robot's next action command. Due to tactile characteristics... Having already contained rich physical dynamic information, policy networks can more easily learn to handle contact-sensitive tasks.

[0048] Through specific practical experiments, this method was validated on a robot platform using a FlexivRizon4 collaborative robotic arm, a FlexivGrav adaptive gripper, and a single GoPro camera as the visual input source: First, the agent task was trained and evaluated on a dataset of more than 1 million frames of human play, with a training hardware environment of 4 NVIDIA H100 GPUs and a total training time of 120 hours.

[0049] Intermediate experimental data show that when historical tactile sensation, complete future and historical action sequences, and current visual images are integrated as all conditional inputs to the model, the model has the lowest mean squared error (MSE) for predicting future tactile sensation, reaching 0.0099. This proves that the motion perception prediction framework of this invention can capture physical contact dynamics most accurately.

[0050] Building upon this, the pre-trained and parameter-frozen haptic encoder is integrated into the downstream imitation learning diffusion policy network, and then... Figure 5 The nine tasks shown were comprehensively evaluated on real-world robotic tasks of varying difficulty and scenario. These tasks included not only routine tasks such as picking up different objects (e.g., cubes, broccoli), inserting pens into tubes, and stacking blocks with sides of 5 cm, but also contact-sensitive tasks requiring high force perception, such as placing soft balls, unscrewing bottle caps, pulling drawers containing random weights, and precisely inserting a 4 cm × 3 cm block into a 4.3 cm × 3.3 cm slot.

[0051] The final results are shown in Table 1. The tactile representation pre-trained using this invention significantly improves the robot's task success rate, especially in the aforementioned challenging tasks. For example, in the drawer-pulling task, which requires sensing heavy objects inside and adjusting the strategy, the success rate of this method reaches 95%, while the baseline method (vision + ordinary tactile feedback) only reaches 50%. In the precise insertion stage of the block-in-the-hole task, the success rate of this method reaches 80%, significantly higher than the baseline method's 60%. In other contact-sensitive tasks, this method also generally brings performance gains of over 15%.

[0052] Table 1

[0053] Compared to existing technologies that typically treat tactile signals as independent, static observation data or attempt to align them with vision, thus ignoring the dynamic nature of touch as a result of interaction between action and environment, this invention considers future action sequences... and current visual image As an explicit condition, to predict future tactile image sequences This innovative step forces neural network models to move beyond simply encoding static tactile patterns and instead learn and understand the deeper physical causal relationships—the physical consequences of performing a certain action in the current environment. Therefore, the tactile encoder pre-trained using this method... The output feature vector embodies this understanding of physical dynamics. When this high-quality, task-independent feature vector is applied to the downstream task of step 3.2, the imitation learning policy network can make decisions based on richer and more physically meaningful information, thereby achieving a success rate far exceeding the baseline method in tasks with extremely high requirements for force perception, sliding, and precise contact (such as pulling a heavy drawer or inserting a block into a hole).

[0054] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A robot control method based on tactile prediction pre-training, characterized in that, In the offline phase, a human play dataset consisting of three-channel image tensors is collected and generated to train a conditional diffusion model containing tactile encoders, decoders, and action and visual encoders. In the online phase, the trained conditional diffusion model is integrated into a standard imitation learning policy network to generate the robot's action commands based on the robot's own state, current visual features, and tactile feature vectors extracted by the imitation learning policy network.

2. The robot control method based on tactile prediction pre-training according to claim 1, characterized in that, specifically... include: Step 1: Collect data using a portable handheld teaching device, and then construct and preprocess the dataset after synchronizing the multimodal data timestamps; Step 2: Construct a conditional diffusion model, specifically including: 2.1 Constructing a probabilistic model to predict the future The tactile signals from each step are used to learn a general tactile encoder, specifically: Where: T is the tactile image sequence, and A is the action sequence. V represents the visual image at the current moment, where t is the current time tick and V is the visual image. As a historical time window, For the future time window; 2.2 Data parameterization and input representation, specifically: For the input tactile image sequence T, the resolution of the 3-channel tactile image at each time step is unified to 224×224 pixels; for the input motion sequence A, the motion at each time step is parameterized into a 7-dimensional vector, including the relative pose changes of the robot's end effector in the world coordinate system (Δx, Δy, Δz, Δroll, Δpitch, Δyaw) and the opening and closing state of the gripper (scalar); for the input visual image V, the resolution is unified to 224×224 pixels; 2.3 Construct a haptic prediction pre-training network that includes a multimodal coding part and a core prediction part; Step 3: Offline pre-training and downstream task application stage.

3. The robot control method based on tactile prediction pre-training according to claim 2, characterized in that, The multimodal data timestamp synchronization is performed once before each data acquisition task begins, specifically including: 1.1) Calibration action execution: The operator holds the teaching device and performs several periodic horizontal sweeps above the stationary ArUco visual marker; 1.2) Trajectory data extraction: The system simultaneously records two one-dimensional motion trajectories: one is the X-axis displacement trajectory f(t) from the AR pose tracking module; the other is the X-axis displacement trajectory g(t) of the ArUco visual marker in the image coordinate system, which is detected in real time by the video stream of the visual perception module. 1.3) Delay optimization calculation: based on the time offset, i.e., delay The objective is to minimize the two trajectories, which are then solved using a binary search algorithm. The algorithm aims to find the closest trajectories after temporal alignment, specifically by considering the delay. ,in: The sampling time point is used to perform iterative search within a preset delay range. In each iteration, the current search interval is divided into M points, the MSE loss of each point is calculated, and then the point with the smallest loss and its neighborhood are selected as the new search interval until the interval range is less than the preset precision threshold. 1.4) Data alignment application: Apply the delay calculated in step 1.3 It is applied to the alignment of all subsequent data streams and generates multimodal data frames that are precisely synchronized in time.

4. The robot control method based on tactile prediction pre-training according to claim 2, characterized in that, The aforementioned dataset construction and preprocessing specifically include: A) Dataset Construction: By having the instructor hold the above-mentioned device and interact with more than 300 different objects by grabbing, holding, pushing and pulling, a human play dataset containing more than 1 million frames of data was constructed, of which the proportion of effective tactile contact frames exceeded 60%, providing rich samples for the model to learn contact dynamics. B) Tactile Image Processing: To provide the model with richer tactile information, the original tactile images at each time step in the dataset obtained in step A are processed as follows: First, they are converted into grayscale images; then, by comparing them pixel-level with a pre-recorded reference image in a non-contact state, two binary images are generated: one is a concave pixel image, with a pixel value of 1 indicating that pressure indentation has occurred at that location; the other is a convex pixel image, with a pixel value of 1 indicating that deformation convexity has occurred at that location due to lateral force. Finally, the original grayscale image, concave image, and convex image are stacked in the channel dimension to form a 3-channel image tensor, which serves as the standard input for subsequent models.

5. The robot control method based on tactile prediction pre-training according to claim 2, characterized in that, The multimodal coding section includes: a tactile encoder and decoder for converting complex image information into easily processed low-dimensional manifolds, and an action and visual encoder for generating feature embedding sequences and feature embedding vectors, wherein: the tactile encoder High-dimensional tactile images at each time step Compressed into a low-dimensional latent space vector ; Haptic decoder Reconstructing tactile images from latent vectors, motion encoder The sequence of actions for the entire window Encoding as a sequence of feature embeddings; visual encoder Current visual image Encode it as a single feature embedding vector.

6. The robot control method based on tactile prediction pre-training according to claim 2, characterized in that, The core prediction component includes: a historical information fusion unit, a conditional information construction unit, and a conditional denoising prediction unit, wherein: the historical information fusion unit converts the tactile latent vector sequence of historical time steps... The embedding sequence of future actions is fused with the corresponding historical action embedding sequence; the conditional information construction unit fuses the embedding sequence of future actions with the current visual embedding to construct a conditional vector containing a description of the future intent and the current scene. The conditional denoising prediction unit obtains the predicted tactile representation vector for future moments, i.e., the latent vector sequence, through iterative processing of adding noise, denoising, and conditional injection. .

7. The robot control method based on tactile prediction pre-training according to claim 5, characterized in that, The conditional denoising prediction unit includes a noise-adding scheduler unit and a U-Net-based Transformer noise prediction unit, wherein the noise-adding scheduler unit receives the fused feature vector. At the current time step s, noise e is added to the feature vector to obtain a noisy feature vector; the Transformer noise prediction unit receives the noisy features at any time step, and performs cross-attention calculation with the image features and robot state features as conditions and the noisy multimodal features. The Transformer module obtains the predicted value of the noise added at the current time step s, and calculates the tactile features with noise removed based on the predicted value of the noise.

8. The robot control method based on tactile prediction pre-training according to claim 6 or 7, characterized in that, The iterative process specifically includes: i. Adding noise: During the training phase, the real future tactile latent vector sequence is processed by a noisy scheduler unit. Gaussian noise is gradually added within a preset discrete time step S until its distribution is close to pure Gaussian noise; ii. Denoising: During the generation phase, noise prediction is performed using the Transformer noise prediction unit. Starting with a random Gaussian noise tensor, noise is iteratively removed in S steps: in each denoising iteration s, The system receives three inputs: the noisy latent variable of the current step, the encoding of the current denoising step number s, and the conditional vector constructed above. ; iii. Conditional Injection: This involves injecting information about future actions and the current visual context. As a condition, noise prediction units of the Transformer are injected via a cross-attention mechanism. In each layer of U-Net, the model's prediction process is explicitly guided to predict tactile changes that are more in line with physical laws.

9. The robot control method based on tactile prediction pre-training according to claim 2, characterized in that, In the offline pre-training phase, the model is pre-trained on a large scale using a human play dataset to minimize the mixture loss function. The training objective is denoted as , where: is used to optimize the Transformer noise prediction unit. diffusion loss , The vector represents the true future tactile latent vector, ϵ represents random noise, and s represents the number of denoising steps. These are preset noise scheduling coefficients; used to optimize the reconstruction loss of the VAE. = λ is the weighting coefficient that balances the two losses.

10. The robot control method based on tactile prediction pre-training according to claim 2, characterized in that, In the downstream task application stage, the tactile encoder in the pre-trained conditional diffusion model is frozen. After obtaining the network parameters, the pre-trained visual encoder will be... As a plug-and-play feature extraction module, it is integrated into a standard imitation learning policy network, which takes at each time step the robot's own state, the current visual features extracted by an independent visual backbone network, and the tactile encoder as input. High-quality tactile feature vectors extracted It outputs the robot's next action command.

Images