Robot control method based on cross-modal information fusion coding

Abstract

The invention relates to a robot control method based on cross-modal information fusion coding, and the method comprises the steps: collecting visual data and touch data which are synchronous with an operation behavior, carrying out the preprocessing, and obtaining a matched data pair; the matched data pair is input into an MVTE-DP model, visual tactile fusion potential features are extracted through a pre-trained MVTE model, the visual tactile fusion potential features are input into a diffusion strategy model, a future action trajectory is obtained, and the pre-trained MVTE model is obtained by training the MVTE model through a visual tactile alignment data set; and real-time control over the robot in the grabbing process is achieved according to the future action track and in combination with tactile feedback. Based on a real-time and high-precision behavior strategy of tactile feedback, fusion features are embedded into a diffusion strategy framework, a real-time correction mechanism driven by tactile feedback is designed, and stable and high-generalization control over complex objects by the robot dexterous hand is achieved.

Description

Technical Field

[0001] This invention relates to the field of visual-tactile fusion technology, and in particular to a robot control method based on cross-modal information fusion coding. Background Technology

[0002] When performing grasping tasks, robots often face challenges due to the complexity of object physical properties and visual occlusion. Traditional vision-based perception solutions lack sensitivity to soft objects, and when vision is obstructed or recognition is inaccurate, the agent cannot accurately perceive the environment and complete the task. Tactile sensors provide contact information, allowing the agent to perceive whether an object is sliding or has been firmly grasped, serving as an important supplement to visual perception. Imitation learning, which can learn from human hand movement data, offers a promising approach to endowing robots with human-like dexterity.

[0003] Recent work has demonstrated that representations learned from large-scale human manipulation data can generalize well to new manipulation tasks and environments. However, these methods primarily focus on human visual or natural language modalities, neglecting tactile feedback.

[0004] Existing visual-tactile fusion methods: (1) F / T sensor and vision fusion: such as Lee et al. and VTT combined tactile signals collected by F / T sensors with other visual pattern learning tasks (such as plugging in a plug).

[0005] (2) Vision-based tactile sensors and vision fusion: such as GelSight's tactile images combined with visual images, applied to tasks such as pouring liquids and handling cables.

[0006] (3) Fusion of tactile array sensors and vision: For example, the T-Dex method uses array sensors to collect data, and learns strategies by combining tactile representations with vision through self-supervised pre-training.

[0007] Existing data acquisition systems, such as motion capture systems, can provide accurate 3D information and are reliable in the face of visual occlusion, but they suffer from poor portability and high cost. Array sensors designed for robot dexterity hands are also typically expensive, hindering the possibility of large-scale data acquisition. Summary of the Invention

[0008] The purpose of this invention is to provide a robot control method based on cross-modal information fusion coding, which effectively integrates visual and sparse tactile modalities and learns joint latent features with strong generalization ability to assist downstream robot operation tasks.

[0009] To achieve the above objectives, the present invention provides the following solution: A robot control method based on cross-modal information fusion coding includes: Collect and preprocess visual and tactile data synchronized with the operation behavior to obtain matching data pairs; The matched data pairs are input into the MVTE-DP model, and the latent features of visual-tactile fusion are extracted through the pre-trained MVTE model. The latent features of visual-tactile fusion are then input into the diffusion strategy model to obtain the future action trajectory. The pre-trained MVTE model is obtained by training the MVTE model through the visual-tactile alignment dataset. Based on the predicted future motion trajectory and combined with tactile feedback, real-time control of the robot during the grasping process is achieved.

[0010] Optionally, acquiring and preprocessing visual and tactile data synchronized with the operational behavior includes: The visual data and tactile data are collected synchronously by a tactile glove and a visual sensor. The tactile glove collects tactile data including pressure data and bending data through thin-film piezoresistive sensors distributed in the fingertips and palm areas and bending sensors distributed in the finger joints. Align the visual and tactile data according to their timestamps to obtain matching data pairs.

[0011] Optionally, extracting latent features for visual-haptic fusion using a pre-trained MVTE model includes: The matching data pairs are divided into visual patches by a visual encoder and 2D sine and cosine position encoding is added to output the visual patch embedding vector. The tactile encoder converts the tactile signal matrix of the matched data pair into a high-dimensional tactile token, adds 1D sine and cosine position coding, and outputs a tactile token embedding vector. The tactile encoder is composed of a multilayer perceptron array. The visual patch embedding vector and the tactile token embedding vector are input into the fusion module, and cross-modal information interaction is performed through an attention mechanism to generate the visual-tactile fusion potential features.

[0012] Optionally, the fusion module is built based on Transformer blocks, which consist of several stacked Transformer standard blocks, each containing a multi-head self-attention layer and a feedforward neural network layer.

[0013] Optionally, training the MVTE model using a visual-haptic alignment dataset includes: The visual encoder and the tactile encoder extract visual tokens and tactile tokens from the visual-tactile alignment dataset, perform random masking, and train the dataset using a joint reconstruction loss function.

[0014] Optionally, the joint reconstruction loss function for ; in, The mean square error between the original image and the reconstructed image. The mean square error between the original tactile signal and the reconstructed tactile signal is denoted as . and As weight.

[0015] Optionally, obtaining future motion trajectories includes: The potential features of visual-tactile fusion, the position and velocity information of all joints of the robot's dexterous hand, and the action sequence of the previous K frames are used as policy inputs. The policy inputs are then input into the diffusion policy model, and the future action trajectory is output.

[0016] Optionally, obtaining future motion trajectories also includes: when the change in tactile force exceeds a preset threshold, asynchronously triggering the diffusion strategy model to predict new motion sequences.

[0017] The beneficial effects of the present invention are: (1) Low cost and high frequency data acquisition: Compared with expensive commercial dexterous hand array sensors or large visual tactile sensors, the present invention uses 8 sparse resistance pressure sensors and bending sensors, which significantly reduces the hardware cost. At the same time, the high-speed acquisition circuit realizes high-frequency tactile data capture of 200Hz, ensuring the practicality and scalability of the data source.

[0018] (2) Significantly improves robot grasping success rate: The MVTE-DP framework effectively integrates the two complementary modalities of vision and touch through MAE-style self-supervised pre-training. Compared with pre-training using only a single visual modality (such as V-Only, MVP), the grasping accuracy on both "visible" and "occluded" objects reaches the highest (94% / 92%), proving the superiority of multimodal fusion, especially in visually restricted scenarios (such as occlusion) where the effect is significantly improved.

[0019] (3) Efficient feature extraction and policy generation decoupling: The MVTP (Masked Visual-Tactile Pre-training) encoder is frozen as a feature extractor and integrated with the Diffusion Policy model. This decouples "perceptual learning" from "behavior generation", greatly reduces the difficulty of policy learning, and allows the policy network to directly utilize high-quality perceptual semantic information, thereby improving training efficiency.

[0020] (4) Tactile-priority real-time robust control: A tactile feedback-driven asynchronous correction mechanism was designed, that is, Diffusion Policy is triggered only when the tactile change exceeds the threshold to predict and correct the motion. This enables a fast, tactile-priority response to changes in the grasping state, and improves the stability and success rate of the robot in dealing with object slippage and insufficient contact force during the grasping process.

[0021] (5) Effective utilization of sparse tactile information: For sparse piezoresistive signals, a dedicated MLP mapping and 1D position encoding were designed to transform them into high-dimensional features that can be processed by Transformer, solving the problem that sparse and low-dimensional tactile data are difficult to be directly integrated into deep learning models. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a robot control method based on cross-modal information fusion coding, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the data acquisition process according to an embodiment of the present invention. Figure 3 This is a working framework diagram of the MVTE-DP model in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] like Figure 1 As shown, this embodiment proposes a robot control method based on cross-modal information fusion coding, including: Collect and preprocess visual and tactile data synchronized with the operation behavior to obtain matching data pairs; The matched data pairs are input into the MVTE-DP (Masked Visual-Tactile Encoder - DiffusionPolicy) model. The MVTE model extracts the latent visual-tactile fusion features, and the latent visual-tactile fusion features are input into the diffusion policy model to obtain the future action trajectory. The pre-trained MVTE model is obtained by training the MVTE model on the visual-tactile alignment dataset. Based on the future motion trajectory and combined with tactile feedback, real-time control of the robot during the grasping process can be achieved.

[0027] Specifically, this embodiment uses a low-cost visual-haptic glove system to collect visual and sparse tactile data from human operations. Then, using the MVTE-DP framework—which learns joint representations of the data through self-supervised pre-training—visual and tactile information is encoded into highly semantic latent features. Finally, these latent features are used as input to drive a Diffusion Policy model to generate the robot's motion trajectory, and combined with tactile feedback, real-time control is achieved during the grasping process.

[0028] Furthermore, the acquisition and preprocessing of visual and tactile data synchronized with the operational behavior includes: Visual and tactile data are collected simultaneously through tactile gloves and visual sensors. The tactile gloves collect tactile data, including pressure data and bending data, through thin-film piezoresistive sensors distributed in the fingertips and palm areas and bending sensors distributed in the finger joints. Align the visual and tactile data based on their timestamps to obtain matching data pairs.

[0029] Specifically, this embodiment provides a dexterous operation glove system based on vision and piezoresistive-bending tactile feedback, including: Haptic gloves: The core component integrates 8 thin-film piezoresistive sensors (5 for fingertips and 3 for palms) and 5 flexure sensors (for finger joints).

[0030] Main control chip: Seeed Studio XIAO ESP32S3 is selected, which is responsible for data acquisition and wireless communication.

[0031] Peripheral circuit: The CD4051 3-to-8 multiplexer is used in a cascaded design to expand the GPIO of the ESP32, which can connect up to 32 analog signals for acquiring multi-channel sensor data.

[0032] Visual sensor: High-resolution RGB camera (e.g., Intel RealSense) is responsible for capturing the operation scene from an egocentric perspective.

[0033] This embodiment aims to address the challenge of collecting tactile data during human manipulation and effectively integrating visual and tactile modalities. To this end, an advanced dexterous manipulator glove system based on vision and piezoresistive-bending tactile sensing was designed and developed, unifying piezoresistive and bending sensors into a single tactile modality. Array sensors designed for dexterous hands are typically expensive. The core component of this system is a specially designed, low-cost tactile glove integrating eight small thin-film piezoresistive sensors and five bending sensors. These sensors are distributed across different parts of the glove, enabling precise detection of subtle changes in hand-object contact. Furthermore, the system includes a visual sensor responsible for capturing the operator's egocentric perspective of the manipulation scene. By synchronizing the glove with the visual sensor, visual and tactile data synchronized with human manipulation behavior can be collected, providing a solid data foundation for subsequent research on vision-tactile pre-trained models.

[0034] The main control chip used is the ESP2S3. The Seeed Studio XIAO ESP32S3 is an embedded development board with various peripheral interfaces and GPIO pins, supporting 2.4GHz Wi-Fi-802.11b / g / n and Bluetooth Low Energy (BLE) dual wireless communication, and has good RF performance.

[0035] To enable the limited GPIO to access more data channels, an external circuit was designed for the ESP32, using a 3-to-8 multiplexer CD4051.

[0036] The CD4051 consists of three parts: a logic level conversion circuit, an 8-to-1 decoder circuit, and eight CMOS switch units. A, B, and C are 3-bit binary address inputs; the eight combinations of these 3 bits can be used to select eight channels. INH is the address input disable terminal; when it is high, the address input is invalid, meaning no channel is selected. The input levels of A, B, C, and INH are TTL compatible. The CD4051 has eight input / output terminals and one output / input terminal. Digital circuits are powered by +E and -E1, while analog circuits are powered by +E and -E2. Two CD4051s can form a 16-bit analog switch. When unit D3 is low, one CD4051 can output, while the other CD4051 is made high by an inverter.

[0037] The ESP32 peripheral circuit, through the cascading design of CD4051, can accept up to 32 analog signal inputs.

[0038] The tactile data acquisition glove uses eight resistance pressure sensors, five of which are precisely positioned at the fingertips of the five fingers to capture direct contact and sliding information with objects; the other three are positioned in the palm area to capture the overall grip stability and force distribution.

[0039] Each sensor unit has a diameter of 10 mm, except for the distal joint of the thumb and the palm, where the sensor diameter is 18.3 mm due to their larger surface area. These sensors are connected to a resistance-to-voltage conversion module, which converts resistance signals into voltage signals. By adjusting the potentiometer on the conversion module, it is ensured that the voltage values ​​recorded when each sensor is in contact are approximately equal.

[0040] Additionally, a flexibility sensor is installed at each finger joint on the back of the glove; a vision sensor is mounted on the wrist.

[0041] like Figure 2 As shown, the data acquisition process is as follows: 1) Visual Data Acquisition: This embodiment uses a high-resolution RGB camera to capture visual data, specifically a resolution of 424x240 pixels and a frame rate of up to 30 frames per second (fps). Given the large volume of image data, this embodiment records a precise timestamp for each captured image frame. These timestamps are transmitted in real-time to the central computer via User Datagram Protocol (UDP) to ensure efficient and real-time data transmission.

[0042] 2) Timestamp Recording and Synchronization: After receiving timestamps from the cameras, the central computer records this information and appends the corresponding local timestamp. This process ensures the time synchronization of the visual data, providing an accurate time reference for subsequent data alignment.

[0043] 3) Tactile Data Acquisition: Tactile signal acquisition relies on a set of resistance-to-voltage conversion modules, which convert resistance changes into voltage signals. In this embodiment, an AD7606 module is used in conjunction with an ESP32 microcontroller to sample the tactile voltage signal in real time at a sampling rate of 200 Hz. The acquired tactile signals are stored in a first-in-first-out (FIFO) buffer for subsequent data processing.

[0044] 4) Retrieval and Timestamp Recording of Tactile Signals: The central computer periodically retrieves tactile signals from the FIFO buffer via a serial connection. During the retrieval process, the central computer records the start and end timestamps of the tactile signals, ensuring accurate time stamping of the tactile data. Simultaneously, interpolation is used to estimate a more precise time point between two known timestamps.

[0045] 5) Alignment of visual and tactile data: The data alignment process relies on timestamps recorded by a central computer from the visual and tactile acquisition sources. Each image frame is synchronized with its corresponding tactile frame to minimize the time difference between them, thereby forming a matched data pair.

[0046] Furthermore, the latent features for visual-tactile fusion extracted through a pre-trained MVTE model include: The visual encoder divides the RGB visual image in the matched data pair into visual patches and adds 2D sine and cosine position encoding, and outputs the visual patch embedding vector. The tactile encoder converts the tactile signal matrix of the matched data pair into a high-dimensional tactile token, adds 1D sine and cosine position coding, and outputs a tactile token embedding vector. The tactile encoder is composed of a multilayer perceptron array. Visual patch embedding vectors and haptic token embedding vectors are input into the fusion module, and cross-modal information interaction is performed through an attention mechanism to generate latent features for visual-haptic fusion.

[0047] Furthermore, training the MVTE model using the visual-haptic alignment dataset includes: Visual and tactile tokens are extracted from the visual-tactile alignment dataset using a visual encoder and a tactile encoder, and then randomly masked. The dataset is then trained using a joint reconstruction loss function.

[0048] Furthermore, obtaining future action trajectories includes: The potential features of visual-tactile fusion, the position and velocity information of all joints of the robot's dexterous hand, and the action sequence of the previous K frames are used as policy inputs. The policy inputs are then fed into the diffusion policy model to output the future action trajectory.

[0049] Obtaining future motion trajectories also includes: when the change in tactile force exceeds a preset threshold, asynchronously triggering the diffusion strategy model to predict new motion sequences.

[0050] Specifically, this embodiment constructs an efficient visual-tactile fusion imitation learning strategy and deploys it on a robot platform to achieve real-time control based on tactile feedback.

[0051] like Figure 3 As shown, this embodiment introduces and improves the MVTP (Masked Visual-Tactile Pre-training) framework for learning a general visual-tactile joint representation from human manipulation data. It employs a Transformer architecture similar to MAE (Masked Autoencoder), including a visual encoder, a tactile encoder, and a fusion module. For sparse 8-dimensional binary tactile signals and 5-dimensional curvature signals, high-dimensional embedding is performed using dedicated MLPs, with 1D positional encoding added to capture contextual information of contact at different parts of the hand. A masking mechanism forces the model to reconstruct the masked original image and tactile signals using visible visual and tactile information, thereby learning visual-tactile latent features with strong generalization capabilities.

[0052] After the pre-training phase is completed, the pre-trained model is applied in the Diffusion Policy: the parameters of the MVTP fusion encoder are frozen and embedded as a feature extractor into the downstream Diffusion Policy framework to generate the robot's motion trajectory.

[0053] The diffusion strategy model is a trajectory generation method based on a denoised diffusion model. It can learn the mapping from complex multimodal inputs to multi-step action sequences, making it naturally suitable for high-dimensional, multi-degree-of-freedom robot control tasks. The input state of the diffusion strategy consists of three parts: perceptual features: latent visual-tactile fusion features output from the frozen MVTP encoder, which contain semantic information from the robot's current viewpoint image V and the binary tactile signal C from the robot's fingertips. Proprioception: Position and velocity information P of the robot's dexterous hand joints. The output of the diffusion strategy is the sequence of joint angles over a future period (e.g., the 20 degrees of freedom of the Shadow Hand).

[0054] The masked vision-tactile pre-trained (MVTP) model consists of a visual encoder, a tactile encoder, and a fusion module (the encoder in the transformer).

[0055] 1) Visual encoder ( This is based on convolution operations and the MAE (Masked Autoencoder) framework. The input is an H×W×3 RGB image. The image is divided into segments after convolution. Each visual patch is then processed using 2D positional encoding to preserve spatial information. Finally, a set of visual patch embedding vectors is output.

[0056] 2) Tactile encoder ( It consists of a multilayer perceptron (MLP) array. The input is... sparse binary tactile signal matrix (in (Representing 8 tactile sensor points). The contact value of each tactile point is independently mapped and converted into a high-dimensional tactile token through a dedicated MLP, and then 1D sine and cosine position codes are added to identify the physical identity information of different finger / palm sensors. Finally, a set of tactile token embedding vectors is output.

[0057] 3) Fusion module ( This is built upon Transformer Blocks. The input is a concatenated set of visual patch embedding vectors and haptic token embedding vectors. The output is a sequence of visual-haptic latent features with cross-modal interaction information. The attention mechanism enables tokens from each modality to utilize information from another modality to extract semantically complementary joint features.

[0058] A Transformer block consists of multiple stacked Transformer standard blocks. Each block contains a Multi-Head Self-Attention (MHSA) layer and a Feed-Forward Network (FFN) layer. The processing flow is as follows: the input undergoes layer normalization before entering the MHSA layer, and residual connections are used to prevent gradient vanishing.

[0059] During the pre-training phase, this invention employs an asymmetric encoder-decoder structure, which is based on the core self-supervised learning logic of masked autoencoders (MAE): 1) Encoder: Processes only a very small number of tokens that are not visible. Its purpose is to map incomplete, cross-modal inputs into a highly condensed latent space. Because the input information is incomplete, the encoder is forced to learn how to use vision to infer missing tactile information, or to use tactile information to complete the object features in visually occluded areas.

[0060] 2) Decoder: Consists of another set of lightweight Transformer blocks. Visible features of the encoder output. This is combined with learnable mask tokens representing the masked location. The decoder's sole objective is "reconstruction." Based on the global semantics extracted by the encoder, it attempts to reconstruct the original pixel values ​​of the masked area. and primitive tactile contact state The training process is completed by calculating the loss.

[0061] To acquire input features, high-dimensional embedding is performed using dedicated MLPs on sparse 8-dimensional piezoresistive signals and 5-dimensional curvature signals, with 1D positional encoding added to capture contextual information about contact between different parts of the hand. Through MLPs and positional encoding, low-dimensional, discrete binary tactile point data is transformed into high-dimensional semantic tokens containing spatial topological relationships (e.g., which finger is making contact). This allows the model to capture the inherent correlations between contact states in different areas of the hand (e.g., simultaneous contact of the thumb and forefinger may indicate a pinching action). The extracted tactile tokens are directly input into the fusion module. In the process, the visual tokens participate in the calculation of self-attention, and the resulting fused features will be sent to the decoder (pre-training stage) or the diffusion strategy model (downstream task stage).

[0062] A dataset was constructed containing 120 sets of human bottle-tightening operation sequences, totaling approximately 30,000 frames of visual-tactile alignment data. Visual images were uniformly cropped to 224×224 pixels; tactile signals were converted into binary sequences based on a 0.3V threshold and preprocessed. During the pre-training phase, visual tokens were subjected to a high-ratio random masking (mask rate...). Masking the haptic token (masking rate) A joint reconstruction loss function is used: ; in, The mean square error between the original image and the reconstructed image. This represents the mean square error between the original tactile signal and the reconstructed tactile signal. Weight settings. The sparsity of tactile data is compensated by increasing the tactile weight.

[0063] The goal of the model is to reconstruct the original masked image and tactile signals using the unmasked visible information. This forces the model to learn the inherent correlation and complementarity between vision and touch, thereby generating general, highly generalizable latent features for visual-tactile fusion.

[0064] MVTP-based diffusion policies include: During the transfer learning phase, all parameters of the pre-trained MVTP model are completely frozen, including all weights for visual extraction, tactile extraction, and the fusion Transformer block. This encoder is embedded at the top layer of the observer in the diffusion policy model, serving as the backbone feature extraction network. Its output, consisting of 512 dimensions (or equivalent dimensions) of fused latent features, is directly input into the MLP or CNN network of the policy network.

[0065] The policy input is a description of the environmental state observed by the diffused policy network at each decision time, consisting of three cascaded parts: perceptual features. : The image from the current frame is frozen by the MVTP encoder. and tactile signals Real-time extraction of visual-tactile fusion features. Proprioception. This contains the current position and velocity information of all 24 joints of the robot's dexterous hand (after linear layer dimensionality reduction projection). Historical information: Previous... The action sequence of the frame is buffered to enhance the temporal consistency of decision-making. The policy output is a sequence of joint angle changes predicted by the diffusion policy over a future time period (e.g., 8-16 steps) through a denoising process.

[0066] The system continuously monitors the tactile sensor readings. When the tactile force changes... When a preset threshold is exceeded, i.e., when object slippage or a drastic change in contact force is detected, the Diffusion Policy model is asynchronously triggered to perform a new motion sequence prediction and correction. The policy model uses the absolute joint angle predicted in the previous step as a benchmark to predict the amount of joint angle change that needs to be executed in the next step.

[0067] The control process is as follows: 1. Receding Horizon Control: The diffusion strategy model no longer outputs a single action in each inference, but instead outputs a result containing future actions. A sequence of steps (such as the changes in joint angles over the next 8 frames). The robot typically executes this sequence smoothly to avoid the judder caused by single-step control.

[0068] 2. A haptic-prioritized "interruption-reinference" mechanism: Normal state: When the tactile sensor readings are stable or change only slightly, the robot moves along the predicted trajectory, reducing computational energy consumption.

[0069] Abnormal Trigger: The system monitors the signals from 8 piezoresistive sensors and 5 curvature sensors in real time at a frequency of 200Hz. If a change in tactile pressure gradient is detected... If a preset threshold is exceeded (e.g., a slight displacement or sudden decrease in pressure is detected at the physical contact surface of an object), the system determines that a "slip" or "collision" has occurred.

[0070] Real-time correction: At this point, the system immediately interrupts the old action sequence that is being executed. It captures the visual image and tactile features of the current instant, extracts the fused high semantic features through the frozen MVTP encoder, and uses them as the new initial state input to the Diffusion Policy model to regenerate a future action sequence corrected for the current emergency state.

[0071] The following is the overall process for grabbing and moving a water-filled plastic bottle: 1. Initial planning: When the robot recognizes the plastic bottle, the Diffusion Policy algorithm in MVTE-DP generates a sequence of future motion trajectories: "open hand - approach the bottle - close fingers".

[0072] 2. Tactile Feedback Intervention: The moment a finger touches the bottle, the pressure of the piezoresistive sensor surges and exceeds the trigger threshold. The system senses "contact" and asynchronously triggers a new inference, integrating "contact sensation" into the feature and outputting a more precise "pressure grip" trajectory.

[0073] 3. Real-time correction (handling slippage): During the lifting of the bottle, the increased water flow causes a change in gravity, resulting in a slight slippage of the bottle. At this moment, the piezoresistive sensor at the fingertip captures the rapid displacement of the center of pressure (CoP).

[0074] Detection: A sudden change in tactile sensation triggers a threshold alarm. Response: Without waiting for the previous trajectory to complete, the system immediately re-predicts based on the "visual image of the sliding bottle" + "tactile sensation of downward shearing force." Result: The newly generated future trajectory sequence automatically includes compensatory actions such as "increasing grip strength" and "finger upward fine-tuning," thus completing a robust grasp before the bottle falls, achieving truly real-time robust control.

[0075] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A robot control method based on cross-modal information fusion coding, characterized in that, include: Collect and preprocess visual and tactile data synchronized with the operation behavior to obtain matching data pairs; The matched data pairs are input into the MVTE-DP model, and the latent features of visual-tactile fusion are extracted through the pre-trained MVTE model. The latent features of visual-tactile fusion are then input into the diffusion strategy model to obtain the future action trajectory. The pre-trained MVTE model is obtained by training the MVTE model through the visual-tactile alignment dataset. Based on the predicted future motion trajectory and combined with tactile feedback, real-time control of the robot during the grasping process is achieved.

2. The robot control method based on cross-modal information fusion coding according to claim 1, characterized in that, The acquisition and preprocessing of visual and tactile data synchronized with the operational behavior includes: The visual data and tactile data are collected synchronously by a tactile glove and a visual sensor. The tactile glove collects tactile data including pressure data and bending data through thin-film piezoresistive sensors distributed in the fingertips and palm areas and bending sensors distributed in the finger joints. Align the visual and tactile data according to their timestamps to obtain matching data pairs.

3. The robot control method based on cross-modal information fusion coding according to claim 1, characterized in that, The latent features for visual-tactile fusion extracted through a pre-trained MVTE model include: The matching data pairs are divided into visual patches by a visual encoder and 2D sine and cosine position encoding is added to output the visual patch embedding vector. The tactile encoder converts the tactile signal matrix of the matched data pair into a high-dimensional tactile token, adds 1D sine and cosine position coding, and outputs a tactile token embedding vector. The tactile encoder is composed of a multilayer perceptron array. The visual patch embedding vector and the tactile token embedding vector are input into the fusion module, and cross-modal information interaction is performed through an attention mechanism to generate the visual-tactile fusion potential features.

4. The robot control method based on cross-modal information fusion coding according to claim 3, characterized in that, The fusion module is built on Transformer blocks, which consist of several stacked Transformer standard blocks. Each standard block contains a multi-head self-attention layer and a feedforward neural network layer.

5. The robot control method based on cross-modal information fusion coding according to claim 3, characterized in that, Training an MVTE model using a visual-tactile alignment dataset includes: The visual encoder and the tactile encoder extract visual tokens and tactile tokens from the visual-tactile alignment dataset, perform random masking, and train the dataset using a joint reconstruction loss function.

6. The robot control method based on cross-modal information fusion coding according to claim 5, characterized in that, The joint reconstruction loss function for ； in, The mean square error between the original image and the reconstructed image. The mean square error between the original tactile signal and the reconstructed tactile signal is denoted as . and As weight.

7. The robot control method based on cross-modal information fusion coding according to claim 1, characterized in that, Obtaining future action trajectories includes: The potential features of visual-tactile fusion, the position and velocity information of all joints of the robot's dexterous hand, and the action sequence of the previous K frames are used as policy inputs. The policy inputs are then input into the diffusion policy model, and the future action trajectory is output.

8. The robot control method based on cross-modal information fusion coding according to claim 1, characterized in that, Obtaining future motion trajectories also includes: when the change in tactile force exceeds a preset threshold, asynchronously triggering the diffusion strategy model to predict new motion sequences.

Images