Robot perception enhancement method based on vision-touch fusion codec

Abstract

The invention discloses a robot perception enhancement method based on a vision-touch fusion codec, and belongs to the technical field of robot perception and artificial intelligence. Aiming at the problems that a single visual perception scheme cannot directly observe contact information, the robustness is poor under environment change and multi-modal fusion is difficult, the method is characterized by comprising the following steps: synchronously collecting visual and tactile data, and carrying out time sequence alignment and preprocessing; respectively converting the image blocks and the tactile signals into feature sequences and adding position codes; feature interaction is carried out through cross-modal attention in a unified encoder, and training is carried out by adopting a joint random masking and reconstruction mechanism to obtain shared representation; the shared characterization is used for a downstream robotic operation task. According to the method, the observability of the contact behavior is realized, the perception robustness under shielding and illumination changes is improved, and more stable perception input is provided for a smart operation task.

Description

Technical Field

[0001] This invention belongs to the field of robot perception and artificial intelligence technology, and in particular relates to a robot perception enhancement method based on a vision-touch fusion codec. Background Technology

[0002] In the field of robot perception technology, especially in scenarios involving automated assembly, intelligent services, and medical care that require precise physical interaction with the environment, extremely high demands are placed on the ability to perceive stable and nuanced environmental states. Currently, mainstream technical solutions are mostly built around a single visual modality. These solutions acquire images through cameras and utilize visual representation networks such as object detection, semantic segmentation, pose estimation, or convolutional neural networks and visual Transformers to understand the environment, thereby driving the robot to perform operations such as grasping and handling. These solutions are widely used due to their relatively simple structure and low deployment threshold. To further improve robustness and data utilization efficiency in complex scenarios, advanced technologies such as visual encoding methods based on mask self-supervised pre-training have been proposed. The most representative work is to use a masked autoencoder framework to pre-train a visual encoder on large-scale data and freeze the encoder in downstream tasks. Only a small amount of demonstration data is needed to significantly improve the success rate of various dexterity tasks through behavior cloning. This is considered the existing technical route closest to this invention.

[0003] However, despite the progress made by the aforementioned vision-based pre-training methods, several key shortcomings remain from the perspective of enhancing overall robot perception. These shortcomings limit the system's performance and reliability in practical applications. First, these methods rely entirely on image signals during training and inference, making it impossible to directly perceive whether contact occurs between the robot's hand and the environment, the specific location of the contact, and the distribution of contact force. This lack of contact information makes it difficult for the system to achieve precise and safe control when facing tasks that are highly sensitive to tactile feedback, such as grasping soft objects, manipulating fragile items, or pouring liquids. Second, these solutions are extremely sensitive to changes in ambient lighting, partial occlusion of target objects, or loss of video frames. Incomplete or temporary missing information in the visual channel directly leads to inaccurate environmental state estimation and fluctuations in control strategies, resulting in insufficient overall system robustness. Furthermore, in existing engineering practices, attempts to introduce tactile sensors outside the vision system to compensate for the aforementioned shortcomings often lack an integrated modeling path starting from data acquisition. Fusion is typically achieved through simple feature stitching in the later stages, which is highly susceptible to temporal misalignment and fusion noise due to inaccurate timestamp alignment, making it difficult to achieve truly collaborative and complementary multimodal perception. In addition, to overcome the inherent limitations of a single visual modality, existing solutions usually require larger datasets and more complex models for training, significantly increasing computational and data acquisition costs and raising the barrier to technology reproduction and deployment. The difficulty in solving these problems lies in achieving high-precision synchronization and alignment of multi-source heterogeneous signals at the data level, designing a unified architecture at the model level that can naturally fuse and understand the intrinsic relationship between visual and tactile information, and finding an efficient learning mechanism that enables the model to complete information in the event of modality loss. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a robot perception enhancement method based on a vision-tactile fusion codec. This method enables the observability of contact behavior, improves perception robustness under occlusion and lighting changes, and provides more stable perception input for dexterous maneuvering tasks.

[0005] To achieve the above objectives, this invention provides a robot perception enhancement method based on a vision-tactile fusion codec, comprising:

[0006] Acquire aligned visual-tactile multimodal data synchronously acquired by visual and tactile acquisition devices;

[0007] The visual data and the tactile data are converted into visual feature sequences and tactile feature sequences, respectively;

[0008] The visual feature sequence and the tactile feature sequence are input into a unified encoder model, and feature interaction is performed through a cross-modal attention mechanism to form a fused shared representation.

[0009] After applying random occlusion to the visual feature sequence and the tactile feature sequence, the input is used to train the encoder model, and the occluded parts are reconstructed using the decoding heads corresponding to vision and touch respectively, so as to drive model learning with joint reconstruction loss.

[0010] The shared representation output by the trained encoder model is used as the state input for downstream robot tasks to enhance robot perception.

[0011] Optionally, acquiring aligned visual-tactile multimodal data includes:

[0012] Use a camera to capture continuous image frames during the operation;

[0013] Use tactile gloves to collect pressure and bending sensor signal sequences during the operation;

[0014] Record a unified timestamp for the image frame and the sensor signal sequence;

[0015] Based on the unified timestamp, the sensing signal sequence is aligned temporally with the image frame.

[0016] Optionally, acquiring aligned visual-tactile multimodal data also includes data preprocessing:

[0017] The image frames are cropped, sized, and have their pixel values ​​standardized.

[0018] The pressure sensing signal is filtered, smoothed, and divided into discrete contact force levels;

[0019] The curvature sensing signal is normalized.

[0020] Add spatial coding to the tactile data based on the sensor's location on the glove.

[0021] Optionally, converting visual and tactile data into feature sequences includes:

[0022] The image is divided into image blocks, and each image block is mapped to a visual feature vector through linear projection. Then, two-dimensional positional encoding is superimposed to form the visual feature sequence.

[0023] The pressure signal and curvature signal are projected into a high-dimensional space through a multilayer perceptron, and then a one-dimensional position code is superimposed to form the tactile feature sequence.

[0024] Optionally, a random occlusion process is applied to the visual feature sequence and the tactile feature sequence, using different occlusion ratios to randomly occlude the visual features and the tactile features respectively.

[0025] Optionally, the joint reconstruction loss includes visual reconstruction loss and tactile reconstruction loss;

[0026] The visual reconstruction loss is calculated based on the mean square error between the original image and the reconstructed image;

[0027] The tactile reconstruction loss is calculated based on the cross-entropy loss between the original pressure signal level and the predicted pressure signal level, as well as the mean absolute error between the original curvature signal and the predicted curvature signal.

[0028] Optionally, the downstream robot task is a reinforcement learning task performed in a simulation environment, and the shared representation serves as the state observation input for the reinforcement learning agent.

[0029] An electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0030] When the processor executes the computer program instructions, it implements the robot perception enhancement method based on the vision-tactile fusion codec.

[0031] A computer storage medium storing computer program instructions, which, when executed by a processor, implement the robot perception enhancement method based on a vision-tactile fusion codec.

[0032] Technical Effects of this Invention: This invention discloses a robot perception enhancement method based on a vision-tactile fusion codec. By constructing a high-precision aligned acquisition pipeline for visual and tactile data, a clean and spatiotemporally consistent multimodal dataset is obtained, laying a high-quality data foundation for model learning. The proposed vision-tactile fusion codec model, through a unified coding framework and joint occlusion reconstruction mechanism, achieves deep fusion and complementarity of cross-modal features. It can effectively complete observations incomplete due to visual occlusion, lighting changes, or missing tactile signals by utilizing information from another modality, thereby obtaining a more robust and complete shared environmental state representation. This shared representation explicitly incorporates contact details and global visual priors, significantly enhancing the direct observation capability of robot-environment interaction behavior. This provides a more stable and nuanced perceptual input for downstream reinforcement learning strategies or real-machine operation tasks, thereby improving the overall success rate, adaptability, and system reliability in tactilely sensitive tasks and complex dynamic environments. Attached Figure Description

[0033] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 This is a flowchart illustrating a robot perception enhancement method based on a vision-touch fusion codec according to an embodiment of the present invention.

[0035] Figure 2 This is a flowchart illustrating the detailed modules of the visual-tactile fusion codec according to an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of the visual-tactile fusion codec network according to an embodiment of the present invention;

[0037] Figure 4 This is a Loss-Epoch graph on the verification set of an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram illustrating the image reconstruction effect of an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram of the tactile reconstruction effect of the eight pressure channels in an embodiment of the present invention;

[0040] Figure 7 This is a schematic diagram of the tactile reconstruction effect of five curvature channels in an embodiment of the present invention;

[0041] Figure 8 This is a schematic diagram of a real-machine zero-sample classification task according to an embodiment of the present invention. Detailed Implementation

[0042] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0044] like Figure 1 As shown, this embodiment provides a robot perception enhancement method based on a vision-tactile fusion codec, including:

[0045] Acquire aligned visual-tactile multimodal data synchronously acquired by visual and tactile acquisition devices;

[0046] The visual data and the tactile data are converted into visual feature sequences and tactile feature sequences, respectively;

[0047] The visual feature sequence and the tactile feature sequence are input into a unified encoder model, and feature interaction is performed through a cross-modal attention mechanism to form a fused shared representation.

[0048] After applying random occlusion to the visual feature sequence and the tactile feature sequence, the input is used to train the encoder model, and the occluded parts are reconstructed using the decoding heads corresponding to vision and touch respectively, so as to drive model learning with joint reconstruction loss.

[0049] The shared representation output by the trained encoder model is used as the state input for downstream robot tasks to enhance robot perception.

[0050] Furthermore, acquiring aligned visual-tactile multimodal data includes:

[0051] Use a camera to capture continuous image frames during the operation;

[0052] Use tactile gloves to collect pressure and bending sensor signal sequences during the operation;

[0053] Record a unified timestamp for the image frame and the sensor signal sequence;

[0054] Based on the unified timestamp, the sensing signal sequence is aligned temporally with the image frame.

[0055] Specifically, the implementation process of this embodiment includes:

[0056] This embodiment utilizes an Intel RealSense camera and a self-developed tactile data acquisition glove to simultaneously acquire visual and tactile information, constructing a high-quality visual-tactile multimodal dataset. The camera is fixed at the researcher's head height, recording video data of hand operations from a first-person perspective. The self-developed glove is equipped with eight pressure sensors on the fingertips and palms, and five bend sensors on the backs of the fingers, used to capture contact pressure and finger movements. A timestamp function is added to the camera and glove data acquisition modules to record the acquisition time for each frame of image and tactile data stream, ensuring precise alignment between visual frames and tactile signals. Linear interpolation is used to time-complement the tactile data, ensuring data synchronization. Standardized operational tasks are divided into two main categories: grasping soft objects and pouring liquids. Objects in the task scenarios include yogurt containers, fresh fruit, and plastic water cups, enriching the dimensions of data acquisition. During the acquisition process, environmental variables such as bright and dim lighting conditions, clean backgrounds, and interfering backgrounds are manually controlled to simulate various operational scenarios. Specific hand-object interactions during the tasks are recorded, including contact force and movement trajectory information. Such a data collection setup will help build a high-quality visual-tactile multimodal dataset, providing strong support for subsequent model training and task validation.

[0057] Furthermore, acquiring aligned visual-tactile multimodal data also includes data preprocessing:

[0058] The image frames are cropped, sized, and have their pixel values ​​standardized.

[0059] The pressure sensing signal is filtered, smoothed, and divided into discrete contact force levels;

[0060] The curvature sensing signal is normalized.

[0061] Add spatial coding to the tactile data based on the sensor's location on the glove.

[0062] Specifically, the implementation process of this embodiment includes:

[0063] For video data, firstly, Region of Interest (ROI) is used to crop the data to the center and adjust the resolution to ensure that the acquired visual data corresponds to the area required for the target task, maintaining consistent visual focus. The image size is then standardized to match the model input size (224×224). Secondly, image pixel values ​​are standardized using Z-score normalization to adapt to the model input requirements. Furthermore, image enhancement techniques such as flipping, rotating, and brightness adjustment are used to increase data diversity and simulate different visual conditions that may be encountered in real-world scenarios. Finally, to ensure visual-tactile data alignment, for image data, a frame rate synchronized with the tactile data is selected, and keyframes are chosen to avoid computational burden from redundant images.

[0064] For tactile data, a low-pass filter is first used to remove high-frequency noise from the signal, and then a sliding window averaging method is used to smooth the signal, reducing interference from instantaneous fluctuations. Secondly, for pressure sensor signals, pressure thresholds are set to classify the signals into several contact force levels such as "light touch," "light press," and "hard pressure." For curvature sensor signals, the range is normalized to between 0 and 1 to enhance their numerical expressive power. Since multiple sensor signals are available, they are encoded according to the sensor's position on the hand, and these codes are incorporated into the tactile features, enabling the model to understand the spatial relationship between different tactile signals during learning.

[0065] Furthermore, converting visual and tactile data into feature sequences includes:

[0066] The image is divided into image blocks, and each image block is mapped to a visual feature vector through linear projection. Then, two-dimensional positional encoding is superimposed to form the visual feature sequence.

[0067] The pressure signal and curvature signal are projected into a high-dimensional space through a multilayer perceptron, and then a one-dimensional position code is superimposed to form the tactile feature sequence.

[0068] Specifically, the implementation process of this embodiment includes:

[0069] like Figure 2 As shown, the visual-tactile fusion encoder proposed in this embodiment uses five key modules to collaboratively achieve multimodal feature extraction, fusion, and reconstruction. The visual-tactile fusion encoder-decoder network is as follows: Figure 3 As shown, the visual feature extraction module: The input image is first divided into 16×16 pixel patches using patchify, which decomposes a large image into several local regions to facilitate individual feature extraction for each small region. Then, patch embedding is performed on each patch, which is equivalent to a 2D convolution with a kernel size of 16×16 and a stride of 16 (corresponding to...). Figure 3 The green Conv2D bounding box maps each local patch to a feature vector of the same dimension. All these vectors are concatenated into a sequence according to the scan order, and then superimposed with a 2D positional embedding (2D PE) based on sine and cosine to form a visual token (corresponding to...). Figure 3 The 2D PE in the model preserves the spatial structure priors to provide structured input for subsequent fusion and global modeling.

[0070] Tactile feature extraction module: Pressure sensor data and curvature sensor data are projected into a high-dimensional space (corresponding to) by two independent multi-layer perceptrons (MLPs). Figure 3 The purple MLPs box in the middle shows the high-dimensional space aligned with the dimensions after image patch embedding. Then, a one-dimensional sine / cosine positional encoding (1D Positional Embedding, 1D PE) reflecting the distribution of tactile sensors on the glove is introduced to form a tactile token (corresponding to...). Figure 3 The 1D PE in the image indicates the spatial ownership and order of each channel, which facilitates subsequent cross-modal attention to accurately utilize tactile information such as "which finger or which joint area".

[0071] Furthermore, the process of applying random occlusion to the visual feature sequence and the tactile feature sequence involves randomly occluding the visual features and the tactile features with different occlusion ratios.

[0072] Specifically, the implementation process of this embodiment includes:

[0073] Random Masking Encoder / Decoder: After generating visual tokens, pressure tokens, and bending tactile tokens, independent random masking is performed on each of the three sequences. Specifically, a portion of tokens are randomly retained from each sequence according to a certain ratio, while the remaining positions are marked as masked. This is achieved by generating random scores for each sample sequence along its dimension and sorting them, selecting the top n as the retained subsequences, and simultaneously generating two types of metadata: a binary mask ("0" indicates retention, "1" indicates masking) and a restoration index (used to reassemble the retained and masked tokens back into their original order during the decoding stage). This process reduces the total number of tokens entering the shared backbone. Self-attention computational complexity decreases The loss function employs multimodal reconstruction loss, calculating only the occluded locations. Specifically, it uses Mean Squared Error (MSE) loss for the image and a hybrid loss of Cross-Entropy (CE) and Mean Absolute Error (MAE) for the tactile signal, with appropriate weights set to balance the training contributions of the two modalities. The calculation formula is as follows:

[0074] ;

[0075] in, and These represent the loss weights for the visual and tactile modalities, respectively. and These represent the original image information and the predicted image information, respectively. and These represent the raw pressure information and the predicted pressure information, respectively. and These represent the original curvature information and the predicted curvature information, respectively. The mixing coefficient is the internal mixing coefficient of tactile information.

[0076] Furthermore, the joint reconstruction loss includes visual reconstruction loss and tactile reconstruction loss;

[0077] The visual reconstruction loss is calculated based on the mean square error between the original image and the reconstructed image;

[0078] The tactile reconstruction loss is calculated based on the cross-entropy loss between the original pressure signal level and the predicted pressure signal level, as well as the mean absolute error between the original curvature signal and the predicted curvature signal.

[0079] Specifically, the implementation process of this embodiment includes:

[0080] Visual reconstruction module: The preserved token and occluded token output by the encoder are reassembled into their original order based on the reconstruction index, then the positional encoding is removed, and the result is input into the lightweight decoder. The decoding head linearly projects each visual token back into the patch pixel space, equivalent to an MLP without hidden layers. Figure 3 The green MLP box in the middle is used to reassemble all patches into a complete image using unpatchify. During inference, the restored image is generated by "using predictions for occluded areas and ground truth for the rest", which intuitively demonstrates the occlusion reconstruction capability and spatial consistency.

[0081] The tactile reconstruction module follows the same processing flow as the visual reconstruction module, differing only in the final output. It employs a dual-head design: one is a pressure grading and classification head, providing four prediction categories for each pressure channel ("no touch," "light touch," "light press," and "heavy pressure"); the other is a bending regression head, outputting the continuous amplitude of the corresponding channel. Both, weighted according to their respective values, are combined with the visual reconstruction to form the total loss. During inference and display, the probability of the pressure level and the estimated bending degree are output.

[0082] Random occlusion forces the model to perform intra-modal and inter-modal feature completion under incomplete observations such as lighting changes, occlusion, frame drops, or missing single modalities, thereby obtaining robust and transferable shared representations. These fused joint features can be directly used for downstream tasks: firstly, for visualization and quantitative evaluation of visual-tactile reconstruction quality; secondly, as input representations for reinforcement learning policy networks, providing more stable decision support in visually constrained or tactilely dependent scenarios. Through this design, the encoder, while maintaining global visual priors, explicitly incorporates tactile contact details and temporal consistency, significantly enhancing its adaptability and robustness to complex operational tasks. Figures 4-7 The figures show the training loss, visual and tactile information reconstruction results of the model on the validation set.

[0083] Furthermore, the downstream robot task is a reinforcement learning task performed in a simulation environment, and the shared representation serves as the state observation input for the reinforcement learning agent.

[0084] Specifically, the implementation process of this embodiment includes:

[0085] Example of zero-shot transfer learning evaluation on a real machine: A zero-shot classification task is constructed on a real robotic arm platform to evaluate the transferability of the pre-trained visual-tactile representations of this invention under unlabeled conditions. Specifically, as follows... Figure 8 As shown, the fusion codec is migrated and deployed to a dexterous hand system with force control mode. In a "blind" scenario without visual information, the force control mode is used to grasp objects. Subsequently, a lightweight classification network is constructed on an image set containing multiple candidate objects, using pre-trained shared representations as feature inputs to predict and identify the category of the grasped object. The entire process requires no manual annotation, belonging to a completely unsupervised learning paradigm. By combining tactile-driven grasping with image-domain category inference, the cross-modal transfer capability and perception enhancement effect of the invention's representation in a real-device zero-shot scenario are verified.

[0086] An electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0087] When the processor executes the computer program instructions, it implements the robot perception enhancement method based on the vision-tactile fusion codec.

[0088] A computer storage medium storing computer program instructions, which, when executed by a processor, implement the robot perception enhancement method based on a vision-tactile fusion codec.

[0089] Compared with the prior art, the advantages of this invention are:

[0090] This invention first establishes a multimodal acquisition and synchronization pipeline with vision as the primary timescale: Vision and tactile data are time-stamped uniformly throughout the process, tactile sequences are aligned to frame-by-frame vision, and alignment logs and missing data markers are structurally stored along with the samples; necessary standardization and enhancement are also performed. On the tactile side, pressure grading and bending normalization are employed, combined with glove topology annotations, to form learnable spatiotemporal priors. Therefore, the training data is more consistent in terms of temporal reference, spatial semantics, and numerical scale, directly reducing fusion errors caused by misalignment, noise, and scale inconsistencies. Compared to data construction methods relying solely on a single vision, the resulting dataset is "clean and aligned," supporting more stable reconstruction quality even under incomplete observations such as occlusion and frame loss.

[0091] Designing a visual-tactile fusion model and implementing joint occlusion reconstruction: At the model level, this invention proposes a unified coding framework for a visual-tactile fusion encoder and adopts a joint occlusion reconstruction paradigm: during training, visual and tactile functions are occluded separately, and corresponding information is reconstructed by dual decoders (visual functions reconstruct pixels, while tactile functions simultaneously perform multi-class pressure recognition and bending regression). This mechanism forces the model to learn to use the other modality to complete key cues when one modality is constrained. Therefore, compared to a single-vision MAE, this invention achieves quantifiable improvements in contact observability, occlusion and frame loss tolerance, and fusion stability; and under the same hardware conditions, it achieves stronger robustness with lower computational cost, providing transferable joint representations for downstream policy learning.

[0092] A reproducible multimodal data acquisition and preprocessing pipeline: Tactile sequences are aligned using visual frames as the primary timescale. Alignment logs and missing data markers are stored in a structured manner along with the samples, and ROI, size uniformity, and standardization are achieved. On the tactile side, pressure grading and curvature normalization are employed, combined with glove position annotations to form learnable spatiotemporal priors. This pipeline normalizes multi-source data to a unified temporal reference and spatial semantics, significantly reducing fusion misalignment and scale drift, providing a "clean and aligned" data foundation for stable model learning.

[0093] A vision-tactile fusion method is proposed: images and tactile sensations are tokenized separately and then superimposed with frozen 2D and 1D sine and cosine positional codes. Training employs joint occlusion reconstruction, occluding both visual and tactile sensations separately, and calculating the loss only at the occluded locations (MSE for vision, multi-class CE for tactile sensation and MAE for curvature). Reconstruction is performed separately by dual decoders. This method can perform intra-modal and inter-modal feature completion, balancing contact observability, occlusion or frame loss tolerance, and computational overhead.

[0094] Without changing the core idea of ​​"temporal alignment + unified encoding + joint occlusion reconstruction", Kalman, spline, and Dynamic Time Warping (DTW) can be used to replace linear interpolation for alignment; learnable, relative position encoding can replace frozen sine and cosine position encoding; and convolution-augmented Transformer (Conformer) or CNN + Hybrid CNN with Cross-Attention can replace the Transformer backbone.

[0095] The visual-tactile multimodal dataset constructed in this invention possesses high quality, strong consistency, and explicit spatiotemporal priors, significantly reducing noise and misalignment risks in traditional multimodal fusion. The fusion encoder can simultaneously recover image and tactile signals, maintaining stable reconstruction accuracy even under complex conditions such as occlusion, lighting changes, and video frame drops, demonstrating significantly better robustness than a single visual MAE model. Shared representations provide unified features encompassing both visual global structure and tactile contact details, offering more stable state input for downstream reinforcement learning, resulting in faster policy convergence and higher task success rates, particularly excelling in tactile-dependent tasks. Simultaneously, the random occlusion strategy reduces backbone computation, enabling higher overall performance and deployability under the same hardware conditions.

[0096] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot perception enhancement method based on a vision-tactile fusion codec, characterized in that, include: Acquire aligned visual-tactile multimodal data synchronously acquired by visual and tactile acquisition devices; The visual data and the tactile data are converted into visual feature sequences and tactile feature sequences, respectively; The visual feature sequence and the tactile feature sequence are input into a unified encoder model, and feature interaction is performed through a cross-modal attention mechanism to form a fused shared representation. After applying random occlusion to the visual feature sequence and the tactile feature sequence, the input is used to train the encoder model, and the occluded parts are reconstructed using the decoding heads corresponding to vision and touch respectively, so as to drive model learning with joint reconstruction loss. The shared representation output by the trained encoder model is used as the state input for downstream robot tasks to enhance robot perception.

2. The robot perception enhancement method based on a vision-tactile fusion codec as described in claim 1, characterized in that, Acquiring aligned visual-tactile multimodal data includes: Use a camera to capture continuous image frames during the operation; Use tactile gloves to collect pressure and bending sensor signal sequences during the operation; Record a unified timestamp for the image frame and the sensor signal sequence; Based on the unified timestamp, the sensing signal sequence is aligned temporally with the image frame.

3. The robot perception enhancement method based on a vision-tactile fusion codec as described in claim 2, characterized in that, Acquiring aligned visual-tactile multimodal data also includes data preprocessing: The image frames are cropped, sized, and have their pixel values ​​standardized. The pressure sensing signal is filtered, smoothed, and divided into discrete contact force levels; The curvature sensing signal is normalized. Add spatial coding to the tactile data based on the sensor's location on the glove.

4. The robot perception enhancement method based on a vision-tactile fusion codec as described in claim 1, characterized in that, Converting visual and tactile data into feature sequences includes: The image is divided into image blocks, and each image block is mapped to a visual feature vector through linear projection. Then, two-dimensional positional encoding is superimposed to form the visual feature sequence. The pressure signal and curvature signal are projected into a high-dimensional space through a multilayer perceptron, and then a one-dimensional position code is superimposed to form the tactile feature sequence.

5. The robot perception enhancement method based on a vision-tactile fusion codec as described in claim 1, characterized in that, The process of applying random occlusion to visual feature sequences and tactile feature sequences involves randomly occluding visual features and tactile features using different occlusion ratios.

6. The robot perception enhancement method based on a vision-tactile fusion codec as described in claim 1, characterized in that, The combined reconstruction loss includes visual reconstruction loss and tactile reconstruction loss; The visual reconstruction loss is calculated based on the mean square error between the original image and the reconstructed image; The tactile reconstruction loss is calculated based on the cross-entropy loss between the original pressure signal level and the predicted pressure signal level, as well as the mean absolute error between the original curvature signal and the predicted curvature signal.

7. The robot perception enhancement method based on a vision-tactile fusion codec as described in claim 1, characterized in that, The downstream robot task is a reinforcement learning task performed in a simulation environment, and the shared representation serves as the state observation input for the reinforcement learning agent.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the robot perception enhancement method based on the vision-tactile fusion codec as described in any one of claims 1-7.

9. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the robot perception enhancement method based on a vision-tactile fusion codec as described in any one of claims 1-7.

Images