A transformer and contrast learning based robot vision-haptic perception method and system

Abstract

This invention discloses a robot visual-tactile perception method and system based on transformer and contrastive learning, belonging to the field of robotics technology. The method introduces an enhanced Swing Transformer model as the perception model. This enhanced model uses the Swing Transformer as the backbone network and incorporates a coordinate attention mechanism. Global information is extracted through the sliding window attention mechanism of the backbone network, while local feature extraction is enhanced through the coordinate attention mechanism. This effectively extracts and fuses global and local features from visual and tactile data. In the pre-training stage, the enhanced Swing Transformer model undergoes multimodal contrastive learning training, specifically intra-modal and inter-modal contrastive learning, generating richer and more accurate discriminative features, thereby effectively improving the robot's intelligent perception and manipulation capabilities. This method can effectively extract features in multimodal data fusion and utilize the correlation and synergistic effects of features between different modalities to achieve deeper multimodal data fusion.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a robot visual-touch perception method and system based on transformer and contrastive learning. Background Technology

[0002] In the field of robotics, intelligent perception and manipulation play a crucial role in multiple application scenarios. In industrial automation, robots need to accurately identify and manipulate objects of various complex shapes and materials to improve production efficiency and product quality. In the medical field, surgical robots rely on precise perception and manipulation capabilities to perform minimally invasive surgeries and delicate operations, reducing surgical risks. In service robots, home robots and nursing robots need to be able to accurately identify and handle everyday objects to provide safe and efficient services. Achieving intelligent perception and manipulation is a key objective in robotics. Visual sensors provide global perception of the environment through cameras, capturing the overall layout and dynamic changes of a scene, but may have limitations in recognition and localization in complex environments. Tactile sensors provide local contact information through sensor arrays, such as the shape, texture, and hardness of objects; these details are crucial for precise manipulation and perception. However, due to the significant differences in feature representation between visual and tactile data, effectively fusing these modal data to improve robot perception and manipulation capabilities remains a challenge.

[0003] Invention patent application CN117435973A proposes a deep learning-based visual-tactile multimodal object recognition method. The design utilizes a fiber optic grating sensor encapsulated in a silicone sleeve, mounted at the end effector of a robotic arm. It collects tactile and slip signals by contacting objects, while simultaneously acquiring visual data using an industrial camera. The tactile and slip signals are demodulated to obtain pressure and vibration data, and the visual images are preprocessed and labeled with surface characteristics. The tactile and slip data are fused with the visual data to form a dataset, which is then input into a trained multimodal neural network model to achieve accurate object recognition. This method, through feature-level fusion, combines tactile and visual information, improving recognition accuracy and making it suitable for object recognition tasks in complex environments for intelligent robots.

[0004] Invention patent application CN117611919A proposes a method for classifying graspable objects based on visual-tactile fusion, which is implemented through four steps: data preprocessing, feature extraction, feature fusion, and classification decision. First, the original visual and tactile image data are standardized and preprocessed. Next, a residual network and convolutional attention mechanism module are used to extract and enhance visual and tactile features. Then, the two types of features are concatenated and fused along the channel dimension to enhance their representational power. Finally, the fused features are input into the classifier module, and the classification result is output through a fully connected layer and activation function. The advantage of this method lies in fully utilizing the complementarity of visual and tactile data, improving feature acquisition capabilities through the attention mechanism, significantly improving classification accuracy and robustness, and validating its effectiveness on a constructed multimodal dataset.

[0005] While the aforementioned schemes are innovative in multimodal object recognition, they suffer from several limitations. First, their fusion of visual and tactile data may be overly simplistic, failing to delve deeper into the complementarity and intrinsic connections between different modalities, thus restricting the performance of the recognition system. Second, while these methods may perform well under specific environments or conditions, their adaptability and robustness may be insufficient when faced with changing environments or complex scenarios. Furthermore, the reliance on large amounts of labeled data limits the widespread application of the models in the real world, particularly in scenarios where data acquisition is costly or labeling is difficult. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a robot visual-touch perception method and system based on transformer and contrastive learning, so as to solve the problems of insufficient complementarity between different modalities and insufficient model robustness in the process of multimodal object recognition in the prior art.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] A robot vision-touch perception method based on transformer and contrastive learning includes:

[0009] Acquire visual and tactile images of an object;

[0010] Visual feature vectors are extracted from visual images using a visual encoder; the visual encoder is a first enhanced Swin Transformer model, in which a coordinate attention mechanism is set before the Swin Transformer block; the first enhanced Swin Transformer model is obtained through multimodal contrastive learning training.

[0011] Tactile feature vectors are extracted from tactile images using a tactile encoder; the tactile encoder is a second enhanced Swin Transformer model, in which a coordinate attention mechanism is set before the Swin Transformer block; the second enhanced Swin Transformer model is obtained through multimodal contrastive learning training;

[0012] During the multimodal contrastive learning training process, visual feature vectors and tactile feature vectors undergo intramodal contrastive learning, while visual feature vectors and tactile feature vectors undergo intermodal contrastive learning.

[0013] By fusing visual feature vectors and tactile feature vectors, a fused feature is obtained;

[0014] The fused features are processed through a linear classification layer to obtain the classification result of the object.

[0015] A further improvement of the present invention is that:

[0016] Preferably, the intramodal contrastive learning includes contrastive learning of two visual feature vectors and contrastive learning of two tactile feature vectors. The two visual feature vectors are extracted by a visual encoder and a momentum visual encoder, respectively, and the two tactile feature vectors are extracted by a tactile encoder and a momentum tactile encoder, respectively.

[0017] The intermodal contrastive learning is a contrastive learning of a visual feature vector extracted by a visual encoder and a tactile feature vector extracted by a momentum tactile encoder, as well as a contrastive learning of a visual feature vector extracted by a momentum visual encoder and a tactile feature vector extracted by a tactile encoder.

[0018] Preferably, after the visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed by feature mapping through two MLP target heads with different parameters; after the momentum visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed by two MLP target heads with different parameters; after the tactile encoder obtains two tactile feature vectors, the two tactile feature vectors are respectively processed by two MLP target heads with different parameters; after the momentum tactile encoder obtains two tactile feature vectors, the two tactile feature vectors are respectively processed by two MLP target heads with different parameters.

[0019] Preferably, the visual encoder and the corresponding MLP target head are trained using the contrastive learning results, which are the sum of the intra-modal contrastive learning results and the inter-modal contrastive learning results. The momentum visual encoder and the corresponding MLP target head are trained using the visual encoder output results.

[0020] The tactile encoder and the corresponding MLP target head are trained using the contrastive learning results, which are the sum of the intramodal and intermodal contrastive learning results. The momentum tactile encoder and the corresponding MLP target head are trained using the output results of the tactile encoder.

[0021] Preferably, the visual image is input into the visual encoder and the momentum visual encoder after data augmentation; the tactile image is input into the tactile encoder and the momentum tactile encoder after data augmentation.

[0022] Preferably, the linear classification layer is obtained through training, and the parameters of the visual encoder, momentum visual encoder, tactile encoder and momentum tactile encoder are fixed during the training process of the linear classification layer.

[0023] Preferably, the contrastive learning process involves processing the two input feature vectors through an information contrast estimation loss function and a supervised contrastive loss function, and then multiplying the results of the two contrastive loss functions by their corresponding weight coefficients and summing them to obtain the loss value.

[0024] Preferably, after the visual encoder or tactile encoder segments the image, it performs linear processing on the segmented image. The linearly processed feature map is then processed through four stages of feature extraction blocks, each stage including several feature extraction blocks. The extraction results are normalized by layers and then average pooled to obtain visual feature vectors or tactile feature vectors. In each feature extraction block, the feature map is first processed by a coordinate attention mechanism and then by a SwingTransformer block. The image is a visual image or a tactile image.

[0025] Preferably, the data used for training the multimodal contrastive learning consists of data from the TAG dataset and the Calandra dataset.

[0026] A robot vision-touch perception system based on transformer and contrastive learning, comprising:

[0027] The acquisition unit is used to acquire visual and tactile images of the object;

[0028] A visual feature unit is used to extract visual feature vectors from a visual image through a visual encoder; the visual encoder is a first enhanced Swing Transformer model, in which a coordinate attention mechanism is set before the Swing Transformer block; the first enhanced Swing Transformer model is obtained through multimodal contrastive learning training.

[0029] A tactile feature unit is used to extract tactile feature vectors from a tactile image through a tactile encoder; the tactile encoder is a second enhanced Swing Transformer model, in which a coordinate attention mechanism is set before the Swing Transformer block; the second enhanced Swing Transformer model is obtained through multimodal contrastive learning training;

[0030] During the multimodal contrastive learning training process, visual feature vectors and tactile feature vectors undergo intramodal contrastive learning, while visual feature vectors and tactile feature vectors undergo intermodal contrastive learning.

[0031] The fusion unit is used to fuse visual feature vectors and tactile feature vectors to obtain fused features;

[0032] The classification unit is used to process the fused features through a linear classification layer to obtain the classification result.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] This invention discloses a robot visual-tactile perception method based on transformer and contrastive learning. The method designs a novel enhanced Swin Transformer model, which uses the Swin Transformer as the backbone network. It extracts global information through a sliding window attention mechanism and introduces a coordinate attention mechanism to enhance local feature extraction. This ensures that the extracted features include comprehensive image information while also selectively capturing local features. It effectively extracts and fuses global and local features from visual and tactile data, significantly improving feature richness and accuracy. During training, the model employs a joint contrastive loss method, combining two types of loss functions as the contrastive loss for multimodal training. Multimodal training includes intra-modal and inter-modal contrastive learning, fully utilizing labeled and unlabeled data to generate richer and more accurate discriminative features, thereby effectively improving the robot's intelligent perception and manipulation capabilities. This method can effectively extract and utilize the correlation and synergistic effects between different modalities in multimodal data fusion, achieving deeper multimodal data fusion. Its framework and core ideas are independent of specific visual and tactile sensor types, exhibiting good generalization ability and enhancing robots' intelligent perception and manipulation capabilities in various application scenarios. This invention also has the following advantages:

[0035] (1) In this invention, a coordinate attention mechanism is introduced into the Swing Transformer model. Swing Transformer can capture richer visual and tactile features. The coordinate attention mechanism is specifically designed to enhance the extraction of local features of tactile data, ensuring that even the most subtle feature differences can be accurately captured by the model, thereby improving the accuracy and detail of data fusion.

[0036] (2) Combining the above-mentioned Swin Transformer and coordinate attention mechanism, this invention proposes a new multimodal representation learning method. This method solves the problem that traditional fusion methods cannot fully utilize the correlation and synergy between modalities by deeply fusing visual and tactile data.

[0037] (3) In this invention, a joint contrastive loss method combining InfoNCE loss and SupCon loss is proposed. This method can utilize both labeled and unlabeled data to promote the generation of richer and more accurate discriminative features in a multimodal environment, improve the generalization ability and environmental adaptability of the model, and significantly improve the robot's intelligent perception and operation capabilities.

[0038] (4) This learning and training method can learn better from a large amount of unlabeled data, improve the generalization ability of the model and its adaptability to environmental changes, reduce the dependence on large-scale labeled data, optimize small sample learning, and make the robot's intelligent perception and operation in complex environments more accurate and flexible, providing an innovative solution for the further development of intelligent robot technology. Attached Figure Description

[0039] Figure 1 This invention presents a robot visual-touch perception method based on transformer and contrastive learning. Figure 1 ;

[0040] Figure 2 The flowchart of robot visual-touch perception method based on transformer and contrastive learning Figure 2 ;

[0041] Figure 3 This is the encoder diagram of the attention-based swin transformer model of the present invention;

[0042] Figure 4 This is a training graph for multimodal contrastive learning.

[0043] Figure 5 This is a multimodal contrast loss diagram;

[0044] Figure 6A flowchart for training a robot visual-touch perception method based on transformer and contrastive learning;

[0045] Figure 7 This is a structural diagram of a robot visual-touch perception system based on transformer and contrastive learning according to the present invention. Detailed Implementation

[0046] Hereinafter, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of that feature.

[0047] The synchronization method provided in this application can be applied to terminal devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.

[0048] It should be noted that the terms "first," "second," etc., used in the specification and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0049] As mentioned in the background section, intelligent perception and manipulation of robots require visual sensors to perceive the environment and tactile sensors to provide detailed information. However, due to the differences in data distribution and feature representation between visual and tactile data, effectively fusing these modal features remains challenging. Existing methods typically employ simple fusion approaches, failing to fully utilize the correlations and synergies between different modalities, resulting in insufficient richness and accuracy of the fused features. This invention aims to address the problem of visual and tactile data fusion in the field of robotics. To this end, a method combining the enhanced Transformer mechanism and contrastive learning is proposed for multimodal representation learning of vision and touch.

[0050] See Figure 1 and Figure 2 The first aspect of this invention provides a robot visual-touch perception method based on transformer and contrastive learning, which specifically includes the following steps:

[0051] S1, acquire the visual and tactile images of the object;

[0052] S2, visual feature vectors are extracted from visual images through a visual encoder; the visual encoder is a first enhanced Swing Transformer model, which has a coordinate attention mechanism set before the Swing Transformer block; the first enhanced Swing Transformer model is obtained through multimodal contrastive learning training.

[0053] Tactile feature vectors are extracted from tactile images using a tactile encoder; the tactile encoder is a second enhanced Swing Transformer model, which has a coordinate attention mechanism set before the Swing Transformer block; the second enhanced Swing Transformer model is obtained through multimodal contrastive learning training;

[0054] In the multimodal contrastive learning training process described above, visual feature vectors and tactile feature vectors undergo intramodal contrastive learning, while visual feature vectors and tactile feature vectors undergo intermodal contrastive learning.

[0055] S3, fuses visual feature vectors and tactile feature vectors to obtain fused features;

[0056] S4 processes the fused features through a linear classification layer to obtain the classification result of the object.

[0057] Some embodiments of the present invention are described in detail below. Figure 3This is a unified system framework for visual and tactile encoders, where the first and second enhanced Swin Transformer models are both enhanced Swin Transformer models with the same structure. Based on the Swin Transformer model, this model can effectively extract global features, offering advantages over traditional convolutional networks. To enhance the model's ability to extract tactile and visual features, a coordinate attention mechanism is introduced. Specifically, after segmenting the image, the visual or tactile encoder performs linear processing on the segmented image. The resulting feature map is then processed through four stages of feature extraction blocks, each stage containing several feature extraction blocks. The extraction results are normalized by layers and then averaged to obtain visual or tactile feature vectors. Within each feature extraction block, the feature map is first processed by the coordinate attention mechanism and then by the Swin Transformer block. The image can be either a visual image or a tactile image, processed by the corresponding visual encoder and tactile encoder, respectively.

[0058] In some embodiments, the preprocessed visual and tactile images are 224×224 pixels in size. The image segmentation module divides the image into 4×4 blocks and flattens them to 48 in the channel direction. In stage 1, a linear embedding model is used to perform a linear transformation on the image pixel channels, changing the number of feature map channels from 48 to 96. Then, the feature map of size 56×56 with 96 channels is processed sequentially through four feature extraction blocks. Each feature extraction block includes a coordinate attention mechanism and a Swin Transformer block. The feature map after linear transformation is processed by the coordinate attention mechanism and then fed into the Swin Transformer block. Stages 2, 3, and 4 contain 2, 6, and 2 Swin Transformer blocks and an image patch merging module, respectively, with the number of channels being 192, 384, and 768, and the feature map sizes being 28×28, 14×14, and 7×7. Finally, the 7×7 feature map with 768 channels undergoes l-layer normalization and average pooling operations to obtain feature vectors. Image patches are then merged and downsampled between Swin Transformer blocks to reduce the resolution of the feature map and increase the number of channels, thereby achieving hierarchical feature extraction. This method significantly improves the model's feature extraction capability for complex data through multi-stage processing and attention mechanisms.

[0059] See Figure 4 This invention also discloses the training process of an enhanced Swin Transformer model, which is obtained through learning and training. The training model can be found in [link to training documentation]. Figure 3This learning and training model employs the self-supervised learning framework MOCO-v2 to extract intra-modal features. Instead of maintaining a memory queue, other samples from the same batch are treated as negative samples to reduce storage costs and computation time. The core of this model is intra-modal and inter-modal comparative learning through comparisons between eight feature vectors.

[0060] The model uses eight feature vectors for learning, including four visual feature vectors and four tactile feature vectors. Of the four visual feature vectors, two are extracted using a visual encoder and two are extracted using a momentum visual encoder. The tactile feature vectors consist of four tactile feature vectors, with two extracted using a tactile encoder and two using a momentum tactile encoder.

[0061] It should be noted that the network structures of the visual encoder, momentum visual encoder, tactile encoder, and momentum tactile encoder are all... Figure 3 The encoder structure.

[0062] The intramodal contrastive learning includes contrastive learning of two visual feature vectors and contrastive learning of two tactile feature vectors. The two visual feature vectors are extracted by a visual encoder and a momentum visual encoder, respectively, and the two tactile feature vectors are extracted by a tactile encoder and a momentum tactile encoder, respectively.

[0063] The intermodal contrastive learning involves comparing a visual feature vector extracted by the visual encoder with a tactile feature vector extracted by the momentum tactile encoder, and comparing a visual feature vector extracted by the momentum visual encoder with a tactile feature vector extracted by the tactile encoder. In this process, features from both modalities are independently extracted by the visual encoder and the tactile encoder, providing fundamental feature representations for subsequent intramodal and intermodal contrastive learning.

[0064] As a preferred embodiment, after the visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed through feature mapping using two MLP target heads with different parameters; after the momentum visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed through two MLP target heads with different parameters; after the haptic encoder obtains two haptic feature vectors, the two haptic feature vectors are respectively processed through two MLP target heads with different parameters; after the momentum haptic encoder obtains two haptic feature vectors, the two haptic feature vectors are respectively processed through two MLP target heads with different parameters. In this process, a contrastive learning framework is introduced, using MLP target heads to map features, performing intra-modal and inter-modal contrastive learning, and enhancing the representational ability of features through discriminative learning.

[0065] As a preferred embodiment, the visual image is input into the visual encoder and the momentum visual encoder after data augmentation; the tactile image is input into the tactile encoder and the momentum tactile encoder after data augmentation. Data augmentation of both visual and tactile images enhances the model's ability to generalize to image variations.

[0066] As a preferred option, see Figure 5 Contrastive learning processes two input feature vectors using both an information contrastive estimation loss function (InfoNCE) and a supervised contrastive loss function (Supcon loss, SupCon). The results of each loss function are multiplied by their respective weight coefficients and then summed to obtain the total loss value. This joint contrastive loss enables efficient utilization of both labeled and unlabeled data, further improving the discriminative power of features. Information contrastive estimation loss and supervised contrastive loss are two commonly used contrastive learning loss functions. They optimize the model in different ways, enabling it to better learn feature representations in both unsupervised and supervised environments.

[0067] Specifically, the information contrastive estimation loss function is based on the idea of ​​noise contrastive estimation, measuring similarity by maximizing the probability of positive samples and minimizing the probability of negative samples. The supervised contrastive loss function aims to maximize the similarity between samples of the same class and minimize the similarity between samples of different classes. By using these two loss functions together, the feature representation capability of the model can be maximized. The diagram of the joint loss value is shown below. Figure 4 As shown, the formula expression is shown in equation (1).

[0068] L=λL InfoNce +(1-λ)L Supcon (1)

[0069] Where L is the joint loss, L InfoNce The loss value of the loss function is estimated by information comparison, where λ is the weighting coefficient of this loss value, and L Supcon To supervise the loss value of the contrastive loss function, 1-λ is the weight coefficient of the loss function.

[0070] The information contrastive estimation loss function is a commonly used unsupervised loss function in contrastive learning, and it is defined as follows:

[0071]

[0072] Where: z i and z j It is the feature representation of positive sample pairs, z k It is the feature representation of the negative samples, containing all positive sample pairs and negative samples, and τ is a temperature parameter used to control the sensitivity of the contrast.

[0073] The supervised contrastive loss function is a type of supervised contrastive learning loss function that extends unsupervised contrastive learning methods by using label information to guide feature learning. It is defined as follows.

[0074]

[0075] Where: P(i) is the set of positive samples belonging to the same class as sample i, and A(i) is the set of all samples excluding sample i itself. i z p and z a These are the feature representations of sample i, positive sample p, and negative sample a, respectively. τ is the temperature parameter.

[0076] It should be noted that each pair of feature vectors is trained and learned individually using a multimodal contrastive loss. Therefore, four loss values ​​are obtained by formula (1). The four loss values ​​are summed to obtain a total loss value.

[0077] Furthermore, based on the total loss value obtained above, the visual encoder, tactile encoder, and MLP target head are each trained by comparative learning using the total loss value; the momentum visual encoder is trained using the output results of the visual encoder, and the momentum tactile encoder is trained using the output results of the tactile encoder.

[0078] As a preferred approach, the linear classification layer is obtained through training throughout the entire model training process, during which the parameters of the visual encoder, momentum visual encoder, tactile encoder, and momentum tactile encoder are fixed. Based on the fixed encoder parameters, feature fusion is performed and input into the linear classification layer to classify downstream tasks and obtain classification labels. Simultaneously, enhanced Transformer and coordinate attention mechanisms are utilized to improve the richness and accuracy of feature extraction, thereby enhancing the robot's intelligent perception and operational capabilities.

[0079] In some embodiments of the present invention, the data used in the training process consists of data from the TAG dataset and the Calandra dataset. Both datasets are divided into training and test sets in a 7:3 ratio during application. This division method helps to evaluate the model's performance and generalization ability, ensuring stable model performance across different datasets. The training set is used for model training, and the test set is used to verify the model's effectiveness and accuracy.

[0080] It should be noted that the Calandra dataset is a publicly available dataset provided by Roberto Calandra at the University of California, Berkeley. It contains tactile data acquired using a robotic arm and tactile sensors, as well as visual data acquired using a depth camera. Data acquisition was conducted in three timeframes: initial position, gripping but not lifting, and hovering for 2 seconds. Each experimental routine contains six images and their corresponding grasping result labels. To enhance the model's generalization performance, the images were resized to 256×256 and randomly cropped to 224×224, while also being randomly rotated to increase data diversity and improve model robustness. The dataset contains 106 different grasped items. These preprocessing and data augmentation measures aim to improve the model's performance in different environments.

[0081] It's important to note that the TAG dataset consists of egocentric viewpoint videos recorded by human collectors exploring objects in natural environments using Gelsight tactile sensors. It contains approximately 13.9k detected touches and about 3971 individual object instances. Collectors selected a variety of objects to press in each scene, including chairs, walls, floors, sofas, and tables in indoor scenes, and grass, rocks, trees, and sand in outdoor scenes, encompassing 20 categories. Each object was pressed approximately three times, with each press lasting an average of 0.7 seconds. Compared to previous datasets limited to laboratory or simulated environments, the TAG dataset is more diverse, covering more outdoor objects and scenes, and closely resembling the complexity of the real world. It includes various tactile features, helping to distinguish material properties such as soft / hard surfaces and smooth / rough surfaces. The task of this experiment was to distinguish different material categories. To improve the model's generalization performance, the researchers processed the images, resizing them to 256×256 pixels, then randomly cropping them to 224×224 pixels and rotating them randomly. The diversity and complexity of TAG datasets provide a broader application context for tactile sensor research.

[0082] In the pre-training phase of contrastive learning, the above process involves data augmentation of visual and tactile images, followed by feature extraction using visual and tactile encoders, respectively. For intra-modal and inter-modal contrastive learning, different MLP target heads are used for feature mapping. This yields four features for the visual modality and four features for the tactile modality. Two features from the visual and tactile modalities are used for intra-modal contrastive learning, while the other two features from the visual modality and the two features from the tactile modality are used for inter-modal contrastive learning. The proposed joint loss is employed for the multimodal contrastive loss.

[0083] See Figure 6One embodiment of the present invention discloses a specific model training process. In the pre-training stage of contrastive learning, data augmentation is performed on visual and tactile images to obtain augmented visual and tactile images. After the visual image is processed by a visual encoder, the resulting feature map is processed by two MLP target heads with different parameters to obtain two different visual feature vectors, designated as a first visual feature vector and a second visual feature vector, respectively. After the visual image is processed by a momentum visual encoder, the resulting feature map is processed by two MLP target heads with different parameters to obtain two different visual feature vectors, designated as a third visual feature vector and a fourth visual feature vector, respectively. After the tactile image is processed by a tactile encoder, the resulting feature map is processed by two MLP target heads with different parameters to obtain two different tactile feature vectors, designated as a first tactile feature vector and a second tactile feature vector, respectively. After the tactile image is processed by a momentum tactile encoder, the resulting feature map is processed by two MLP target heads with different parameters to obtain two different tactile feature vectors, designated as a third tactile feature vector and a fourth tactile feature vector, respectively.

[0084] Based on the visual and tactile feature vectors obtained above, multimodal contrastive learning is performed. Specifically, intramodal contrastive learning is performed between the first and third visual feature vectors, intermodal contrastive learning is performed between the second and fourth tactile feature vectors, intermodal contrastive learning is performed between the fourth and second visual feature vectors, and intramodal contrastive learning is performed between the first and third tactile feature vectors.

[0085] The contrastive learning described above all uses multimodal contrastive loss. Multimodal contrastive loss learning processes the two input feature vectors through the information comparison estimation loss function of formula (2) and the supervised contrastive loss function of formula (3) to obtain two loss function values ​​L. InfoNce and L Supcon The two loss function values ​​are summed using formula (1) to obtain the loss values ​​of the two input feature vectors. The above four sets of feature vectors can obtain a total of four loss values. After summing the four loss values, a total loss value is obtained.

[0086] Based on the total loss value, the visual encoder, tactile encoder, and MLP target head in the contrastive learning model are trained respectively. The momentum visual encoder is trained using the training results of the visual encoder, and the momentum tactile encoder is trained using the tactile encoder, until the final set number of iterations is reached, or the function loss value is less than the predetermined target.

[0087] See Figure 7The second aspect of this invention discloses a robot visual-touch perception system based on transformer and contrastive learning, comprising:

[0088] The acquisition unit is used to acquire visual and tactile images of the object;

[0089] A visual feature unit is used to extract visual feature vectors from a visual image through a visual encoder; the visual encoder is an enhanced Swing Transformer model, which has a coordinate attention mechanism set before the Swing Transformer block; the enhanced Swing Transformer model is obtained through multimodal contrastive learning training.

[0090] A tactile feature unit is used to extract tactile feature vectors from a tactile image through a tactile encoder. The tactile encoder is an enhanced Swin Transformer model, which has a coordinate attention mechanism set before the Swin Transformer block. The enhanced Swin Transformer model is obtained through multimodal contrastive learning training. During the multimodal contrastive learning training process, visual feature vectors and tactile feature vectors undergo intramodal contrastive learning, and visual feature vectors and tactile feature vectors undergo intermodal contrastive learning.

[0091] The fusion unit is used to fuse visual feature vectors and tactile feature vectors to obtain fused features;

[0092] The classification unit is used to process the fused features through a linear classification layer to obtain the classification result.

[0093] The network structure proposed in this invention has been well validated on the public datasets Calandra and Touch and Go, and its prediction performance has been improved compared with the original model, proving that the method is feasible.

[0094] The backbone network used in this invention can be replaced by networks other than the Swing transformer, such as ResNet, Vision transformer, VGG, AlexNet, etc. The visual-based tactile modal information input in this invention can theoretically be replaced by other modal data, such as visual image information.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot visual-touch perception method based on transformer and contrastive learning, characterized in that, include: Acquire visual and tactile images of an object; Visual feature vectors are extracted from visual images using a visual encoder; the visual encoder is a first enhanced SwinTransformer model, in which a coordinate attention mechanism is set before the Swin Transformer block; the first enhanced Swin Transformer model is obtained through multimodal contrastive learning training. Tactile feature vectors are extracted from tactile images using a tactile encoder; the tactile encoder is a second enhanced SwinTransformer model, in which a coordinate attention mechanism is set before the Swin Transformer block; the second enhanced SwinTransformer model is obtained through multimodal contrastive learning training; During the multimodal contrastive learning training process, visual feature vectors and tactile feature vectors undergo intramodal contrastive learning, while visual feature vectors and tactile feature vectors undergo intermodal contrastive learning. By fusing visual feature vectors and tactile feature vectors, a fused feature is obtained; The fused features are processed through a linear classification layer to obtain the classification result of the object; The intramodal contrastive learning includes contrastive learning of two visual feature vectors and contrastive learning of two tactile feature vectors. The two visual feature vectors are extracted by a visual encoder and a momentum visual encoder, respectively, and the two tactile feature vectors are extracted by a tactile encoder and a momentum tactile encoder, respectively. The intermodal contrast learning is a contrast learning between a visual feature vector extracted by a visual encoder and a tactile feature vector extracted by a momentum tactile encoder, and a contrast learning between a visual feature vector extracted by a momentum visual encoder and a tactile feature vector extracted by a tactile encoder. After obtaining two visual feature vectors, the visual encoder performs feature mapping processing on the two visual feature vectors through two MLP target heads with different parameters; after obtaining two visual feature vectors, the momentum visual encoder performs feature mapping processing on the two visual feature vectors through two MLP target heads with different parameters; after obtaining two tactile feature vectors, the tactile encoder performs feature mapping processing on the two tactile feature vectors through two MLP target heads with different parameters; after obtaining two tactile feature vectors, the momentum tactile encoder performs feature mapping processing on the two tactile feature vectors through two MLP target heads with different parameters.

2. The robot visual-touch perception method based on transformer and contrastive learning according to claim 1, characterized in that, The visual encoder and the corresponding MLP target head are trained using the contrastive learning results, which are the sum of the intramodal contrastive learning results and the intermodal contrastive learning results. The momentum visual encoder and the corresponding MLP target head are trained using the output results of the visual encoder. The tactile encoder and the corresponding MLP target head are trained using the contrastive learning results, which are the sum of the intramodal and intermodal contrastive learning results. The momentum tactile encoder and the corresponding MLP target head are trained using the output results of the tactile encoder.

3. The robot visual-touch perception method based on transformer and contrastive learning according to claim 1, characterized in that, The visual image is input into the visual encoder and the momentum visual encoder after data augmentation; the tactile image is input into the tactile encoder and the momentum tactile encoder after data augmentation.

4. The robot visual-touch perception method based on transformer and contrastive learning according to claim 1, characterized in that, The linear classification layer is obtained through training, and the parameters of the visual encoder, momentum visual encoder, tactile encoder and momentum tactile encoder are fixed during the training process.

5. The robot visual-touch perception method based on transformer and contrastive learning according to claim 1, characterized in that, Contrastive learning involves processing two input feature vectors through an information comparison estimation loss function and a supervised contrastive loss function. The results of the two contrastive loss functions are multiplied by their corresponding weight coefficients and then summed to obtain the loss value.

6. The robot visual-touch perception method based on transformer and contrastive learning according to claim 1, characterized in that, After the visual encoder or tactile encoder segments the image, it performs linear processing on the segmented image. The linearly processed feature map is then processed through four stages of feature extraction blocks, each stage including several feature extraction blocks. The extraction results are normalized by layers and then average pooled to obtain visual feature vectors or tactile feature vectors. In each feature extraction block, the feature map is first processed by a coordinate attention mechanism and then by a Swing Transformer block. The image is a visual image or a tactile image.

7. The robot visual-touch perception method based on transformer and contrastive learning according to claim 1, characterized in that, The data used for the multimodal contrastive learning training consists of data from the TAG dataset and the Calandra dataset.

8. A robot visual-touch perception system based on transformer and contrastive learning for implementing the robot visual-touch perception method of claim 1, characterized in that, include: The acquisition unit is used to acquire visual and tactile images of the object; A visual feature unit is used to extract visual feature vectors from a visual image through a visual encoder; the visual encoder is a first enhanced Swing Transformer model, in which a coordinate attention mechanism is set before the Swing Transformer block; the first enhanced Swing Transformer model is obtained through multimodal contrastive learning training. A tactile feature unit is used to extract tactile feature vectors from a tactile image through a tactile encoder; the tactile encoder is a second enhanced Swing Transformer model, in which a coordinate attention mechanism is set before the Swing Transformer block; the second enhanced Swing Transformer model is obtained through multimodal contrastive learning training; During the multimodal contrastive learning training process, visual feature vectors and tactile feature vectors undergo intramodal contrastive learning, while visual feature vectors and tactile feature vectors undergo intermodal contrastive learning. The fusion unit is used to fuse visual feature vectors and tactile feature vectors to obtain fused features; The classification unit is used to process the fused features through a linear classification layer to obtain the classification result.

Images