Robot visual touch perception method and system based on transformer and comparative learning

Abstract

The invention discloses a robot visual touch sensing method and system based on transformer and comparative learning, and belongs to the technical field of robots. According to the method, an enhanced Swin Transform model is introduced to serve as a sensing model, the enhanced model takes Swin Transform as a backbone network, a coordinate attention mechanism is introduced into the backbone network, global information is extracted through a sliding window attention mechanism of the backbone network, local feature extraction is enhanced through the coordinate attention mechanism, and the global information is extracted through the sliding window attention mechanism. The global and local features in the visual and tactile data can be effectively extracted and fused. According to the enhanced Swin Transformer model, in-modal comparative learning and inter-modal comparative learning are specifically performed through multi-modal comparative learning training in a pre-training stage of an earlier stage, and richer and more accurate discrimination features are generated, so that the intelligent sensing and operation capability of the robot to the environment is effectively improved. According to the method, the features can be effectively extracted in multi-modal data fusion, and deeper multi-modal data fusion is realized by utilizing the correlation and synergistic effect of the features among different modals.

Description

Technical Field

[0001] The present invention belongs to the technical field of robots, and in particular relates to a robot visual-tactile perception method and system based on transformer and contrastive learning. Background Art

[0002] In the field of robotics, intelligent perception and manipulation play a key role in multiple application scenarios. In industrial automation, robots need to accurately identify and manipulate objects of various complex shapes and materials to improve production efficiency and product quality. In the medical field, surgical robots rely on precise perception and manipulation capabilities to perform minimally invasive surgery and delicate operations to reduce surgical risks. In service robots, home robots and nursing robots need to be able to accurately identify and handle everyday objects to provide safe and efficient services. In the field of robotics, achieving intelligent perception and manipulation is a key goal. Vision sensors provide global perception of the environment through cameras, capturing the overall layout and dynamic changes of the scene, but may have limitations in recognition and positioning in complex environments. Tactile sensors provide local contact information such as the shape, texture, and hardness of objects through sensor arrays, and these details are crucial for precise manipulation and perception. However, due to the significant differences in feature representation between visual and tactile data, it remains challenging to effectively fuse these modal data to improve robot perception and manipulation capabilities.

[0003] Invention patent CN117435973A proposes a visual-tactile multimodal object recognition method based on deep learning. The scheme design uses a fiber grating sensor encapsulated in a silicone sleeve and installed at the end of the robot arm. It collects touch and slip signals by contacting the object, and uses an industrial camera to collect visual data. The touch and slip signals are demodulated to obtain pressure and vibration data, and the visual image is preprocessed and the surface characteristics are annotated. The touch and slip data is fused with the visual data to form a data set, which is input into a well-trained multimodal neural network model to achieve accurate object recognition. This method improves recognition accuracy by combining tactile and visual information through feature-level fusion, and is suitable for object recognition tasks of intelligent robots in complex environments.

[0004] Invention patent CN117611919A proposes a grasping object classification method based on visual-tactile fusion, which is implemented through four steps: data preprocessing, feature extraction, feature fusion, and classification decision. First, the original visual and tactile image data are standardized and preprocessed; then, the residual network and convolutional attention mechanism module are used to extract and enhance the visual and tactile features; then, the two features are spliced ​​and fused in the channel dimension to enhance the representation ability of the features; finally, the fused features are input to the classifier module, and the classification results are output through the fully connected layer and activation function. The advantage of this method is that it makes full use of the complementarity of visual and tactile data, improves the feature acquisition capability through the attention mechanism, significantly improves the accuracy and robustness of classification, and verifies its effectiveness on the constructed multimodal dataset.

[0005] Although the above schemes are innovative in multimodal object recognition, they may be too simple in the fusion of visual and tactile data, and fail to deeply explore the complementarity and intrinsic connection between different modalities, which limits the performance of the recognition system. Secondly, the above methods may perform well in specific environments or conditions, but their adaptability and robustness may be insufficient when the environment changes or when facing complex scenes. In addition, the reliance on a large amount of labeled data limits the widespread application of the model in the real world, especially in scenarios where data acquisition is costly or difficult to label. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide a robot visual-tactile perception method and system based on transformer and contrastive learning, so as to solve the problems of insufficient complementarity between different modalities and insufficient model robustness in the process of multimodal object recognition in the prior art.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A robot visual-tactile perception method based on transformer and contrastive learning, comprising:

[0009] Acquire visual and tactile images of objects;

[0010] A visual feature vector is extracted from a visual image through a visual encoder; the visual encoder is a first enhanced Swin Transformer model, wherein a coordinate attention mechanism is arranged before a Swin Transformer block therein; the first enhanced Swin Transformer model is obtained through multimodal contrastive learning training;

[0011] A tactile feature vector is extracted from a tactile image through a tactile encoder; the tactile encoder is a second enhanced Swin Transformer model, wherein a coordinate attention mechanism is arranged before a Swin Transformer block therein; the second enhanced Swin Transformer model is obtained through multimodal contrastive learning training;

[0012] In the multimodal contrastive learning training process, the visual feature vector and the tactile feature vector are respectively subjected to intra-modal contrastive learning, and the visual feature vector and the tactile feature vector are subjected to inter-modal contrastive learning;

[0013] Fusing the visual feature vector and the tactile feature vector to obtain a fused feature;

[0014] After processing the fused features through the linear classification layer, the classification results of the object are obtained.

[0015] A further improvement of the present invention is:

[0016] Preferably, the intra-modality contrast learning includes contrast learning of two visual feature vectors and contrast learning of two tactile feature vectors, the two visual feature vectors are respectively extracted by a visual encoder and a momentum visual encoder, and the two tactile feature vectors are respectively extracted by a tactile encoder and a momentum tactile encoder;

[0017] The inter-modality contrast learning is contrast learning of a visual feature vector extracted by a visual encoder and a tactile feature vector extracted by a momentum tactile encoder, and contrast learning of a visual feature vector extracted by a momentum visual encoder and a tactile feature vector extracted by a tactile encoder.

[0018] Preferably, after the visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed by two MLP target heads with different parameters for feature mapping; after the momentum visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed by two MLP target heads with different parameters; after the tactile encoder obtains two tactile feature vectors, the two tactile feature vectors are respectively processed by two MLP target heads with different parameters; after the momentum tactile encoder obtains two tactile feature vectors, the two tactile feature vectors are respectively processed by two MLP target heads with different parameters.

[0019] Preferably, the visual encoder and the corresponding MLP target head are trained by contrastive learning results, the contrastive learning result is the sum of contrastive learning results within two modalities and contrastive learning results between two modalities, and the momentum visual encoder and the corresponding MLP target head are trained by visual encoder output results;

[0020] The tactile encoder and the corresponding MLP target head are trained by contrastive learning results, where the contrastive learning results are the sum of contrastive learning results within two modalities and contrastive learning results between two modalities; the momentum tactile encoder and the corresponding MLP target head are trained by tactile encoder output results.

[0021] Preferably, the visual image is input into the visual encoder and the momentum visual encoder after data enhancement; the tactile image is input into the tactile encoder and the momentum tactile encoder after data enhancement.

[0022] Preferably, the linear classification layer is obtained through training, and during the training process of the linear classification layer, the parameters of the visual encoder, the momentum visual encoder, the tactile encoder and the momentum tactile encoder are fixed.

[0023] Preferably, contrastive learning is to process the two input feature vectors through an information contrast estimation loss function and a supervised contrast loss function, and multiply the results of the two contrast loss functions by corresponding weight coefficients respectively and then add them together to obtain a loss value.

[0024] Preferably, after the visual encoder or tactile encoder segments the image, the segmented image is linearly processed, and the feature map after linear processing is processed by four stages of feature extraction blocks in sequence, each stage includes several feature extraction blocks, and the extraction results are normalized by layers and then average pooled to obtain a visual feature vector or a tactile feature vector; the feature map is first processed by a coordinate attention mechanism in each of the feature extraction blocks, and then processed by a SwinTransformer block; the image is a visual image or a tactile image.

[0025] Preferably, the data for multimodal contrastive learning training consists of data from the TAG dataset and the Calandra dataset.

[0026] A robot visual-tactile perception system based on transformer and contrastive learning, comprising:

[0027] A collection unit, used for acquiring visual and tactile images of the object;

[0028] A visual feature unit, configured to extract a visual feature vector from a visual image through a visual encoder; the visual encoder is a first enhanced Swin Transformer model, wherein a coordinate attention mechanism is provided before a Swin Transformer block therein; the first enhanced Swin Transformer model is obtained through multimodal contrastive learning training;

[0029] A tactile feature unit, configured to extract a tactile feature vector from a tactile image through a tactile encoder; the tactile encoder is a second enhanced Swin Transformer model, wherein a coordinate attention mechanism is arranged before a Swin Transformer block therein; the second enhanced Swin Transformer model is obtained through multimodal contrastive learning training;

[0030] During the multimodal contrastive learning training process, the visual feature vector and the tactile feature vector are subjected to intra-modal contrastive learning respectively, and the visual feature vector and the tactile feature vector are subjected to inter-modal contrastive learning;

[0031] A fusion unit, used for fusing the visual feature vector and the tactile feature vector to obtain a fusion feature;

[0032] The classification unit is used to process the fused features through the linear classification layer to obtain the classification results.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention discloses a robot visual-tactile perception method based on transformer and contrastive learning. The method designs a new enhanced Swin Transformer model. The model uses Swin Transformer as the backbone network, extracts global information through a sliding window attention mechanism, and introduces a coordinate attention mechanism to enhance local feature extraction, so that the extracted features include both comprehensive image information and targeted local features; effectively extracts and fuses global and local features in visual and tactile data, and significantly improves the richness and accuracy of features. During training, the model is trained by a joint contrastive loss method, combining two types of loss functions as contrastive losses for multimodal training. Multimodal training includes intra-modal contrastive learning and inter-modal contrastive learning, which can make full use of labeled and unlabeled data to generate richer and more accurate discriminant features, thereby effectively improving the robot's intelligent perception and operation capabilities of the environment. The method can effectively extract and utilize the correlation and synergy between different modalities in multimodal data fusion, and achieve deeper multimodal data fusion. Its framework and core idea are independent of the specific types of visual and tactile sensors, have good generalization, and can improve the robot's intelligent perception and operation capabilities of the environment in different application scenarios. The present invention also has the following advantages:

[0035] (1) The present invention introduces a coordinate attention mechanism into the Swin Transformer model. Swin Transformer can capture richer visual and tactile features. The coordinate attention mechanism specifically enhances the extraction of local features of tactile data, ensuring that even the slightest feature differences can be accurately captured by the model, thereby improving the accuracy of data fusion and the expression of details.

[0036] (2) Combining the above-mentioned Swin Transformer and coordinate attention mechanism, the present invention proposes a new multimodal representation learning method. This method solves the problem that traditional fusion methods cannot fully utilize the correlation and synergy between modalities by deeply fusing visual and tactile data.

[0037] (3) This paper innovatively proposes a joint contrast loss method that combines InfoNCE loss and SupCon loss. This method can simultaneously utilize labeled and unlabeled data, promote the model to generate richer and more accurate discriminant features in a multimodal environment, improve the generalization ability and environmental adaptability of the model, and significantly improve the intelligent perception and operation capabilities of the robot.

[0038] (4) This learning and training method can better learn from a large amount of unlabeled data, improve the model's generalization ability and adaptability to environmental changes, reduce dependence on large-scale labeled data, optimize small sample learning, and make the robot's intelligent perception and operation in complex environments more accurate and flexible, providing an innovative solution for the further development of intelligent robot technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The present invention is a robot visual touch perception method based on transformer and contrast learning process Figure 1 ;

[0040] Figure 2 The process of robot visual touch perception method based on transformer and contrast learning Figure 2 ;

[0041] Figure 3 The encoder diagram of the swin transformer model based on the attention mechanism of the present invention;

[0042] Figure 4 Training graph for multimodal contrastive learning;

[0043] Figure 5 It is the multimodal contrast loss map;

[0044] Figure 6A training flowchart for a robot visual-tactile perception method based on transformer and contrastive learning;

[0045] Figure 7 This is a structural diagram of a robot visual-tactile perception system based on transformer and contrastive learning according to the present invention. DETAILED DESCRIPTION

[0046] In the following, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first", "second", "third", and "fourth" may explicitly or implicitly include one or more of the features.

[0047] The co-shooting method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0048] It should be noted that the terms "first", "second", etc. in the specification and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] As mentioned in the background technology, robot intelligent perception and operation require visual sensors to perceive the environment and tactile sensors to provide detailed information. However, due to the differences in data distribution and feature representation between visual and tactile data, it is still challenging to effectively fuse these modal features. Existing methods usually adopt a simple fusion method, which cannot fully utilize the correlation and synergy between different modalities, resulting in insufficient richness and accuracy of fused features. The present invention aims to solve the problem of visual and tactile data fusion in the field of robotics. To this end, a method combining an enhanced Transformer mechanism and contrastive learning is proposed for multimodal representation learning of vision and touch.

[0050] See also Figure 1 and Figure 2 The first aspect of the present invention provides a robot visual-tactile perception method based on transformer and contrastive learning, the method specifically comprising the following steps:

[0051] S1, obtaining visual and tactile images of objects;

[0052] S2, extracting a visual feature vector from a visual image through a visual encoder; the visual encoder is a first enhanced Swin Transformer model, and the first enhanced Swin Transformer model is provided with a coordinate attention mechanism before the Swin Transformer block; the first enhanced Swin Transformer model is obtained through multimodal contrastive learning training;

[0053] A tactile feature vector is extracted from a tactile image through a tactile encoder; the tactile encoder is a second enhanced Swin Transformer model, and a coordinate attention mechanism is set before the Swin Transformer block of the second enhanced Swin Transformer model; the second enhanced Swin Transformer model is obtained through multimodal contrastive learning training;

[0054] In the above multimodal contrastive learning training process, the visual feature vector and the tactile feature vector are respectively subjected to intra-modal contrastive learning, and the visual feature vector and the tactile feature vector are subjected to inter-modal contrastive learning;

[0055] S3, fuses the visual feature vector and the tactile feature vector to obtain a fused feature;

[0056] S4, after processing the fused features through the linear classification layer, the classification result of the object is obtained.

[0057] In some embodiments of the present invention, see Figure 3, which is a unified system framework of the visual encoder and the tactile encoder, that is, the first enhanced Swin Transformer model and the second Swin Transformer model are enhanced Swin Transformer models with the same structure. The model is based on the Swin Transformer model, can effectively extract global features, and has advantages over traditional convolutional networks. In order to enhance the model's ability to extract tactile and visual features, a coordinate attention mechanism is introduced. Specifically, after the visual encoder or tactile encoder segments the image, the segmented image is linearly processed, and the linearly processed feature map is processed by feature extraction blocks in four stages in sequence. Several feature extraction blocks are set in each stage, and the extraction results are averaged and pooled after layer normalization to obtain a visual feature vector or a tactile feature vector; the feature map is first processed by the coordinate attention mechanism in each of the feature extraction blocks, and then processed by the Swin Transformer block. The above image can be a visual image or a tactile image, which is processed by the corresponding visual encoder and tactile encoder respectively.

[0058] In some embodiments, the size of the visual image and the tactile image after preprocessing is 224×224, and the image segmentation module segments the image into blocks of size 4×4 and flattens them to 48 in the channel direction. In stage 1, a linear embedding model is used to linearly transform the image pixel channels, and the number of feature map channels is changed from 48 to 96; then, the feature map of size 56×56 and number of channels 96 is processed by four feature extraction blocks in sequence, each of which includes a coordinate attention mechanism and a Swin Transformer block. The feature map after linear transformation is processed by the coordinate attention mechanism and then sent to the Swin Transformer block. In stage 2, stage 3 and stage 4, the four stages contain 2, 6 and 2 Swin Transformer blocks and image patch merging modules respectively, the number of channels is 192, 384 and 768 respectively, and the size of the feature map is 28×28, 14×14 and 7×7. Finally, the feature map with a size of 7×7 and 768 channels is normalized and averaged to obtain a feature vector, and the image patches are merged between the Swin Transformer blocks for downsampling to reduce the resolution of the feature map and increase the number of channels, thereby achieving hierarchical feature extraction. This method significantly improves the model's feature extraction capability for complex data through multi-stage processing and attention mechanism.

[0059] See also Figure 4 The present invention also discloses a training process of an enhanced Swin Transformer model, which is obtained through learning and training. The training model is shown in Figure 3The learning and training model uses the self-supervised learning framework MOCO-v2 to extract intra-modal features, in which other samples in the same batch are used as negative samples instead of maintaining a memory queue to reduce storage costs and computing time. The core of the learning and training model is to perform intra-modal contrast learning and inter-modal contrast learning between eight feature vectors.

[0060] The eight feature vectors learned in the model include four visual feature vectors and four tactile feature vectors. Among the four visual feature vectors, two visual feature vectors are extracted by visual encoders, and two visual feature vectors are extracted by momentum visual encoders; the tactile feature vectors include four tactile feature vectors, two tactile feature vectors are extracted by tactile encoders, and two tactile feature vectors are extracted by momentum tactile encoders.

[0061] It should be noted that the network structures of the visual encoder, momentum visual encoder, tactile encoder and momentum tactile encoder are Figure 3 The encoder structure.

[0062] The intra-modality contrastive learning includes contrastive learning of two visual feature vectors and contrastive learning of two tactile feature vectors. The two visual feature vectors are respectively extracted by a visual encoder and a momentum visual encoder, and the two tactile feature vectors are respectively extracted by a tactile encoder and a momentum tactile encoder.

[0063] The inter-modal contrastive learning is the contrastive learning of a visual feature vector extracted by the visual encoder and a tactile feature vector extracted by the momentum tactile encoder, and the contrastive learning of a visual feature vector extracted by the momentum visual encoder and a tactile feature vector extracted by the tactile encoder. In the above process, the features of the two modalities are independently extracted by the visual encoder and the tactile encoder, providing basic feature representation for subsequent intra-modal and inter-modal contrastive learning.

[0064] As a preferred solution, after the visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed by two MLP target heads with different parameters for feature mapping; after the momentum visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed by two MLP target heads with different parameters; after the tactile encoder obtains two tactile feature vectors, the two tactile feature vectors are respectively processed by two MLP target heads with different parameters; after the momentum tactile encoder obtains two tactile feature vectors, the two tactile feature vectors are respectively processed by two MLP target heads with different parameters. In this process, a contrastive learning framework is introduced, an MLP target head is used to map features, and contrastive learning within and between modalities is performed, and the representation ability of features is enhanced through discriminative learning.

[0065] As a preferred solution, the visual image is input into the visual encoder and the momentum visual encoder after data enhancement; the tactile image is input into the tactile encoder and the momentum tactile encoder after data enhancement. Data augmentation is performed on the visual and tactile images to enhance the model's generalization ability to image changes.

[0066] As a preferred solution, see Figure 5 , contrastive learning is to process the two input feature vectors through the information contrast estimation loss function (Infonce loss, abbreviated as InfoNCE) and the supervised contrast loss function (Supcon loss, SupCon), and multiply the results of the two contrast loss functions by the corresponding weight coefficients and add them together to obtain the loss value; this joint contrast loss can achieve efficient use of labeled and unlabeled data and further improve the discriminability of features. Information contrast estimation loss function and supervised contrast loss function are two commonly used contrastive learning loss functions. They optimize the model in different ways, so that the model can better perform feature representation learning in unsupervised and supervised environments.

[0067] Specifically, the information contrast estimation loss function is based on the idea of ​​noise contrast estimation, which measures similarity by maximizing the probability of positive samples and minimizing the probability of negative samples. The supervised contrast loss function aims to maximize the similarity of samples of the same category and minimize the similarity of samples of different categories. By combining these two loss functions, the feature representation ability of the model can be maximized. The block diagram of the joint loss value is shown in the figure below. Figure 4 As shown, the formula expression is shown in formula (1).

[0068] L=λL InfoNce +(1-λ)L Supcon (1)

[0069] Among them, L is the joint loss, L InfoNce is the loss value of the information comparison estimation loss function, λ is the weight coefficient of the loss value, L Supcon is the loss value of the supervised contrast loss function, and 1-λ is the weight coefficient of the loss function.

[0070] The information contrast estimation loss function is an unsupervised loss function commonly used in contrastive learning and is defined as follows:

[0071]

[0072] Where: z i and z j is the feature representation of the positive sample pair, z k is the feature representation of negative samples, including all positive sample pairs and negative samples, and τ is the temperature parameter used to control the sensitivity of the contrast.

[0073] The supervised contrastive loss function is a supervised contrastive learning loss function that extends the unsupervised contrastive learning method by using label information to guide feature learning, and is defined as follows.

[0074]

[0075] Where: P(i) is the set of positive samples belonging to the same class as sample i, and A(i) is the set of all samples excluding sample i itself. i 、z p and z a are the feature representations of sample i, positive sample p, and negative sample a respectively. τ is the temperature parameter.

[0076] It should be noted that the above two feature vectors are learned separately through a multimodal contrast loss training, so four loss values ​​calculated by formula (1) are finally obtained. After adding the four loss values, a total loss value is obtained.

[0077] Furthermore, based on the above-mentioned total loss value, the visual encoder, tactile encoder and MLP target head are each trained by comparative learning through the total loss value; the momentum visual encoder is trained through the visual encoder output result, and the momentum tactile encoder is trained through the tactile encoder output result.

[0078] As a preferred solution, during the entire model training process, the linear classification layer is obtained through training, and the parameters of the visual encoder, momentum visual encoder, tactile encoder and momentum tactile encoder are fixed during the training process. Based on the fixed encoder parameters, feature fusion is performed and input into the linear classification layer to achieve classification of downstream tasks and obtain classification labels. At the same time, the enhanced Transformer mechanism and coordinate attention mechanism are used to improve the richness and accuracy of feature extraction, thereby enhancing the intelligent perception and operation capabilities of the robot.

[0079] In some embodiments of the present invention, the data used in the training and learning process consists of data from the TAG dataset and the Calandra dataset. In the application process, both types of datasets are divided into training sets and test sets in a ratio of 7:3. This division method helps to evaluate the performance and generalization ability of the model and ensure that the performance of the model on different data is stable. The training set is used for model training, and the test set is used to verify the effect and accuracy of the model.

[0080] It should be noted that the Calandra dataset is a public dataset provided by Roberto Calandra of the University of California, Berkeley. It contains tactile data collected using a robotic arm and tactile sensors, as well as visual data collected by a depth camera. Data collection is divided into three moments: initial position, clamped but not lifted, and hovering for 2 seconds. Each experimental routine contains six images and their corresponding grasping result labels. In order to enhance the generalization performance of the model, the images are resized to 256×256 and randomly cropped to 224×224, and randomly rotated to increase the diversity of the data and improve the robustness of the model. The dataset grasps a total of 106 different items. Through these preprocessing and data enhancement measures, the aim is to improve the performance of the model in different environments.

[0081] It should be noted that the TAG dataset consists of egocentric perspective videos recorded by human collectors when they use Gelsight tactile sensors to explore objects in natural environments, containing about 13.9k detected touches and about 3971 individual object instances. The data collectors selected a variety of objects to press in each scene, including chairs, walls, ground, sofas, tables, etc. in indoor scenes, and grass, rocks, trees, sand, etc. in outdoor scenes, totaling 20 categories. Each object was pressed about 3 times, and each press lasted an average of 0.7 seconds. Compared with previous datasets limited to laboratories or simulated environments, the TAG dataset is more diverse, covering more outdoor objects and scenes, and close to the complexity of the real world. It contains a variety of tactile features that help distinguish material properties, such as soft and hard surfaces and smooth and rough surfaces. The task of this experiment is to distinguish different material categories. In order to improve the generalization performance of the model, the researchers processed the images, resized them to 256×256, and then randomly cropped them to 224×224 and randomly rotated them. The diversity and complexity of the TAG dataset provides a broader application context for tactile sensor research.

[0082] In the pre-training stage of contrastive learning, the above process performs data enhancement on visual and tactile images, and then uses the visual encoder and tactile encoder to extract corresponding features respectively. In order to perform intra-modal and inter-modal contrastive learning, different MLP target heads are used for feature mapping. Four features of the visual modality and four features of the tactile modality are obtained, of which two features of the visual modality and the tactile modality are used for intra-modal contrastive learning, while the other two features of the visual modality and the two features of the tactile modality are used for inter-modal contrastive learning. The proposed joint loss is used for the multi-modal contrastive loss.

[0083] See also Figure 6In one embodiment of the present invention, a specific model training process is disclosed. In the pre-training stage of contrastive learning, data enhancement is performed on visual and tactile images to obtain visual images and tactile images after data enhancement; after the visual image is processed by the visual encoder, the obtained feature map is processed by two different MLP target heads with different parameters, thereby obtaining two different visual feature vectors, which are set as the first visual feature vector and the second visual feature vector respectively; after the visual image is processed by the momentum visual encoder, the obtained feature map is processed by two different MLP target heads with different parameters, thereby obtaining two different visual feature vectors, which are set as the third visual feature vector and the fourth visual feature vector respectively. After the tactile image is processed by the tactile encoder, the obtained feature map is processed by two different MLP target heads with different parameters, thereby obtaining two different tactile feature vectors, which are set as the first tactile feature vector and the second tactile feature vector respectively; after the tactile image is processed by the momentum tactile encoder, the obtained feature map is processed by two different MLP target heads with different parameters, thereby obtaining two different tactile feature vectors, which are set as the third tactile feature vector and the fourth tactile feature vector respectively.

[0084] Based on the visual feature vectors and tactile feature vectors obtained above, multimodal contrast learning is performed. Specifically, the first visual feature vector and the third visual feature vector perform intra-modal contrast learning, the second visual feature vector and the fourth tactile feature vector perform inter-modal contrast learning, the fourth visual feature vector and the second tactile feature vector perform inter-modal contrast learning, and the first tactile feature vector and the third tactile feature vector perform intra-modal contrast learning.

[0085] The above contrastive learning is all performed through multimodal contrastive loss. Multimodal contrastive loss learning processes the two input feature vectors through the information contrast estimation loss function of formula (2) and the supervised contrast loss function of formula (3) to obtain two loss function values ​​L InfoNce and L Supcon , the two loss function values ​​are added together through formula (1) to obtain the loss values ​​of the two input feature vectors; the above four groups of feature vectors can obtain a total of four loss values, and after adding the four loss values, a total loss value is obtained.

[0086] Based on the total loss value, the visual encoder, tactile encoder and MLP target head in the contrastive learning model are trained respectively. The momentum visual encoder is trained by the training results of the visual encoder, and the momentum tactile encoder is trained by the tactile encoder until the final set number of cycles is reached, or the function loss value is less than the predetermined target.

[0087] See also Figure 7The second aspect of the present invention discloses a robot visual-tactile perception system based on transformer and contrastive learning, comprising:

[0088] A collection unit, used for acquiring visual and tactile images of the object;

[0089] A visual feature unit, used for extracting a visual feature vector from a visual image through a visual encoder; the visual encoder is an enhanced Swin Transformer model, and the enhanced Swin Transformer model is provided with a coordinate attention mechanism before the Swin Transformer block; the enhanced Swin Transformer model is obtained through multimodal contrastive learning training;

[0090] A tactile feature unit is used to extract a tactile feature vector from a tactile image through a tactile encoder; the tactile encoder is an enhanced Swin Transformer model, and a coordinate attention mechanism is set before the Swin Transformer block of the enhanced Swin Transformer model; the enhanced Swin Transformer model is obtained through multimodal contrastive learning training; during the multimodal contrastive learning training, the visual feature vector and the tactile feature vector are respectively subjected to intra-modal contrastive learning, and the visual feature vector and the tactile feature vector are subjected to inter-modal contrastive learning;

[0091] A fusion unit, used for fusing the visual feature vector and the tactile feature vector to obtain a fusion feature;

[0092] The classification unit is used to process the fused features through the linear classification layer to obtain the classification results.

[0093] The network structure proposed in this invention has been well verified on the Calandra and Touch and go public datasets, and the prediction performance has been improved compared with the original model, proving that the method is feasible.

[0094] The backbone network used in the present invention can be replaced by networks other than Swin transformer, such as ResNet, Vision transformer, VGG, AlexNet, etc. The vision-based tactile modal information input in the present invention can theoretically be replaced by other modal data, such as visual photography information, etc.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot visual-tactile perception method based on transformer and contrastive learning, characterized in that: include: Acquire visual and tactile images of objects; A visual feature vector is extracted from a visual image through a visual encoder; the visual encoder is a first enhanced Swin Transformer model, wherein a coordinate attention mechanism is arranged before a Swin Transformer block therein; the first enhanced Swin Transformer model is obtained through multimodal contrastive learning training; A tactile feature vector is extracted from a tactile image through a tactile encoder; the tactile encoder is a second enhanced Swin Transformer model, wherein a coordinate attention mechanism is arranged before a Swin Transformer block therein; the second enhanced Swin Transformer model is obtained through multimodal contrastive learning training; In the multimodal contrastive learning training process, the visual feature vector and the tactile feature vector are respectively subjected to intra-modal contrastive learning, and the visual feature vector and the tactile feature vector are subjected to inter-modal contrastive learning; Fusing the visual feature vector and the tactile feature vector to obtain a fused feature; After processing the fused features through the linear classification layer, the classification results of the object are obtained.

2. The robot visual-tactile perception method based on transformer and contrastive learning according to claim 1, characterized in that: The intra-modality contrast learning includes contrast learning of two visual feature vectors and contrast learning of two tactile feature vectors, the two visual feature vectors are respectively extracted by a visual encoder and a momentum visual encoder, and the two tactile feature vectors are respectively extracted by a tactile encoder and a momentum tactile encoder; The inter-modality contrast learning is contrast learning of a visual feature vector extracted by a visual encoder and a tactile feature vector extracted by a momentum tactile encoder, and contrast learning of a visual feature vector extracted by a momentum visual encoder and a tactile feature vector extracted by a tactile encoder.

3. The robot visual-tactile perception method based on transformer and contrastive learning according to claim 2, characterized in that: After the visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed by two MLP target heads with different parameters for feature mapping; after the momentum visual encoder obtains two visual feature vectors, the two visual feature vectors are respectively processed by two MLP target heads with different parameters; after the tactile encoder obtains two tactile feature vectors, the two tactile feature vectors are respectively processed by two MLP target heads with different parameters; after the momentum tactile encoder obtains two tactile feature vectors, the two tactile feature vectors are respectively processed by two MLP target heads with different parameters.

4. The robot visual-tactile perception method based on transformer and contrastive learning according to claim 3, characterized in that: The visual encoder and the corresponding MLP target head are trained by contrastive learning results, wherein the contrastive learning result is the sum of contrastive learning results within two modalities and contrastive learning results between two modalities, and the momentum visual encoder and the corresponding MLP target head are trained by visual encoder output results; The tactile encoder and the corresponding MLP target head are trained by contrastive learning results, where the contrastive learning results are the sum of contrastive learning results within two modalities and contrastive learning results between two modalities; the momentum tactile encoder and the corresponding MLP target head are trained by tactile encoder output results.

5. The robot visual-tactile perception method based on transformer and contrastive learning according to claim 2, characterized in that: The visual image is input into the visual encoder and the momentum visual encoder after data enhancement; the tactile image is input into the tactile encoder and the momentum tactile encoder after data enhancement.

6. The robot visual-tactile perception method based on transformer and contrastive learning according to claim 2, characterized in that: The linear classification layer is obtained through training, and during the training process of the linear classification layer, the parameters of the visual encoder, the momentum visual encoder, the tactile encoder and the momentum tactile encoder are fixed.

7. The robot visual-tactile perception method based on transformer and contrastive learning according to claim 1, characterized in that: Contrastive learning is to process the two input feature vectors through the information contrast estimation loss function and the supervised contrast loss function, multiply the results of the two contrast loss functions by the corresponding weight coefficients and add them together to obtain the loss value.

8. The robot visual-tactile perception method based on transformer and contrastive learning according to claim 1, characterized in that: After the visual encoder or tactile encoder segments the image, the segmented image is linearly processed, and the feature map after linear processing is sequentially processed by feature extraction blocks in four stages, each stage includes several feature extraction blocks, and the extraction results are layer-normalized and average pooled to obtain a visual feature vector or a tactile feature vector; the feature map is first processed by a coordinate attention mechanism in each of the feature extraction blocks, and then processed by a Swin Transformer block; the image is a visual image or a tactile image.

9. The robot visual-tactile perception method based on transformer and contrastive learning according to claim 1, characterized in that: The data for the multimodal contrastive learning training consists of data from the TAG dataset and the Calandra dataset.

10. A robot visual-tactile perception system based on transformer and contrastive learning, characterized in that: include: A collection unit, used for acquiring visual and tactile images of the object; A visual feature unit, configured to extract a visual feature vector from a visual image through a visual encoder; the visual encoder is a first enhanced Swin Transformer model, wherein a coordinate attention mechanism is provided before a Swin Transformer block therein; the first enhanced Swin Transformer model is obtained through multimodal contrastive learning training; A tactile feature unit, configured to extract a tactile feature vector from a tactile image through a tactile encoder; the tactile encoder is a second enhanced Swin Transformer model, wherein a coordinate attention mechanism is arranged before a Swin Transformer block therein; the second enhanced Swin Transformer model is obtained through multimodal contrastive learning training; During the multimodal contrastive learning training process, the visual feature vector and the tactile feature vector are subjected to intra-modal contrastive learning respectively, and the visual feature vector and the tactile feature vector are subjected to inter-modal contrastive learning; A fusion unit, used for fusing the visual feature vector and the tactile feature vector to obtain a fusion feature; The classification unit is used to process the fused features through the linear classification layer to obtain the classification results.

Images