Tactile guidance-based diffusion adaptive fusion object identification method

Abstract

The invention relates to the technical field of computer vision, and discloses a tactile guidance-based diffusion adaptive fusion object recognition method, which comprises the following steps of: acquiring visual and tactile data as input of an offline trained tactile guidance-based diffusion adaptive fusion object recognition model; visual data and tactile data respectively pass through a feature extraction network, obtained features respectively pass through a full-connection layer with a self-adaptive dropout algorithm and are subjected to respe operation, and obtained visual features and tactile features pass through a visual-tactile fusion module, the full-connection layer with the self-adaptive dropout algorithm and the full-connection layer with the self-adaptive dropout algorithm and the full-connection layer with the self-adaptive dropout algorithm. And obtaining an object identification result and confidence. According to the method, the unique advantages of the visual information and the tactile information are fully utilized, effective integration of two perception modes is promoted, the decision precision of the model facing samples with high certainty is ensured, and the exploration capability and robustness of the model in an uncertain environment are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an object recognition method based on tactile-guided diffusion adaptive fusion. Background Art

[0002] Object recognition is an important research direction in the field of computer vision and artificial intelligence. Object recognition is usually achieved by capturing object images with RGB cameras and using machine learning algorithms for image classification. At present, visual learning methods have been widely studied, and many advanced algorithms and large visual learning datasets have been born. However, visual sensors can only capture the appearance characteristics of objects, cannot perceive their material properties, and are easily affected by external environments such as light and background, which leads to low recognition rates in industrial scenarios. To make up for the shortcomings of visual sensors, tactile sensors can be used to obtain various properties of objects (such as texture, roughness, and friction). Tactile learning can provide important information about the properties of objects, which is particularly effective in distinguishing between many visually similar objects. Object recognition methods using visual or tactile information have made significant progress, but how to effectively combine visual and tactile learning methods in object recognition is still a very attractive research topic. Although visual information and tactile information can complement each other, in the field of perception, visual perception focuses on the perception of shape and local color, while tactile perception emphasizes pressure and roughness. All these differences make it challenging to directly fuse the information of the two modalities. Li et al. [1] A prediction model was constructed using a generative adversarial network, demonstrating the potential for mutual conversion between visual and tactile modalities. The model can synthesize realistic tactile images from visual inputs and vice versa. Zheng et al. [2] By establishing a knowledge forgetting mechanism, the problem of visual-tactile cross-modal learning in a lifelong learning environment is solved. This method can continuously learn the knowledge acquired from visual and tactile data. Liu et al. [3]A visual-tactile fusion framework is developed, which designs a sparse coding method to alleviate the weak pairing problem to achieve alignment of the two modalities. Although these methods utilize the alignment information from the visual and tactile modalities, their fusion operations only focus on concatenation and summation. How to balance the information alignment of the two modalities and the fusion of deep interaction features, and how to capture and enhance the complementary information between the two modalities of visual information and tactile information are still important issues that plague current researchers. The anisotropic diffusion model has shown significant advantages in the field of computer vision, especially in image denoising, edge protection, and feature enhancement. The model can adaptively adjust the direction and intensity of the diffusion process according to the local characteristics of pixels in the image, such as gradients and textures, thereby effectively protecting edge details while smoothing image noise. This feature makes the anisotropic diffusion model a powerful tool that has achieved excellent performance in image processing and computer vision tasks. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an object recognition method based on tactile-guided diffusion adaptive fusion, which is used to solve the problem that traditional visual information and tactile information fusion operations only focus on the series and summation of features, and to achieve a more accurate object recognition effect by capturing and enhancing the complementary information between the two modalities of visual information and tactile information.

[0004] In order to solve the above technical problems, the present invention provides an object recognition method based on tactile-guided diffusion adaptive fusion, including collecting visual data and tactile data, adjusting the image size and converting it into a tensor and performing normalization processing, as the input of an offline trained object recognition model based on tactile-guided diffusion adaptive fusion, and obtaining an object recognition result and confidence;

[0005] The object recognition model based on tactile-guided diffusion adaptive fusion includes a feature extraction network and a visual-tactile fusion module. The standardized visual data and tactile data are each passed through a feature extraction network, and then the extracted visual features and tactile features are each passed through a fully connected layer with an adaptive dropout algorithm and a reshape operation is performed. The visual features V and tactile features T obtained after the reshape operation are input into the visual-tactile fusion module to obtain the fusion feature F; the fusion feature F is then passed through a fully connected layer with an adaptive dropout algorithm and a fully connected layer.

[0006] As an improvement of the object recognition method based on tactile-guided diffusion adaptive fusion of the present invention:

[0007] The feature extraction network is a Resnet18 network.

[0008] As a further improvement of the object recognition method based on tactile-guided diffusion adaptive fusion of the present invention:

[0009] The visual-tactile fusion module achieves cross-modal fusion through an anisotropic diffusion model:

[0010] (1) The tactile feature T is used as the guiding feature map to determine the diffusion rate. The diffusion rate is multiplied by the feature difference of the visual feature V and a learnable convolution kernel to obtain the primary visual-tactile fusion feature:

[0011] Y (m,n) =∑ (i,j)∈N S T (T (i,j) ,T (m,n) )×D V (V (i,j) ,V (m,n) )×W (i,j) (1)

[0012] Among them, the tuple (i, j) enumerates the index of each entry in the 3′3 local area window N centered at (m, n), W is a learnable convolution kernel with a size of 3′3;

[0013] S T (T (i,j) ,T (m,n) )=exp(-|T (i,j) -T (m,n) | 2 / s 2 ) (2)

[0014] D v (V (i,j) ,V (m,n) )=V (i,j) -V (m,n) (3)

[0015] Where s is a hyperparameter, set to 2; T (i,j) and T (m,n) Respectively represent the features at (i, j) and (m, n) of the tactile feature map, V (i,j) and V (m,n) Represent the features at positions (i, j) and (m, n) in the visual feature map respectively.

[0016] As a further improvement of the object recognition method based on tactile-guided diffusion adaptive fusion of the present invention:

[0017] The adaptive dropout algorithm is:

[0018] (1) After the input features pass through the fully connected layer, a Softmax operation is performed to generate the prediction confidence λitem ;

[0019] (2) Update the Dropout parameter p:

[0020]

[0021] Among them, Z i (t) refers to the connection strength of a single neuron in the network, w ij represents the weight between neurons i and j in the tth iteration, π ij represents the activation state of any neuron connected to neuron i in the network;

[0022] (3) Using the updated p-value, randomly decide whether to “close” each neuron in the fully connected layer according to the Bernoulli distribution;

[0023] (4) Dropout is not enabled during testing and online use, and p is set to 0.

[0024] As a further improvement of the object recognition method based on tactile-guided diffusion adaptive fusion of the present invention:

[0025] The training process of the object recognition model based on tactile-guided diffusion adaptive fusion is as follows:

[0026] The visual image data and tactile data in the AU dataset are used as training datasets. The visual image data is enhanced by randomly rotating, translating, changing contrast, brightness or saturation. The tactile data is converted into a waveform graph. The visual image and tactile image are resized and converted into tensors and standardized. The AU dataset is split at a ratio of 1:4.

[0027] After initializing the model parameters, the training set is input into the object recognition model based on tactile-guided diffusion adaptive fusion to obtain the recognition results. During the training process, the adaptive dropout algorithm measures the prediction confidence through probability distribution, updates the dropout parameter p according to the confidence and the neuron connection strength, and then uses the Bernoulli distribution to randomly exclude neurons in the dropout layer with probability p; the cross entropy loss is used to calculate the loss between the recognition result and the true label, and the gradient of the parameters in the network is calculated according to the calculated loss, and the network parameters are updated through the optimizer until the training is completed.

[0028] The beneficial effects of the present invention are mainly reflected in:

[0029] 1. The method of the present invention draws on the anisotropic diffusion model and designs a novel fusion strategy to capture and enhance the complementary information between the two modalities of visual information and tactile information. By using the local details and semantic information of the tactile features, key structures such as edges and contours in the visual features can be strengthened, thereby improving the quality of the final fused feature map. Connecting the primary fused features with the original visual features and further fusing them with convolutions not only retains the original visual clues, but also strengthens the cross-modal fusion information carried by the fused features, which not only maintains the integrity of the information, but also enhances the discriminability of the feature representation, ensuring that the model can learn more abstract and discriminative cross-modal features.

[0030] 2. The present invention makes full use of the unique advantages of visual information and tactile information, promotes the effective integration of the two perception modes, ensures the decision-making accuracy of the model when facing samples with high certainty, and enhances its exploration ability and robustness in uncertain environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0032] Figure 1 A schematic diagram of an object recognition model based on tactile-guided diffusion adaptive fusion of the present invention;

[0033] Figure 2 This is a flow chart of the adaptive dropout algorithm;

[0034] Figure 3 An example of classification for the OBJECTFOLDER2.0 dataset 8;

[0035] Figure 4 This is a schematic diagram of the ablation experiment results of the visual-tactile fusion module and the adaptive dropout algorithm ((a) is the test result on AUDataset, (b) is the test result on OBJECTFOLDER2.0). DETAILED DESCRIPTION

[0036] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto:

[0037] Embodiment 1: An object recognition method based on tactile-guided diffusion adaptive fusion, such as Figure 1 shown.

[0038] Step 1: Construct an object recognition model based on tactile-guided diffusion adaptive fusion (referred to as object recognition model), including feature extraction, visual-tactile fusion and classification prediction operations.

[0039] Step 1.1: Feature extraction

[0040] The present invention uses a Resnet network as a feature extraction network, and extracts the visual features and tactile features of an object from the visual image and tactile graph of the object respectively through two parallel resnet18s, and then passes the obtained different visual and tactile features through a fully connected layer with an adaptive dropout algorithm, and then reshapes the visual and tactile features after passing through the fully connected layer to adjust the dimensions and shapes of the tactile features and the visual features to feature maps of the same size (visual features V and tactile features T).

[0041] Step 1.2: Visual-tactile fusion

[0042] The anisotropic diffusion model is used to realize cross-modal feature fusion. The anisotropic diffusion model is approximated in a discrete manner. During the discretization process, the continuous partial differential equation (PDE) is converted into a difference equation. The tactile feature map is used as a guide feature map to determine the diffusion rate. The diffusion rate is multiplied by the feature difference of the visual feature map and a learnable convolution kernel to obtain the primary visual tactile fusion feature. The primary fusion feature map is further fused with the visual feature map through connection and convolution operations to form the final feature map F.

[0043] Step 1.2.1. Use the tactile feature map as the guiding feature map, and calculate the similarity between each pixel on the tactile feature map based on it, and then determine the anisotropic diffusion rate. Subsequently, we combine this diffusion rate with the feature difference in the visual feature map, and weight it through a learnable convolution kernel to obtain the primary visual-tactile fusion feature. This process is similar to the concept of differential convolution, which can introduce tactile information to enhance the expressiveness of visual features while maintaining the original feature details. For the obtained visual feature map and tactile feature map, feature fusion is achieved in a similar way to differential convolution. Specifically, we set a 3′3 local area window N centered on (m,n), calculate the fusion feature on the visual feature V and the tactile feature T, and output the primary fusion feature map primary fusion feature Y. Each component of Y (denoted by Y (m,n) It is expressed by the formula:

[0044] Y (m,n) =∑ (i,j)∈N S T (T (i,j) ,T (m,n) )×D V (V (i,j) ,V (m,n) )×W (i,j) (1)

[0045] The tuple (i, j) enumerates the index of each entry in the local window N, and W is a learnable convolution kernel of size 3′3;

[0046] The first term S in formula (1) T (T (i,j) ,T (m,n) ), is a scalar used to measure the feature T at (i, j) and (m, n) in the tactile feature map (i,j) and feature T (m,n) Similarities between:

[0047] S T (T (i,j) ,T (m,n) )=exp(-|T (i,j) -T (m,n) | 2 / s 2 ) (2)

[0048] Where s is a hyperparameter, which is 2.

[0049] The second term D in formula (1) V (V (i,j) ,V (m,n) )=V (i,j) -V (m,n) is the visual pixel difference term, and calculates the feature V at the visual feature map position (i, j) and (m, n) (i,j) and feature V (m,n) The characteristic difference between .

[0050] This method not only considers the direct matching of the two modal features, but also implicitly includes their differences and synergies in local areas. By processing in this way, the local details and semantic information of the tactile features can be used to strengthen key structures such as edges and contours in the visual features, thereby improving the quality of the final fused feature map. Connecting the primary fused features with the original visual features and further fusing them with convolutions not only retains the original visual cues, but also strengthens the cross-modal fusion information carried by the fused features, which not only maintains the integrity of the information, but also enhances the discriminability of the feature representation, ensuring that the model can learn more abstract and discriminative cross-modal features.

[0051] Step 1.2.2: After obtaining the primary fusion feature map, connect it with the visual feature V. Then use a simple but effective 1′1 convolution to achieve channel-level feature fusion, thereby obtaining the fusion feature F, that is, F=Conv(V,Y).

[0052] Step 1.2.3: The fused feature F passes through a fully connected layer with an adaptive dropout algorithm and another fully connected layer (without an adaptive dropout algorithm) to obtain the object recognition result and confidence. Note that the dropout parameter p is set to 0 in actual use.

[0053] Step 1.3: Adaptive dropout algorithm

[0054] The adaptive dropout algorithm will adjust the confidence λ of the network prediction results during model training. item Dynamically adjust the dropout parameter p. The algorithm flow is as follows Figure 2 As shown in the figure, the core is to use the model's own prediction results to dynamically adjust the Dropout parameter p, so as to determine which neurons should be temporarily excluded according to the current model's "confidence" in the input data. The specific implementation can be divided into the following steps: Calculate the prediction confidence: First, perform a Softmax operation on the output vector obtained after the current input sample passes through the fully connected layer to generate a probability distribution. This probability distribution reflects the model's confidence in the prediction results of each possible category, that is, the prediction confidence λ item . Update the Dropout parameter p: p is updated by combining the prediction confidence λ item The update rule of p is completed by the neuron connection strength:

[0055]

[0056] Among them, λ item Refers to the confidence of the recognition network output, Z i (t) refers to the connection strength of a single neuron in the network, w ij represents the weight between neurons i and j in the tth iteration, π ij It represents the activation state of any neuron connected to neuron i in the network, and its value is set to 0 or 1, where 1 indicates activation and 0 indicates inactivation.

[0057] Next, using the updated p-value, the Bernoulli distribution is used to randomly decide whether to "turn off" (i.e. set its activation value to 0) each neuron in the fully connected layer. Finally, the model continues to perform the normal forward propagation process, but the above steps are repeated at each iteration to dynamically adjust the p-value and randomly discard some neurons accordingly.

[0058] The integrated adaptive dropout algorithm intelligently controls the activation state of neurons during model training and dynamically adjusts the learning process according to the confidence level of each prediction, which not only ensures the decision-making accuracy of the model when facing samples with high certainty, but also enhances its exploration ability and robustness in uncertain environments. This dynamic optimization mechanism effectively suppresses the risk of overfitting, allowing the model to demonstrate excellent generalization performance on a variety of tasks and data sets.

[0059] The overall block diagram of the object recognition model is as follows Figure 1As shown in the figure, the two ResNet18 branches focus on extracting deep and high-quality feature representations from visual and tactile images respectively. The extracted visual feature representation and tactile feature representation are then reshaped through a fully connected layer with an adaptive dropout algorithm to eliminate the differences in spatial structure between the two and ensure that the visual features and tactile features are consistent in dimension. The visual feature V and tactile feature T obtained after the reshape operation are then input into the visual tactile fusion module, and cross-modal fusion is achieved through the anisotropic diffusion model. The diffusion rate is determined by using the tactile feature T as a guide feature map, and the diffusion rate is multiplied by the feature difference of the visual feature V and a learnable convolution kernel to obtain the primary visual tactile fusion feature. Then, the primary fusion feature map is further fused with the visual feature V through connection and convolution operations to form the final fusion feature F. The fusion feature F is input into the fully connected layer to obtain the prediction result and confidence. During the model training process, the adaptive dropout algorithm measures the prediction confidence λ by the probability distribution obtained by performing a softmax operation on the output of the fully connected layer item , the adaptive dropout algorithm updates the dropout parameter p according to the confidence of the prediction result and the strength of the connection with the neuron, and then uses the Bernoulli distribution to exclude some neurons in the abstention layer with probability p. It should be noted that the model does not enable dropout when it is actually used, that is, p is set to 0.

[0060] Step 2: Model training

[0061] Step 2.1: Training and testing set data

[0062] The present invention uses the AU dataset to train and test the model. The AU dataset contains visual, tactile and kinesthetic data, with a total of 63 objects, including objects with ambiguous vision and touch. Each object was photographed from three angles with a camera to generate three visual data. During the data collection process of tactile and kinesthetic modalities, each object was observed five times. Among them, the tactile data involves "feel" and "poke", both of which are collected simultaneously through 5 microphones. The data of the kinesthetic mode includes "closure", "extra", "pressure squeeze" and "unsupported hold". These four types of information are collected by the integrated current acquisition chip, infrared sensor and position sensor inside the robot, and measured three times each time. Therefore, each object forms 50 tactile data and 60 kinesthetic data each time. We only use the visual data and tactile data of the AU dataset.

[0063] The visual images of AUDATASET were augmented to 50 by random rotation, translation, contrast, brightness or saturation changes to obtain visual modal information in various lighting environments. The csv file of tactile data was used to generate a waveform as the tactile input of the object recognition model. The dataset was preprocessed, and the visual and tactile images were adjusted to a fixed size (224, 224) using bilinear interpolation. They were then converted into pytorch tensors and saved as npy files after standardization. The AU dataset was split in a ratio of 1:4, that is, 10 samples of each object were randomly selected as the test set, with a total of 630 samples in the test set, and the remaining 4 / 5, i.e., 5670 visual images and corresponding tactile images were used as training sets. We used the Adam optimizer with an initial learning rate of 1×10 -4 , momentum is 0.9, and cross entropy is used as the loss function. The batch size is set to 8 and trained for 200 epochs, and it is trained on NVIDIA A100 GPU.

[0064] Step 2.2: Training process

[0065] Initialize all parameters of the model. Prepare a training dataset containing visual image data and tactile image data.

[0066] Forward propagation: Each batch of data is sent to two ResNet18 branches to extract visual feature representation and tactile feature representation. Dropout operations are performed on the extracted feature representations to increase the generalization ability of the model. The reshape operation is used to adjust the dimension of the feature so that the visual feature and the tactile feature are consistent in dimension. The reshaped visual feature V and tactile feature T are sent to the visual tactile fusion module for cross-modal fusion through the anisotropic diffusion model. The diffusion rate is determined according to the tactile feature T, and the feature difference of the visual feature V and a learnable convolution kernel are combined to generate the primary visual tactile fusion feature. The primary fusion feature is further fused with the visual feature V to form the final fusion feature F.

[0067] Adaptive dropout: The fused feature F is sent to the fully connected layer with adaptive dropout, and the probability distribution is obtained through softmax operation to measure the prediction confidence. The dropout parameter p is updated according to the confidence and the strength of the neuron connection, and then the Bernoulli distribution is used to exclude some neurons in the dropout layer with probability p. This mechanism can effectively simulate different sub-network structures and enhance the model's adaptability to the diversity of input data. Therefore, the model can not only avoid overfitting in the training phase, but also improve the prediction accuracy of unseen data in the testing phase, thereby significantly improving the overall generalization performance.

[0068] Loss calculation: The feature vector after adaptive dropout processing is sent to the fully connected layer to obtain the final recognition result. Use cross entropy as the loss function to calculate the loss between the recognition result and the true label:

[0069]

[0070] Where N is the number of samples and C is the total number of categories. ic is the true label of the cth category corresponding to the ith sample, usually in the form of one-hot encoding, that is, the position corresponding to the true category is 1, and the other positions are 0. It is the probability that the i-th sample predicted by the model belongs to the c-th category, which is actually the output of the softmax function.

[0071] Back propagation: The gradient of the parameters in the network is calculated according to the calculated loss, and the network parameters are updated through the optimizer (Adam).

[0072] Iteration: Repeat the above steps until all training data are iterated or a predetermined stopping condition (such as the maximum number of iterations) is reached.

[0073] Step 2.3, testing process

[0074] The same forward propagation steps as the training process are performed on the test data, but no dropout operation is required, and no gradient calculation and parameter update are required.

[0075] Use the trained model parameters to extract features and perform fusion. The final recognition result is obtained through the fully connected layer and compared with the true label of the test data to evaluate the model performance (such as accuracy, F-1score indicators).

[0076] Step 3: Use online

[0077] Specific scenarios: For example, in an industrial production line, robots need to identify different types of parts based on visual and tactile information.

[0078] Data collection: Visual data is captured in real time by a camera installed on a robot or other device. Tactile data is captured by a sensor installed at the end of a robotic arm when an object is in contact. For example, a gelsight sensor is used to directly contact an object to capture the object's tactile image, or a microphone is used to capture the sound waveform generated by the movement of an object when it is in contact. These waveforms can be converted into images.

[0079] Data preprocessing: The visual image and tactile image are resized to a fixed size (224, 224) through bilinear interpolation, and then the resized images are converted to pytorch tensors and normalized. The preprocessed visual image data and tactile image data are saved as npy files for model use.

[0080] The object recognition model trained and tested in step 2 is integrated into the robot control system. The preprocessed visual image data and tactile image data are input into the object recognition model. First, each of them is subjected to a Resnet18 network for feature extraction. Then, the extracted visual features and tactile features are respectively passed through a fully connected layer with dropout and a reshape operation. Then, the visual features V and tactile features T obtained after the reshape operation are input into the visual-tactile fusion module to obtain the fused feature F. The fused feature F is then passed through two fully connected layers to obtain the object recognition result and confidence, thereby realizing the automatic part recognition function.

[0081] experiment:

[0082] The effectiveness of the present invention is demonstrated by comparing the proposed method (object recognition model) with three mainstream multimodal fusion methods. The comparison models include MMM

[10] , CRNN-SA

[11] and CRNN-CA

[11] In addition, we also compared with some methods that only use visual images for classification, that is, we compared the proposed method (object recognition model) with 9 mainstream methods on the AU dataset, including EfficientViT [4] 、Swin Transformer V2 [5] 、ConvNeXt [6] 、FocalNet [7] 、EfficientNetV2 [8] and MaxViT [9] , to prove the effectiveness of the method of the present invention. The results are shown in Table 1.

[0083] From the table, we can intuitively see that compared with the comparison methods (including single-modal methods and multi-modal fusion methods), our model achieves the best object recognition performance. The accuracy of our method is 95.23% and the F1-score is 95.29%. Compared with the second best method FocalNet on the AU dataset, the accuracy is improved by about 1.58% and the F1-score is improved by 3.27%. Compared with the second best F1-score method Swin Transformer V2, the accuracy is improved by about 1.74% and the F1-score is improved by about 3.2%. Comparing our model with the single visual modality method, our method performs better, which shows that the effective fusion of visual and tactile data does improve the performance of object recognition compared with the use of information from a single visual modality. Compared with the multi-modal fusion method, our model still performs better. The accuracy of the model is about 3.96% higher than the best performing CRNN-SA, and the F1-SCORE is about 4.68% higher. CRNN-SA uses convolutional neural networks (CNN) to learn spatial representations, recurrent neural networks (RNN) to model temporal relationships, and Transformer's self-attention and cross-attention structures to focus on global and cross-modal information. Although it achieves alignment of visual, tactile, and kinesthetic data, there is limited discussion on the effective integration of cross-modal data. As a result, although the number of parameters is much smaller than that of single-visual modality methods, its accuracy and F1-SCORE are only better than EfficientNetV2, the worst performing single-visual modality method. Although the number of parameters of our proposed method is larger than that of MMM and CRNN-SA, its performance is better than the 6 mainstream visual methods compared.

[0084] Table 1: Comparison results of different methods on the AU dataset (V, T and K represent visual, tactile and kinesthetic modes respectively)

[0085]

[0086]

[0087] 2. Verify the effectiveness of the visual-tactile fusion module and the adaptive dropout algorithm:

[0088] In order to verify the effect of the visual-tactile fusion module in step 1.2 and the adaptive dropout algorithm in step 1.3, we conducted experiments on two datasets by removing the visual-tactile fusion part and disabling the adaptive dropout algorithm during training to show its accuracy performance, as shown in the attached figure. Figure 4As shown in the figure, it can be clearly observed that, compared with the complete model, on the same dataset, the model without the visual-tactile fusion part or the adaptive dropout algorithm has a lower accuracy and F1-score in object recognition, and the model performance is reduced, while the performance of the model with both removed is even more reduced. This shows that the fusion method we proposed effectively integrates visual-tactile information and improves the efficiency of cross-modal features in expressing high-level semantics; the proposed loss mechanism effectively enhances the flexibility and robustness of the model and improves the recognition performance.

[0089] The present invention applies the anisotropic diffusion model to the fusion of visual tactile features. By using the tactile feature map as the guiding feature map and using the local details and texture information of the tactile features to guide the diffusion of the visual features, the semantic information of the tactile features can be effectively integrated into the visual features. It can not only make up for the shortcomings of single modal information and enhance the robustness and representation ability of the features, but also adaptively adjust the diffusion direction and intensity during the fusion process to better adapt to different scenes and object surface characteristics.

[0090] In addition, in order to further optimize the model performance and generalization ability, an adaptive dropout algorithm is introduced in the fully connected layer before and after the feature fusion q. Traditional dropout technology prevents overfitting by randomly shutting down some neurons in the neural network during model training, while our adaptive dropout algorithm mechanism goes a step further. It does not only rely on randomness, but dynamically adjusts the dropout ratio according to the prediction confidence of the model output, that is, it determines the activation degree of neurons in the fully connected layer according to the certainty of the prediction.

[0091] Finally, it should be noted that the above examples are only some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there are many variations. All variations that can be directly derived or associated with the content disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

[0092] References:

[0093] [1]Li Y, Zhu JY, Tedrake R, et al. Connecting touch and vision via cross-modal prediction[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and PatternRecognition.2019:10609-10618.

[0094] [2]Zhang T,Cong Y,Sun G,et al.Visual–tactile fused graph learning forobject clustering[J].IEEE Transactions on Cybernetics,2021,52(11):12275-12289.

[0095] [3]Liu H,Wang F,Sun F,et al.Active visual-tactile cross-modalmatching[J].IEEETransactions on Cognitive and Developmental Systems,2018,11(2):176-187.

[0096] [4]Cai H,Li J,Hu M,et al.Efficientvit:Lightweight multi-scaleattention for high-resolutiondense prediction[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2023:17302-17313.

[0097] [5]Liu Z,Hu H,Lin Y,et al.Swin transformer v2:Scaling up capacityandresolution[C] / / Proceedings of the IEEE / CVF conference on computer visionand pattern recognition.2022:12009-12019.

[0098] [6]Liu Z,Mao H,Wu C Y,et al.A convnet for the 2020s[C] / / Proceedingsof the IEEE / CVFconference on computer vision and pattern recognition.2022:11976-11986.

[0099] [7]Yang J,Li C,Dai X,et al.Focal modulation networks[J].Advances inNeural InformationProcessing Systems,2022,35:4203-4217.

[0100] [8]Tan M,Le Q.Efficientnetv2:Smaller models and faster training[C] / / Internationalconference on machine learning.PMLR,2021:10096-10106.

[0101] [9]Tu Z,Talebi H,Zhang H,et al.Maxvit:Multi-axis vision transformer[C] / / Europeanconference on computer vision.Cham:Springer Nature Switzerland,2022:459-479.

[0102]

[10] Zhang P,Zhou M,Shan D,et al.Object description using visual andtactile data[J].IEEEAccess,2022,10:54525-54536.

[0103]

[11] Zhou X,Lan S,Wang W,et al.Visual-Haptic-Kinesthetic ObjectRecognition withMultimodal Transformer[C] / / International Conference onArtificial Neural Networks.Cham:Springer Nature Switzerland,2023:233-245。

Claims

1. An object recognition method based on tactile-guided diffusion adaptive fusion, characterized in that: It includes collecting visual data and tactile data, adjusting the image size, converting it into a tensor and performing normalization processing, and using it as the input of an offline trained object recognition model based on tactile-guided diffusion adaptive fusion to obtain object recognition results and confidence. The object recognition model based on tactile-guided diffusion adaptive fusion includes a feature extraction network and a visual-tactile fusion module. The standardized visual data and tactile data are each passed through a feature extraction network, and then the extracted visual features and tactile features are each passed through a fully connected layer with an adaptive dropout algorithm and a reshape operation. The visual features V and tactile features T obtained after the reshape operation are input into the visual-tactile fusion module to obtain the fusion feature F. The fused feature F then passes through a fully connected layer with an adaptive dropout algorithm and a fully connected layer.

2. The object recognition method based on tactile-guided diffusion adaptive fusion according to claim 1, characterized in that: The feature extraction network is a Resnet18 network.

3. The object recognition method based on tactile-guided diffusion adaptive fusion according to claim 2, characterized in that: The visual-tactile fusion module achieves cross-modal fusion through an anisotropic diffusion model: (1) The tactile feature T is used as the guiding feature map to determine the diffusion rate. The diffusion rate is multiplied by the feature difference of the visual feature V and a learnable convolution kernel to obtain the primary visual-tactile fusion feature: Y (m,n) =∑ (i,j)∈N S T (T (i,j) ,T (m,n) )·D V (V (i,j) ,V (m,n) )·W (i,j) (1) Among them, the tuple (i, j) enumerates the index of each entry in the 3′3 local area window N centered at (m, n), W is a learnable convolution kernel with a size of 3′3; S T (T (i,j) ,T (m,n) )=exp(-|T (i,j) -T (m,n) | 2 / σ 2 ) (2) D v (V (i,j) ,V (m,n) )=V (i,j) -V (m,n) (3) Where σ is a hyperparameter, set to 2; T (i,j) and T (m,n) Respectively represent the features at (i, j) and (m, n) of the tactile feature map, V (i,j) and V (m,n) Represent the features at positions (i, j) and (m, n) in the visual feature map respectively.

4. The object recognition method based on tactile-guided diffusion adaptive fusion according to claim 3 is characterized in that: The adaptive dropout algorithm is: (1) After the input features pass through the fully connected layer, a Softmax operation is performed to generate the prediction confidence λ item ; (2) Update the Dropout parameter p: Among them, Z i (t) refers to the connection strength of a single neuron in the network, w ij represents the weight between neurons i and j in the tth iteration, π ij represents the activation state of any neuron connected to neuron i in the network; (3) Using the updated p-value, randomly decide whether to "close" each neuron in the fully connected layer according to the Bernoulli distribution; (4) Dropout is not enabled during testing and online use, and p is set to 0.

5. The object recognition method based on tactile-guided diffusion adaptive fusion according to claim 4, characterized in that: The training process of the object recognition model based on tactile-guided diffusion adaptive fusion is as follows: The visual image data and tactile data in the AU dataset are used as training datasets. The visual image data is enhanced by randomly rotating, translating, changing contrast, brightness or saturation. The tactile data is converted into a waveform graph. The visual image and tactile image are resized and converted into tensors and standardized. The AU dataset is split at a ratio of 1:

4. After initializing the model parameters, the training set is input into the object recognition model based on tactile-guided diffusion adaptive fusion to obtain the recognition results. During the training process, the adaptive dropout algorithm measures the prediction confidence through probability distribution, updates the dropout parameter p according to the confidence and the neuron connection strength, and then uses the Bernoulli distribution to randomly exclude neurons in the dropout layer with probability p; the cross entropy loss is used to calculate the loss between the recognition result and the true label, and the gradient of the parameters in the network is calculated according to the calculated loss, and the network parameters are updated through the optimizer until the training is completed.

Images