Object identification method based on bimodal dual-channel attention

Abstract

The invention relates to the technical field of article identification and classification and machine vision processing, and discloses a bimodal dual-channel attention-based article identification method, which comprises the following steps of: preprocessing collected visual data and standardizing tactile data to serve as input of a visual and tactile fusion network; the preprocessed visual data and the standardized tactile data respectively pass through a feature extraction network to obtain a visual feature map and a tactile feature map, the visual feature map is subjected to visual channel attention extraction, and the tactile feature map is subjected to tactile channel attention extraction; and carrying out space channel attention fusion on the extracted features to obtain an object recognition result. The visual and tactile fusion network adopts a two-stage offline training method. The multi-modal feature extraction and fusion strategy provided by the invention not only strengthens the processing of the tactile feature, but also can deeply interact with the visual feature, realizes the complementation and integration of information, fully utilizes the visual and tactile information, and improves the recognition precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of object recognition classification and machine vision processing, and in particular relates to an object recognition method based on dual-modal dual-channel attention. Background Art

[0002] Object recognition technology initially relied on traditional image processing methods, such as edge detection and feature matching, which are easily affected by factors such as lighting, posture, and occlusion, limiting the accuracy of recognition. With the development of deep learning technology, especially the application of convolutional neural networks (CNNs), object recognition has made significant progress. However, when processing complex scenes, CNNs still have difficulty in effectively dealing with background interference and subtle differences.

[0003] To solve these problems, the attention mechanism was introduced, which imitates the way human visual attention is allocated and automatically focuses on the key feature areas in the image, thereby improving the robustness and efficiency of the recognition system. Nevertheless, visual recognition still has limitations in some cases, especially when dealing with local details of objects and low-density visual areas.

[0004] Tactile sensors can capture information such as the texture and roughness of an object, which is difficult for the visual system to obtain. Tactile learning can provide a unique perspective on the properties of objects, especially when visual similarities cannot be distinguished, and effective classification can be achieved through the differences in tactile features. Therefore, tactile information has significant application value in object shape reconstruction, category identification, etc.

[0005] Given the advantages and disadvantages of visual and tactile information, visual-tactile fusion technology has become a new direction in the field of object recognition. This technology aims to integrate visual and tactile information to enhance the accuracy and robustness of object recognition through complementarity. Specifically, visual information provides macroscopic contours and color attributes, while tactile information supplements details such as the material, texture, and shape of the object.

[0006] In order to achieve visual-tactile fusion, the present invention proposes a visual-tactile fusion network based on dual-modal dual-channel attention. The network uses the high sensitivity of the tactile channel and combines the channel attention mechanism to accurately capture and strengthen the key information in the tactile features. At the same time, by processing the grayscale covariance of the tactile features, the spatial learning rate expression of the visual features is enhanced, which effectively makes up for the shortcomings of the visual attention mechanism when processing sparse areas. In this way, the network can better recognize and understand objects, especially in complex environments. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide an object recognition method based on dual-modal dual-channel attention, which is used to effectively improve the accuracy of object recognition by fusing visual and tactile features.

[0008] In order to solve the above technical problems, the present invention provides an object recognition method based on dual-modal dual-channel attention, comprising: collecting visual data and tactile data of an object, preprocessing the visual data and standardizing the tactile data, and using them as inputs of a visual-tactile fusion network trained offline in two stages;

[0009] The visual tactile fusion network includes a feature extraction network. The pre-processed visual data and the standardized tactile data are respectively processed through the feature extraction network to obtain a visual feature map. and tactile feature maps Then the visual feature map Perform visual channel attention extraction and tactile feature map Perform tactile channel attention extraction and then perform spatial channel attention fusion to obtain object recognition results.

[0010] As an improvement of the object recognition method based on dual-modal dual-channel attention of the present invention:

[0011] The feature extraction network is an AlexNet network.

[0012] As a further improvement of the object recognition method based on dual-modal dual-channel attention of the present invention:

[0013] The specific process of visual channel attention extraction is:

[0014] (1) The visual feature map Group and decompose into two parallel eigenvectors v c and x c ;

[0015] (2) Global average pooling operation:

[0016]

[0017] Among them, v c represents the input features of the channel along the horizontal dimension at height H, C represents the number of input channels, and H and W represent the spatial learning rate of the input features;

[0018]

[0019] Among them, x c Represents the input features along the vertical dimension at width W;

[0020] (3) After the grayscale value of each channel of the preprocessed visual data is normalized, the grayscale feature map is obtained through the GLCM module, and then the visual feature map is obtained through the AlexNet network.

[0021] The visual matrix feature map With the visual feature map Concatenate and pass through the convolution layer to obtain the channel descriptor V g ￠ lcm ;

[0022] (4) Encode the feature vector and feature encoding vector The concatenated feature vectors are merged along the spatial dimension, and then a set of shared convolution kernels are used to perform convolution operations on the concatenated feature vectors; then, they are split into two independent feature vectors along the spatial direction and divided into two parallel branches, which are fed into the nonlinear Sigmoid function to generate adaptive weight representation; at the same time, the channel descriptor V g ￠ lcm are also synchronously fed to the same Sigmoid function to generate another set of adaptive weight representations; the three sets of parallel adaptive weight representations are used to perform Re-weight operations on the original feature map, and finally the fused visual channel attention feature V is obtained. fused .

[0023] As a further improvement of the object recognition method based on dual-modal dual-channel attention of the present invention:

[0024] The specific process of tactile channel attention extraction is:

[0025] (1) Average pooling and maximum pooling

[0026] The tactile feature map After feature grouping, the maximum pooling layer and average pooling layer operations are performed separately:

[0027]

[0028] Where T(i,j,c) represents the value of the input tactile feature map at position (i,j) and channel c, O max (c) is the maximum pooling output of the cth channel;

[0029]

[0030] Among them, O avg (c) is the average pooling output of the cth channel;

[0031] (2) O max (c) and O avg (c) Pass through a fully connected layer with shared parameters and perform matrix addition:

[0032] Tfused =W×(O max (c)+O avg (c))+b (9)

[0033] Among them, W and b are the weight matrix and bias vector of the fully connected layer, respectively, T fused is the fused tactile channel attention feature vector.

[0034] As a further improvement of the object recognition method based on dual-modal dual-channel attention of the present invention:

[0035] The specific process of the spatial channel attention fusion is:

[0036] The visual channel attention feature V fused and the tactile channel attention feature T fused After normalization, the visual channel attention feature V￠ and the tactile channel attention feature T￠ are obtained;

[0037] The visual channel attention feature V￠ is encoded by a two-dimensional global flat pooling of visual global spatial information:

[0038]

[0039] in,

[0040] For feature V agp,bc The natural nonlinear function Softmax of the two-dimensional Gaussian mapping is used for global output to obtain V softmax , and then perform matrix dot product with the tactile channel attention T￠:

[0041] Z TV =T￠×V softmax (11)

[0042] The tactile channel attention feature T￠ is reweighted and normalized, encoded by two-dimensional global average pooling, and then matrix multiplied with the visual channel attention feature V￠:

[0043] Z VT =V￠×T softmax (12)

[0044] Among them, T softmax Represents the features of the tactile channel attention feature T￠ after global average pooling and softmax operation;

[0045] Next, the visual space descriptor Z that incorporates the tactile channel information is VT and the tactile space descriptor Z that integrates visual channel information TVPerform the addition operation, and then apply the nonlinear Sigmoid function to generate the spatial channel weight W; the original visual feature map after the spatial channel weight W is used to group the features and tactile feature maps Perform feature modulation, realize feature reweighting through element-level multiplication operation, obtain the adjusted visual feature V￠￠ and tactile feature T￠￠, and perform matrix addition of the visual feature V￠￠ and the tactile feature T￠￠ to obtain the fusion feature F fusion ; Fusion feature F fusion After passing through the fully connected layer, the predicted classification results are output.

[0046] As a further improvement of the object recognition method based on dual-modal dual-channel attention of the present invention:

[0047] The specific process of the two-stage offline training is as follows:

[0048] (1) Offline training on the OBJECTFOLDER2.0 dataset:

[0049] The visual data samples in the dataset OBJECTFOLDER2.0 were subjected to data enhancement operations such as random cropping, image scaling, color space perturbation, and horizontal flipping. The tactile data samples were converted into RGB images through rendering technology. Then they were divided into the first-stage training set and the first-stage test set. The visual-tactile fusion network was trained and tested in the first stage. The cross entropy loss function and Adam optimizer were used to update the parameters in the network through the back-propagation algorithm to obtain the first-stage optimal network weights of the visual-tactile fusion network.

[0050] (2) Offline training on the dataset AUDataset

[0051] The visual image of each sample in the dataset AUDataset is expanded according to different lighting conditions, and then divided into the second-stage training set and the second-stage test set. The visual-tactile fusion network trained in the first stage is trained and tested in the second stage. The cross entropy loss function and Adam optimizer are used to update the parameters in the network through the back-propagation algorithm to obtain the second-stage optimal network weights of the visual-tactile fusion network.

[0052] The visual-tactile fusion network trained in the second stage is used as a network that can be used online.

[0053] The beneficial effects of the present invention are mainly reflected in:

[0054] 1. This paper proposes an effective modal fusion: the proposed multimodal feature fusion strategy, especially the tactile attention module, not only strengthens the processing of tactile features, but also deeply interacts with visual features to achieve information complementarity and integration. This enables the model to make full use of visual and tactile information and improve recognition accuracy when processing complex environmental perception tasks;

[0055] 2. The present invention can make up for the deficiency of the visual attention mechanism in the perception of local details by introducing tactile global features and grayscale covariance matrix analysis, especially in areas with low feature intensity, thus improving the generalization ability and detail capture ability of the model;

[0056] 3. Enhanced robustness and generalization: Through the integration of multiple mechanisms, including feature standardization, dynamic allocation of attention weights, and deep analysis of tactile images, the present invention can more stably process diverse inputs and improve robustness and generalization in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0058] Figure 1 It is a schematic diagram of the structure of the dual-modal dual-channel visual-tactile fusion network (DMDA-NET) of the present invention;

[0059] Figure 2 Schematic diagram of the visual channel attention extraction process;

[0060] Figure 3 Schematic diagram of the attention extraction process of the tactile channel;

[0061] Figure 4 Schematic diagram of the spatial channel attention fusion process for visual and tactile features of the present invention. DETAILED DESCRIPTION

[0062] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto:

[0063] Embodiment 1,

[0064] The recognition of unknown features requires the fusion of visual and tactile perception data to achieve a greater degree of object recognition. The present invention proposes a method for object recognition based on a dual-modal dual-channel attention visual tactile fusion network (DMDA-NET).

[0065] Step 1: Construct a visual-tactile fusion network (DMDA-NET)

[0066] Dual-modal dual-channel attention-based visual-tactile fusion network (DMDA-NET) Figure 1 As shown, specifically:

[0067] Step 1.1: Feature extraction

[0068] A pre-trained AlexNet classic convolutional neural network architecture is used to perform parallel feature extraction on the input visual and tactile raw images, respectively, to extract meaningful features from the data of the two modalities and obtain a visual feature map. and tactile feature maps Among them, the pre-trained AlexNet network uses alexnet=models.alexnet(pretrained=True) to load the pre-trained weights 【1】 .

[0069] Step 1.2: Visual channel attention extraction

[0070] The schematic diagram of the visual channel attention extraction process is as follows Figure 2 shown.

[0071] (1) Channel grouping

[0072] After feature extraction, the visual features and tactile features are further grouped on the channel learning rate, the so-called channel grouping, in order to better organize and process the feature information.

[0073] First, the input visual feature map Grouping, visual feature map after grouping Decomposed into two parallel 1D feature vectors to establish cross-channel dependencies and spatial position information, where C represents the number of input channels, and H and W represent the spatial learning rate of the input features;

[0074] (2) Global Average Pooling

[0075] One of the two parallel 1D feature vectors obtained in step 1.1 is subjected to 1D global average pooling along the horizontal learning rate direction to capture the global information in the horizontal learning rate direction. The 1D global average pooling operation that encodes the global information along the horizontal learning rate direction of C at height H can be expressed as:

[0076]

[0077] where v cRepresents the input features along the horizontal dimension at height H, and outputs the feature encoding vector in the H direction (size is C′1′W). This route uses the average pooling of position information in the vertical learning rate direction to capture long-range interactions in space, obtains global dependencies in the vertical direction, and retains precise position information along the horizontal learning rate direction;

[0078] Similarly, the other of the two parallel 1D feature vectors obtained in step 1.1 is subjected to 1D global average pooling along the direction perpendicular to the learning rate to capture the global information in the direction perpendicular to the learning rate. At width W, the pooling output in C can be expressed as:

[0079]

[0080] where x c Represents the input features along the vertical dimension at width W, and outputs the feature encoding vector in the W direction (size is C′1′H). Similarly, the global dependency line in the horizontal direction retains the precise position information along the vertical learning rate direction;

[0081] (3) GLCM module

[0082] First, for the input visual original image The grayscale value of each channel of each sample is normalized to remove the influence of brightness offset and ensure that the covariance matrix reflects the relative changes between pixels. The normalization process is as follows:

[0083]

[0084] in, represents the normalized tensor, μ bc is the average gray value of channel c in sample b, and the calculation formula is:

[0085]

[0086] Here V bcij Represents the grayscale value of channel c at position (i, j) of the normalized image tensor sample b.

[0087] Next, the covariance between each channel of each sample and other channels is calculated based on the standardized image tensor. This step is actually calculating the statistics of the gray-level co-occurrence matrix (GLCM). Specifically, for each sample b and each channel pair (c, k), an element of the covariance matrix Cov bk It can be expressed as:

[0088]

[0089] By directly using the standardized The covariance is calculated by using the tensor of , which implicitly handles the elimination of the mean value. The resulting covariance matrix tensor It is processed through a convolution layer to further extract features. The kernel size of the convolution layer is 3′3, the amplitude is 1, and the padding is 1 to keep the size of the output feature map the same as the input. The convolution operation can be expressed as:

[0090]

[0091] Among them, V glcm,bcij Represents the grayscale feature map, that is, the value of the output tensor at channel c of sample b at position (i, j);

[0092] Subsequently, this set of grayscale feature maps is input into the pre-trained AlexNet network and feature grouped to further refine high-level abstract features and obtain visual feature maps.

[0093] Next, from the grouped visual matrix feature map Visual feature map obtained by grouping in step 1 In order to reduce the feature learning rate while keeping the key information intact, the composite feature is reduced in dimension through a 1′1 convolutional layer and a channel descriptor V is obtained that combines texture details and spatial context information. g ￠ lcm ;

[0094] (4) The feature encoding vector in the H direction and the feature encoding vector in the W direction Merge along the spatial learning rate. Then use a set of shared convolution kernels to perform convolution operations on the concatenated feature vectors in order to fuse the information from different channels.

[0095] Subsequently, the processed features are split again along the spatial direction into two independent feature vectors with sizes C / / G′1′W and C / / G′H′1 respectively. The two feature vectors are divided into two parallel branches, which are then fed into the nonlinear Sigmoid function to generate an adaptive weight representation.

[0096] At the same time, the channel descriptor V g ￠ lcm It is also synchronously fed into the same Sigmoid function to generate another set of adaptive weight representations.

[0097] The three sets of parallel adaptive weight representations for aggregating the original intermediate feature maps are used to perform Re-weight operations on the original feature maps, and the original feature maps are adaptively adjusted at the channel level to achieve effective regulation of network attention and optimize the accuracy of feature expression. Finally, the fused visual channel attention feature V is obtained. fused .

[0098] Here, the adaptive weights adjusted by Sigmoid are regarded as channel weights after feature selection, and then weighted to the previous features channel by channel through multiplication to complete the recalibration of the original features on the channel learning rate;

[0099] Step 1.3: Tactile channel attention extraction

[0100] Since tactile features are more sensitive to local features and touch itself has multi-channel characteristics, tactile images will be processed in a more direct and efficient way. The schematic diagram of the tactile channel attention extraction process is shown in Figure 3 shown.

[0101] (1) Average pooling and maximum pooling

[0102] The tactile feature map obtained in step 1.1 Perform feature grouping and visual feature map after grouping It is fed into two parallel paths for further abstraction and refinement through the maximum pooling layer and the average pooling layer respectively to capture and strengthen information at different levels.

[0103] Specifically, the tactile feature vector of one of the branches is adaptively max-pooled to focus on the key local changes in the tactile signal, and its mathematical expression is:

[0104]

[0105] Where T(i,j,c) represents the value of the input tactile feature map at position (i,j) and channel c, and O max (c) is the maximum pooling output of the cth channel;

[0106] The other branch is dedicated to maintaining the global consistency of tactile features through the adaptive average pooling layer, and its mathematical expression is:

[0107]

[0108] Here, H and W are the height and width of the input tactile feature map, respectively. avg (c) is the average pooling output of the c-th channel.

[0109] (2) Obtain the two parallel channel descriptors Omax (c) and O avg (c) Pass through a fully connected layer with shared parameters and perform matrix addition. Mathematically, this operation can be formalized as:

[0110] T fused =W×(O max (c)+O avg (c))+b (9)

[0111] Among them, W and b are the weight matrix and bias vector of the fully connected layer, respectively, T fused is the fused tactile channel attention feature vector.

[0112] After that, the tactile channel attention feature T fused The channel attention weights generated by the Sigmoid activation function are dynamically allocated according to the importance of the tactile signal and are consistent with the tactile feature map. After multiplication, the tactile feature map Perform channel-level adaptive adjustments.

[0113] Tactile channel attention fusion uses shared parameters and linear combination strategies to more accurately capture the subtle differences in tactile signals, achieving in-depth understanding and efficient processing of tactile signals. The Sigmoid function is also used to strengthen the expression of important tactile information and weaken the unimportant parts, ultimately enabling the model to more effectively highlight the key information in tactile features when processing tactile data, while suppressing irrelevant or redundant information;

[0114] Step 1.4: Fusion of visual and tactile features using spatial channel attention

[0115] The schematic diagram of the spatial channel attention fusion process of visual and tactile features is as follows: Figure 4 As shown, specifically:

[0116] The visual channel attention feature V obtained in step 1.2 fused And the tactile channel attention feature T obtained in step 1.3 fused After normalization, the standardized visual channel attention feature V￠ and tactile channel attention feature T￠ are obtained.

[0117] The standardized visual channel attention feature V￠ is encoded by a two-dimensional global flat pooling visual global space information, and its mathematical representation is:

[0118]

[0119] in,

[0120] For efficient calculation, the feature V after global average pooling isagp,bc The natural nonlinear function Softmax of the two-dimensional Gaussian mapping is used for global output to obtain V softmax , and then perform matrix dot product with the standardized tactile channel attention T￠ to accurately capture the pairwise correlation of visual-tactile features at the pixel level. A tactile space descriptor that integrates visual channel information is obtained, which can be mathematically expressed as:

[0121] Z TV =T￠×V softmax (11)

[0122] Similarly, using a similar method, the tactile channel attention feature T￠ is reweighted and standardized, and then its spatial information is encoded through two-dimensional global average pooling, and finally matrix multiplication is performed with the standardized visual channel attention feature V￠ to obtain the visual spatial descriptor Z that integrates the tactile channel information VT :

[0123] Z VT =V￠×T softmax (12)

[0124] Among them, T softmax Represents the features of the tactile channel attention feature T￠ after global average pooling and softmax operation.

[0125] Since there is an intrinsic correlation between channels and spatial positions, the present invention adopts the above-mentioned novel multimodal feature fusion strategy to achieve deep fusion of visual and tactile features at the channel level and information complementarity between channel features through an efficient spatial interactive aggregation method.

[0126] Step 1.5: To further deepen the interaction between visual and tactile features, the visual space descriptor Z that incorporates the tactile channel information is constructed. VT and the tactile space descriptor Z that integrates visual channel information TV The addition operation is performed to achieve deep synergy between the two modalities in the spatial domain. Then the nonlinear Sigmoid function is applied to generate the spatial channel weights W, which intuitively reflect the relative importance of visual and tactile features at a specific spatial position.

[0127] Step 1.6, global fusion

[0128] The original visual feature map after the spatial channel weights W derived by the spatial attention mechanism are grouped and tactile feature maps Feature modulation is performed, and feature reweighting is achieved through element-level multiplication operations to obtain the adjusted visual features V￠￠ and tactile features T￠￠. Finally, V￠￠ and T￠￠ are fused by matrix addition to obtain the fusion feature F that integrates visual and tactile information. fusion .

[0129] This process not only effectively improves the expressiveness of features, but also highlights key visual information and suppresses redundant or irrelevant background interference. Thus, while maintaining the details of tactile perception, it improves the expressiveness and recognition of visual features, providing a richer and more comprehensive representation basis for subsequent classification.

[0130] Step 1.7, finally, fusion feature F fusion After the dropout operation to suppress overfitting, the extracted features are passed to the fully connected layer, which is responsible for mapping the feature space to the classification space, thereby outputting the final predicted classification results of the tactile image.

[0131] In summary, the visual-tactile fusion network based on dual-modal dual-channel attention:

[0132] (1) It strengthens the deep attention feature extraction and processing of visual and tactile information, enhances the overall perception ability of the visual attention mechanism and the fine recognition ability of the tactile perception system for the material and texture of objects.

[0133] (2) By introducing global visual features and grayscale covariance matrix analysis, the insufficiency of the visual attention mechanism in local detail perception can be compensated, especially in areas with low feature intensity, thus improving the model’s generalization ability and detail capture ability;

[0134] (3) An interactive multimodal feature fusion strategy is proposed, which not only strengthens the processing of tactile features, but also deeply interacts with visual features to achieve information complementarity and integration. This enables the model to make full use of visual and tactile information and improve recognition accuracy when dealing with complex environmental perception tasks.

[0135] Step 2: Training

[0136] The present invention adopts the data set OBJECTFOLDER2.0 and the data set AUDataset to perform secondary offline training on the visual tactile fusion network (DMDA-NET) respectively, the purpose is to enhance the adaptability and generalization ability of the network on different data sets. Through this staged training, the network can improve its understanding and processing ability of different tactile data features, thereby improving the overall application effect. First, the data set OBJECTFOLDER2.0 is trained offline to optimize its performance and obtain the optimal network weight, and then the visual tactile fusion network (DMDA-NET) with the optimal weight trained on OBJECTFOLDER2.0 is trained offline again using the data set AUDataset, and finally the training visual tactile fusion network (DMDA-NET) with the optimal weight trained in AUDataset is used as a network that can be used online. Since tactile data has the characteristics of various forms, the tactile data used in the two data sets of the present invention are inconsistent, in order to face different forms of tactile data, still maintain good recognition accuracy in online use.

[0137] 2.1. First stage offline training

[0138] OBJECTFOLDER2.0 is a large-scale multisensory dataset containing comprehensive visual, audio, and tactile information of 1,000 objects. Each object is virtualized through an implicit neural representation, fully reflecting its texture, material, and 3D structure.

[0139] The characteristics of the visual dataset are that all images are taken in an environment without background interference, and 34 images are obtained from multiple angles to increase the diversity of perspectives. In the end, the training set has 81 objects of each category (27 original images are data enhanced), and the test set retains 7 original images to ensure the independence of the evaluation.

[0140] The tactile data uses the GeSight sensor to capture the surface details of the object, record the contact point information and convert the tactile information into RGB images through rendering technology. The division method of the training set and the test set is consistent with the visual data set.

[0141] In order to improve the generalization ability of the model, the samples of visual data in the dataset OBJECTFOLDER2.0 were subjected to various enhancement techniques such as random cropping, image scaling, color space perturbation, and horizontal flipping. These operations enhanced the diversity of the dataset, improved the recognition stability of the model, and improved the ability to learn object features. The enhanced dataset OBJECTFOLDER2.0 was randomly split into an 80:20 ratio as the training set and test set for the first stage of training.

[0142] In the first phase of offline training, the batch size is set to 16, which ensures that the model can access a rich and representative sample set in each training iteration, effectively promoting the model's ability to recognize and understand complex information, and laying a solid foundation for improving its generalization performance on unknown data. The first phase of offline training of the present invention selects the cross entropy loss function as the optimization target, and uses the Adam optimization algorithm to adjust the model parameters. The training plan is set to 300 cycles, i.e., epochs, and the starting learning rate is set to 0.000001.

[0143] After initializing the model parameters, the training set of the first stage is first input into the visual-tactile fusion network (DMDA-NET). During the training process, the feature vector after dropout processing will be passed to the fully connected layer to generate the final recognition result. Next, the loss between the recognition result and the true label is calculated, and the cross entropy loss function is used to quantify the accuracy of the model's prediction results. Based on the calculated loss, the gradient is calculated layer by layer from the output layer to the input layer through the back propagation algorithm. The parameters in the network are updated and the optimizer (such as Adam) is used for gradient descent. The forward propagation and back propagation steps are repeated until one of the following stop conditions is met:

[0144] 1. The predetermined maximum number of iterations is reached.

[0145] 2. The loss value no longer decreases significantly within multiple consecutive epochs.

[0146] After completing the first stage of training, the trained model parameters are used on the test set to extract features and fuse visual and tactile data. The previously trained model and its parameters need to be loaded. Next, in the test setup phase, it is necessary to define evaluation indicators such as accuracy, precision, recall, F1 score or mean square error, and ensure that the test environment is consistent with the training environment to avoid performance impact due to environmental differences. Then, the final recognition result is obtained through the fully connected layer, the predicted output is obtained, and it is compared with the true label of the test data to obtain the desired evaluation model performance indicator. Throughout the process, it is necessary to ensure that the test data set is large enough and representative, avoid using training data or validation data as test data, and prevent overfitting or evaluation bias.

[0147] Step 2.2: Second stage offline training

[0148] The first stage of training is to obtain the optimal weights obtained by the network when processing tactile data as images. In the second stage of training, the present invention will use the weights obtained in the first stage to process the waveform images obtained by the Gelsight tactile sensor, thereby obtaining the optimal weights with good accuracy when facing different forms of visual and tactile data for online use.

[0149] The present invention uses the AUDataset data set when training the visual tactile fusion network (DMDA-NET) in the second stage 【2】 The dataset combines visual, tactile and kinesthetic information and contains 63 object samples, covering visual textures and touches from clear to blurred. Each object is photographed synchronously from multiple angles by three fixed high-definition cameras to ensure the integrity and diversity of the visual data.

[0150] At the same time, tactile data is collected by Gelsight sensors, and five rounds of repeated observations are carried out using refined measures to improve the reliability of the data. Tactile collection focuses on the two interaction modes of "feeling" and "probing". The kinesthetic modality is carried out around the four actions of "encircling grasping", "extra support", "pressure squeezing" and "unsupported suspension". The equipment is used to record the mechanical response and spatial posture changes in dynamic interaction, ensuring three measurements / experiments to obtain more comprehensive data.

[0151] In order to improve the generalization ability of the model, the number of visual images in the training set was expanded to 50 by changing the illumination angle, covering a variety of lighting conditions. In the data preprocessing stage, fine division was implemented, and the training and test sets were allocated in an 80-20 ratio to ensure reasonable sample distribution and information security. Similarly, in the second stage of training, the cross entropy loss function was selected as the optimization target, and the Adam optimization algorithm was used to adjust the model parameters. The training plan was set to 60 cycles, i.e. epochs, and the starting learning rate was set to 0.000005. All parameters of the model were initialized.

[0152] Each batch of data from the second stage training set is sent to the visual tactile fusion network (DMDA-NET) to obtain the final fusion feature map F fusion . The feature vector after adaptive dropout processing is sent to the fully connected layer to obtain the final recognition result. The loss between the recognition result and the true label is calculated (using cross entropy loss). Then the gradient of the parameters in the network is calculated according to the calculated loss, and the network parameters are updated through the optimizer (Adam) for back propagation. Repeat the above steps until all iterations of training data are completed, or the predetermined stopping condition (such as the maximum number of iterations) is reached.

[0153] The same forward propagation steps as the training process are performed on the second-stage test data, but no dropout operation is required, and no gradient calculation and parameter update are required. Use the trained second-stage model parameters to extract features and fuse them. The final recognition result is obtained through the fully connected layer and compared with the true label of the test data to evaluate the model performance (such as accuracy, F-1score indicators).

[0154] 3. Online use

[0155] In the automation application of industrial production lines, robots can efficiently and accurately identify and classify different types of items by combining visual and tactile sensors, thereby improving production efficiency and product quality. The specific implementation steps are as follows:

[0156] Data collection

[0157] On the production line, high-precision cameras are installed to collect visual data in real time. These cameras can work under various lighting conditions to ensure the capture of clear images, and have image processing capabilities to overcome environmental interference. To enhance reliability, the system adopts a multi-camera layout to cover multiple key angles of the production line.

[0158] In addition, the Gelsight tactile sensor installed at the end of the robotic arm captures the tactile data of the object in real time. The data obtained is in the same form as the tactile data type when using AUDataset for training in the second phase, both of which are waveforms. The tactile sensor has high sensitivity and can recognize slight touch changes, thereby improving the accuracy of object recognition.

[0159] 3.2. Data preprocessing and standardization

[0160] After data collection, the visual data is first pre-processed, including image denoising, background removal, and standardization, to ensure the accuracy and consistency of the image information. At the same time, the tactile data also needs to be standardized, including normalizing the perception data of different force levels.

[0161] After preprocessing, visual and tactile data will be integrated into a unified data format, ensuring that both types of information can be effectively processed in the same model, thus providing support for comprehensive analysis.

[0162] 3.3 The preprocessed visual data and tactile data are used as the visual tactile fusion network (DMDA-NET) trained offline in the second stage of step 2.2. The specific process is as follows:

[0163] (1) Load the trained DMDA-NET model and set the model to evaluation mode, thereby turning off the dropout layer and disabling the training behavior in Batch Normalization; turn off gradient calculation, which can reduce memory consumption and increase inference speed;

[0164] (2) Loading preprocessed visual data and tactile data;

[0165] (3) The preprocessed visual data and the standardized tactile data are respectively subjected to feature extraction to obtain the network visual feature map and tactile feature maps

[0166] (4) Visual feature map Perform visual channel attention extraction to obtain the visual channel attention feature V fused ; Tactile feature map Perform tactile channel attention extraction to obtain tactile channel attention feature T fused ;

[0167] (5) The visual channel attention feature V fused and tactile channel attention feature T fused Perform spatial channel attention fusion to obtain the fusion feature F fusion ;

[0168] (6) Fusion feature F fusion After fully connecting the output distribution probability, find the index of the maximum value in the output probability distribution, which is the recognition result of the model.

[0169] Through the above method, the present invention can not only improve the application efficiency of industrial robots in object identification, but also adapt to diverse production environments and contribute new technical solutions to the field of intelligent manufacturing.

[0170] experiment:

[0171] 1. Performance comparison of different attention mechanism networks

[0172] Based on the analysis of ObjectFolder2.0, this paper selected five representative attention mechanism networks in the field of computer vision for performance evaluation and comparison, including: Squeeze-and-Excitation Networks (SENet), Selective Kernel Networks (SKNet), Convolutional Block Attention Module (CBAM), Channel Attention (CA), and Efficient Multi-Scale Attention (EMA) networks. These network designs are all models of visual information processing, optimized for enhancing image feature representation, but do not consider tactile information in their original form.

[0173] In the research of the present invention, in order to ensure a fair and consistent basis for performance comparison, all selected attention network models use the same backbone network architecture, the classic AlexNet, as their underlying feature extraction. AlexNet has become a widely recognized basic model choice because of its simple structure and good performance in a variety of visual tasks. In the experiment, the present invention selects Accuracy, Precision, F1 Score and Parameter as performance indicators.

[0174] Table 1 Performance comparison under different attention networks

[0175]

[0176]

[0177] The experimental data analysis shown in Table 1 reveals a clear trend: Through the data in the table, we can intuitively observe that compared with the single visual modality, our model has achieved the best performance in the object recognition task, with an accuracy of 96.857% and an F1-score of 0.9658. Compared with the second-performing single visual modality method EMA on ObjectFolder 2.0, our method has improved the accuracy by about 2.286% and the F1-score by 0.0238. At the same time, although our model is second only to EMA in the number of parameters, it effectively eliminates the interference caused by high accuracy and shows excellent performance.

[0178] Our model is compared with the traditional single visual modality method, and the results show that our method shows better performance. The proposed visual-tactile fusion model rationally utilizes the advantages of each modality network and effectively integrates visual and tactile data, thereby significantly improving the overall recognition ability. Experimental results show that our multimodal fusion strategy successfully achieves information complementarity and integration, enabling the model to fully utilize visual and tactile information when dealing with complex environmental perception tasks, improving recognition accuracy, while maintaining a low number of parameters, reflecting a good balance between efficiency and effect.

[0179] 2. Performance comparison of different multimodal networks

[0180] The DMDA-Net of the present invention is compared with several mainstream multimodal fusion methods on the AUDataset dataset, including CRNN-SA (ICANN 2023), CRNN-CA (ICANN 2023), MMM and MulT (ACL 2019). For a fair comparison, the present invention uses the public code released by the authors to compare the method of the present invention with the most advanced deep learning-based multimodal fusion methods and visual methods. In particular, the present invention uses visual modality data to test the performance of visual methods.

[0181] In the experiment, the present invention selects Accuracy, F1 Score and Parameter as performance indicators.

[0182] Table 2 Performance comparison under different multimodal networks

[0183]

[0184]

[0185] As can be seen from Table 2, DMDA-NET performs well in terms of accuracy, reaching 92.54%, significantly surpassing other models. This result shows that the model has a high accuracy in classification tasks and can effectively distinguish different categories. Similarly, as a multimodal model, the CRNN-SA model uses convolutional neural networks (CNN) to learn spatial representations and recurrent neural networks (RNN) to model temporal relationships, while combining the self-attention mechanism and cross-attention mechanism of Transformer to focus on global and cross-modal information. Although CRNN-SA achieves alignment of visual, tactile, and kinesthetic data, the discussion on the effective integration of cross-modal data is still relatively limited. This model is closely followed by DMDA-NET with an accuracy of 91.27%. DMDA-NET also achieved the highest score in F1 score, reaching 0.92023, which shows that the model has achieved a good balance between precision and recall, especially when dealing with unbalanced data sets, it can effectively control the ratio of true positives to false negatives. CRNN-SA's F1 score is 0.9061, closely following. The F1 scores of other models decrease in turn, among which MulT has the lowest F1 score of only 0.8026, which may be related to its limited generalization ability when processing specific tasks. Overall, DMDA-NET performs best in both accuracy and F1 score, showing its strong ability in integrating visual and tactile information processing.

[0186] The results in Table 2 show that in the experiment compared with other multimodal models, the multimodal fusion strategy adopted by our DMDA-NET is more advantageous, with better performance in accuracy and F1 Score, which effectively solves the pain point that visual and tactile data are difficult to fuse, and highlights the necessity and effectiveness of adopting a reasonable fusion strategy to improve model performance when dealing with complex scenarios.

[0187] 3. Ablation experiment

[0188] In order to verify that the object recognition results obtained by visual-tactile fusion are better than those obtained using single visual or single tactile information, ablation experiments were performed.

[0189] We selected SENET and EMA, which had the best single vision effect in the comparative experiments, to conduct single vision and single touch experiments on the ObjectFolder2.0 dataset and AUDataset dataset, and distinguished them with prefixes V and T respectively.

[0190] Table 3. Results of visual tactile ablation experiment

[0191]

[0192]

[0193] As shown in Table 3, when performing classification tasks on the OB2.0 dataset, DMDA-NET showed excellent classification performance with an accuracy of 96.857% when integrating visual and tactile information. When performing classification tasks on the AUDataset, its accuracy also reached 92.54%. At the same time, the V_SENet and V_EMA models achieved accuracies of 94.857% and 94.571% respectively under pure visual information conditions, which is about 2% lower than the recognition accuracy of our visual and tactile fusion model. It is worth noting that the recognition accuracies of the T_SENET and T_EMA models on the OB2.0 dataset were 51.166% and 47.791% respectively when using only tactile information. On the AUDataset, even after 60 rounds of incomplete fitting, the accuracies were only 10.467% and 11.111%. This low accuracy can be attributed to the existence of single background, narrow objects, and lighting effects in tactile data.

[0194] The data in Table 3 show that although tactile information is indispensable in certain specific tasks, in most cases, visual information is still the key to achieving high-precision classification, and the recognition accuracy of single tactile information is still much lower than that of single visual data. However, the model with the correct fusion strategy does combine the advantages of the two modal data, making the accuracy of the fusion model higher than that of single visual recognition. This finding also highlights the importance of cross-modal information fusion, especially the combination of tactile information and visual information. This fusion strategy can extract more comprehensive and delicate object features, thereby significantly improving the accuracy and reliability of recognition. The results show that through the complementarity and integration of cross-modal information, even when a certain modality (such as vision or touch) faces limitations, the fusion strategy using DMDA-NET can also show strong robustness and target recognition capabilities, ensuring that accurate target detection and classification can still be achieved in complex or changing environments.

[0195] Our experiments verified the significant advantages of DMDA-NET in visual-tactile fusion in object recognition tasks, proving that the model that integrates visual and tactile information is far more accurate than the model that relies solely on a single perception mode.

[0196] Systematic evaluation further consolidates the significant advantages of DMDA-Net in visual-tactile fusion in object recognition tasks, proving that models that fuse visual and tactile information are far more accurate than models that rely solely on a single perception mode. This result not only strongly supports the complementary theory of multimodal information in object recognition tasks, but also highlights the necessity and effectiveness of adopting reasonable fusion strategies to improve model performance when dealing with complex scenes. In short, the research of this invention not only verifies the effectiveness of integrating visual and tactile information, but also emphasizes the great significance of this multimodal fusion method in promoting the development of intelligent perception technology under diverse and challenging conditions in the real world.

[0197] In summary, the present invention is committed to building an innovative solution to integrate visual and tactile information, targeting the limitations of current technology in the field of object recognition. The core of the solution proposed by the present invention is to design a novel visual-tactile fusion network, which is particularly integrated with the attention mechanism of spatial interactive learning, and cleverly solves the problem that visual and tactile data are difficult to effectively combine due to significant differences. This unique fusion strategy not only significantly enhances the accuracy of object recognition, but also relies on the efficient characteristics of DMDA-NET. This method not only improves the robustness of the recognition system, but also makes full use of the miniaturized computing advantages of DMDA-NET, so that the entire model can maintain high performance while reducing resource consumption.

[0198] Therefore, the visual-tactile fusion network of the present invention, relying on advanced spatial interactive attention learning technology, successfully breaks the barriers of visual and tactile data fusion, achieves a higher level of object recognition capability, and opens up a new path for the development of robot perception and interaction technology.

[0199] Finally, it should be noted that the above examples are only some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there are many variations. All variations that can be directly derived or associated with the content disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

[0200] 【1】Website: https: / / download.pytorch.org / models / alexnet-owt-4df8aa71.pth ;

[0201] 【2】Bonner, LER, Buhl, DD, Kristensen, K., & Navarro-Guerrero, N. (2021). AU Dataset for Visuo-Haptic Object Recognition for Robots.figshare. https: / / doi.org / 10.6084 / m9.figshare .14222486.

Claims

1. A method for object recognition based on dual-modal dual-channel attention, characterized in that: It includes collecting visual data and tactile data of the object, preprocessing the visual data and standardizing the tactile data, and using them as inputs of the visual-tactile fusion network trained offline in the second stage; The visual tactile fusion network includes a feature extraction network. The pre-processed visual data and the standardized tactile data are respectively processed through the feature extraction network to obtain a visual feature map. and tactile feature maps Then the visual feature map Perform visual channel attention extraction and tactile feature map Perform tactile channel attention extraction and then perform spatial channel attention fusion to obtain object recognition results.

2. The object recognition method based on dual-modal dual-channel attention according to claim 1, characterized in that: The feature extraction network is an AlexNet network.

3. The object recognition method based on dual-modal dual-channel attention according to claim 2, characterized in that: The specific process of visual channel attention extraction is: (1) The visual feature map Group and decompose into two parallel eigenvectors v c and x c ; (2) Global average pooling operation: Among them, v c represents the input features of the channel along the horizontal dimension at height H, C represents the number of input channels, and H and W represent the spatial learning rate of the input features; Among them, x c Represents the input features along the vertical dimension at width W; (3) After the grayscale value of each channel of the preprocessed visual data is normalized, the grayscale feature map is obtained through the GLCM module, and then the visual feature map is obtained through the AlexNet network. The visual matrix feature map With the visual feature map Concatenate and pass through the convolution layer to obtain the channel descriptor V g ￠ lcm ; (4) Encode the feature vector and feature encoding vector The concatenated feature vectors are merged along the spatial dimension, and then a set of shared convolution kernels are used to perform convolution operations on the concatenated feature vectors; then, they are split into two independent feature vectors along the spatial direction and divided into two parallel branches, which are fed into the nonlinear Sigmoid function to generate adaptive weight representation; at the same time, the channel descriptor V g ￠ lcm are also synchronously fed to the same Sigmoid function to generate another set of adaptive weight representations; the three sets of parallel adaptive weight representations are used to perform Re-weight operations on the original feature map, and finally the fused visual channel attention feature V is obtained. fused .

4. The object recognition method based on dual-modal dual-channel attention according to claim 3, characterized in that: The specific process of tactile channel attention extraction is: (1) Average pooling and maximum pooling The tactile feature map After feature grouping, the maximum pooling layer and average pooling layer operations are performed separately: Where T(i,j,c) represents the value of the input tactile feature map at position (i,j) and channel c, O max (c) is the maximum pooling output of the cth channel; Among them, O avg (c) is the average pooling output of the cth channel; (2) O max (c) and O avg (c) Pass through a fully connected layer with shared parameters and perform matrix addition: T fused =W×(O max (c)+O avg (c))+b (9) Among them, W and b are the weight matrix and bias vector of the fully connected layer, respectively, T fused is the fused tactile channel attention feature vector.

5. The object recognition method based on dual-modal dual-channel attention according to claim 4, characterized in that: The specific process of the spatial channel attention fusion is: The visual channel attention feature V fused and the tactile channel attention feature T fused After normalization, the visual channel attention feature V￠ and the tactile channel attention feature T￠ are obtained; The visual channel attention feature V￠ is encoded by a two-dimensional global flat pooling of visual global spatial information: in, For feature V agp,bc The natural nonlinear function Softmax of the two-dimensional Gaussian mapping is used for global output to obtain V softmax , and then perform matrix dot product with the tactile channel attention T￠: Z TV =T￠×V softmax (11) The tactile channel attention feature T￠ is reweighted and normalized, encoded by two-dimensional global average pooling, and then matrix multiplied with the visual channel attention feature V￠: With VT =V￠×T softmax (12) Among them, T softmax Represents the features of the tactile channel attention feature T￠ after global average pooling and softmax operation; Next, the visual space descriptor Z that incorporates the tactile channel information is VT and the tactile space descriptor Z that integrates visual channel information TV Perform the addition operation, and then apply the nonlinear Sigmoid function to generate the spatial channel weight W; the original visual feature map after the spatial channel weight W is used to group the features and tactile feature maps Perform feature modulation, realize feature reweighting through element-level multiplication operation, obtain the adjusted visual feature V￠￠ and tactile feature T￠￠, and perform matrix addition of the visual feature V￠￠ and the tactile feature T￠￠ to obtain the fusion feature F fusion ; Fusion feature F fusion After passing through the fully connected layer, the predicted classification results are output.

6. The method for object recognition based on dual-modal dual-channel attention according to claim 5, characterized in that: The specific process of the two-stage offline training is as follows: (1) Offline training on the OBJECTFOLDER2.0 dataset: The visual data samples in the dataset OBJECTFOLDER2.0 were subjected to data enhancement operations such as random cropping, image scaling, color space perturbation, and horizontal flipping. The tactile data samples were converted into RGB images through rendering technology. Then they were divided into the first-stage training set and the first-stage test set. The visual-tactile fusion network was trained and tested in the first stage. The cross entropy loss function and Adam optimizer were used to update the parameters in the network through the back-propagation algorithm to obtain the first-stage optimal network weights of the visual-tactile fusion network. (2) Offline training on the dataset AUDataset The visual image of each sample in the dataset AUDataset is expanded according to different lighting conditions, and then divided into the second-stage training set and the second-stage test set. The visual-tactile fusion network trained in the first stage is trained and tested in the second stage. The cross entropy loss function and Adam optimizer are used to update the parameters in the network through the back-propagation algorithm to obtain the second-stage optimal network weights of the visual-tactile fusion network. The visual-tactile fusion network trained in the second stage is used as a network that can be used online.

Images