An item recognition method based on dual-mode dual-channel attention

Abstract

The application relates to the technical field of object recognition classification and machine vision processing, and discloses an object recognition method based on bimodal double-channel attention, which comprises the following steps: after the collected visual data is preprocessed and the tactile data is standardized, the visual data and the tactile data are taken as the input of a visual-tactile fusion network; the preprocessed visual data and the standardized tactile data respectively pass through a feature extraction network to obtain a visual feature map and a tactile feature map; the visual feature map is subjected to visual channel attention extraction, and the tactile feature map is subjected to tactile channel attention extraction; and the extracted features are subjected to spatial channel attention fusion to obtain an object recognition result. The visual-tactile fusion network adopts a two-stage offline training method. The multi-modal feature extraction and fusion strategy proposed by the application not only strengthens the processing of tactile features, but also realizes deep interaction with visual features, achieves information complementation and integration, fully utilizes visual and tactile information, and improves recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of object recognition and classification and machine vision processing, and in particular relates to an object recognition method based on dual-modal dual-channel attention. BACKGROUND

[0002] Object recognition technology initially relied on traditional image processing methods such as edge detection and feature matching, which are susceptible to factors such as lighting, pose, and occlusion, limiting the accuracy of recognition. With the development of deep learning technology, especially the application of convolutional neural networks (CNNs), object recognition has made significant progress. However, CNNs still struggle to effectively deal with background interference and subtle differences when dealing with complex scenes.

[0003] To address these issues, attention mechanisms have been introduced, which mimic the way human visual attention is distributed, automatically focusing on key feature areas in images, thereby improving the robustness and efficiency of recognition systems. Despite this, visual recognition still has limitations in certain situations, especially when dealing with local details and low-density visual areas of objects.

[0004] Haptic sensors can capture information such as texture and roughness of objects, which are difficult for visual systems to obtain. Haptic learning can provide a unique perspective on object attributes, especially when visual similarity cannot be distinguished, effectively classifying through differences in haptic features. Therefore, haptic information has significant application value in object shape reconstruction, category identification, etc.

[0005] Given the strengths and weaknesses of visual and haptic information, visual-haptic fusion technology has become a new direction in object recognition. This technology aims to integrate visual and haptic information, enhancing the accuracy and robustness of object recognition through complementarity. Specifically, visual information provides macroscopic contours and color attributes, while haptic information supplements details such as material, texture, and shape of objects.

[0006] To achieve visual-haptic fusion, the present application proposes a visual-haptic fusion network based on dual-modal dual-channel attention. This network utilizes the high sensitivity of the haptic channel, combined with a channel attention mechanism, to accurately capture and enhance key information in haptic features. At the same time, by processing the gray covariance of haptic features, it strengthens the spatial learning rate expression of visual features, effectively compensating for the shortcomings of visual attention mechanisms in dealing with sparse areas. In this way, the network can better recognize and understand objects, especially in complex environments. SUMMARY

[0007] The technical problem to be solved by the present application is to provide an object recognition method based on dual-modal dual-channel attention, which effectively improves the accuracy of object recognition through visual and haptic feature fusion.

[0008] To solve the above technical problems, the application provides an object recognition method based on dual-modal dual-channel attention, comprising: collecting visual data and tactile data of an object, pre-processing the visual data and standardizing the tactile data, and taking the pre-processed visual data and the standardized tactile data as inputs of a visual-tactile fusion network trained offline in two stages;

[0009] The visual-tactile fusion network comprises a feature extraction network, and the pre-processed visual data and the standardized tactile data are respectively input into the feature extraction network to obtain a visual feature map and a tactile feature map Then, the visual feature map is subjected to visual channel attention extraction, the tactile feature map is subjected to tactile channel attention extraction, and spatial channel attention fusion is performed to obtain an object recognition result.

[0010] As an improvement of the object recognition method based on dual-modal dual-channel attention, the application provides the following:

[0011] The feature extraction network is an AlexNet network.

[0012] As a further improvement of the object recognition method based on dual-modal dual-channel attention, the application provides the following:

[0013] The specific process of the visual channel attention extraction is as follows:

[0014] (1) The visual feature map is grouped and decomposed into two parallel feature vectors v c and x c .

[0015] (2) Global average pooling operation:

[0016]

[0017] wherein v c represents an input feature along a horizontal dimension direction channel at a height H, C represents a number of input channels, and H and W represent spatial learning rates of the input feature;

[0018]

[0019] wherein x c represents an input feature along a vertical dimension direction at a width W.

[0020] (3) After performing standardization of gray values on each channel of the pre-processed visual data, a gray feature map is obtained through a GLCM module, and then a visual feature map is obtained through an AlexNet network.

[0021] The visual matrix feature map is spliced and a channel descriptor V is obtained through a convolution layer g lcm

[0022] (4) The feature encoding vectors and the feature encoding vectors are merged along the spatial dimension, and then a convolution operation is performed on the spliced feature vectors using a set of shared convolution kernels; subsequently, the feature vectors are split into two independent feature vectors along the spatial direction, and are divided into two parallel branches, and the two parallel branches are fed to a nonlinear Sigmoid function to generate adaptive weight representations; at the same time, the channel descriptor V g lcm is also synchronously fed to the same Sigmoid function to generate another set of adaptive weight representations; the three sets of parallel adaptive weight representations are used for Re-weight operation on the original feature map, and finally the fused visual channel attention feature V fused is obtained.

[0023] As a further improvement of the item recognition method based on the dual-modal dual-channel attention of the present application:

[0024] The specific process of the tactile channel attention extraction is as follows:

[0025] (1) Average pooling and maximum pooling

[0026] The tactile feature map is grouped by features, and then maximum pooling and average pooling operations are performed respectively:

[0027]

[0028] Wherein, T(i,j,c) represents the value of the input tactile feature map at position (i,j) and channel c, O max (c) is the maximum pooling output of the cth channel;

[0029]

[0030] Wherein, O avg (c) is the average pooling output of the cth channel;

[0031] (2) O max (c) and O avg (c) are added through a fully connected layer with shared parameters and matrix addition:

[0032] T​​​​fused = W x (O max (c) + O avg (c)) + b (9)

[0033] where W and b are the weight matrix and bias vector of the fully connected layer, respectively, and T fused is the fused tactile channel attention feature vector.

[0034] As a further improvement of the item recognition method based on dual-modal dual-channel attention of the present application:

[0035] The specific process of the spatial channel attention fusion is as follows:

[0036] The visual channel attention feature V fused and the tactile channel attention feature T fused are respectively normalized to obtain the visual channel attention feature V￠ and the tactile channel attention feature T￠;

[0037] The visual channel attention feature V￠ is encoded by a two-dimensional global average pooling visual global spatial information:

[0038]

[0039] where,

[0040] The feature V agp,bc is globally outputted by using a natural nonlinear function Softmax of two-dimensional Gaussian mapping to obtain V softmax , and then matrix dot product is performed with the tactile channel attention T￠:

[0041] Z TV = T￠ x V softmax (11)

[0042] The tactile channel attention feature T￠ is re-weighted and standardized, and then encoded by two-dimensional global average pooling, and then matrix multiplication operation is performed with the visual channel attention feature V￠:

[0043] Z VT = V￠ x T softmax (12)

[0044] where T softmax represents the feature of the tactile channel attention feature T￠ after global average pooling and softmax operation;

[0045] Then, the visual spatial descriptor Z VT fused with the tactile channel information is combined with the tactile spatial descriptor Z TVThe addition operation is performed, and then a nonlinear Sigmoid function is applied to generate a spatial channel weight W; the spatial channel weight W is respectively applied to the original visual feature map after feature grouping And a tactile feature map Feature modulation is performed, the feature is reweighted through element-level multiplication operation, to obtain adjusted visual feature V and tactile feature T, and the visual feature V and the tactile feature T are added in matrix to obtain a fusion feature F fusion ; the fusion feature F fusion Pass through a fully connected layer to output a predicted classification result.

[0046] As a further improvement of the item recognition method based on the dual-modal dual-channel attention of the application:

[0047] The specific process of the two-stage offline training is as follows:

[0048] (1) Offline training on the data set OBJECTFOLDER2.0:

[0049] The samples of visual data in the data set OBJECTFOLDER2.0 are subjected to data enhancement operations such as random cropping, image scaling, color space disturbance and horizontal flipping, the samples of tactile data are converted into RGB images through rendering technology, and then divided into a first-stage training set and a first-stage test set, the visual-tactile fusion network is trained and tested in the first stage, the cross-entropy loss function and the Adam optimizer are used, the parameters in the network are updated through the back propagation algorithm, and the first-stage optimal network weight of the visual-tactile fusion network is obtained;

[0050] (2) Offline training on the data set AUDataset

[0051] The visual images of each sample in the data set AUDataset are expanded under different light conditions, and then divided into a second-stage training set and a second-stage test set, the visual-tactile fusion network trained in the first stage is trained and tested in the second stage, the cross-entropy loss function and the Adam optimizer are used, the parameters in the network are updated through the back propagation algorithm, and the second-stage optimal network weight of the visual-tactile fusion network is obtained;

[0052] The visual-tactile fusion network trained in the second stage is used as an online available network.

[0053] The beneficial effects of the application mainly include:

[0054] 1.The present application proposes an effective modal fusion: the proposed multi-modal feature fusion strategy, especially the tactile attention module, not only strengthens the processing of tactile features, but also deeply interacts with visual features, realizing the complementation and integration of information. This enables the model to fully utilize visual and tactile information when processing complex environmental perception tasks, improving recognition accuracy;

[0055] 2.The present application can make up for the deficiency of visual attention mechanism in local detail perception by introducing tactile global feature and gray covariance matrix analysis, especially in processing areas with low feature intensity, which improves the generalization ability and detail capturing ability of the model;

[0056] 3.Enhanced robustness and generalization ability: the present application can more stably process diversified inputs and improve robustness and generalization ability in different scenes through the fusion of multiple mechanisms, including feature standardization, dynamic allocation of attention weight, and deep analysis of tactile images. BRIEF DESCRIPTION OF DRAWINGS

[0057] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0058] Figure 1 It is a structural diagram of the dual-modal dual-channel visual-tactile fusion network (DMDA-NET) of the present application.

[0059] Figure 2 It is a schematic diagram of the visual channel attention extraction process.

[0060] Figure 3 It is a schematic diagram of the tactile channel attention extraction process.

[0061] Figure 4 It is a schematic diagram of the spatial channel attention fusion process of visual and tactile features of the present application. DETAILED DESCRIPTION

[0062] The present application will be further described below in conjunction with specific embodiments, but the scope of protection of the present application is not limited thereto:

[0063] Example 1,

[0064] Feature-unknown recognition needs to fuse visual and tactile two modal perception data to realize greater object recognition. The present application proposes a method for object recognition based on a dual-modal dual-channel attention visual-tactile fusion network (DMDA-NET).

[0065] Step 1, construct a visual-tactile fusion network (DMDA-NET)

[0066] The visual-haptic fusion network based on dual-modal dual-channel attention (DMDA-NET) is specifically as shown in Figure 1

[0067] Step 1.1, feature extraction

[0068] The input visual and haptic raw images are respectively subjected to parallel feature extraction by using a pre-trained AlexNet classic convolutional neural network architecture, so as to extract meaningful features from the data of the two modalities, thereby obtaining visual feature maps and haptic feature maps wherein the pre-trained AlexNet network uses alexnet = models.alexnet(pretrained=True) to load the pre-trained weights 【1】 .

[0069] Step 1.2, visual channel attention extraction

[0070] A schematic diagram of the visual channel attention extraction process is shown in Figure 2 .

[0071] (1) Channel grouping

[0072] After feature extraction, the visual features and the haptic features are further grouped in terms of channel learning rate, i.e., so-called channel grouping, so as to better organize and process the feature information.

[0073] First, the input visual feature maps are grouped, and the grouped visual feature maps are decomposed into two parallel 1D feature vectors for establishing cross-channel dependency and spatial position information, wherein C represents the number of input channels, and H and W represent the spatial learning rates of the input features;

[0074] (2) Global average pooling

[0075] One of the two parallel 1D feature vectors obtained in step 1.1 is subjected to 1D global average pooling along the horizontal learning rate direction, so as to capture global information in the horizontal learning rate direction, and the 1D global average pooling operation for encoding the global information along the C horizontal learning rate direction at the height H can be represented as:

[0076]

[0077] wherein v c ​represents the input features along the horizontal dimension at height H and outputs the feature encoding vector in the H direction (size C'1'W), which captures the long-range interactions in space using the position information average pooling in the vertical learning rate direction, gets the global dependency in the vertical direction, and preserves the precise position information along the horizontal learning rate direction;

[0078] Similarly, the other of the two parallel 1D feature vectors obtained in step 1.1 is captured by performing 1D global average pooling along the vertical learning rate direction, thereby capturing the global information in the vertical learning rate direction. At width W, the pooling output in C can be expressed as:

[0079]

[0080] where x c represents the input features along the vertical dimension at width W and outputs the feature encoding vector in the W direction (size C'1'H). Similarly, the global dependency line in the horizontal direction preserves the precise position information along the vertical learning rate direction;

[0081] (3) GLCM module

[0082] First, for the input visual raw image The standardization of gray value is performed for each channel of each sample to remove the influence of brightness offset and ensure that the covariance matrix reflects the relative changes between pixels. The standardization process is as follows:

[0083]

[0084] where, represents the normalized tensor, μ bc is the average gray value of channel c in sample b, and the calculation formula is:

[0085]

[0086] Here V bcij represents the gray value of channel c of the normalized image tensor sample b at position (i,j).

[0087] Then, the covariance between each channel and other channels of each sample is calculated based on the normalized image tensor, which actually calculates the statistics of the gray level co-occurrence matrix (GLCM). Specifically, for each sample b and each channel pair (c, k), an element Cov bk of the covariance matrix can be expressed as:

[0088]

[0089] By directly using the normalized The covariance is calculated by the tensor of the covariance, which implicitly handles the elimination of the mean in the calculation. The resulting covariance matrix tensor Further feature extraction is performed by a convolution layer. The kernel size of the convolution layer is 3'3, the stride is 1, and the padding is 1, so that the size of the output feature map is the same as the input. The convolution operation can be represented as:

[0090]

[0091] where V glcm,bcij represents the gray feature map, i.e., the value of the output tensor at the channel c of the sample b at the position (i,j);

[0092] Subsequently, this set of gray feature maps is input into the pre-trained AlexNet network and grouped into features to further refine high-order abstract features, and the visual feature map V

[0093] Next, the grouped visual matrix feature map V is concatenated with the visual feature map V obtained by grouping in step 1 to form a composite feature representation. In order to reduce the feature learning rate while keeping the key information from being lost, the composite feature is reduced in dimension by a 1'1 convolution layer and obtains a channel descriptor V g ￠ lcm ;

[0094] (4) The feature encoding vectors in the H direction V and the feature encoding vectors in the W direction V are combined along the spatial learning rate. Then, a set of shared convolution kernels is used to perform convolution operation on the concatenated feature vector, aiming to fuse the information of different channels.

[0095] Subsequently, the processed feature is again divided into two independent feature vectors along the spatial direction, with sizes of C / / G'1'W and C / / G'1'H, respectively. The two feature vectors are divided into two parallel branches, and the two parallel branches are fed into a nonlinear Sigmoid function to generate an adaptive weight representation.

[0096] At the same time, the channel descriptor V g ￠ lcm is also fed into the same Sigmoid function to generate another set of adaptive weight representations.

[0097] The three parallel adaptive weight representations obtained for aggregating the original intermediate feature maps are used for the Re-weight operation on the original feature maps, which adaptively adjusts the channel level of the original feature maps to effectively regulate network attention and optimize the precision of feature expression. Finally, the fused visual channel attention features V fused .

[0098] Here, the Sigmoid-adjusted adaptive weights are regarded as channel weights after feature selection, and then multiplied by the previous features to complete the reweighting of the original features in the channel learning rate.

[0099] Step 1.3, tactile channel attention extraction

[0100] Since tactile features are more sensitive to local features, and tactile itself has the characteristics of multi-channel, tactile images will adopt a more direct and efficient processing method. The schematic diagram of the tactile channel attention extraction process is shown in Figure 3 .

[0101] (1) Average pooling and max pooling

[0102] The tactile feature map obtained in step 1.1 is grouped into features, and the visual feature map after grouping is fed into two parallel branches, which are further abstracted and refined by max pooling and average pooling layers respectively to capture and strengthen information at different levels.

[0103] Specifically, the tactile feature vector in one branch is focused on the key local changes in the tactile signal through adaptive max pooling, and its mathematical expression is:

[0104]

[0105] where T(i,j,c) represents the value of the input tactile feature map at position (i,j) and channel c, and O max (c) is the max-pooled output of the cth channel.

[0106] The other branch passes through an adaptive average pooling layer, which aims to maintain the global consistency of the tactile features, and its mathematical expression is:

[0107]

[0108] Here, H and W are the height and width of the input tactile feature map, respectively, and O avg (c) is the average-pooled output of the cth channel.

[0109] (2) The two parallel channel descriptors Omax (c) and O avg (c) by a fully connected layer with shared parameters and matrix addition. Mathematically, this operation can be formalized as:

[0110] T fused = W x (O max (c) + O avg (c)) + b (9)

[0111] where W and b are the weight matrix and bias vector of the fully connected layer, respectively, and T fused is the fused tactile channel attention feature vector.

[0112] Thereafter, the tactile channel attention feature T fused is multiplied by the channel attention weight generated by the Sigmoid activation function, which dynamically allocates the importance of the tactile signal, and then adjusts the tactile feature map at the channel level.

[0113] The tactile channel attention fusion utilizes shared parameters and linear combination strategies to capture subtle differences in tactile signals more finely, enabling in-depth understanding and efficient processing of tactile signals. Similarly, the use of the Sigmoid function strengthens the expression of important tactile information and weakens unimportant parts, ultimately enabling the model to more effectively highlight key information in tactile features while suppressing irrelevant or redundant information when processing tactile data.

[0114] Step 1.4, spatial channel attention fusion of visual and tactile features

[0115] The schematic diagram of the spatial channel attention fusion process of visual and tactile features is shown in Figure 4 , which is specifically as follows:

[0116] After standardizing the visual channel attention feature V fused obtained in step 1.2 and the tactile channel attention feature T fused obtained in step 1.3, the standardized visual channel attention feature V￠ and tactile channel attention feature T￠ are obtained.

[0117] The standardized visual channel attention feature V￠ is encoded by a two-dimensional global average pooling visual global spatial information, which is mathematically represented as:

[0118]

[0119] where,

[0120] To efficiently calculate, the feature V​agp,bc The global output is obtained by using the natural nonlinear function Softmax of the two-dimensional Gaussian mapping softmax , and the matrix dot product is performed with the normalized tactile channel attention T

[0121] Z TV = T softmax × V

[0122] Similarly, by using a similar method, the tactile channel attention feature T VT is reweighted and standardized, and then the spatial information thereof is encoded via two-dimensional global average pooling, and finally the matrix multiplication operation is performed with the standardized visual channel attention feature V

[0123] Z VT = V softmax × T

[0124] wherein T softmax represents the feature of the tactile channel attention feature T

[0125] Since there is an inherent correlation between the channel and the spatial position, in view of this, the application adopts the above-mentioned novel multi-modal feature fusion strategy, and realizes the deep fusion of visual and tactile features in the channel level and the information complementation between the channel features by using an efficient spatial interaction aggregation mode.

[0126] Step 1.5, in order to further deepen the interaction of visual and tactile features, the visual spatial descriptor Z VT fused with tactile channel information is added to the tactile spatial descriptor Z TV fused with visual channel information, so as to realize the deep cooperation of the two modalities in the spatial domain. Then, the nonlinear Sigmoid function is applied to generate the spatial channel weight W, which directly reflects the relative importance of visual and tactile features at a specific spatial position.

[0127] Step 1.6, global fusion

[0128] The spatial channel weight W derived by using the spatial attention mechanism is respectively applied to the feature grouped original visual feature map and the tactile feature map The feature modulation is performed, the feature is reweighted through element-level multiplication operation, and the adjusted visual feature V￠￠ and tactile feature T￠￠ are obtained. Finally, V￠￠ and T￠￠ are fused by matrix addition to obtain the fusion feature F which integrates visual and tactile information fusion .

[0129] This process not only effectively improves the expression of the feature, highlights the key visual information, and suppresses the redundant or irrelevant background interference, thereby maintaining the tactile perception details while improving the expression and recognition of the visual feature, providing a more rich and comprehensive representation basis for subsequent classification;

[0130] Step 1.7, finally, the fusion feature F fusion After the dropout operation to suppress the overfitting phenomenon, the extracted features are passed to the fully connected layer, which is responsible for mapping the feature space to the classification space, thereby outputting the final predicted classification result of the tactile image.

[0131] In summary, the visual-tactile fusion network based on dual-modal and dual-channel attention:

[0132] (1) strengthens the deep attention feature extraction and processing of visual and tactile information, enhances the overall perceptual ability of visual attention mechanism and the fine recognition ability of tactile perception system for object material and texture.

[0133] (2) By introducing visual global feature and gray covariance matrix analysis, it can make up for the deficiency of visual attention mechanism in local detail perception, especially in processing areas with low feature intensity, and improve the generalization ability and detail capturing ability of the model;

[0134] (3) An interactive multi-modal feature fusion strategy is proposed, which not only strengthens the processing of tactile features, but also deeply interacts with visual features, realizes the complementation and integration of information. This makes the model can make full use of visual and tactile information when dealing with complex environment perception tasks, and improve the recognition accuracy.

[0135] Step 2, training

[0136] The application adopts the data set OBJECTFOLDER2.0 and the data set AUDataset to respectively perform secondary offline training on the visual tactile fusion network (DMDA-NET), the purpose is to enhance the adaptability and generalization ability of the network on different data sets. Through this phased training, the understanding and processing ability of the network to different tactile data characteristics can be improved, so as to improve the overall application effect. First, the data set OBJECTFOLDER2.0 is offline trained to optimize its performance and obtain the optimal network weight, then the visual tactile fusion network (DMDA-NET) with the optimal weight trained on the OBJECTFOLDER2.0 is offline trained again using the data set AUDataset, and finally the trained visual tactile fusion network (DMDA-NET) with the optimal weight on the AUDataset is used as the network that can be used online. Since the tactile data has the characteristics of various forms, the tactile data of the two data sets of the application is inconsistent, so as to expect to face different forms of tactile data and still maintain good recognition accuracy in online use.

[0137] 2.1, first stage offline training

[0138] OBJECTFOLDER2.0 is a large-scale multi-sensory data set, which contains comprehensive visual, audio and tactile information of 1,000 objects. Each object is virtualized by implicit neural representation, fully embodying its texture, material and three-dimensional structure.

[0139] The visual data set is characterized in that all images are taken in a background-free environment, and 34 images are obtained from multiple angles to increase the diversity of viewing angles. Finally, the training set has 81 images per category (27 original images are enhanced by data enhancement), and the test set retains 7 original images to ensure the independence of evaluation.

[0140] The tactile data adopts the GeSight sensor to capture the surface details of the object, records the contact point information and converts the tactile information into an RGB image through rendering technology, and the division method of the training set and the test set is consistent with the visual data set.

[0141] In order to improve the generalization ability of the model, the samples of the visual data in the data set OBJECTFOLDER2.0 are subjected to a variety of enhancement techniques such as random cropping, image scaling, color space disturbance and horizontal flipping. These operations enhance the diversity of the data set, improve the recognition stability of the model and the learning ability of the object features. The enhanced data set OBJECTFOLDER2.0 is randomly divided into a training set and a test set according to an eight-two division ratio for the first stage training.

[0142] In the first stage offline training process, the batch-size is set to 16, ensuring that the model can access a rich and representative sample set in each training iteration, effectively promoting the model's ability to recognize and understand complex information, and laying a solid foundation for improving its generalization performance on unknown data. The first stage offline training of the present application selects the cross-entropy loss function as the optimization objective and uses the Adam optimization algorithm to adjust the model parameters. The training plan is set to perform 300 epochs, and the starting learning rate is set to 0.000001.

[0143] After initializing the model parameters, the first stage training set is first input into the visual-haptic fusion network (DMDA-NET). During the training process, the feature vectors after dropout processing will be passed to the fully connected layer to generate the final recognition result. Then, the loss between the recognition result and the true label is calculated, and the cross-entropy loss function is used to quantify the accuracy of the model's prediction results. Based on the calculated loss, the gradient is calculated layer by layer from the output layer to the input layer through the backpropagation algorithm. Update the parameters in the network and use the optimizer (such as Adam) for gradient descent. Repeat the forward propagation and backpropagation steps until one of the following stopping conditions is met:

[0144] 1. Reach the predetermined maximum number of iterations.

[0145] 2. The loss value no longer significantly decreases over consecutive epochs.

[0146] After completing the first stage of training, the trained model parameters are used to extract features and fuse visual and haptic data on the test set. The previously trained model and its parameters need to be loaded. Then, in the test setting stage, evaluation metrics such as accuracy, precision, recall, F1 score, or mean square error need to be defined, and the test environment needs to be consistent with the training environment to avoid performance differences due to environmental differences. Then, the final recognition result is obtained through the fully connected layer, the prediction output is obtained, and it is compared with the true label of the test data to obtain the desired evaluation model performance indicators. Throughout the process, it is necessary to ensure that the test data set is large and representative, and to avoid using training data or validation data as test data to prevent overfitting or evaluation bias.

[0147] Step 2.2, second stage offline training

[0148] The first stage of training is to obtain the optimal weight of the network when processing haptic data as images. In the second stage of training, the present application will use the weight obtained in the first stage to process the waveform images obtained by the Gelsight haptic sensor, thereby obtaining the optimal weight that has good accuracy when facing different forms of visual-haptic data for online use.

[0149] In the second phase of training the visual-haptic fusion network (DMDA-NET), the AUDataset dataset is used 【2】 , which combines visual, haptic, and dynamic information, contains 63 samples of items, covering visual textures and tactile sensations from clear to ambiguous. Each item is synchronously captured from multiple angles by three fixed high-definition cameras to ensure the integrity and diversity of visual data.

[0150] At the same time, haptic data is collected by Gelsight sensors, with five rounds of repeated observation to improve data reliability. Haptic collection focuses on "feeling" and "exploring" two interaction modes. Dynamic modalities revolve around "enclosing grip", "additional support", "pressure squeezing" and "non-support suspension", using equipment to record the mechanical response and spatial pose changes in dynamic interaction, ensuring three measurements / experiments to obtain more comprehensive data.

[0151] To improve the generalization ability of the model, the number of visual images in the training set is expanded to 50 by changing the light intensity angle, covering various lighting conditions. Fine division is implemented in the data preprocessing stage, and the eight-two division ratio is followed for training and testing set allocation to ensure reasonable sample distribution and information security. Similarly, in the second phase of training, the cross-entropy loss function is selected as the optimization objective, and the Adam optimization algorithm is used to adjust the model parameters. The training plan is set to 60 epochs, and the initial learning rate is set to 0.000005. All parameters of the initialized model are initialized.

[0152] Each batch of data in the second phase of training is sent into the visual-haptic fusion network (DMDA-NET) to obtain the final fusion feature map F fusion . The feature vector after adaptive dropout processing is sent to the fully connected layer to obtain the final recognition result. The loss between the recognition result and the true label is calculated (using cross-entropy loss). Then, according to the calculated loss, the parameters in the network are calculated by gradient, and the network parameters are updated by the optimizer (Adam) for backpropagation. Repeat the above steps until all training data iterations are completed, or the predetermined stopping condition (such as maximum iteration number) is reached.

[0153] The same forward propagation steps as in the training process are performed on the second phase test data, but without dropout operation, gradient calculation and parameter update. The trained second phase model parameters are used to extract features and perform fusion. The final recognition result is obtained through the fully connected layer, and compared with the true label of the test data to evaluate the model performance (such as accuracy, F-1 score indicators).

[0154] 3. Online use

[0155] In the automation application of industrial production lines, robots can efficiently and accurately identify and classify different types of objects by combining visual and tactile sensors, thereby improving production efficiency and product quality. The specific implementation steps are as follows:

[0156] 3.1. Data collection

[0157] On the production line, high-precision cameras are installed to collect visual data in real time. These cameras can work under various lighting conditions to ensure clear image capture, and have image processing capabilities to overcome environmental interference. To enhance reliability, the system uses a multi-camera layout to cover multiple key angles of the production line.

[0158] In addition, a Gelsight tactile sensor is installed at the end of the robotic arm. This sensor captures real-time tactile data of objects, and the data obtained is in the form of waveforms, consistent with the tactile data type used in the second stage of training using the AUDataset. The tactile sensor has high sensitivity and can identify slight touch changes, thereby improving the accuracy of object recognition.

[0159] 3.2. Data preprocessing and standardization

[0160] After data collection, the visual data is first preprocessed, including image denoising, background removal, and standardization processing, to ensure the accuracy and consistency of image information. At the same time, tactile data also needs to be standardized, including normalizing perception data of different force levels.

[0161] After preprocessing, visual and tactile data will be integrated into a unified data format, ensuring that both types of information can be effectively processed in the same model, thereby supporting comprehensive analysis.

[0162] 3.3. The preprocessed visual data and tactile data are used as inputs to the DMDA-NET model trained in the second stage offline in step 2.2. The specific process is as follows:

[0163] (1) Load the trained DMDA-NET model and set the model to evaluation mode to close the dropout layer and disable the training behavior of Batch Normalization during inference; turn off gradient calculation to reduce memory consumption and improve inference speed.

[0164] (2) Load the preprocessed visual data and tactile data.

[0165] (3) The preprocessed visual data and standardized tactile data are respectively extracted to obtain network visual feature maps. and tactile feature map

[0166] (4), visual channel attention extraction is performed on the visual feature map to obtain visual channel attention features V fused ; tactile channel attention extraction is performed on the tactile feature map to obtain tactile channel attention features T fused ;

[0167] (5), spatial channel attention fusion is performed on the visual channel attention features V fused and the tactile channel attention features T fused to obtain fusion features F fusion ;

[0168] (6), the fusion features F fusion pass through a full connection to output a distribution probability, and the index of the maximum value in the output probability distribution is the recognition result of the model.

[0169] Through the above method, the application of the industrial robot in the identification of the article can be improved in efficiency, and the diversified production environment can be adapted, so as to contribute new technical solutions to the field of intelligent manufacturing.

[0170] Experiment:

[0171] 1. Performance comparison of different attention mechanism networks

[0172] For the analysis of ObejctFolder2.0, the application selects five representative attention mechanism networks in the field of computer vision to evaluate and compare their performance, which includes: Squeeze-and-Excitation Networks (SENet), Selective Kernel Networks (SKNet), Convolutional Block Attention Module (CBAM), Channel Attention (CA), and Efficient Multi-Scale Attention (EMA) network. These network designs are all models for visual information processing, optimized for enhancing image feature representation, but do not consider tactile information in their original form.

[0173] In the research of the present application, in order to ensure a fair and consistent performance comparison basis, all selected attention network models use the same backbone network architecture, namely the classic AlexNet, as its underlying feature extraction. AlexNet has become a widely recognized base model choice due to its simple structure and good performance in various visual tasks. In the experiment, the present application selects Accuracy, Precision, F1 Score and Parameter as performance indicators.

[0174] Table 1 Performance comparison under different attention networks

[0175]

[0176]

[0177] As shown in Table 1, the experimental data analysis reveals a clear trend: through the data in the table, we can intuitively observe that our model has achieved the best performance in the object recognition task compared to single visual modalities, with an accuracy of 96.857% and an F1-score of 0.9658. Compared with EMA, which is the second best single visual modality method on ObjectFolder 2.0, our method improves the accuracy by about 2.286% and the F1-score by 0.0238. At the same time, although our model has fewer parameters than EMA, it effectively eliminates the interference caused by high accuracy, demonstrating excellent performance.

[0178] Comparing our model with traditional single visual modality methods, the results show that our method shows better performance. The proposed visual-haptic fusion model reasonably utilizes the advantages of each modality network, effectively integrates visual and haptic data, and significantly improves the overall recognition ability. Experimental results show that our multi-modal fusion strategy successfully realizes the complementation and integration of information, enabling the model to fully utilize visual and haptic information when processing complex environmental perception tasks, improving recognition accuracy while maintaining a low parameter quantity, reflecting a good balance between efficiency and effectiveness.

[0179] 2. Performance comparison of different multi-modal networks

[0180] The DMDA-Net of the present application is compared with several mainstream multi-modal fusion methods on the AUDataset dataset, including CRNN-SA (ICANN 2023), CRNN-CA (ICANN 2023), MMM, and MulT (ACL 2019). For fair comparison, the author's published open code is used to compare the method of the present application with the most advanced deep learning-based multi-modal fusion methods and visual methods. In particular, the visual method is tested using visual modal data to test its performance.

[0181] In the experiment, Accuracy, F1 Score, and Parameter are selected as performance indicators.

[0182] Table 2 Performance comparison under different multi-modal networks

[0183]

[0184]

[0185] As can be seen from Table 2, DMDA-NET performs excellently in accuracy, reaching 92.54%, significantly surpassing other models. This result shows that the model has a high correct rate in the classification task and can effectively distinguish different categories. Similarly, as a multi-modal model, the CRNN-SA model uses a convolutional neural network (CNN) to learn spatial representation and uses a recurrent neural network (RNN) to model temporal relationships, while combining the self-attention mechanism and cross-attention mechanism of the Transformer to focus on global and cross-modal information. Although CRNN-SA achieves alignment of visual, tactile, and kinesthetic data, the effective integration of cross-modal data is still limited. This model follows DMDA-NET with an accuracy of 91.27%. DMDA-NET also achieves the highest F1 score, reaching 0.92023, which indicates that the model has achieved a good balance between precision and recall, especially when dealing with unbalanced datasets, it can effectively control the ratio of true positives and false negatives. The F1 score of CRNN-SA is 0.9061, followed by it. The F1 scores of other models decrease in turn, with the F1 score of MulT being the lowest, only 0.8026, which may be related to its limited generalization ability when dealing with specific tasks. Overall, DMDA-NET performs best in both accuracy and F1 score, showing its strong ability in processing visual and tactile information.

[0186] As shown by the results in Table 2, in the experiment compared with other multi-modal models, our DMDA-NET takes a multi-modal fusion strategy that is more advantageous, and has better accuracy and F1 Score performance, effectively solving the pain point of difficult fusion of visual and tactile data, and highlighting the necessity and effectiveness of using a reasonable fusion strategy to improve model performance when dealing with complex scenarios.

[0187] 3. Ablation experiment

[0188] In order to verify that the object recognition result obtained by visual-tactile fusion is better than the result obtained using single visual or single tactile information, an ablation experiment is performed.

[0189] The single visual effect best in the comparative experiment, SENET and EMA, are selected for single visual and single tactile experiments on the ObjectFolder2.0 dataset and AUDataset dataset, distinguished by the prefixes V and T, respectively.

[0190] Table 3, visual-tactile ablation experiment results

[0191]

[0192]

[0193] As shown in Table 3, in the classification task on the OB2.0 dataset, DMDA-NET exhibits excellent classification performance when fusing visual and tactile information, with an accuracy of 96.857%, and when classifying AUDataset, its accuracy reaches 92.54%. At the same time, the V_SENet and V_EMA models achieve an accuracy of 94.857% and 94.571%, respectively, under pure visual information conditions, which is about 2% lower than the recognition accuracy of our visual-tactile fusion model. It is worth noting that the T_SENET and T_EMA models achieve an accuracy of 51.166% and 47.791%, respectively, on the OB2.0 dataset using only tactile information, and on the AUDataset, even after 60 rounds of fitting, the accuracy is only 10.467% and 11.111%, respectively. This low accuracy can be attributed to factors such as single background, narrow object, and light effects in tactile data.

[0194] As shown by the data in Table 3, although tactile information is indispensable in certain specific tasks, in most cases, visual information is still the key to achieving high-precision classification, and the recognition accuracy of single tactile information is still much lower than that of single visual data. However, the model using the correct fusion strategy combines the advantages of the two modal data, so that the accuracy of the fusion model is higher than that of single visual recognition. This finding also highlights the importance of cross-modal information fusion, especially the combination of tactile information and visual information. This fusion strategy can extract more comprehensive and delicate object features, thereby significantly improving the recognition accuracy and reliability. The results show that through the complementarity and integration of cross-modal information, even in the case of limitations in a certain modality (such as vision or touch), the fusion strategy using DMDA-NET can still exhibit strong robustness and target recognition ability, ensuring that in complex or changing environments, accurate target detection and classification can still be achieved.

[0195] Our experiments verify the significant advantages of DMDA-NET in visual-tactile fusion in object recognition tasks, and prove that the model that fuses visual and tactile information is much more accurate than the model that relies only on a single perception mode.

[0196] Systematic evaluation further consolidates the significant advantages of DMDA-Net in visual-tactile fusion in object recognition tasks, and proves that the model that fuses visual and tactile information is much more accurate than the model that relies only on a single perception mode. This achievement not only strongly supports the complementary theory of multi-modal information in object recognition tasks, but also highlights the necessity and effectiveness of using a reasonable fusion strategy to improve the performance of the model when dealing with complex scenes. In summary, the research of the present invention not only verifies the effectiveness of the integration of visual and tactile information, but also emphasizes the great significance of this multi-modal fusion method for promoting the development of intelligent perception technology under the conditions of the diversity and challenges of the real world.

[0197] In summary, the present invention is dedicated to building an innovative solution to fuse visual and tactile information to address the limitations of current technology in the field of object recognition. The core of the solution proposed by the present invention is to design a novel visual-tactile fusion network that particularly incorporates a spatial interaction learning attention mechanism, which cleverly solves the problem of effectively combining visual and tactile data due to significant differences. This unique fusion strategy not only significantly enhances the accuracy of object recognition, but also takes advantage of the efficient characteristics of DMDA-NET. This method not only improves the robustness of the recognition system, but also fully utilizes the miniaturization computing advantage of DMDA-NET, so that the entire model maintains high performance while reducing resource consumption.

[0198] Therefore, the visual-haptic fusion network of the present application relies on advanced spatial interactive attention learning technology, successfully breaking the barrier of visual and tactile data fusion, realizing higher level object recognition ability, and opening up a new path for the development of robot perception and interaction technology.

[0199] Finally, it should also be noted that the above enumeration is only several specific embodiments of the present application. Obviously, the present application is not limited to the above embodiments, but can also have many variations. All variations that can be directly derived or inferred from the disclosed content by those of ordinary skill in the art should be considered within the scope of the present application.

[0200] [1] URL: https: / / download.pytorch.org / models / alexnet-owt-4df8aa71.pth ;

[0201] [2] Bonner, L. E. R., Buhl, D. D., Kristensen, K., & Navarro-Guerrero, N. (2021). A U Dataset for Visuo-Haptic Object Recognition for Robots. figshare. https: / / doi.org / 10.6084 / m9.figshare .14222486.

Claims

1. An item recognition method based on bimodal dual-channel attention, characterized in that: The visual data and the tactile data of the object are collected, the visual data is preprocessed, and the tactile data is standardized, and then the visual and tactile fusion network trained offline in two stages is inputted; The visual-haptic fusion network comprises a feature extraction network, the preprocessed visual data and the standardized haptic data are respectively subjected to the feature extraction network to obtain visual feature maps and haptic feature maps The visual feature maps are subjected to visual channel attention extraction, the haptic feature maps are subjected to haptic channel attention extraction, and then spatial channel attention fusion is performed to obtain an object recognition result; The specific process of the spatial channel attention fusion is as follows: normalizing the visual channel attention feature V fused and the tactile channel attention feature T fused to obtain the normalized visual channel attention feature V' and the normalized tactile channel attention feature T', respectively; The visual channel attention feature V' is encoded by a two-dimensional global average pooling visual global spatial information: wherein, For feature V agp,bc Global output is obtained by using the natural nonlinear function Softmax of two-dimensional Gaussian mapping to get V softmax And the tactile channel attention T' matrix dot product: Z TV = T' · V softmax (11) The tactile channel attention feature T' is reweighted and standardized, and then encoded by a two-dimensional global average pooling, and then matrix multiplication is performed with the visual channel attention feature V': Z VT = V' · T softmax (12) wherein T softmax represents the features of the tactile channel attention feature T' after global pooling and softmax operation; Then, the visual spatial descriptor Z fused with the tactile channel information VT The tactile spatial descriptor Z fused with the visual channel information TV The addition operation is performed, and then a nonlinear Sigmoid function is applied to generate the spatial channel weight W; the spatial channel weight W is respectively applied to the original visual feature map of the feature group and the tactile feature map The feature modulation is performed, the feature is reweighted through element-level multiplication operation, the adjusted visual feature V" and tactile feature T" are obtained, and the visual feature V" and the tactile feature T" are added in matrix to obtain the fused feature F fusion ; the fused feature F fusion is output through the full connection layer to output the predicted classification result.

2. The object recognition method based on the dual-modal dual-channel attention according to claim 1, wherein: The feature extraction network is an AlexNet network.

3. The object recognition method based on the dual-modal dual-channel attention according to claim 2, wherein: The specific process of the visual channel attention extraction is as follows: (1) the visual feature map is divided into two parallel feature vectors v grouped and decomposed into two parallel feature vectors v c and x c ; (2) Global average pooling operation: where v c represents the input features of the channel along the horizontal dimension at height H, C represents the number of input channels, and H and W represent the spatial learning rates of the input features, respectively; wherein x c represents an input feature along the vertical dimension direction at the width W; (3) After the standardization of the gray value of each channel of the pre-processed visual data, the gray feature map is obtained through the GLCM module, and then the visual feature map is obtained through the AlexNet network concatenate the visual matrix feature maps with the visual feature maps and obtain channel descriptors V' by convolution layers glcm ; (4) the feature encoding vector and the feature encoding vector are merged along the spatial dimension, and a set of shared convolution kernels are used to implement convolution operation on the concatenated feature vector; subsequently, the concatenated feature vector is split into two independent feature vectors along the spatial direction, and is divided into two parallel branches, and the two parallel branches are fed into a nonlinear Sigmoid function to generate adaptive weight representations; at the same time, the channel descriptor V′ glcm is also fed into the same Sigmoid function to generate another set of adaptive weight representations; the three sets of parallel adaptive weight representations are used to implement Re-weight operation on the original feature map, and finally the fused visual channel attention feature V fused is obtained.

4. The object recognition method based on the dual-modal dual-channel attention according to claim 3, wherein: The specific process of the tactile channel attention extraction is as follows: (1) Average pooling and maximum pooling The haptic signature map is generated After feature grouping, max-pooling and average-pooling layers are applied respectively: where T(i,j,c) represents the value of the input tactile feature map at position (i,j) and channel c, O max (c) is the max-pooling output of the c-th channel; where O avg (c) is the average pooling output of the cth channel. (2) O max (c) and O avg (c) by a fully connected layer with a shared parameter and matrix addition: T fused = W - (O max (c) + O avg (c)) + b (9) where W and b are the weight matrix and bias vector of the fully connected layer, respectively, T fused is the fused tactile channel attention feature vector.

5. The object recognition method based on the dual-modal dual-channel attention according to claim 4, wherein: The specific process of the two-stage offline training is as follows: (1) Offline training on the dataset OBJECTFOLDER2.0: The samples of the visual data in the dataset OBJECTFOLDER2.0 are subjected to data enhancement operations such as random cropping, image scaling, color space disturbance and horizontal flipping, the samples of the tactile data are converted into RGB images by rendering technology, and then the visual and tactile fusion network is trained and tested in the first stage, the cross-entropy loss function and the Adam optimizer are adopted, the parameters in the network are updated by the back propagation algorithm, and the first-stage optimal network weight of the visual and tactile fusion network is obtained; (2) Offline training on the dataset AUDataset The visual images of each sample in the dataset AUDataset are expanded under different light conditions, and then the visual and tactile fusion network trained in the first stage is trained and tested in the second stage, the cross-entropy loss function and the Adam optimizer are adopted, the parameters in the network are updated by the back propagation algorithm, and the second-stage optimal network weight of the visual and tactile fusion network is obtained; The visual and tactile fusion network trained in the second stage is used as an online available network.

Images