Visual touch fusion object material identification method and device based on channel switching

Abstract

The invention discloses a visual touch fusion object material identification method and device based on channel switching, and the method comprises the steps: obtaining a paired data sample which comprises an object and is composed of a visual image and a touch image, carrying out the preprocessing of the paired data sample, and constructing a training data set; constructing a visual touch fusion object material identification network, wherein the visual touch fusion object material identification network comprises a visual feature extraction sub-network, a touch feature extraction sub-network, a channel switching gating fusion module, a space attention module, a feature pyramid fusion module and a classification module; training the visual touch fusion object material identification network by using the training data set; and deploying the trained visual touch fusion object material identification network to a device end, and performing object material identification by using the trained visual touch fusion object material identification network to obtain an identification result. According to the invention, on the basis of the lightweight design of the model, the substance identification accuracy is obviously improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot perception and multi-modal information processing, and particularly relates to a visual-tactile fusion object material recognition method and device based on channel switching. BACKGROUND

[0002] Object material recognition is one of the important tasks in robot perception and physical interaction. At present, it mainly relies on single modal information of vision or touch for classification: the method based on vision uses the color, texture and reflectivity characteristics of camera image to judge the object material, while the method based on touch perceives the mechanical feedback (such as hardness, roughness, temperature change, etc.) of the object surface through the touch sensor to infer the material type. However, there are obvious limitations in using single modal: visual perception is easily affected by light changes, occlusions and object appearance similarity, for example, different materials may have similar appearances, which leads to visual misjudgment; tactile perception is limited by the contact range and force of the sensor, if the sensor does not contact the key part or the contact is not sufficient, it is difficult to fully judge the material by relying solely on tactile information. In addition, the data dimensions and characteristics of visual and tactile modalities are different, and direct simple splicing and fusion of the two cannot fully play their respective advantages: for example, vision provides global appearance information and touch provides local details, and the misalignment of the two information will lead to redundancy or conflict; different modalities are different in the importance of material discrimination in certain situations, and fixed fusion methods may overweight the wrong modal information in some cases. Some existing research works try to use deep learning to fuse visual and tactile data, for example, a double-branch neural network is constructed to process vision and touch respectively and merge the feature vectors for classification in the later stage. But most of the traditional fusion methods give fixed or experience set weights to each modality when merging, lack of dynamic adjustment strategy according to the input situation, and rarely explicitly focus on the importance of key channel features or spatial contact area, so the recognition performance is still not ideal in complex scenes.

[0003] In recent years, attention mechanism has made significant progress in the field of computer vision, and has been proven to effectively improve the relevance and discriminability of feature expression. Among them, channel attention enhances important features and suppresses secondary features by learning the weights of each feature channel. Typical methods such as the "Squeeze-and-Excitation" mechanism proposed by SENet and its improved form ECA-Net (Efficient Channel Attention) propose to use one-dimensional convolution to efficiently model channel correlation. Spatial attention focuses on different positions in the feature map and is used to highlight target regions and suppress background noise in tasks such as semantic segmentation and object detection. At the same time, multi-scale feature fusion (such as Feature Pyramid Network FPN) is widely used in detection and recognition tasks to balance high-level semantics and low-level detail information. However, the above technologies are rarely applied to the field of material identification in visual-haptic modal fusion. Therefore, how to comprehensively use attention mechanism and multi-scale features to improve the multi-modal fusion method, so that the model can adaptively assign modal weights and highlight key areas according to the input, to overcome the shortcomings of existing methods and improve the material classification accuracy and robustness in complex environments, is still a technical problem to be solved in the field. SUMMARY

[0004] The application provides a visual-haptic fusion object material identification method and device based on channel switching to solve the technical problems mentioned in the background art.

[0005] To achieve the above purpose, the technical scheme of the application is as follows:

[0006] The application provides a visual-haptic fusion object material identification method based on channel switching, comprising the following steps:

[0007] S1, obtaining a pair of data samples containing an object composed of a visual image and a haptic image, preprocessing the pair of data samples, and constructing a training data set;

[0008] S2, constructing a visual-haptic fusion object material identification network, the visual-haptic fusion object material identification network comprising a visual feature extraction subnetwork, a haptic feature extraction subnetwork, a channel switching gate fusion module, a spatial attention module, a feature pyramid fusion module, and a classification module;

[0009] The visual feature extraction subnetwork and the haptic feature extraction subnetwork are connected in parallel and then connected with the spatial attention module, the feature pyramid fusion module, and the classification module in sequence;

[0010] S3, training the visual-haptic fusion object material identification network using the training data set to obtain a trained visual-haptic fusion object material identification network;

[0011] S4, deploy the trained visual-haptic fusion object material recognition network to the device end, and use the trained visual-haptic fusion object material recognition network to perform object material recognition to obtain a recognition result.

[0012] The application also provides a visual-haptic fusion object material recognition device for performing object material recognition by using the visual-haptic fusion object material recognition method.

[0013] The simulation visual-haptic rendering generation module is configured to acquire visual images and haptic images of the object.

[0014] The feature extraction module includes a visual feature extraction subnetwork and a haptic feature extraction subnetwork in parallel, both of which adopt a ResNet-50 backbone network structure and are configured to extract visual multi-level features and haptic multi-level features from the visual images and the haptic images, respectively.

[0015] The channel switching gate fusion module is configured to receive the visual feature maps and the haptic feature maps and perform cross-modal channel replacement fusion based on attention weights of the channels.

[0016] The spatial attention module is configured to generate attention weights and adaptively fuse information of different modalities according to the attention weights to obtain visual final multi-level fusion features and haptic final multi-level fusion features.

[0017] The feature pyramid fusion module adopts a feature pyramid network (FPN) structure and is configured to fuse feature maps of different scales.

[0018] The classification module is configured to perform pooling and full connection operations on the fusion features output by the feature pyramid fusion module and output a material recognition result of the object.

[0019] The application has the following beneficial effects:

[0020] The application overcomes the limitations of existing visual-haptic fusion material recognition methods and provides a visual-haptic fusion object material recognition method based on channel switching, which internally discloses a visual-haptic fusion object material recognition network. The visual-haptic fusion object material recognition network realizes deep fusion of visual and haptic information through the internal channel switching gate fusion module and the spatial attention module, can dynamically balance the contributions of the two modalities to the recognition result for different objects and environments, and improves the extraction ability of the application for material features of different sizes and different detail levels by combining multi-scale fusion of the feature pyramid fusion module, thereby significantly improving the accuracy and robustness of the material recognition of the application in complex scenes and having important application value. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 FIG. 1 is a structural block diagram of a visual-haptic fusion object material recognition network in the application.

[0022] Figure 2 A flow chart of the object material quality identification method in the visual-haptic fusion is shown in the figure;

[0023] Figure 3 A visual image rendering schematic diagram in the embodiment of the present application is shown in the figure;

[0024] Figure 4 A haptic image rendering schematic diagram in the embodiment of the present application is shown in the figure;

[0025] Figure 5 An object visual image and haptic image instance schematic diagram based on a simulation platform constructed in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0026] In order to facilitate the understanding of the present application, the present application will be described in more detail below with reference to the relevant drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be realized in many other different forms, and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0027] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly specified.

[0028] Reference Figure 1 and Figure 2 The embodiment of the present application provides a visual-haptic fusion object material quality identification method based on channel switching, which comprises the following steps:

[0029] S1, obtaining a pair of data samples composed of visual images and haptic images containing objects by a multi-modal acquisition method based on a simulation platform, pre-processing the pair of data samples, and constructing a training data set;

[0030] S2, constructing a visual-haptic fusion object material quality identification network, the visual-haptic fusion object material quality identification network comprising a visual feature extraction sub-network, a haptic feature extraction sub-network, a channel switching gate fusion module, a spatial attention module, a feature pyramid fusion module and a classification module;

[0031] Among them, the visual feature extraction sub-network and the haptic feature extraction sub-network are connected in parallel, and then connected with the spatial attention module, the feature pyramid fusion module and the classification module in sequence;

[0032] S3, training the visual-haptic fusion object material recognition network using the training data set to obtain a trained visual-haptic fusion object material recognition network;

[0033] S4, deploying the trained visual-haptic fusion object material recognition network to the device end, and using the trained visual-haptic fusion object material recognition network for object material recognition to obtain a recognition result.

[0034] Specifically, the trained visual-haptic fusion object material recognition network is deployed to an actual application environment for online identification of object materials. The deployment form can be that a robot's on-board computing unit executes the model parameters trained offline, or the visual-haptic fusion object material recognition network can be integrated into the software of an industrial detection device. When a new object material needs to be identified, the current visual image and haptic image of the object are obtained as input and fed to the visual-haptic fusion object material recognition network. The visual-haptic fusion object material recognition network sequentially passes through a convolution feature extraction module, a channel gating fusion module, a spatial attention module, a feature pyramid fusion module (FPN) fusion module, and a classification module, and finally outputs the predicted material category of the object. For example, in a robot grasping scene, the camera of the mechanical hand obtains the surface image of the object, and the haptic sensor records the pressure distribution graph at the moment of contacting the object. After inputting the two kinds of data into the model, the system gives a real-time identification result of the object material (such as metal, wood, plastic, fabric, etc.). The identification result can be used to guide the subsequent actions of the robot (such as adjusting the holding force, selecting a different placement area, etc.), and can also be fed back to the human-machine interaction interface for reference by the operator.

[0035] In some embodiments, S1 specifically includes the following steps:

[0036] S11, setting the positions of the camera and the light source in the three-dimensional space to obtain visual images of the object under different viewing angles and illuminations; the visual images reflect the global appearance, color, texture, and other visual properties of the object; the rendering process and schematic diagram of the visual image are shown in Figure 3 and Figure 5

[0037] S12, modeling the contact process on the surface of the object based on the principle of optical haptic sensing, and generating a two-dimensional haptic image through multi-dimensional contact vector input;

[0038] ​Specifically, first, the contact process of the object surface is modeled based on the optical haptic sensing principle of the haptic rendering network, and the rendering parameters include the three-dimensional coordinates of the contact point, the contact direction, the contact depth, the width and height of the local haptic image, forming a multi-dimensional contact vector, and then the multi-contact vector is mapped to the pixel value of the local haptic image, and the image is rendered into a simulated haptic image through the Taxim engine, and the haptic image reflects the microscopic deformation, friction texture and local hardness response of the material; the rendering process and schematic diagram of the haptic image are shown in Figure 4 and Figure 5 ;

[0039] S13, the visual image and the haptic image of the same object are taken as a pair of data samples, and then the pair of data samples is preprocessed, and the preprocessing at least includes one or more of image size unification, (color equalization or) brightness normalization processing, denoising operation and image normalization coding (such as normalization to [0, 1] or [-1, 1] interval);

[0040] Among them, the image size unification is used to unify the image size to a preset standard size (such as 224x224 pixels); the color equalization or brightness normalization processing is used to suppress the influence of light change; the denoising operation is used to remove the background interference texture;

[0041] S14, a training data set is constructed by using the preprocessed pair of data samples.

[0042] In some embodiments, the visual feature extraction sub-network and the haptic feature extraction sub-network both adopt a residual network ResNet-50 as a backbone structure, and the two have the same hierarchical structure, but the network parameters of the two are independent of each other and are not shared; the visual feature extraction sub-network and the haptic feature extraction sub-network respectively output visual multi-level features and haptic multi-level features;

[0043] The channel switching gate fusion module sequentially includes a channel attention mechanism and a channel switching module, wherein the channel attention mechanism is used to generate attention weights for each channel; the channel attention mechanism first performs channel average pooling and channel maximum pooling on the visual multi-level features and the haptic multi-level features, then splices the obtained visual feature map and the haptic feature map to generate a spatial attention map through a 1x1 convolution layer and a Sigmoid activation function, so as to strengthen the significant area in the feature map in a way without additional learnable parameters;

[0044] The feature pyramid fusion module includes three parallel convolution layers and two up-sampling layers, which are used to integrate the fusion features of different scales from top to bottom;

[0045] The classification module includes a pooling layer and a fully connected layer, which are used to perform final material classification on the fused features.

[0046] In some embodiments, the S3 specifically comprises the following steps:

[0047] S31, selecting a pair of data samples from the training data set, containing a visual image and a tactile image, inputting the visual image into a visual feature extraction subnetwork; the visual feature extraction subnetwork adopts ResNet-50 as the backbone network, extracts multi-level features of the visual image through the first three residual blocks of the backbone network, obtains visual multi-level features, and the visual multi-level features include visual high-level features, visual middle-level features and visual low-level features corresponding to different spatial resolutions;

[0048] S32, inputting the pair of data samples selected in S31 into a tactile feature extraction subnetwork to extract tactile multi-level features, and the tactile multi-level features include tactile high-level features, tactile middle-level features and tactile low-level features;

[0049] S33, a channel switching gate fusion module generates attention weights ω of each channel according to the visual multi-level features and the tactile multi-level features m , then performs cross-modal exchange on the channels with importance lower than the set threshold according to the weights of the channels, replaces and fuses according to the set cross-modal information injection ratio λ, and the channels with importance higher than the set threshold retain the original modal features, thereby obtaining visual multi-level fusion features and tactile multi-level fusion features;

[0050] S34, inputting the visual multi-level fusion features and the tactile multi-level fusion features obtained in S33 into a spatial attention module to adaptively fuse information of different modalities, and obtaining visual final multi-level fusion features and tactile final multi-level fusion features;

[0051] S35, inputting the visual final multi-level fusion features and the tactile final multi-level fusion features generated in S34 into a feature pyramid fusion module, and the feature pyramid fusion module adopts a feature pyramid network (FPN) structure, integrates the fusion features of different scales from top to bottom through multi-layer convolution and up-sampling operation, and outputs the fused multi-scale feature maps;

[0052] S36, inputting the fused multi-scale feature maps into a classification module to obtain a predicted label

[0053] S37, calculating a cross-entropy loss according to the predicted label and the real label;

[0054] S38, iterating S31 to S37, and in the process of iteration, adjusting the weight parameters of the visual-tactile fusion object material recognition network through back propagation to minimize the cross-entropy loss, and obtaining the trained visual-tactile fusion object material recognition network.

[0055] In some embodiments, the channel attention mechanism uses the Efficient Channel Attention (ECA) module, which includes a Global Average Pooling (GAP) layer, a one-dimensional convolutional layer, and a Sigmoid activation function connected in sequence.

[0056] In some embodiments, S33 specifically includes the following steps:

[0057] S331. Input the visual multi-level features and tactile multi-level features into the efficient channel attention ECA module, perform global average pooling on each feature map in the channel dimension, and predict the attention weight ω of each channel through one-dimensional convolution. m The calculation formula is as follows:

[0058] ω m =σ(f(GAP(X) m )))

[0059] in, This represents the feature map extracted from modality m, i.e., visual multi-level features or tactile multi-level features; modality represents vision or tactile. Let f(·) represent the set of real numbers; f(·) is a one-dimensional convolutional layer; σ(·) represents the sigmoid activation function; GAP(.) represents the global average pooling operation of the global average pooling layer GAP, calculated as follows:

[0060]

[0061] Where H and W represent the height and width of the feature map, respectively; i and j represent the height and width of the current feature map, respectively; X ij This represents a feature map with height i and width j.

[0062] S332, assign attention weight ω to each channel. m The input is fed into the channel switching module for gating and replacement operations to complete the visual and tactile information, resulting in visual multi-level fusion features and tactile multi-level fusion features. The expression for the channel switching mechanism is as follows:

[0063]

[0064] Among them, X m，c This represents the feature map on the c-th channel of mode m, when ω m,c When k ≥ k, the channel characteristics of this mode are preserved; X m′，c Let m' represent another mode m' opposite to mode m. If mode m is visual, then mode m' is tactile, and vice versa. The feature map on the c-th channel, when ω... m,cWhen ω < k, the channel features of the current mode are replaced with features from another mode; k is a set threshold; ω m，c The attention weight ω represents the c-th channel. m ;

[0065] Among them, visual multi-level fusion features include high-level visual fusion features, mid-level visual fusion features, and low-level visual fusion features; tactile multi-level fusion features include high-level tactile fusion features, mid-level tactile fusion features, and low-level tactile fusion features.

[0066] The channel switching gating fusion module in this invention plays a crucial role in multimodal fusion, enhancing the discriminative power and information richness of feature representation. This module not only helps extract shared information between modalities but also preserves the important features unique to each modality. Without introducing additional parameters, the module replaces weak, redundant channels in the current modality with semantically consistent, higher-contributing channels in another modality, thereby constructing a heterogeneous and complementary fusion channel group. Its core idea is to dynamically adjust the channel information of different modalities based on channel importance.

[0067] In some embodiments, S34 specifically includes the following steps:

[0068] S34 specifically includes the following steps:

[0069] S341. Input the visual multi-level fusion features and the tactile multi-level fusion features into the spatial attention module. The spatial attention module concatenates the visual multi-level fusion features and the tactile multi-level fusion features at the same level to obtain the fusion features.

[0070] S342, Integrating Features The inputs are respectively fed into the channel average pooling layer (CAP) and the channel max pooling layer (CMP) and fused with features. Element-wise multiplication yields visual secondary multi-level fusion features. and tactile secondary multi-level fusion features The specific formula is as follows:

[0071]

[0072] in, This indicates element-wise multiplication; Represents visual secondary multi-level fusion features Or tactile secondary multi-level fusion features

[0073] The formulas for calculating the channel average pooling layer (CAP) and the channel maximum pooling layer (CMP) are as follows:

[0074]

[0075] Among them, X ij This represents a feature map with height i, width j, and number of channels c; C represents the total number of channels.

[0076] S343, Visual secondary multi-layer fusion features and tactile secondary multi-layer fusion features The input is fed into the adaptive modal fusion mechanism to achieve on-demand modal switching, resulting in the final multi-level fusion features of vision and tactile sensation. The specific formulas are as follows:

[0077]

[0078] Among them, X fused This represents the final multi-level fusion feature of the visual modality or the final multi-level fusion feature of the tactile modality; 'a' represents the weight of the visual modality; 1-a is the supplementary weight of the tactile modality; the formula for calculating the weight 'a' of the visual modality is as follows:

[0079]

[0080] Where [·] represents the concatenation of feature vectors, and MLP(.) represents a gated network containing two fully connected layers.

[0081] The spatial attention module in this invention aims to highlight spatial regions containing discriminative information in the feature map, thereby enhancing the ability of the visual-touch fusion object material recognition network to focus on local structures. The spatial attention module further improves the network's response to key regions, contributing to the construction of more compact and discriminative feature representations. To maintain the lightweight nature of the fusion structure, the spatial attention module is based on a parameter-free spatial compression design. The module utilizes two basic operations: Channel Average Pooling (CAP) and Channel Max Pooling (CMP), to extract cross-channel spatial response patterns from the fused features without introducing additional learnable parameters. CAP reflects the overall response trend of the feature map, while CMP captures the strongest local response regions; the two complement each other and collaboratively generate a discriminative spatial attention map.

[0082] In some embodiments, S35 specifically includes the following steps:

[0083] S351. Input the final multi-level fusion features of vision and the final multi-level fusion features of touch into the feature pyramid fusion module;

[0084] S352. Within the Feature Pyramid Fusion module, a Feature Pyramid Network (FPN) structure is used for multi-scale feature fusion: the three convolutional layers within the FPN structure process fusion features at different scales, and two bilinear interpolation upsampling layers upsample the mid-level and low-level features to the high-level dimensions, respectively, and add them to the high-level features to finally generate the output fused feature map P. i The calculation formula is as follows:

[0085] P i =Conv1×1(C i )+Up(C i+1 )

[0086] Among them, C i With C i+1 X is the final multi-level fusion feature of vision or the final multi-level fusion feature of touch. fused The two different levels of features; the multi-level fusion features of the input visual and tactile senses. For the output features of the feature pyramid fusion module at the i-th layer, Conv 1×1 (·) indicates a linear transformation operation using a 1×1 convolutional layer, and Up(·) indicates an upsampling layer implemented using bilinear interpolation.

[0087] In some embodiments, the calculation formula for the cross-entropy loss in S38 is as follows:

[0088]

[0089] Where L represents the cross-entropy loss; mu represents the mu-th material; MU represents the total number of categories; y represents the probability of classifying it as material of type mu; ture，mu This indicates the actual label for the mu-type material.

[0090] This invention overcomes the limitations of existing visual-tactile fusion material recognition methods and provides a visual-tactile fusion object material recognition method based on channel switching. Internally, it discloses a visual-tactile fusion object material recognition network. This network achieves deep fusion of visual and tactile information through an internal channel switching gating fusion module and a spatial attention module, enabling it to dynamically balance the contributions of the two modalities to the recognition results for different objects and environments. Simultaneously, by combining multi-scale fusion with a feature pyramid fusion module, it improves the invention's ability to extract material features of different sizes and levels of detail, thereby significantly enhancing the accuracy and robustness of material recognition in complex scenes and possessing significant application value.

[0091] To verify the effectiveness of the present invention, the following two experiments were specifically conducted:

[0092] Experiment 1: Performance Comparison Experiment of the Invention in Single-Mode and Multi-Mode

[0093] This embodiment conducts comparative experiments on the publicly available dataset ObjectFolder 2.0 to compare the performance of the present invention in both unimodal and multimodal modes. The classification performance of existing visual models, tactile models, and the present invention was evaluated on seven material categories (ceramics, glass, wood, plastic, iron, polycarbonate, and steel) in ObjectFolder 2.0. The main quantitative evaluation metrics (including accuracy, precision, and recall for each category) are summarized in Table 1.

[0094] Table 1: Performance comparison of existing visual models, tactile models and the classification effect of the present invention;

[0095]

[0096] Experiment 2: Performance Comparison with Existing Methods

[0097] This embodiment conducts comparative experiments on the publicly available dataset ObjectFolder 2.0 to evaluate the accuracy of the present invention against existing representative methods. The comparison methods include multimodal deep fusion models such as DLTU, DLSMC, MAMUDA, and VITO-former. The experiments are performed on seven typical material categories (ceramics, glass, wood, plastic, iron, foam, and plaster) for recognition tasks. Evaluation metrics include accuracy, precision, and F1 score. The experimental results are shown in Table 2.

[0098] Table 2: Performance comparison between the present invention and existing representative methods;

[0099]

[0100]

[0101] As shown in Table 2, this invention outperforms existing methods in all three dimensions: accuracy, precision, and F1 score, achieving higher recognition accuracy in material recognition tasks. Specifically, it achieves 92.25% accuracy, 92.32% precision, and 93.22% F1 score on the ObjectFolder2.0 dataset, significantly outperforming state-of-the-art methods such as VTS and VITO-former.

[0102] In another aspect, the present invention provides a visual-touch fusion object material recognition device, which uses the above-described visual-touch fusion object material recognition method to recognize object materials, including:

[0103] The simulation visual and tactile rendering generation module is used to acquire visual and tactile images of objects.

[0104] The feature extraction module includes a parallel visual feature extraction sub-network and a tactile feature extraction sub-network, both of which adopt the ResNet-50 backbone network structure to extract visual multi-level features and tactile multi-level features from visual images and tactile images, respectively.

[0105] The channel switching gating fusion module is used to receive the visual feature map and tactile feature map, and perform cross-modal channel replacement fusion based on the attention weight of each channel;

[0106] The spatial attention module is used to generate attention weights and adaptively fuse information from different modalities based on the attention weights to enhance the feature representation of key regions, thereby obtaining the final multi-level fusion features of vision and the final multi-level fusion features of touch.

[0107] The Feature Pyramid Fusion Module employs a Feature Pyramid Network (FPN) structure to fuse feature maps of different scales.

[0108] The classification module performs pooling and fully connected operations on the fused features output by the feature pyramid fusion module to output the material recognition results of the object.

[0109] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for recognizing object material through visual-touch fusion based on channel switching, characterized in that, Includes the following steps: S1. Obtain paired data samples consisting of visual and tactile images of objects, preprocess the paired data samples, and construct the training dataset. S2. Construct a visual-touch fusion object material recognition network, which includes a visual feature extraction sub-network, a tactile feature extraction sub-network, a channel switching gating fusion module, a spatial attention module, a feature pyramid fusion module, and a classification module. Among them, the visual feature extraction subnetwork and the tactile feature extraction subnetwork are connected in parallel and then connected to the spatial attention module, the feature pyramid fusion module and the classification module in turn. S3. Train the visual-touch fusion object material recognition network using the training dataset to obtain the trained visual-touch fusion object material recognition network. S4. Deploy the trained visual-touch fusion object material recognition network to the device and use the trained visual-touch fusion object material recognition network to perform object material recognition and obtain the recognition results.

2. The method for object material recognition based on channel switching and visual-touch fusion according to claim 1, characterized in that, S1 specifically includes the following steps: S11. Set the positions of the camera and light source in three-dimensional space to obtain visual images of the object under different viewpoints and lighting conditions; S12. The contact process of the object surface is modeled based on the principle of optical tactile sensing. A two-dimensional tactile image is generated by mapping through multi-dimensional contact vector input. S13. Take the visual image and tactile image of the same object as a pair of data samples, and then preprocess the pair of data samples. The preprocessing includes at least one or more of the following: image size unification, brightness normalization, noise reduction, and image normalization coding. S14. Use the preprocessed paired data samples to construct the training dataset.

3. The method for object material recognition based on channel switching and visual-touch fusion according to claim 1, characterized in that, Both the visual feature extraction subnetwork and the tactile feature extraction subnetwork use ResNet-50 as the backbone structure. They have the same hierarchical structure, but their network parameters are independent and do not share each other. The visual feature extraction subnetwork and the tactile feature extraction subnetwork output visual multi-level features and tactile multi-level features, respectively. The channel switching gating fusion module includes a channel attention mechanism and a channel switching module. The channel attention mechanism is used to generate attention weights for each channel. The channel attention mechanism first performs channel average pooling and channel max pooling on the visual multi-level features and tactile multi-level features. Then, the obtained visual feature map and tactile feature map are concatenated and then a spatial attention map is generated through a 1×1 convolutional layer and a Sigmoid activation function. The feature pyramid fusion module includes three parallel convolutional layers and two upsampling layers, which are used to integrate fusion features of different scales from top to bottom. The classification module includes a pooling layer and a fully connected layer, which are used to perform the final material classification on the fused features.

4. The method for object material recognition based on channel switching and visual-touch fusion according to claim 3, characterized in that, S3 specifically includes the following steps: S31. Select a pair of data samples from the training dataset, including visual images and tactile images, and input the visual images into the visual feature extraction sub-network. The visual feature extraction sub-network uses ResNet-50 as the backbone network and extracts multi-level features of the visual images through the first three levels of residual blocks of the backbone network to obtain multi-level visual features. The multi-level visual features include high-level visual features, mid-level visual features, and low-level visual features, corresponding to different spatial resolutions. S32. Input the paired data samples selected in S31 into the tactile feature extraction sub-network to extract tactile multi-level features, which include high-level tactile features, mid-level tactile features and low-level tactile features. S33, the channel switching gating fusion module generates the attention weight ω for each channel based on visual multi-level features and tactile multi-level features. m Then, based on the weight of each channel, cross-modal exchange is performed on channels whose importance is lower than a set threshold. During replacement, fusion is performed according to the set cross-modal information injection ratio λ. Channels whose importance is higher than the set threshold retain the original modal features, thereby obtaining visual multi-level fusion features and tactile multi-level fusion features. S34. Input the visual multi-level fusion features and tactile multi-level fusion features obtained in S33 into the spatial attention module, and adaptively fuse information from different modalities to obtain the final visual multi-level fusion features and the final tactile multi-level fusion features. S35. Input the final visual multi-level fusion features and the final tactile multi-level fusion features generated in S34 into the feature pyramid fusion module. The feature pyramid fusion module adopts the feature pyramid network FPN structure and integrates the fusion features of different scales from top to bottom through multi-layer convolution and upsampling operations, and outputs the fused multi-scale feature map. S36. Input the fused multi-scale feature map into the classification module to obtain the predicted label. S37, Based on predicted labels Calculate the cross-entropy loss with the real labels; S38. Iterate from S31 to S37. During the iteration process, adjust the weight parameters of the visual-touch fusion object material recognition network through backpropagation to minimize the cross-entropy loss and obtain the trained visual-touch fusion object material recognition network.

5. The method for object material recognition based on channel switching and visual-touch fusion according to claim 4, characterized in that, The channel attention mechanism uses an efficient channel attention (ECA) module, which includes a global average pooling layer (GAP), a one-dimensional convolutional layer, and a sigmoid activation function connected in sequence.

6. The method for object material recognition based on channel switching in visual-touch fusion according to claim 5, characterized in that, S33 specifically includes the following steps: S331. Input the visual multi-level features and tactile multi-level features into the efficient channel attention ECA module, perform global average pooling on each feature map in the channel dimension, and predict the attention weight ω of each channel through one-dimensional convolution. m The calculation formula is as follows: oh m =σ(f(GAP(X m ))) in, This represents the feature map extracted from modality m, i.e., visual multi-level features or tactile multi-level features; modality represents vision or tactile. Let f(·) represent the set of real numbers; f(·) is a one-dimensional convolutional layer; σ(·) represents the sigmoid activation function; GAP(.) represents the global average pooling operation of the global average pooling layer GAP, calculated as follows: Where H and W represent the height and width of the feature map, respectively; i and j represent the height and width of the current feature map, respectively; X ij This represents a feature map with height i and width j. S332, assign attention weight ω to each channel. m The input is fed into the channel switching module for gating and replacement operations to complete the visual and tactile information, resulting in visual multi-level fusion features and tactile multi-level fusion features. The expression for the channel switching mechanism is as follows: Among them, X m，c This represents the feature map on the c-th channel of mode m, when ω m,c When k ≥ k, the channel characteristics of this mode are preserved; X m′，c Let m' represent another mode m' opposite to mode m. If mode m is visual, then mode m' is tactile, and vice versa. The feature map on the c-th channel, when ω... m，c When ω < k, the channel features of the current mode are replaced with features from another mode; k is a set threshold; ω m，c The attention weight ω represents the c-th channel. m ; Among them, visual multi-level fusion features include high-level visual fusion features, mid-level visual fusion features, and low-level visual fusion features; tactile multi-level fusion features include high-level tactile fusion features, mid-level tactile fusion features, and low-level tactile fusion features.

7. The method for object material recognition based on channel switching and visual-touch fusion according to claim 6, characterized in that, S34 specifically includes the following steps: S341. Input the visual multi-level fusion features and the tactile multi-level fusion features into the spatial attention module. The spatial attention module concatenates the visual multi-level fusion features and the tactile multi-level fusion features at the same level to obtain the fusion features. S342, Integrating Features The inputs are respectively fed into the channel average pooling layer (CAP) and the channel max pooling layer (CMP) and fused with features. Element-wise multiplication yields visual secondary multi-level fusion features. and tactile secondary multi-level fusion features The specific formula is as follows: in, This indicates element-wise multiplication; Represents visual secondary multi-level fusion features Or tactile secondary multi-level fusion features The formulas for calculating the channel average pooling layer (CAP) and the channel maximum pooling layer (CMP) are as follows: Among them, X ij This represents a feature map with height i, width j, and number of channels c; C represents the total number of channels. S343, Visual secondary multi-layer fusion features and tactile secondary multi-layer fusion features The input is fed into the adaptive modal fusion mechanism to achieve on-demand modal switching, resulting in the final multi-level fusion features of vision and tactile sensation. The specific formulas are as follows: Among them, X fused This represents the final multi-level fusion feature of the visual modality or the final multi-level fusion feature of the tactile modality; 'a' represents the weight of the visual modality; 1-a is the supplementary weight of the tactile modality; the formula for calculating the weight 'a' of the visual modality is as follows: Where [·] represents the concatenation of feature vectors, and MLP(.) represents a gated network containing two fully connected layers.

8. The method for object material recognition based on channel switching and visual-touch fusion according to claim 7, characterized in that, S35 specifically includes the following steps: S351. Input the final multi-level fusion features of vision and the final multi-level fusion features of touch into the feature pyramid fusion module; S352. Within the Feature Pyramid Fusion module, a Feature Pyramid Network (FPN) structure is used for multi-scale feature fusion: the three convolutional layers within the FPN structure process fusion features at different scales, and two bilinear interpolation upsampling layers upsample the mid-level and low-level features to the high-level dimensions, respectively, and add them to the high-level features to finally generate the output fused feature map P. i The calculation formula is as follows: P i =Conv1×1(C i )+Up(C i+1 ) Among them, C i With C i+1 X is the final multi-level fusion feature of vision or the final multi-level fusion feature of touch. fused Two different levels of features; For the output features of the feature pyramid fusion module at the i-th layer, Conv 1×1 (·) indicates a linear transformation operation using a 1×1 convolutional layer, and Up(·) indicates an upsampling layer implemented using bilinear interpolation.

9. The method for object material recognition based on channel switching in visual-touch fusion according to claim 8, characterized in that, The specific formula for calculating the cross-entropy loss in S38 is as follows: Where L represents the cross-entropy loss; mu represents the mu-th material; MU represents the total number of categories; y represents the probability of classifying it as material of type mu; ture，mu This indicates the actual label for the mu-type material.

10. A visual-touch fusion object material recognition device, characterized in that, The object material recognition method using the visual-touch fusion method according to any one of claims 1 to 9 includes: The simulation visual and tactile rendering generation module is used to acquire visual and tactile images of objects. The feature extraction module includes a parallel visual feature extraction sub-network and a tactile feature extraction sub-network, both of which adopt the ResNet-50 backbone network structure to extract visual multi-level features and tactile multi-level features from visual images and tactile images, respectively. The channel switching gating fusion module is used to receive the visual feature map and tactile feature map, and perform cross-modal channel replacement fusion based on the attention weight of each channel; The spatial attention module is used to generate attention weights and adaptively fuse information from different modalities based on the attention weights to obtain the final multi-level fusion features of vision and tactile sense. The Feature Pyramid Fusion Module employs a Feature Pyramid Network (FPN) structure to fuse feature maps of different scales. The classification module performs pooling and fully connected operations on the fused features output by the feature pyramid fusion module to output the material recognition results of the object.

Images