A multi-modal multi-task workshop target identification method
By constructing a multimodal, multi-task workshop target recognition network and adopting a ResNet50 backbone network and feature fusion module, parallel execution of target detection and instance segmentation in workshop scenes was achieved. This solved the problems of color-similar target recognition and limited computing resources, and achieved high-precision recognition results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI CHENGDIAN FUZHI TECH CO LTD
- Filing Date
- 2022-11-22
- Publication Date
- 2026-05-15
AI Technical Summary
Existing workshop scene target recognition technologies mainly adopt a single modality and single task mode, which makes it difficult to identify machine tool targets with similar colors, and it is difficult to perform target detection and instance segmentation tasks simultaneously under low computing resource conditions.
A multimodal, multi-task workshop target recognition network is constructed, employing two ResNet50 backbone networks and adding four fusion modules. Parallel execution of target detection and instance segmentation is achieved through feature fusion and feature sharing modules in the decoding module, and training is performed using a multi-task learning loss function.
It achieves accurate identification of color-similar targets in workshop scenes, with an accuracy rate of 87% for target detection and 81% for instance segmentation, and solves the problem of limited computing resources in existing technologies.
Smart Images
Figure CN115908848B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an image processing method, and more particularly to a multimodal, multi-task workshop target recognition method. Background Technology
[0002] Existing target recognition networks for workshop scenes mainly adopt a single backbone network structure. This backbone network extracts features from RGB images and feeds them into the decoding network for final result prediction. Its structure is shown in the attached diagram of the specification. Figure 1 As shown. Therefore, existing workshop scene target recognition technology mainly adopts a single-task mode, which uses the features input from the backbone network to perform single-task inference.
[0003] This results in the following drawbacks:
[0004] 1. Existing workshop scene target recognition technology mainly adopts a single modality, that is, it only uses RGB modal features for scene target recognition. However, there are a large number of machine tool targets with similar color and shape features in the workshop scene, which is difficult for recognition networks that only use RGB single modality to identify.
[0005] 2. Existing target recognition technologies for workshop scenes mainly adopt a single-task mode, which cannot simultaneously handle target detection and instance segmentation tasks in a workshop setting. To perform both tasks concurrently, inference needs to be performed on two networks simultaneously, which is difficult to meet the low computational resource requirements of a workshop environment. Summary of the Invention
[0006] The purpose of this invention is to provide a multimodal, multi-task workshop target recognition method that solves the above-mentioned problems and enables accurate identification of color-similar targets in a workshop scene and parallel execution of target detection and instance segmentation tasks in a workshop scene.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a multimodal, multi-task workshop target recognition method, comprising the following steps;
[0008] (1) Construct a sample dataset;
[0009] The workshop site was photographed and captured using a depth camera, and each photograph yielded a set of corresponding color and depth images.
[0010] The categories of targets are determined, including workers, lathes, and material handling robots;
[0011] On color and depth images, object detection and instance segmentation levels are labeled to obtain ground truth bounding boxes for object detection and ground truth instance masks for instance segmentation.
[0012] A set of labeled color and depth images is used as a data sample;
[0013] (2) Construct a multimodal, multi-task workshop target recognition network;
[0014] The multimodal, multi-task workshop target recognition network includes an encoding module and a decoding module;
[0015] The encoding module includes two ResNet50 backbone networks. The ResNet50 backbone network is divided into five stages from the input to the output, namely the first stage to the fifth stage, and outputs the first feature vector to the fifth feature vector respectively.
[0016] Two ResNet50 backbone networks are respectively input to labeled color images and labeled depth images. A fusion module is set between the two second stages, the two third stages, the two fourth stages, and the two fifth stages. From front to back, these are the first to the fourth fusion modules. The input of the first three fusion modules is connected to the output of the two previous stages. The output is split into two paths, which are summed with the output of the two previous stages and then sent to the two next stages.
[0017] The fourth fusion module has its input connected to the outputs of the two fifth stages, and its output is split into two paths, which are then fed into the decoding module.
[0018] The fusion module is used to fuse the two input feature vectors and output the result.
[0019] The decoding module is used to perform target detection and instance segmentation on the output of the encoding module, and output the target detection result and the instance segmentation result.
[0020] (3) Training of the multimodal, multi-task workshop target recognition network;
[0021] The data samples in the sample dataset are input into the multimodal multi-task workshop target recognition network for target detection and instance segmentation. In the first decoding branch, the expected output is the ground truth bounding box corresponding to the target in the data sample. In the second decoding branch, the expected output is the ground truth instance mask of the instance segmentation in the data sample. The training continues until the model converges.
[0022] (4) Multimodal, multi-task workshop target recognition network task recognition;
[0023] A set of color and depth images of the workshop to be tested are acquired and fed into a multimodal, multi-task workshop target recognition network, which outputs the target prediction bounding box and the prediction instance mask corresponding to the target.
[0024] Preferably, the fusion method of the fusion module is as follows:
[0025] (2.1) Color images are RGB C×H×W The depth image is Depth C×H×W ,in C , H and W These represent the number of channels, height, and width of the corresponding image;
[0026] (2.2) The color image and the depth image are stitched together according to the channel dimension to generate the first stitching feature. RGBD 2C×H×W Then, it is further divided into S sub-feature blocks according to the channel dimension, labeled as X0~X100~X200~X30~X40~X50~X60~X7 ... S-1 Each sub-feature block has a dimension of ;
[0027] (2.3) For each sub-feature block, perform convolution operations with different kernel sizes to obtain sub-feature vectors, where the first... i Sub-feature blocks X i Perform the convolution operation according to the following formula;
[0028] F i =Conv i ( X i ),
[0029] In the formula, F i for X i The corresponding sub-feature vector, Conv This represents the convolution operation. i =0~ S ;
[0030] (2.4) Perform global average pooling on the S sub-feature vectors to reduce them to a size of S×1×1, resulting in S weight vectors;
[0031] (2.5) Normalize the S weight vectors to obtain S attention vectors;
[0032] (2.6) Finally, the obtained attention vector is used in conjunction with the first concatenated feature. RGBD Perform element-wise multiplication to obtain the fused features.
[0033] Preferably, the decoding module includes a first decoding branch for target detection and a second decoding branch for instance segmentation;
[0034] The first decoding branch includes a first decoding layer, a second decoding layer, a third decoding layer and an object detection head set in sequence. The first three layers are used to perform upsampling operations, and each time an object detection feature is output. The size of the object detection feature output by the third decoding layer is 1 / 4 of the color image. The object detection head first upsamples the output of the third decoding layer and then predicts the bounding box to obtain the object detection result.
[0035] The second decoding branch includes a first decoding layer, a second decoding layer, a third decoding layer and an instance segmentation head set in sequence. The first three layers are used to perform upsampling operations, and each time a target detection feature is output. The instance segmentation feature output by the third decoding layer is 1 / 4 the size of the color image. The instance segmentation head first upsamples the instance segmentation feature output by the third decoding layer and then predicts the instance mask to obtain the instance segmentation result.
[0036] A feature sharing module is provided between the two first decoding layers;
[0037] The feature sharing module is used to input target detection features and instance segmentation features, concatenate them according to the channel dimension to obtain a second concatenated feature, and then divide the second concatenated feature into two paths: one path is spatially pooled to obtain a spatial attention vector, and the other path is channel pooled to obtain a channel attention vector.
[0038] The spatial attention vector and the channel attention vector are element-wise multiplied with the second concatenation feature, and then the results of the element-wise multiplication are added together to obtain the processed second concatenation feature.
[0039] The processed second concatenated features are split according to the channel dimension to obtain processed target detection features and processed instance segmentation features, which are then sent back to the first decoding layer of the first decoding branch and the first decoding layer of the second decoding branch, respectively, and then sent to the next process by the two first decoding layers.
[0040] Feature sharing modules are also provided between the two second decoding layers and between the two third decoding layers.
[0041] Preferably, the depth camera is an Intel RealSense D455 RGBD camera with a sampling resolution of 640x480.
[0042] Regarding the ResNet50 backbone network: The ResNet50 backbone network consists of five stages, namely stages one through five, also known as layer 0, layer 1, layer 2, layer 3, and layer 4. Stage one, layer 0, does not contain residual blocks and mainly performs convolution, regularization, activation function, and max pooling calculations on the input. The other four stages all contain residual blocks. Each stage corresponds to an output feature map or feature vector.
[0043] In this invention, two ResNet50 backbone networks are used in the encoding module, and four fusion modules are added between the two ResNet50 backbone networks. This is because, due to the large differences in target scale in the workshop scene, using convolutional kernels of the same size on the feature map may ignore the details of small targets, or the receptive field of the convolutional kernel may not be able to capture all the information of large targets. Therefore, referring to the idea of the ESPANet network, the input features are split into multiple sub-feature blocks according to the channel dimension, and then convolutional kernels of different sizes are used to extract features from these sub-feature blocks to obtain attention vectors.
[0044] Compared with the prior art, the advantages of the present invention are as follows:
[0045] In the encoding module, two ResNet50 backbone networks are used, and four fusion modules are added. The second to fifth stages of the ResNet50 backbone network are four downsampling stages. After feature extraction in these four stages, the two types of extracted features are fused and corrected using the fusion modules. This module uses channel attention to highlight its own representative features and suppress noise in the data after inputting color image features and depth image features.
[0046] The fusion module splits the input features into multiple sub-feature blocks according to the channel dimension, and then uses convolution kernels of different sizes to extract features from these sub-feature blocks, thereby making it more adaptable to multi-scale targets.
[0047] In the decoding module, two branches perform parallel inference, simultaneously implementing the scene target detection task and the instance segmentation task. Specifically, in the decoding stages of each branch, the target detection features and instance segmentation features are passed to the feature sharing module at the first, second, and third decoding layers, respectively, enabling mutual optimization between the tasks.
[0048] The decoding module also includes three feature-sharing modules between the first and second decoding branches. Object detection features and instance segmentation features are first concatenated along the channel dimension, and then feature pooling is performed on the concatenated features in both the spatial and channel dimensions to obtain spatial attention vectors and channel attention vectors. Next, the spatial and channel attention vectors are used to highlight representative features at the spatial and channel levels, respectively, while suppressing noise. Finally, the highlighted features from the channel and spatial levels are merged using element-wise summation and then split according to modality to complete the sharing of the two types of features.
[0049] Because a multi-task learning method is used, two types of loss values are generated: object detection and image segmentation. At the same time, due to the differences between tasks, the prediction outputs of each task will have homoscedastic uncertainty. Therefore, a multi-task learning loss function method is adopted to learn regression and classification problems of different scales and quantities at the same time.
[0050] In summary, this invention proposes a novel backbone network and utilizes an attention mechanism for feature fusion. It also proposes a multi-task network for simultaneous instance segmentation and target detection, and designs a feature sharing module to achieve information sharing between the target detection decoding branch and the instance segmentation decoding branch. Ultimately, this invention demonstrates good recognition accuracy for color-similar targets in a workshop scene, achieving instance segmentation and target detection within the same scene. Specifically, it achieves an accuracy of 87% for target detection and 81% for instance segmentation in a workshop scene. Attached Figure Description
[0051] Figure 1 This is a flowchart of the present invention;
[0052] Figure 2 This is a schematic diagram of the fusion module;
[0053] Figure 3 This is a schematic diagram of the decoding module;
[0054] Figure 4 This is a schematic diagram of the feature sharing module. Detailed Implementation
[0055] The invention will now be further described with reference to the accompanying drawings.
[0056] Example 1: See Figures 1-4 A multimodal, multi-task workshop target recognition method includes the following steps;
[0057] (1) Construct a sample dataset;
[0058] The workshop site was photographed and captured using a depth camera, and each photograph yielded a set of corresponding color and depth images.
[0059] The categories of targets are determined, including workers, lathes, and material handling robots;
[0060] On color and depth images, object detection and instance segmentation levels are labeled to obtain ground truth bounding boxes for object detection and ground truth instance masks for instance segmentation.
[0061] A set of labeled color and depth images is used as a data sample;
[0062] (2) Construct a multimodal, multi-task workshop target recognition network;
[0063] The multimodal, multi-task workshop target recognition network includes an encoding module and a decoding module;
[0064] The encoding module includes two ResNet50 backbone networks. The ResNet50 backbone network is divided into five stages from the input to the output, namely the first stage to the fifth stage, and outputs the first feature vector to the fifth feature vector respectively.
[0065] Two ResNet50 backbone networks are respectively input to labeled color images and labeled depth images. A fusion module is set between the two second stages, the two third stages, the two fourth stages, and the two fifth stages. From front to back, these are the first to the fourth fusion modules. The input of the first three fusion modules is connected to the output of the two previous stages. The output is split into two paths, which are summed with the output of the two previous stages and then sent to the two next stages.
[0066] The fourth fusion module has its input connected to the outputs of the two fifth stages, and its output is split into two paths, which are then fed into the decoding module.
[0067] The fusion module is used to fuse the two input feature vectors and output the result.
[0068] The decoding module is used to perform target detection and instance segmentation on the output of the encoding module, and output the target detection result and the instance segmentation result.
[0069] (3) Training of the multimodal, multi-task workshop target recognition network;
[0070] The data samples in the sample dataset are input into the multimodal multi-task workshop target recognition network for target detection and instance segmentation. In the first decoding branch, the expected output is the ground truth bounding box corresponding to the target in the data sample. In the second decoding branch, the expected output is the ground truth instance mask of the instance segmentation in the data sample. The training continues until the model converges.
[0071] (4) Multimodal, multi-task workshop target recognition network task recognition;
[0072] A set of color and depth images of the workshop to be tested are acquired and fed into a multimodal, multi-task workshop target recognition network, which outputs the target prediction bounding box and the prediction instance mask corresponding to the target.
[0073] In this embodiment, the fusion method of the fusion module is as follows:
[0074] (2.1) Color images are RGB C×H×W The depth image is Depth C×H×W ,inC , H and W These represent the number of channels, height, and width of the corresponding image;
[0075] (2.2) The color image and the depth image are stitched together according to the channel dimension to generate the first stitching feature. RGBD 2C×H×W Then, it is further divided into S sub-feature blocks according to the channel dimension, labeled as X0~X100~X200~X30~X40~X50~X60~X7 ... S-1 Each sub-feature block has a dimension of ;
[0076] (2.3) For each sub-feature block, perform convolution operations with different kernel sizes to obtain sub-feature vectors, where the first... i Sub-feature blocks X i Perform the convolution operation according to the following formula;
[0077] F i =Conv i ( X i ),
[0078] In the formula, F i for X i The corresponding sub-feature vector, Conv This represents the convolution operation. i =0~ S ;
[0079] (2.4) Perform global average pooling on the S sub-feature vectors to reduce them to a size of S×1×1, resulting in S weight vectors;
[0080] (2.5) Normalize the S weight vectors to obtain S attention vectors;
[0081] (2.6) Finally, the obtained attention vector is used in conjunction with the first concatenated feature. RGBD Perform element-wise multiplication to obtain the fused features.
[0082] The decoding module includes a first decoding branch for target detection and a second decoding branch for instance segmentation;
[0083] The first decoding branch includes a first decoding layer, a second decoding layer, a third decoding layer and an object detection head set in sequence. The first three layers are used to perform upsampling operations, and each time an object detection feature is output. The size of the object detection feature output by the third decoding layer is 1 / 4 of the color image. The object detection head first upsamples the output of the third decoding layer and then predicts the bounding box to obtain the object detection result.
[0084] The second decoding branch includes a first decoding layer, a second decoding layer, a third decoding layer and an instance segmentation head set in sequence. The first three layers are used to perform upsampling operations, and each time a target detection feature is output. The instance segmentation feature output by the third decoding layer is 1 / 4 the size of the color image. The instance segmentation head first upsamples the instance segmentation feature output by the third decoding layer and then predicts the instance mask to obtain the instance segmentation result.
[0085] A feature sharing module is provided between the two first decoding layers;
[0086] The feature sharing module is used to input target detection features and instance segmentation features, concatenate them according to the channel dimension to obtain a second concatenated feature, and then divide the second concatenated feature into two paths: one path is spatially pooled to obtain a spatial attention vector, and the other path is channel pooled to obtain a channel attention vector.
[0087] The spatial attention vector and the channel attention vector are element-wise multiplied with the second concatenation feature, and then the results of the element-wise multiplication are added together to obtain the processed second concatenation feature.
[0088] The processed second concatenated features are split according to the channel dimension to obtain processed target detection features and processed instance segmentation features, which are then sent back to the first decoding layer of the first decoding branch and the first decoding layer of the second decoding branch, respectively, and then sent to the next process by the two first decoding layers.
[0089] Feature sharing modules are also provided between the two second decoding layers and between the two third decoding layers.
[0090] The depth camera is an Intel RealSense D455 RGBD camera with a sampling resolution of 640x480.
[0091] For information on the fusion module, see [link / reference]. Figure 1 The first three fusion modules have their inputs connected to the outputs of the two previous stages. Their outputs are split into two paths, each summed with the output of the two previous stages before being fed into the two next stages. In other words:
[0092] A fusion module is provided between the two second stages. The input of the fusion module is connected to the output of the two second stages. The output of the fusion module is divided into two paths, which are added to the output of the two second stages respectively, and then sent to the two third stages.
[0093] A fusion module is also provided between the two third stages. The input of the fusion module is connected to the output of the two third stages. The output of the fusion module is divided into two paths, which are added to the output of the two third stages respectively, and then sent to the two fourth stages.
[0094] A fusion module is also provided between the two fourth stages. The input of the fusion module is connected to the output of the two fourth stages. The output of the fusion module is divided into two paths, which are added to the output of the two fourth stages respectively, and then sent to the two fifth stages.
[0095] During network training: The multimodal, multi-task workshop target recognition network is trained using the acquired real target detection labels and instance segmentation labels. Due to the adoption of a multi-task learning method, two types of loss values are generated: target detection and image segmentation. Furthermore, due to the differences between tasks, the prediction outputs of each task exhibit homoscedasticity uncertainty. Therefore, a multi-task learning loss function method is employed to simultaneously learn regression and classification problems at different scales and with varying numbers of elements.
[0096] The joint loss function for multiple tasks is defined to satisfy the following formula.
[0097]
[0098] Let represent the joint loss function of the two tasks, where = This refers to the loss value of the regression task. This refers to the loss in the classification task, where , The actual label value. This is the network prediction value. , These are the noise scalar values output by the two task branches, respectively.
[0099] The general process of this invention is as follows:
[0100] For a given data sample, we label the color image as image A and the depth image as image B. Image A is fed into one ResNet50 backbone network, and image B is fed into another ResNet50 backbone network. See [link to relevant documentation]. Figure 1 After the first and second stages, output the second feature vector A2 corresponding to image A and the second feature vector B2 corresponding to image B;
[0101] A2 and B2 are fed into the fusion module for processing. After steps (2.1)-(2.6), the fused features are obtained. The fused features are divided into two paths and summed with A2 and B2 respectively to obtain two summed features A2' and B2', which are then sent to the next stage, that is, the third stage of the two ResNet50 backbone networks.
[0102] Similarly, there are fusion modules between the two third, fourth, and fifth stages. Following the operation process of the fusion modules, the fourth fusion module finally outputs a fusion feature map, which we call a multimodal fusion feature map; at this point, the encoding module finishes its work.
[0103] The multimodal fusion feature map is divided into two paths, which are fed into the first and second decoding branches of the decoding module, respectively. The first decoding branch outputs the target detection result, and the second decoding branch outputs the instance segmentation result. However, a feature sharing module is also set up between the two branches. The workflow of the feature sharing module between the two first decoding layers is as follows: the target detection feature and the instance segmentation feature are first concatenated along the channel dimension, and then feature pooling operations are performed on the concatenated features in both the spatial and channel dimensions to obtain spatial attention vectors and channel attention vectors. Then, the spatial attention vectors and channel attention vectors are used to highlight representative features at the spatial and channel levels, respectively, while suppressing noise. Finally, the highlighted features at the channel and spatial levels are merged by element-wise summation and split according to modality to complete the sharing of the two types of features. This invention sets up three feature sharing modules in the decoding module, and feature sharing is performed once between each decoding layer.
[0104] During training, we use the ground truth bounding boxes of object detection and the ground truth instance masks of instance segmentation as the expected outputs to correct the predicted bounding boxes and predicted instance masks.
[0105] This invention has good recognition accuracy for color-similar targets in workshop scenes. It can achieve instance segmentation and target detection in the same scene. The accuracy rate for target detection in workshop scenes reaches 87%, and the accuracy rate for instance segmentation reaches 81%.
[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A multimodal, multi-task workshop target recognition method, characterized in that: Includes the following steps; (1) Construct a sample dataset; The workshop site was photographed and captured using a depth camera, and each photograph yielded a set of corresponding color and depth images. The categories of targets are determined, including workers, lathes, and material handling robots; On color and depth images, object detection and instance segmentation levels are labeled to obtain ground truth bounding boxes for object detection and ground truth instance masks for instance segmentation. A set of labeled color and depth images is used as a data sample; (2) Construct a multimodal, multi-task workshop target recognition network; The multimodal, multi-task workshop target recognition network includes an encoding module and a decoding module; The encoding module includes two ResNet50 backbone networks. The ResNet50 backbone network is divided into five stages from the input to the output, namely the first stage to the fifth stage, and outputs the first feature vector to the fifth feature vector respectively. Two ResNet50 backbone networks are respectively input to labeled color images and labeled depth images. A fusion module is set between the two second stages, the two third stages, the two fourth stages, and the two fifth stages. From front to back, these are the first to the fourth fusion modules. The input of the first three fusion modules is connected to the output of the two previous stages. The output is split into two paths, which are summed with the output of the two previous stages and then sent to the two next stages. The fourth fusion module has its input connected to the outputs of the two fifth stages, and its output is split into two paths, which are then fed into the decoding module. The fusion module is used to fuse the two input feature vectors and output the result. The decoding module is used to perform target detection and instance segmentation on the output of the encoding module, and output the target detection result and the instance segmentation result. (3) Training of the multimodal, multi-task workshop target recognition network; The data samples in the sample dataset are input into the multimodal multi-task workshop target recognition network for target detection and instance segmentation. In the first decoding branch, the expected output is the ground truth bounding box corresponding to the target in the data sample. In the second decoding branch, the expected output is the ground truth instance mask of the instance segmentation in the data sample. The training continues until the model converges. (4) Multimodal, multi-task workshop target recognition network task recognition; A set of color and depth images of the workshop to be tested are acquired and fed into a multimodal multi-task workshop target recognition network, which outputs the target prediction box and prediction instance mask corresponding to the target. The decoding module includes a first decoding branch for target detection and a second decoding branch for instance segmentation; The first decoding branch includes a first decoding layer, a second decoding layer, a third decoding layer and an object detection head set in sequence. The first three layers are used to perform upsampling operations, and each time an object detection feature is output. The size of the object detection feature output by the third decoding layer is 1 / 4 of the color image. The object detection head first upsamples the output of the third decoding layer and then predicts the bounding box to obtain the object detection result. The second decoding branch includes a first decoding layer, a second decoding layer, a third decoding layer and an instance segmentation head set in sequence. The first three layers are used to perform upsampling operations, and each time a target detection feature is output. The instance segmentation feature output by the third decoding layer is 1 / 4 the size of the color image. The instance segmentation head first upsamples the instance segmentation feature output by the third decoding layer and then predicts the instance mask to obtain the instance segmentation result. A feature sharing module is provided between the two first decoding layers; The feature sharing module is used to input target detection features and instance segmentation features, concatenate them according to the channel dimension to obtain a second concatenated feature, and then divide the second concatenated feature into two paths: one path is spatially pooled to obtain a spatial attention vector, and the other path is channel pooled to obtain a channel attention vector. The spatial attention vector and the channel attention vector are element-wise multiplied with the second concatenation feature, and then the results of the element-wise multiplication are added together to obtain the processed second concatenation feature. The processed second concatenated features are split according to the channel dimension to obtain processed target detection features and processed instance segmentation features, which are then sent back to the first decoding layer of the first decoding branch and the first decoding layer of the second decoding branch, respectively, and then sent to the next process by the two first decoding layers. Feature sharing modules are also provided between the two second decoding layers and between the two third decoding layers.
2. The multimodal, multi-task workshop target recognition method according to claim 1, characterized in that: The fusion method of the fusion module is as follows: (2.1) Color images are RGB C×H×W The depth image is... C×H×W Where C, H, and W are the number of channels, height, and width of the corresponding image, respectively; (2.2) The color image and the depth image are stitched together according to the channel dimension to generate the first stitching feature RGBD. 2C×H×W Then, it is further divided into S sub-feature blocks according to the channel dimension, labeled as X0~X100~X200~X30~X40~X50~X60~X7 ... S-1 Each sub-feature block has a dimension of ; (2.3) For each sub-feature block, perform convolution operations with different kernel sizes to obtain sub-feature vectors, where the i-th sub-feature block X i Perform the convolution operation according to the following formula; F i =Conv i (X i ), In the formula, F i For X i The corresponding sub-feature vector, Conv represents the convolution operation, i=0~S; (2.4) Perform global average pooling on the S sub-feature vectors to reduce them to a size of S×1×1, resulting in S weight vectors; (2.5) Normalize the S weight vectors to obtain S attention vectors; (2.6) Finally, the obtained attention vector is multiplied element-wise with the first concatenated feature RGBD to obtain the fused feature.
3. The multimodal, multi-task workshop target recognition method according to claim 1, characterized in that: The depth camera is an Intel RealSense D455 RGBD camera with a sampling resolution of 640x480.