RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion

Abstract

The invention provides an RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion, and belongs to the field of computer vision and computer graphics. According to the method, semantic segmentation prediction is performed on an image based on an encoder-decoder network structure, multi-modal feature correction and fusion are realized by using multiple attention mechanisms at each stage of a network, feature representation of input data is optimized and enhanced, and a top-down multi-scale feature fusion strategy is used in a decoder. Through multiple attention mechanisms, the model can effectively capture global features and local features, fusion of different modal information under the same scale is realized, and fine aggregation of features under different scales is promoted. According to the method, cooperation of multi-modal features and fusion of multi-scale features are realized by applying multiple attention mechanisms, and the problems of difference between RGB and depth information, noise interference in data and loss of detail information in a network extraction process are effectively solved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision and computer graphics, and specifically, to an RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion. Background Art

[0002] Image semantic segmentation is a key technology in the field of computer vision. Its goal is to assign each pixel in an image to a category label, thereby understanding the semantic meaning of each pixel in the image. Indoor semantic segmentation focuses on analyzing the indoor environment, performing pixel-level analysis on indoor scene images, assigning the correct category label to each pixel, and achieving detailed recognition of each object and surface in the image. In the process of semantic segmentation of indoor scenes, researchers often need to deal with a series of challenges, such as uneven indoor lighting, numerous object categories, mutual occlusion between objects, and similar surface textures. These factors increase the difficulty of achieving high-precision segmentation.

[0003] Depth images contain rich geometric information, which is helpful for analyzing spatial structure and object layout. Integrating depth images into the semantic segmentation network and combining them with RGB images enriches the input dimension of the network and improves the network's ability to distinguish confusing areas in the image. In addition, the introduction of depth images also helps to overcome the performance degradation of RGB images in low-light or sparse-texture environments. By combining the advantages of the two modalities, the network can handle various environmental conditions more robustly and provide more accurate and reliable segmentation results.

[0004] Attention mechanism is a common method for fusing RGB and depth images. The core idea of ​​attention mechanism is to simulate the function of human visual attention, so that the model can focus on the most important part of the input data, thereby improving processing efficiency and accuracy. Different attention mechanisms each play their own unique functions. Channel attention focuses on the relationship between feature channels, spatial attention emphasizes the spatial position in the image, self-attention focuses on capturing long-distance dependencies within the sequence, and cross attention focuses on information interaction between different sequences. These attention mechanisms can be used alone or in combination. The model combines multiple attention mechanisms to more comprehensively capture the important features of the image to further improve model performance. Summary of the invention

[0005] The main technical problem solved by the present invention is: In view of the problem of loss of detail information when processing multimodal features and extracting features from the network in the RGB-D semantic segmentation task, the present invention provides an RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion. The multimodal features extracted by the dual-branch network are corrected and fused, and a top-down multi-scale fusion method is used in the decoder to effectively integrate the features after fusion of each layer of the network, thereby improving the feature expression capability and enhancing the semantic segmentation effect.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: an RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion, the method comprising the following steps:

[0007] Step 1: Use a dual-branch network to extract RGB and depth features, implement feature interaction layer by layer, and perform multi-scale feature fusion in the decoder to obtain semantic segmentation results;

[0008] Step 2: The multimodal features extracted in each stage are corrected using modal differences and complementary information;

[0009] Step 3: Use the cross-attention mechanism to exchange information and aggregate the corrected multimodal features across modalities;

[0010] Step 4: In the decoder, a top-down multi-scale fusion strategy is used to integrate the fusion features of the networks at each stage to obtain the final feature expression, and prediction is made based on it.

[0011] Furthermore, a dual-branch network is used to extract RGB and depth features, feature interaction is achieved layer by layer, and multi-scale feature fusion is performed in the decoder. The specific steps are as follows:

[0012] Two parallel Segformer encoders are used to extract RGB and depth features respectively, and feature interaction is realized layer by layer. The results of the interaction are passed to the subsequent network layers of each branch for deeper processing. In the decoder, multi-scale feature fusion is performed in a top-down manner to enhance the model's comprehensive processing capabilities for information at different levels.

[0013] Furthermore, the multimodal features extracted in each stage are corrected using modal differences and complementary information. The specific steps are as follows:

[0014] Step 201: Connect the RGB and depth features extracted from each layer of the network along the channel dimension and implement feature correction through a channel attention mechanism;

[0015] Step 202: Use CBAM to correct the connected features using the channel attention mechanism and the spatial attention mechanism in sequence to ensure that the features are fully optimized in both channel and spatial dimensions;

[0016] Step 203 , the initial feature is dynamically fused with another modal feature processed in steps 201 and 202 to obtain a final correction feature.

[0017] Furthermore, the cross-attention mechanism is used to exchange information and aggregate the corrected multimodal features across modalities:

[0018] Step 301: divide the corrected features into residual vectors and interaction vectors, wherein an efficient cross-attention mechanism is used to realize information exchange for the interaction vectors of the two modalities, thereby realizing cross-modal global reasoning;

[0019] Step 302: Use a context fusion module based on local and global information to integrate the features of the two modalities for subsequent multi-scale feature fusion.

[0020] Furthermore, a top-down multi-scale fusion strategy is used in the decoder to integrate the fusion features of the networks at each stage to obtain the final feature expression, as follows:

[0021] The pyramid feature extraction module is used to perform multi-scale feature extraction and integration on the fused features obtained from the last layer of the network. The obtained features are upsampled and combined with the fused features obtained from the previous layer of the network. The features are iterated layer by layer until the first layer of the network to obtain the final fused features, based on which semantic segmentation prediction is performed.

[0022] The beneficial effects of the present invention are as follows: In view of the differences in RGB and Depth information and the noise interference in the data, the network uses a variety of attention mechanisms to achieve multimodal feature correction and fusion, overcomes the optimization and enhances the feature representation of the input data. At the same time, the top-down multi-scale feature fusion strategy is used in the decoder to effectively solve the problem of detail information loss in the feature extraction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the overall network architecture of the present invention;

[0024] Figure 2 It is a schematic diagram of the structure of the feature correction module FCM;

[0025] Figure 3 It is a schematic diagram of the structure of the feature aggregation module FAM;

[0026] Figure 4 Schematic diagram of the context feature fusion module CFFM structure. DETAILED DESCRIPTION

[0027] The present invention is described below in conjunction with the accompanying drawings and specific embodiments. Figure 1is the overall network architecture diagram of the present invention; Figure 2 is a structural diagram of the feature correction module FCM used for cross-modal feature correction in the encoder; Figure 3 This is the structure diagram of the feature aggregation module FAM used for feature interaction and fusion in the encoder; Figure 4 This is the structural diagram of the contextual feature fusion module CFFM used for multimodal fusion and multi-scale fusion.

[0028] like Figure 1 As shown, the overall network of the present invention includes: two parallel groups of Transformer Blocks form an encoder, and two branches extract RGB features and depth features respectively. In each stage of the network, the RGB features and the depth features are corrected by the Feature Calibration Module (FCM), and the corrected features are fed to the next stage network for feature extraction, and are input into the Feature Aggregation Module (FAM) for further information exchange and fusion. The fused features of the last stage network are extracted and fused by the RFB module, followed by upsampling, and fused with the fused features of the previous stage network through the context fusion module (CFFM), iterating the upsampling and fusion process until the final feature expression is obtained by fusion with the features of the first stage network, based on which semantic segmentation prediction is performed and the predicted image is output.

[0029] based on Figure 1 The present invention provides an RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion, comprising the following steps:

[0030] Step 1: Use a dual-branch network to extract RGB and depth features, implement feature interaction layer by layer, and perform multi-scale feature fusion in the decoder;

[0031] Step 2: The multimodal features extracted in each stage are corrected using modal differences and complementary information;

[0032] Step 201: Figure 2 As shown, the input features F of the RGB image and the depth image are RGB ∈R H×W×C and F Depth ∈R H ×W×C Concatenate along the channel dimension to get X∈R H×W×2C , and perform adaptive average pooling and adaptive maximum pooling operations on the concatenated feature tensor to retain more information. Concatenate the two results along the channel dimension to obtain Y∈R4C , and then perform MLP operation on it, and obtain W from Y through sigmoid function C , and then divide them into two weight graphs and The optimized features are obtained by calculation and The above process is as follows:

[0033]

[0034]

[0035]

[0036] Step 202: For the concatenated X, we introduce the CBAM module, which first uses the channel attention mechanism to focus on the extraction of global information. On this basis, the spatial attention mechanism is used to optimize the local features in the spatial dimension. This staged attention guidance strategy enhances the feature expression ability by adaptively adjusting the channel and spatial information in the feature map, and divides the optimized results into and Get the corrected RGB and depth features. The specific process is as follows:

[0037]

[0038] Step 203: Integrate the results of the two stages. The entire calibration process in FAM is as follows:

[0039]

[0040]

[0041] λ a and λ b are two hyperparameters, initially set to 0.5, so that the correction process can be dynamically adjusted through learning during the training process. RGB and F' Depth The fully corrected features are obtained through the FCM module, which are then input into the next stage of feature extraction and simultaneously input into FAM for subsequent operations.

[0042] Step 3: Use the cross-attention mechanism to exchange information and aggregate the corrected multimodal features across modalities;

[0043] Step 301: Figure 3As shown in the figure, the corrected features are flattened and input into the linear projection layer for processing. The output tensor is equally divided into two parts, the residual vector and the interaction vector, along the last dimension. In the cross-modal information interaction stage, for each modality, the global context vector G is first calculated using the interaction vector, and then calculated with the residual vector of the other modality. The process is as follows:

[0044]

[0045]

[0046]

[0047]

[0048] Among them, G represents the global context vector and U represents the attention result.

[0049] The result vector U obtained from each mode is connected to the residual vector along the last dimension and mapped to R through the projection layer H×W×C .

[0050] Step 302: Input the RGB features and depth features after information exchange into the CFFM module for multimodal feature fusion. The CFFM module includes two parts: feature fusion and feature enhancement, both of which are implemented based on the global-local feature fusion module (G-LFFM).

[0051] like Figure 4 As shown in the figure, G-LFFM designs two parallel processing branches to achieve comprehensive capture of global and local feature information. The global information analysis uses a compression and excitation module to capture the global context information of the entire scene. The local information branch uses a series of pointwise convolution and ReLU activation functions to focus on extracting local details and features of small objects, and fuses global information with local information. The calculation process of the feature fusion stage is recorded as:

[0052]

[0053] F a With F b represents two input features, GL(·) represents the global and local information integration operation, represents element-by-element addition, represents element-wise multiplication, Indicates that the weight is negated, and the fused feature F is obtained through the above process. ab .

[0054] F ab Perform a series of operations including convolution, batch normalization, and Relu function to obtain F' ab , followed by F' ab It is fed into the convolutional layer and the maximum pooling layer to obtain the feature F C 、F MP They are input into G-LFFM to further extract global and local information, and then the features of the two branches are added element-wise. Finally, after additional convolution processing, feature fusion is achieved. The above process is shown in the following formula:

[0055] F' ab =CONV(F ab )

[0056]

[0057]

[0058]

[0059] Among them, CONV(·) represents a series of convolution operations, U(·) represents upsampling, and F m The final feature.

[0060] Step 4: Set F i (i=1,2,3,4) is the output feature of the network at the i-th layer FAM module, where In the last stage of the network, the RFB module is used for further multi-scale feature extraction. Subsequently, the features are upsampled and fused with the output features of the previous FAM layer of the network through the CFFM module. This fusion operation is iterated upward layer by layer. Each fused feature is upsampled again and then further fused with the output of the FAM of the lower layer network in the CFFM module until it reaches the first layer of the network. The final fusion result is predicted. The specific process is as follows:

[0061] F 4 =RFB(F 4 )

[0062]

[0063] is the fusion feature of the i-th layer and the i-1-th layer, U(·) represents the upsampling operation, and Linear(·) represents converting the feature into 512 dimensions.

[0064] The multi-scale feature fusion strategy ensures that the model can capture and integrate rich feature information from different levels, thereby significantly improving the performance of the segmentation task.

Claims

1. An RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion, characterized in that: include: Step 1: Use a dual-branch network to extract RGB and depth features, implement feature interaction layer by layer, and perform multi-scale feature fusion in the decoder to obtain semantic segmentation results; Step 2: The multimodal features extracted in each stage are corrected using modal differences and complementary information; Step 3: Use the cross-attention mechanism to exchange information and aggregate the corrected multimodal features across modalities; Step 4: In the decoder, a top-down multi-scale fusion strategy is used to integrate the fusion features of the networks at each stage to obtain the final feature expression, and prediction is made based on it.

2. The RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion according to claim 1, characterized in that: The dual-branch network is used to extract RGB and depth features, implement feature interaction layer by layer, and perform multi-scale feature fusion in the decoder, as follows: Two parallel Segformer encoders are used to extract RGB and depth features respectively, and feature interaction is realized layer by layer. The results of the interaction are passed to the subsequent network layers of each branch for deeper processing. In the decoder, multi-scale feature fusion is performed in a top-down manner to enhance the model's comprehensive processing capabilities for information at different levels.

3. The RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion according to claim 1, characterized in that: The multimodal features extracted at each stage are corrected using modal differences and complementary information, as follows: Step 201: Connect the RGB and depth features extracted from each layer of the network along the channel dimension and implement feature correction through a channel attention mechanism; Step 202: Use CBAM to correct the connected features using the channel attention mechanism and the spatial attention mechanism in sequence to ensure that the features are fully optimized in both channel and spatial dimensions; Step 203 , the initial feature is dynamically fused with another modal feature processed in steps 201 and 202 to obtain a final correction feature.

4. The RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion according to claim 1, characterized in that: The cross-attention mechanism is used to exchange information and aggregate cross-modal features of the corrected multimodal features, as follows: Step 301: divide the corrected features into residual vectors and interaction vectors, wherein an efficient cross-attention mechanism is used to realize information exchange for the interaction vectors of the two modalities, thereby realizing cross-modal global reasoning; Step 302: Use a context fusion module based on local and global information to integrate the features of the two modalities for subsequent multi-scale feature fusion.

5. The RGB-D semantic segmentation method based on feature collaboration and multi-scale feature fusion according to claim 1, characterized in that: In the decoder, a top-down multi-scale fusion strategy is used to integrate the fusion features of the networks at each stage to obtain the final feature expression, as follows: The pyramid feature extraction module is used to perform multi-scale feature extraction and integration on the fused features obtained from the last layer of the network. The obtained features are upsampled and combined with the fused features obtained from the previous layer of the network. The features are iterated layer by layer until the first layer of the network to obtain the final fused features, based on which semantic segmentation prediction is performed.

Images