Semantic segmentation method based on parameter tuning and multi-modal data fusion

Abstract

The invention relates to the field of computer vision and deep learning, and provides a semantic segmentation method based on parameter tuning and multi-modal data fusion. The core of the method lies in that RGB, depth and Mask modal data are fully fused to improve segmentation precision, a frozen SegFormer encoder is adopted to extract features in parallel, a parameter tuning module is combined to realize indoor semantic segmentation task adaptation, RGB and Mask features are dynamically weighted and fused through a Gate gate in the first two layers of a network to enhance boundary information, and the segmentation precision is improved. In the last two layers, the feature semantic information expression ability is improved by utilizing a context information interaction module, a multi-modal cross fusion module is introduced, and effective interaction and enhancement of depth information and RGB features are ensured by adopting efficient cross attention and improved space coordinate attention. According to the method, the calculation burden is reduced, the model efficiency is improved, the accuracy of indoor scene semantic segmentation is remarkably improved, and the method has a wide application prospect.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and specifically to a semantic segmentation method that performs parameter tuning on a SegFormer network architecture and fuses multimodal data to assist segmentation. Background Art

[0002] Semantic segmentation is a core task in the field of computer vision, which aims to classify each pixel in an image and ensure that pixels of the same object are assigned the same label. Traditional semantic segmentation methods mainly rely on convolutional neural networks, which extract local features of the image through layer-by-layer convolution operations, and finally generate a segmented image of the same size as the input image, in which each pixel is assigned to a specific category. In contrast, SegFormer is based on the Transformer architecture, abandons the traditional convolution operation, can capture image context information from a global perspective, and significantly reduces computational overhead. Indoor semantic segmentation focuses on pixel-level classification of indoor scene images, identifying and segmenting objects or regions of different categories. Compared with outdoor semantic segmentation, indoor environments have unique technical challenges and application requirements. Due to the complex scene changes, cluttered object shapes and occlusions indoors, it is difficult to achieve ideal results using only RGB images.

[0003] In the semantic segmentation task, RGB images provide rich color and texture information to help the model distinguish visual features. Depth data provides distance information to the camera for each pixel, adding the third dimension of information to the image, further helping the model distinguish objects at different distances. Mask data provides accurate segmentation boundaries, clearly defines the contours of different objects, and lays a solid foundation for multimodal processing. How to effectively fuse these three modal information and make full use of the information differences between the modalities to assist the model in segmentation is a problem that needs to be solved urgently. How to adapt the semantic segmentation task of indoor scenes based on the existing network architecture is also an important challenge. Summary of the invention

[0004] The technical problem that the present invention needs to solve is how to effectively utilize depth data and Mask data to assist the semantic segmentation of RGB images in indoor scenes, and at the same time explore how to achieve efficient semantic segmentation task adaptation based on the existing network architecture.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a semantic segmentation method based on parameter tuning and multimodal data fusion, characterized in that it includes the following steps:

[0006] S1, using the SegFormer encoder with frozen parameters to extract features from the three modal information in parallel, and using the cross-attention tuning module to assist the model in semantic segmentation;

[0007] S2. In the first two layers of the SegFormer encoder, the Gate gate is used to realize the fusion of RGB features and Mask features. In the last two layers, the RGB features and Mask features are fused through the context information interaction module.

[0008] S3: The RGB features enhanced by the Mask feature are used to interact with the deep features through a multimodal cross fusion module.

[0009] According to a preferred embodiment of the present invention, the cross attention tuning module in S1 assists the model in performing semantic segmentation, and the specific steps are as follows:

[0010] In the Transformer block of the frozen SegFormer network, an additional attention module is configured, in which the Q required for attention calculation comes from the self-attention architecture of the frozen Transformer block, and K and V are obtained by linear mapping the feature data. The features after cross-attention processing are fused into the original Transformer block in the form of element-wise addition.

[0011] According to a preferred embodiment of the present invention, the fusion of the RGB features and the Mask features in S2 adopts different fusion strategies at different layers, and the specific steps are as follows:

[0012] S201. In the first two layers of the encoder, the Mask feature contains rich boundary information. The Gate gate is used to fuse the two modal features. The learnable parameters are set to dynamically adjust the weights of the two modalities. The fused features are obtained by element addition.

[0013] S202. In the last two layers of the encoder, the Mask feature contains rich semantic information. The RGB feature and the Mask feature are connected in the channel direction. The context information interaction module is used to realize the fusion of context information between features, and the semantic information of the RGB feature is enhanced. The context information interaction module contains a simple convolution operation, and the result is normalized and ReLU operated.

[0014] According to a preferred embodiment of the present invention, the multimodal cross fusion module in S3 includes two stages, and the specific steps are:

[0015] S301. In the feature fusion stage, an efficient cross-attention operation is used to interact the two modal information, connect the interacted features along the channel direction, and fuse the two modal information through a convolution operation.

[0016] S302. In the feature enhancement stage, an improved coordinate attention operation is used to perform maximum pooling and average pooling on the fused features along the X-axis and Y-axis directions respectively, the results are connected and linearly mapped, convolution and Sigmoid operations are performed along the X-axis and Y-axis, and the obtained weights are multiplied with the original feature map to obtain enhanced features.

[0017] The beneficial effects of the present invention are as follows:

[0018] In the semantic segmentation method based on parameter tuning and multimodal data fusion of the present invention, the RGB image is automatically segmented by SegmentAnything Model (SAM) to obtain multi-category Mask data. In the first two layers of the network, the Mask features and the RGB image features are element-wise added using the Gate gate to provide rich boundary information for the image features. In the last two layers of the network, rich semantic information is provided for the image features through the interaction of contextual information. The depth information and the RGB image features are subjected to efficient cross-attention operations, which helps to reduce the amount of calculation and realize the interaction of the two modal information, providing rich spatial information for the RGB image. The improved spatial coordinate attention further enhances the features. The SegFormer network architecture using a frozen encoder helps to reduce the amount of calculation and reduce the requirements for hardware equipment. The use of a parameter tuning module can utilize the prior knowledge of the network to better adapt to specific semantic segmentation tasks. Therefore, the present invention can solve the limitations of insufficient modal information and inability to fully integrate in existing indoor scene semantic segmentation tasks, reduce the amount of calculation for model training, and improve the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the overall network framework of the present invention;

[0020] Figure 2 This is a schematic diagram of the CrossAttention Turning Model (CATM) structure;

[0021] Figure 3 It is a schematic diagram of the structure of the Context Information Interaction Module (CIIM);

[0022] Figure 4 Schematic diagram of the structure of the Multi-Modal Cross Attention Model (MCAM). DETAILED DESCRIPTION

[0023] The present invention is described below in conjunction with the accompanying drawings and specific examples. Figure 1The overall process of the semantic segmentation method based on parameter tuning and multimodal data fusion is given. The specific implementation is as follows:

[0024] S1. The RGB data in the RGB-D dataset is processed by SAM to obtain multi-category Mask data, which is converted into a three-channel image to facilitate feature extraction by the SegFormer network. The depth data is processed by program calculation and converted into HHA data and saved in the form of an image;

[0025] S2, using the SegFormer encoder network architecture with frozen parameters to extract features from the three modal information in parallel, using the parameter tuning module to assist the model in adapting to the indoor scene semantic segmentation task, and setting the decoder to a normal learnable state;

[0026] S201, use the network architecture with frozen SegFormer encoder parameters to extract features from these three modal information in parallel:

[0027] X i =Layeri(X i-1 ) i=1,2,3,4

[0028] Where X i represents the features after being processed by the i-th layer of the SegFormer encoder, X i-1 Represents the input features of the previous layer. When i=1, it means feature extraction of the original image data;

[0029] S202. In the present invention, the SegFormer encoder uses the trained parameter weights and freezes the parameters during the training process. It needs to be tuned by the cross-attention tuning module. The specific implementation is as follows: Figure 2 As shown:

[0030] The present invention mainly performs parameter tuning through the interaction between CATM and the Transformer block in the SegFormer network architecture. Assuming that the modal feature data is R x , the operation in the Transfomer block can be expressed by the formula:

[0031] Q t ,K t ,V t =Linear(R x )

[0032] Z x =MSA(Q t ,K t ,V t )

[0033] Linear(·) represents a linear mapping operation, MAS(·) represents a multi-head attention mechanism, and the attention calculation is:

[0034]

[0035] Here is the Q t ,K t ,V t It means from The query, key, and value vectors generated in .

[0036] R x At the same time, as a prompt information for a specific task, it enters the CATM to perform information interaction operations. The operations in CATM can be expressed by the formula:

[0037] K c ,V c =Linear(R x )

[0038] G c =K c T ·V c

[0039] R′ X =Q t Softmax(G c )

[0040] R″ X =Linear([R′ X ,R X ])

[0041] Where Linear(·) represents the modal characteristic data R x After linear mapping, we get the key vector K c Sum value vector V c , K c T With V c Multiply it with the vector Q from the Transformer operation t Multiply to get the vector R′ after cross attention processing x The method used here is an efficient attention method, which reduces the amount of computation while achieving a processing effect similar to that of the standard attention operation, and is suitable for lightweight semantic segmentation tasks. X ,R X ] indicates that R′ x With R x Connect in the channel dimension and then obtain the final output R″ through linear mapping x .

[0042] H=R x +Z x +R″ X

[0043] Out=Mix_FFN(H)

[0044] The processed interaction feature R″ X Integrating into the Transformer framework, the element addition method is adopted here, and the features are output through subsequent operations in the Transformer block.

[0045] S3. In the first two layers of the encoder, the Gate gate is used to realize the fusion of RGB features and Mask features to obtain enhanced RGB features. In the last two layers of the encoder, the RGB features and Mask features are connected along the channel direction, and the connected features are fused through the context information interaction module.

[0046] S301, Gate mechanism can combine context information and adjust the contribution of each modality according to local and global features, which helps improve the model's ability to handle object boundaries and details in complex scenes. Assume that the RGB image feature shape is X R ∈R C×H×W Where C is the number of channels, H is the height, W is the width, and the mask feature shape is X M ∈R C×H×W , Gate is used to control the weighted ratio of the two image features. Combined with X R and X M Generate a dynamic weight matrix G, concatenate the two features, apply a convolutional layer, and then generate Gate weights through the sigmoid activation function:

[0047] G = σ(Conv 1×1 ([X R ,X M ]))

[0048] Where [X R ,X M ] means to convert X into R and X M Connect and generate G∈R C×H×W It is a weight matrix, and the value of each position is between 0 and 1. According to the Gate weight, the two image features are weighted fused, and the Gate weight matrix G controls the contribution of the two features in the fusion process:

[0049] F fusion =G⊙X R + ( 1-G)⊙X M

[0050] Where ⊙ represents element-by-element multiplication, G corresponds to X R The weighted value of (1-G) corresponds to X M In this way, the model can automatically learn to dynamically adjust the fusion ratio of image features at different locations.

[0051] S302, in the last two layers of the encoder, the fusion of RGB features and Mask features is achieved through the context information interaction module. The specific operation is as follows: Figure 3 As shown in Figure 1, the RGB features extracted by the Transformer block and the Mask features are connected in the channel direction, and the RGB features with enhanced semantic content are obtained through convolution operation. Assume that the RGB feature is X RGB ∈R C×H×W , Mask feature is X m ∈R C×H×W The process can be expressed as:

[0052] X′ RGB =σ(Norm(Conv 3×3 ([X RGB ,X m ])))

[0053] Where [X RGB ,X m ] indicates that the two modal features are connected in the channel direction, Norm indicates the normalization operation, and σ indicates the ReLU activation function.

[0054] S4, the enhanced RGB features interact with the deep features through the multimodal cross fusion module to better obtain the spatial features in the image. The multimodal cross fusion module consists of two stages, such as Figure 4 As shown, they are the feature fusion stage and the feature enhancement stage:

[0055] S401, assuming that the enhanced RGB feature is X RGB ∈R C×H×W , the depth information after Transformer feature extraction is X Depth ∈R C×H×W ,The specific operation of MCAM can be expressed by the formula:

[0056] E RGB =Embed(X RGB )

[0057] E Depth =Embed(X Depth )

[0058] Q RGB ,K RGB,V RGB =Linear(E RGB )

[0059] Q Depth ,K Depth ,V Depth =Linear(E Depth )

[0060]

[0061]

[0062] F Depth =Q Depth Softmax(G RGB )

[0063] F RGB =Q RGB Softmax(G Depth )

[0064] F=Conv 3×3 (Reshape([F RGB ,F Depth ]))

[0065] Among them, Embed(·) represents channel embedding of modal information, transforming it from image features into tensor form, and Linear(·) represents linear mapping operation on modal tensor, respectively obtaining query, key and value vectors required for cross-attention operation. The efficient attention operation in MCAM can reduce the calculation of model parameters and is suitable for lightweight semantic segmentation tasks. At the same time, the cross-attention mechanism enables RGB and depth information to complement and model each other more effectively, dynamically adjust the weight of each modality, and improve the model's ability to capture local and global information. This method can significantly improve the performance of the fused multimodal model in complex environments. Finally, the feature maps of the two modalities are connected in the channel direction, and then the two modal information is fused through convolution operation.

[0066] S402, after the two modalities are fused, they enter the improved coordinate attention operation to perform feature enhancement operations on the input feature map F∈R C×H×W Perform average pooling and maximum pooling, where the horizontal direction can be expressed as:

[0067]

[0068]

[0069]

[0070] Similarly, the vertical features are pooled and concatenated, which can be expressed as:

[0071]

[0072]

[0073]

[0074] After completing the concatenation of average pooling and maximum pooling, the feature map undergoes a 1D convolution operation:

[0075]

[0076]

[0077] A h =Sigmod(F′ h ) A h ∈R C×H×1

[0078] A v =Sigmod(F′ v ) A v ∈R C×1×W

[0079] F out (i,j,c)=F(i,j,c)*A h (i,c)*A v (j,c) F out ∈R C×H×W

[0080] Where W h ,W v is a 1D convolution kernel, represents the convolution operation, and σ represents the ReLU activation function.

[0081] S5. The modal features after the multimodal feature fusion processing of the same layer of SegFormer are fused at different scales. By combining features at different levels, the contextual information of different scales in the image can be captured to improve the segmentation accuracy, especially for objects with blurred boundaries. At the same time, multi-scale fusion combines shallow detail information and deep semantic information to avoid feature loss, making the model more adaptable to scale changes in complex scenes, thereby improving the overall semantic segmentation effect. This architecture uses the MLPDecoder decoder. Assuming that the features of different layers are X1, X2, X3, and X4, the general process can be expressed as:

[0082] X out =Conv([X1,up(X2),up(X3),up(X4)])

[0083] Here, up(·) represents the upsampling operation, which adjusts the feature maps of different layers to a uniform size, and finally obtains the classification result through the convolution operation.

Claims

1. A semantic segmentation method based on parameter tuning and multimodal data fusion, characterized in that: S1, using the SegFormer encoder with frozen parameters to extract features from the three modal information in parallel, and using the cross-attention tuning module to assist the model in semantic segmentation; S2. In the first two layers of the SegFormer encoder, the Gate gate is used to realize the fusion of RGB features and Mask features. In the last two layers, the RGB features and Mask features are fused through the context information interaction module. S3: The RGB features enhanced by the Mask feature are used to interact with the deep features through a multimodal cross fusion module.

2. The semantic segmentation method based on parameter tuning and multimodal data fusion according to claim 1, characterized in that: In S1, the cross-attention tuning module is used to assist the model in semantic segmentation. The specific steps are as follows: In the Transformer block with frozen parameters, an additional attention module is configured, in which Q required for attention calculation comes from the self-attention architecture of the frozen Transformer block, K and V are obtained by linear mapping of the feature vectors, and the features after cross-attention processing are fused into the original Transformer block in the form of element-wise addition.

3. The semantic segmentation method based on parameter tuning and multimodal data fusion according to claim 1, characterized in that: In S2, the fusion of RGB features and Mask features adopts different fusion strategies at different layers. The specific steps are as follows: S201. In the first two layers of the encoder, a Gate is used to fuse the features of the two modalities, and learnable parameters are set to dynamically adjust the weights of the two modalities, and the fused features are obtained by element-wise addition. S202. In the last two layers of the encoder, the RGB features and the Mask features are connected in the channel direction, and the context information between the features is fused through the context information interaction module to enhance the semantic information of the RGB features. The context information interaction module includes a simple convolution operation, and the results are normalized and ReLU operated.

4. The semantic segmentation method based on parameter tuning and multimodal data fusion according to claim 1, characterized in that: In S3, the multimodal cross-fusion module consists of two stages: feature fusion stage and feature enhancement stage: S301, in the feature fusion stage, an efficient cross-attention operation is used to interact the two modal information, the interacted features are connected along the channel direction, and the two modal information is fused through a convolution operation; S302: In the feature enhancement stage, an improved coordinate attention operation is used to multiply the obtained weight with the original feature map to obtain the enhanced feature.

Images