Image segmentation method and system fusing sequence modeling and multi-scale attention mechanism

Abstract

The invention discloses an image segmentation method and system fusing sequence modeling and a multi-scale attention mechanism, and relates to the field of computer vision, and the method comprises the steps: carrying out the extraction of multi-scale detail and space information through an encoder based on a multi-scale convolution residual fusion module, and gradually carrying out the downsampling to increase a receptive field; performing multi-scale and global context feature extraction and coding based on a multi-scale context global modeling module through a bridging layer; and through a decoder, encoder features are fused based on up-sampling, convolution and a double-guide gating fusion module, so that recovery and refinement of high-resolution semantic features are realized, and an image segmentation result of fusion sequence modeling and a multi-scale attention mechanism is obtained. According to the method, the multi-scale features are fully extracted and fused, the context sensing capability is enhanced by using sequence modeling, the features of the encoder and the decoder are dynamically fused through a gating mechanism, the segmentation precision and the edge detail performance are improved, and the problem that the model performance and generalization capability are limited in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to an image segmentation method and system fusing sequence modeling and multi-scale attention mechanism. BACKGROUND

[0002] Traditional semantic segmentation models often face the following challenges: Local detail feature extraction is insufficient, resulting in inaccurate segmentation edges; Multi-scale context information fusion effect is not good, affecting the adaptability of the model to complex scenes; The fusion mode of the skip connection between the encoder and the decoder is single, resulting in insufficient feature expression in the decoding stage.

[0003] In the prior art, although there are hollow space pyramid pooling modules, multi-scale convolution modules and gating mechanisms, there is still a lack of an overall network architecture that effectively combines multi-scale convolution, sequence modeling and gating, resulting in limited model performance and generalization ability SUMMARY In view of the above deficiencies in the prior art, the present application provides an image segmentation method and system fusing sequence modeling and multi-scale attention mechanism, which can fully extract and fuse multi-scale features, while using sequence modeling to enhance context awareness, and dynamically fusing encoder and decoder features through a gating mechanism to improve segmentation accuracy and edge detail performance, thereby solving the problem of limited model performance and generalization ability in the prior art.

[0004] In order to achieve the above application purpose, the technical scheme adopted by the present application is as follows: an image segmentation method fusing sequence modeling and multi-scale attention mechanism, adopting an encoder-bridge layer-decoder architecture, comprising the following steps: Through the encoder, based on the multi-scale convolution residual fusion module, multi-scale detail and spatial information is extracted, and the receptive field is gradually increased by down-sampling; Through the bridge layer, based on the multi-scale context global modeling module, multi-scale and global context feature extraction and encoding are performed; Through the decoder, based on the up-sampling, convolution and double-guided gating fusion module, the encoder features are fused to realize high-resolution semantic feature recovery and refinement, and obtain the image segmentation result fusing sequence modeling and multi-scale attention mechanism.

[0005] Further, the processing process of the multi-scale convolution residual fusion module includes: Multi-scale convolution branch processing: using 3x3, 5x5, 7x7 convolution kernels to respectively perform convolution, batch normalization and ReLU activation operations on the input features to obtain three branch outputs , and ;

[0006]

[0007]

[0008] wherein, , and are 3x3, 5x5 and 7x7 convolutions respectively, is batch normalization, is an activation function; feature splicing and fusion: splicing and convolution fusion of , and in the channel dimension to obtain fused features :

[0009]

[0010] wherein, is the spliced feature, is splicing, is the real space where the tensor is located, i.e. the dimension of the feature map, is the batch size, is the number of channels, is the image height, is the image width, is a 1x1 convolution; residual connection processing: 1x1 convolution is performed on the input feature to obtain residual output :

[0011] output result: element-wise addition of the fused feature and the residual output to obtain the final output of the multi-scale convolution residual fusion module : .

[0012] Further, the processing process of the multi-scale context global modeling module includes: three hollow convolution branch processing: branch 1: 1x1 convolution, batch normalization and ReLU activation operation are performed on the input feature to obtain the output of branch 1 :

[0013] Branch 2: 3x3 convolution with dilation rate of 6, batch normalization and ReLU activation are performed on the input feature to obtain the output of Branch 2 :

[0014] Branch 3: 3x3 convolution with dilation rate of 12, batch normalization and ReLU activation are performed on the input feature to obtain the output of Branch 3 :

[0015] Global pooling branch processing: global average pooling, 1x1 convolution and ReLU activation are performed on the input feature, and then up-sampling is performed to restore the original size to obtain the global pooling branch output :

[0016] wherein, is the global average pooling, is the up-sampling, is the original size; Sequence modeling processing: the outputs of the three dilated convolution branches are respectively flattened into sequences, and layer normalization and multilayer perceptron processing are performed, and finally the processing results are restored to feature maps :

[0017]

[0018]

[0019]

[0020] wherein, is the original input image, is the flattened feature sequence, is the layer normalized sequence, is the layer normalization, is the multilayer perceptron, is the activation function, is the weight matrix of the first layer linear transformation, is the weight matrix of the second layer linear transformation; ​​​Feature fusion processing: the three hollow convolution branch outputs and the global pooling branch output are spliced in the channel dimension, and 1x1 convolution dimension reduction fusion is performed to obtain the final output of the multi-scale context global modeling module :

[0021]

[0022] wherein, is the result of splicing in the channel dimension.

[0023] Further, the processing process of the double-guided gating fusion module includes: Jump feature preliminary encoding: performing 1x1 convolution, batch normalization and ReLU activation operation on the jump feature map from the encoder to obtain the new feature map output after projection :

[0024] Decoding feature guided encoding: performing 1x1 convolution, batch normalization and ReLU activation operation on the feature map from the current layer of the decoder to obtain the new feature map output after projection :

[0025] Gating weight calculation: splicing and in the channel dimension, then performing 1x1 convolution, and then calculating the fusion weight through the Sigmoid activation function :

[0026] wherein, is the Sigmoid activation function; Gating weighted fusion: performing element-wise weighting on through the fusion weight , and then adding to obtain the fusion result :

[0027] wherein, represents element-wise multiplication; Fusion enhancement output: performing 3x3 convolution, batch normalization and ReLU activation operation on the fusion result to obtain the final output of the double-guided gating fusion module : .

[0028] Further, the encoder comprises an encoder first layer, an encoder second layer and an encoder third layer connected in sequence: The encoder first layer: based on a multi-scale convolution residual fusion module, using 3x3, 5x5, 7x7 convolution and fusion, introducing residual connection, and then through convolution, the channel number is upgraded to 64, output x1: [B, 64, 256, 256]; The encoder second layer: through maximum pooling and convolution operation, the space is compressed, the receptive field is expanded, and the middle layer semantic information is extracted, output x2: [B, 128, 128, 128]; The encoder third layer: through maximum pooling and convolution operation, the space is further compressed, and the deep layer semantic feature is extracted, output x3: [B, 128, 64, 64].

[0029] Further, the bridge layer is based on a multi-scale context global modeling module, and outputs bridge: [B, 128, 64, 64]; The decoder comprises: The decoder first layer: up-sampling and convolution operation are performed on the bridge layer output bridge, channel compression is realized, and output d1: [B, 64, 128, 128]; The decoding fusion layer: based on a double-guided gating fusion module, d1 and the encoder first layer output x1 are fused, and output d2: [B, 64, 128, 128].

[0030] A system of an image segmentation method fusing sequence modeling and multi-scale attention mechanism, comprising: An encoder based on a multi-scale convolution residual fusion module, used for realizing multi-scale detail and spatial information extraction, and gradually down-sampling to increase the receptive field; A bridge layer based on a multi-scale context global modeling module, used for multi-scale and global context feature extraction and encoding; A decoder based on up-sampling, convolution and double-guided gating fusion module to fuse the encoder features, used for realizing high-resolution semantic feature recovery and refinement, and obtaining the image segmentation result of the fusion sequence modeling and multi-scale attention mechanism.

[0031] The beneficial effects of the present application are: The network structure of the present application effectively improves the detail capture and context understanding ability in image segmentation by fusing multi-scale convolution, sequence modeling and gating fusion mechanism. In various image segmentation tasks, it shows higher accuracy and better edge preservation ability, has good generalization ability and practical application value. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The overall structure diagram of the network of the present application.

[0033] Figure 2 The structure diagram of the multi-scale convolution residual fusion module.

[0034] Figure 3 The structure diagram of the multi-scale context global modeling module.

[0035] Figure 4 The structure diagram of the dual-guided gate fusion module. DETAILED DESCRIPTION

[0036] The present application will be further described below in conjunction with the drawings and specific embodiments.

[0037] Embodiment 1, as shown in the image segmentation method of fusing sequence modeling and multi-scale attention mechanism, adopts an encoder-bridge layer-decoder architecture, including the following steps: Figure 1 Through the encoder, multi-scale details and spatial information are extracted based on the multi-scale convolution residual fusion module, and the receptive field is gradually increased by down-sampling; As shown in the processing process of the multi-scale convolution residual fusion module includes: Multi-scale convolution branch processing: using 3x3, 5x5, 7x7 convolution kernels respectively to perform convolution, batch normalization and ReLU activation operation on the input features Figure 2 , to obtain three branch outputs , and ;

[0038]

[0039]

[0040] Among them, , and are 3x3, 5x5 and 7x7 convolution respectively, is batch normalization, is the activation function; Feature splicing and fusion: splicing and convolution fusion of , and in the channel dimension to obtain the fusion feature :

[0041] ​​

[0042] wherein, is a concatenation feature, is a concatenation, is a real space where the tensor is located, i.e., the dimension of the feature map, is a batch size, is a number of channels, is an image height, is an image width, is a 1x1 convolution; residual connection processing: 1x1 convolution is performed on the input feature to obtain a residual output :

[0043] output result: element-wise addition is performed on the fusion feature and the residual output to obtain the final output of the multi-scale convolution residual fusion module : .

[0044] The encoder comprises an encoder first layer, an encoder second layer and an encoder third layer connected in sequence: The encoder first layer: based on the multi-scale convolution residual fusion module, 3x3, 5x5 and 7x7 convolutions are used and fused, a residual connection is introduced, and the number of channels is then increased to 64 through convolution, and an output x1 is obtained: [B, 64, 256, 256]; The encoder second layer: through maximum pooling and convolution operations, the space is compressed, the receptive field is expanded, and the middle-level semantic information is extracted, and an output x2 is obtained: [B, 128, 128, 128]; The encoder third layer: through maximum pooling and convolution operations, the space is further compressed, and deep-level semantic features are extracted, and an output x3 is obtained: [B, 128, 64, 64].

[0045] Through the bridging layer, based on the multi-scale context global modeling module, multi-scale and global context feature extraction and encoding are performed; As shown in Figure 3 , the processing process of the multi-scale context global modeling module comprises: three hollow convolution branch processing: Branch 1: 1x1 convolution, batch normalization and ReLU activation operations are performed on the input feature to obtain the output of branch 1:

[0046] Branch 2: 1x1 convolution, batch normalization and ReLU activation operations are performed on the input feature A 3x3 convolution with a dilation rate of 6, batch normalization and ReLU activation are performed to obtain the output of branch 2 :

[0047] Branch 3: The input feature A 3x3 convolution with a dilation rate of 12, batch normalization and ReLU activation are performed to obtain the output of branch 3 :

[0048] Global pooling branch processing: The input feature After global average pooling, 1x1 convolution and ReLU activation, up-sampling is performed to restore the original size to obtain the global pooling branch output :

[0049] wherein, is the global average pooling, is the up-sampling, is the original size; Sequence modeling processing: The outputs of the three dilated convolution branches are respectively flattened into sequences, and layer normalization and multilayer perception processing are performed, and finally the processing results are restored to feature maps :

[0050]

[0051]

[0052]

[0053] wherein, is the original input image, is the flattened feature sequence, is the layer normalized sequence, is the layer normalization, is the multilayer perception, is the activation function, is the weight matrix of the first layer linear transformation, is the weight matrix of the second layer linear transformation; Feature fusion processing: The outputs of the three dilated convolution branches and the global pooling branch are spliced in the channel dimension, and 1x1 convolution dimension reduction fusion is performed to obtain the final output of the multi-scale context global modeling module :

[0054]

[0055] wherein, is the result of concatenation in the channel dimension.

[0056] The bridging layer is based on a multi-scale context global modeling module, and outputs bridge: [B, 128, 64, 64].

[0057] Through the decoder, the encoder features are fused based on upsampling, convolution and double-guided gating fusion module to realize the recovery and refinement of high-resolution semantic features, and obtain the image segmentation result of fusion sequence modeling and multi-scale attention mechanism.

[0058] As shown in Figure 4 , the processing process of the double-guided gating fusion module includes: The initial encoding of the skip feature: the skip feature map from the encoder is subjected to 1x1 convolution, batch normalization and ReLU activation operation to obtain the new feature map output after projection:

[0059] The decoding feature guided encoding: the feature map from the current layer of the decoder is subjected to 1x1 convolution, batch normalization and ReLU activation operation to obtain the new feature map output after projection:

[0060] Gating weight calculation: concatenates and in the channel dimension, then performs 1x1 convolution, and then calculates the fusion weight through the Sigmoid activation function:

[0061] wherein, is the Sigmoid activation function; Gating weighted fusion: the fusion result is obtained by element-wise weighting of and adding :

[0062] wherein, represents element-wise product; ​Fusion enhanced output: on the fusion result 3x3 convolution, batch normalization and ReLU activation operation are performed to obtain the final output of the double guided gating fusion module : .

[0063] The decoder comprises: The first layer of the decoder: up-sampling and convolution operation are performed on the bridge layer output bridge to realize channel compression, and the output d1 is [B, 64, 128, 128]; The decoding fusion layer: based on the double guided gating fusion module, d1 is fused with the output x1 of the first layer of the encoder, and the output d2 is [B, 64, 128, 128].

[0064] Finally, the original resolution of the input is restored, and 1x1 convolution is used to reduce the output channel, and the output y is [B, C, H, W].

[0065] In an embodiment of the present application, the experimental environment is as follows: CPU: Intel Xeon Silver 4314 Memory: 128M GPU: NVIDIA A40x2 Operating system: Ubuntu 22.04.3 LTS The network model proposed in the present application has excellent performance on 2 public data sets (multi-class semantic segmentation data set ADE20K and binary medical image segmentation data set ISIC2018), and has been trained for 100 rounds. Good results are obtained on 3 evaluation indicators, verifying the robustness and accuracy of the model in different scenarios, and having wide application potential.

[0066] Table 1 Experimental results

[0067] The present application can extract local and long-distance texture information at the same time through the multi-scale convolution residual fusion module, which fuses three different convolution receptive field branch structures, and realizes multi-scale feature enhancement through channel splicing and convolution fusion.

[0068] Through the multi-scale context global modeling module, a plurality of hollow convolution branches and global pooling branches are used to extract different scale features, and each branch feature is encoded through sequence modeling (composed of layer normalization and multilayer perceptron) to finally generate a multi-scale context rich feature representation.

[0069] Through the double-guided gate fusion module, the features from the skip connection and the decoding end are guided at the same time, the fusion weight is calculated, the dynamic selective enhancement of the skip features is realized, and then the fusion result is residual fused with the decoding features to improve the expression ability of the fusion result.

[0070] In an embodiment 2, a system of an image segmentation method based on fusion sequence modeling and multi-scale attention mechanism comprises: An encoder based on a multi-scale convolution residual fusion module is used to realize multi-scale detail and spatial information extraction and gradually down-sampling to increase the receptive field; A bridge layer based on a multi-scale context global modeling module is used to extract and encode multi-scale and global context features; A decoder based on up-sampling, convolution and double-guided gate fusion module fusion of the encoder features is used to realize high-resolution semantic feature recovery and refinement, and obtain the image segmentation result based on fusion sequence modeling and multi-scale attention mechanism.

[0071] Those skilled in the art will appreciate that the embodiments described herein are presented for the purpose of helping the reader understand the principles of the present application and should be understood as not limiting the scope of protection of the present application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. An image segmentation method integrating sequence modeling and multi-scale attention mechanisms, characterized in that, The encoder-bridge-decoder architecture includes the following steps: Through the encoder, multi-scale details and spatial information are extracted based on the multi-scale convolutional residual fusion module, and the receptive field is gradually downsampled to increase the receptive field. Through the bridging layer, multi-scale and global context features are extracted and encoded based on the multi-scale context global modeling module; By using a decoder, the encoder features are fused based on upsampling, convolution, and dual-guided gating fusion modules to achieve high-resolution semantic feature recovery and refinement, resulting in image segmentation results with fused sequence modeling and multi-scale attention mechanisms.

2. The image segmentation method integrating sequence modeling and multi-scale attention mechanism according to claim 1, characterized in that, The processing steps of the multi-scale convolutional residual fusion module include: Multi-scale convolutional branching: 3×3, 5×5, and 7×7 convolutional kernels are used to process the input features respectively. Perform convolution, batch normalization, and ReLU activation operations to obtain three branch outputs. , and ; in, , and Convolutions of 3×3, 5×5, and 7×7 respectively. For batch normalization, For activation functions; Feature splicing and fusion: , and The features are obtained by concatenating and convolutional fusion along the channel dimension. : in, For splicing features, For splicing, Let be the real space in which the tensor resides, i.e., the dimension of the feature map. For batch size, For the number of channels, Image height, Image width, A 1×1 convolution; Residual connection processing: processing input features Perform a 1×1 convolution to obtain the residual output. : Output: fused features With residual output Element-wise addition is performed to obtain the final output of the multi-scale convolutional residual fusion module. : 。 3. The image segmentation method integrating sequence modeling and multi-scale attention mechanism according to claim 2, characterized in that, The processing steps of the multi-scale context global modeling module include: Three dilated convolution branches are used for processing: Branch 1: Input features Perform 1×1 convolution, batch normalization, and ReLU activation to obtain the output of branch 1. : Branch 2: Input features Perform a 3×3 convolution with a dilation rate of 6, batch normalization, and ReLU activation to obtain the output of branch 2. : Branch 3: Input features Perform a 3×3 convolution with a dilation rate of 12, batch normalization, and ReLU activation to obtain the output of branch 3. : Global pooling branch processing: processing input features After performing global average pooling, 1×1 convolution, and ReLU activation, the sample size is upsampled back to the original size to obtain the output of the global pooling branch. : in, For global average pooling, For upsampling, Original dimensions; Sequence modeling processing: The outputs of the three dilated convolution branches are flattened into sequences, layer normalization and multilayer perceptron processing are performed, and finally the processing results are restored to feature maps. : in, The original input image, The flattened feature sequence, The sequence is after layer normalization. For layer normalization, It is a multilayer perceptron. For activation function, The weight matrix is ​​the first-level linear transformation. This is the weight matrix for the second-level linear transformation; Feature fusion processing: The outputs of the three dilated convolution branches and the global pooling branch are concatenated along the channel dimension and then fused using 1×1 convolution dimensionality reduction to obtain the final output of the multi-scale context global modeling module. : in, This is the result of splicing along the channel dimension.

4. The image segmentation method integrating sequence modeling and multi-scale attention mechanism according to claim 3, characterized in that, The processing steps of the dual-guided gating fusion module include: Preliminary encoding of skip features: skip feature maps from the encoder Perform 1×1 convolution, batch normalization, and ReLU activation to obtain a new feature map output after projection. : Decoding Feature-Guided Encoding: Encoding the feature map from the current layer of the decoder Perform 1×1 convolution, batch normalization, and ReLU activation to obtain a new feature map output after projection. : Gating weight calculation: and After concatenating the channels, a 1×1 convolution is performed, followed by calculation of the fusion weights using the Sigmoid activation function. : in, Use the Sigmoid activation function; Gated weighted fusion: By fusing weights right Perform element-wise weighting, then combine with Adding them together yields the fusion result. : in, Represents element-wise product; Fusion Enhancement Output: This refers to the output of the fusion result. Perform 3×3 convolution, batch normalization, and ReLU activation operations to obtain the final output of the dual-guided gated fusion module. : 。 5. The image segmentation method integrating sequence modeling and multi-scale attention mechanism according to claim 4, characterized in that, The encoder comprises a first encoder layer, a second encoder layer, and a third encoder layer connected in sequence: The encoder's first layer: Based on a multi-scale convolutional residual fusion module, 3×3, 5×5, and 7×7 convolutions are used and fused to introduce residual connections. Then, the number of channels is increased to 64 through convolution, and the output x1 is [B,64,256,256]. The second layer of the encoder compresses the space, expands the receptive field, and extracts semantic information from the middle layer through max pooling and convolution operations, outputting x2: [B,128,128,128]; Encoder third layer: The space is further compressed by max pooling and convolution operations to extract deep semantic features, and the output is x3: [B,128,64,64].

6. The image segmentation method fusing sequence modeling and multi-scale attention mechanism according to claim 5, characterized in that, The bridging layer is based on a multi-scale context global modeling module and outputs bridge: [B, 128, 64, 64]; The decoder includes: Decoder layer 1: Upsamples and convolutions the bridge output to achieve channel compression, outputting d1: [B,64,128,128]; Decoding fusion layer: Based on the dual-guided gating fusion module, d1 is fused with the encoder first layer output x1, and the output d2 is [B,64,128,128].

7. A system for image segmentation using the fusion sequence modeling and multi-scale attention mechanism as described in any one of claims 1-6, comprising: The encoder, based on a multi-scale convolutional residual fusion module, is used to extract multi-scale details and spatial information, and gradually downsample to increase the receptive field; The bridging layer, based on the multi-scale context global modeling module, is used for multi-scale and global context feature extraction and encoding; The decoder, based on upsampling, convolution, and dual-guided gating fusion modules, fuses encoder features to achieve high-resolution semantic feature recovery and refinement, obtaining image segmentation results with fused sequence modeling and multi-scale attention mechanisms.

Images