Semantic segmentation method and system based on dual decoder

Abstract

The present invention belongs to the technical field of semantic segmentation, and specifically discloses a semantic segmentation method and system based on a dual decoder, which utilizes multiple spatial attention modules connected in series with a channel attention module to construct an AOA decoder; or multiple spatial attention modules connected in parallel with a channel attention module to construct an AIA decoder; if an AOA decoder is used, the feature information output by the encoder at different stages corresponds to the spatial attention module input into the AOA decoder, and the spatial information of the encoder is fused and input into the channel attention module of the AOA decoder to extract important features; if an AIA decoder is used, the feature information output by the encoder is input into the spatial attention module and the channel attention module of the AIA decoder, and the important features in different spatial feature information in the AIA decoder are calculated and determined; the obtained important features are output as semantic segmentation results. By adopting this technical solution, spatial and semantic information can be learned to improve the accuracy of semantic segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic segmentation, and relates to a semantic segmentation method and system based on a dual decoder. Background Art

[0002] Semantic segmentation is a basic task in the computer vision community and an important research direction in the field of computer vision. Its main goal is to combine pixels in an image based on semantic information, which can be regarded as a pixel-by-pixel classification task. With the rapid development of deep learning technology, semantic segmentation has also made great progress.

[0003] FCN (Fully Convolutional Networks for semantic Segmentation) is one of the most successful semantic segmentation algorithms based on deep learning. It makes full use of CNN (convolutional neural network) as a visual model to learn hierarchical features. Existing classic classification models can be easily converted into fully convolutional models, such as deep learning models such as VGGNet, GoogLeNet and ResNet (residual model). The FCN-based method causes the network to lose some spatial and semantic information during the forward propagation process. In addition, the receptive field is small during the convolution process, resulting in insufficient learned semantic information.

[0004] To solve these problems, the prior art introduces a large number of skip connections to utilize contextual information. Atrous convolution and conditional random fields are introduced to increase the receptive field during the convolution process and refine the segmentation results. The prior art proposes a Pyramid Pooling Module to aggregate different scales and contextual information.

[0005] At present, more and more studies are applying Transformer (visual transformer) to computer vision tasks. In the field of semantic segmentation, many Transformer-based models have also been proposed, such as Swin Transformer, Twins, Segmenter, SegFormer, etc. Transformer-based methods can achieve new and state-of-the-art results mainly from the multi-head self-attention mechanism. Its main advantage is that it can establish long-distance relationships between pixels and the calculation process can be parallelized. However, these Transformer-based methods focus more on the encoder design of the model rather than the decoder, resulting in the decoder's insufficient accuracy in semantic segmentation and affecting its use. Summary of the invention

[0006] The purpose of the present invention is to provide a semantic segmentation method and system based on dual decoders to improve the accuracy of semantic segmentation.

[0007] In order to achieve the above object, the basic scheme of the present invention is: a semantic segmentation method based on dual decoders, comprising the following steps:

[0008] Obtain feature information output by the encoder;

[0009] Use multiple spatial attention modules in series with a channel attention module to build an AOA decoder; or connect multiple spatial attention modules in parallel with a channel attention module to build an AIA decoder;

[0010] If the AOA decoder is used, the feature information output at different stages of the encoder corresponds to multiple spatial attention modules input into the AOA decoder, and the spatial information in the feature information at different stages of the encoder is extracted; the spatial information in the feature information at different stages of the encoder is fused and input into the channel attention module of the AOA decoder, and the important features in the spatial information at different stages are extracted;

[0011] If an AIA decoder is used, the feature information output at different stages of the encoder corresponds to multiple spatial attention modules input into the AIA decoder, and at least three channels are input, of which at least two channels are used to obtain the spatial information of the feature, and one channel adds the semantic information of the feature, and the softmax output of the feature information output at different stages of the encoder is obtained; the feature information output at different stages of the encoder is fused and input into the channel attention module of the AIA decoder to obtain multiple weights; the weight is multiplied by the softmax output to determine the important features in different spatial feature information in the AIA decoder;

[0012] The important features are output as semantic segmentation results.

[0013] The working principle and beneficial effects of this basic scheme are as follows: the attention mechanism shows great advantages in visual tasks. This scheme customizes the spatial attention module and channel attention module to effectively learn the spatial information and semantic information of the features, realize important feature extraction, and improve the accuracy of semantic segmentation.

[0014] Further, the encoding method of the encoder is as follows:

[0015] Setting the FaPN* module improves the FaPN module by replacing the feature selection module with a convolution module with a kernel size of 1:

[0016]

[0017]

[0018]

[0019] Among them, Fl and F v are the inputs of FaPN* from the horizontal connection and vertical pathway, respectively; δ(·) represents 1x1 convolution, up(·) represents the upsampling function; f a (·) represents the feature alignment module, which aims to solve the problem of feature misalignment; represents the output after 1x1 convolution, F fam represents the output after feature alignment, F out Indicates the final splicing and F fam The subsequent output, concat represents the concatenation function.

[0020] The FaPN* module solves the feature errors in the feature pyramid constructed in the encoder, reducing the computational complexity and improving the segmentation accuracy.

[0021] Furthermore, the spatial attention module includes three input channels, two of which are used to obtain the spatial information of the feature, and an additional skip connection is set to increase the semantic information of the feature:

[0022] X sa =soft max(reshape(conv 1×1 (X in )))

[0023]

[0024] Among them, X in ∈R C′×H×W is the input feature map, X sa ∈R C / 2×H×W represents the softmax output of the spatial attention module, X out ∈R (C′+C / 2)×H×W represents the output of the spatial attention module, R C / 2×H×W represents the characteristic space of C / 2xHxW over the real number field, and the other two are the same; represents element-wise multiplication, represents concatenation; softmax represents the normalized exponential function, and reshape represents the transformation of the dimension of the tensor;

[0025] In order to reduce the computational complexity, when calculating the spatial attention, a convolution kernel with a size of 1 is used. 1×1 X in The number of channels is reduced to C / 2, where C is the number of channels of the output feature map after fusion.

[0026] Set up an appropriate spatial attention module for easy use.

[0027] Furthermore, the channel attention module uses global average pooling and softmax function to model the importance information of different channels of input features and assign different importance to feature maps of different channels.

[0028] Simple structure and easy to use.

[0029] The present invention also provides a semantic segmentation system based on the method of the present invention, comprising an encoder, and an AOA decoder or an AIA decoder, wherein the output end of the encoder is connected to the input end of the AOA decoder or the input end of the AIA decoder.

[0030] The system uses AOA decoder or AIA decoder to extract important features and optimize semantic segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a structural diagram of an AOA decoder of a semantic segmentation method based on a dual decoder of the present invention;

[0032] Figure 2 It is a schematic diagram of the structure of the AIA decoder of the semantic segmentation method based on dual decoders of the present invention;

[0033] Figure 3 It is a schematic diagram of the structure of an encoder of a semantic segmentation method based on dual decoders of the present invention;

[0034] Figure 4 Schematic diagram of the structure of the spatial attention module of the semantic segmentation method based on dual decoders of the present invention;

[0035] Figure 5 It is a structural diagram of the channel attention module of the semantic segmentation method based on dual decoders of the present invention. DETAILED DESCRIPTION

[0036] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0037] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0038] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0039] The present invention discloses a semantic segmentation method based on a dual decoder, such as Figure 3 As shown, the following steps are included:

[0040] Obtain feature information output by the encoder; the encoder may be, but is not limited to, the encoder of SegFormer.

[0041] Using multiple spatial attention modules (SA) in series with a channel attention module (CA), SA and CA are arranged in sequence to form an AOA module, such as Figure 1 As shown in Figure 1, an AOA (Attention On Attention) decoder is constructed. The AOA decoder is a serial structure. Alternatively, multiple spatial attention modules are connected in parallel with a channel attention module, and SA and CA are arranged in parallel to form an AIA module, as shown in Figure 1. Figure 2 As shown in the figure, an AIA (Attention In Attention) decoder is constructed, and the AIA decoder is a parallel structure; the two attention-based decoders make full use of the spatial information and channel information of the features in the decoder.

[0042] If an AOA decoder is used, the feature information outputted by the encoder at different stages corresponds to the multiple spatial attention modules inputted into the AOA decoder, and the spatial information in the feature information at different stages of the encoder is extracted; the spatial information in the feature information at different stages of the encoder is fused and inputted into the channel attention module of the AOA decoder, and the important features in the spatial information at different stages are extracted, and redundant information is reduced; in this embodiment, four SA modules are used to learn spatial information from different stages of the encoder respectively, and after fusing the outputs of the four stages, the importance of the spatial information learned between different stages is modeled through the CA module, and its redundant information is reduced at the same time. On the whole, the SA module and the CA module constitute the AOA module in sequence.

[0043] If the AIA decoder is used, the feature information output at different stages of the encoder corresponds to multiple spatial attention modules input into the AIA decoder, and at least three channels are input, of which at least two channels are used to obtain the spatial information of the features, and one channel adds the semantic information of the features to obtain the softmax output of the feature information output at different stages of the encoder; softmax is a normalized exponential function:

[0044]

[0045] Among them, e i represents the natural exponential with base e, ∑ jej represents the sum of the natural exponentials of each j, S i Represents the value obtained by normalizing i. The feature information output by different stages of the encoder is fused and input into the channel attention module of the AIA decoder to obtain multiple weights; the weights are multiplied by the softmax output to determine the important features in different spatial feature information in the AIA decoder; four weights α, β, γ and λ can be used to dynamically determine the importance of the spatial information learned by the four SA modules through CA learning. In order to better integrate SA and CA, the weights learned by softmax in the SA module are directly multiplied by the spatial attention, which is why it is called AIA. The role of the CA module in AIA is different from that in AOA: in AOA, CA is used to extract important features after feature fusion and reduce redundant information. In AIA, CA is used to learn different weights, which are directly applied to the softmax outputs in the four SA modules to dynamically determine the importance of different spatial attention.

[0046] The important features are output as semantic segmentation results.

[0047] In a preferred embodiment of the present invention, the encoding method of the encoder is as follows:

[0048] A feature pyramid is constructed in the encoder to learn features of different scales, and the FaPN* module is set to alleviate the feature misalignment problem when fusion of features of different scales, reducing the computational complexity and improving the segmentation accuracy. In the encoder, the feature map of each stage is fused with the feature map from the lateral connection. Although the small-scale feature map is upsampled, the fusion of small-scale and large-scale feature maps still inevitably has feature misalignment, which will lead to the loss of segmentation accuracy.

[0049] FaPN consists of two modules, the Feature Selection Module (FSM) and FAM. FAM is implemented through a deformable convolutional network to learn the position deviation between features. FSM was proposed to model the importance of different channels in features and suppress redundant information. However, through the ablation experiment of FSM on Cityscapes, it was found that FSM had little effect on improving segmentation accuracy. It only brought an improvement of 0.3% in mIoU, but caused the number of model parameters to increase by 1.4 million. Therefore, FaPN* (improved FaPN module) was proposed, and the FaPN* module was set to improve the FaPN module. It replaced the feature selection module FSM with a convolution module with a kernel size of 1:

[0050]

[0051]

[0052]

[0053] Among them, F l and F v are the inputs of FaPN* from the horizontal connection and vertical pathway, respectively; δ(·) represents 1x1 convolution, up(·) represents the upsampling function; f a (·) represents the Feature Alignment Module (FAM), which aims to solve the problem of feature misalignment; represents the output after 1x1 convolution, F fam represents the output after feature alignment, F out Indicates the final splicing and F fam The subsequent output; concat represents the concatenation function, which concatenates the feature tensors in a certain dimension.

[0054] In a preferred embodiment of the present invention, Figure 4 As shown in the figure, the spatial attention module includes three input channels, two of which are used to obtain the spatial information of the feature, and an additional jump connection is set to increase the semantic information of the feature:

[0055] X sa =soft max(reshape(conv 1×1 (X in )))

[0056]

[0057] Among them, X in ∈R C′×H×W is the input feature map, Xsa ∈R C / 2×H×W represents the softmax output of the spatial attention module, X out ∈R (C′+C / 2)×H×W represents the output of the spatial attention module, R C / 2×H×W represents the characteristic space of C / 2xHxW over the real number field, and the other two are the same; represents element-wise multiplication, Indicates concatenation; softmax indicates a normalized exponential function; reshape indicates transforming the dimension of a tensor, for example, reshape a 3x4x5 tensor into a 3x2x10 tensor;

[0058] In order to reduce the computational complexity, when calculating the spatial attention, a convolution kernel with a size of 1 is used. 1×1 (Convolution with kernel size 1) X in The number of channels is reduced to C / 2, where C is the number of channels of the output feature map after fusion.

[0059] like Figure 5 As shown in Figure 1, the channel attention module uses global average pooling and softmax function to model the importance information of different channels of the input features and assign different importance to the feature maps of different channels.

[0060] The present invention also provides a semantic segmentation system based on the method of the present invention, comprising an encoder, and an AOA decoder or an AIA decoder, wherein the output of the encoder is connected to the input of the AOA decoder or the input of the AIA decoder. The system uses the AOA decoder or the AIA decoder to extract important features and optimize the semantic segmentation accuracy.

[0061] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0062] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A semantic segmentation method based on dual decoders, characterized in that: The steps include: Obtain feature information output by the encoder; Use multiple spatial attention modules in series with a channel attention module to build an AOA decoder; or connect multiple spatial attention modules in parallel with a channel attention module to build an AIA decoder; If the AOA decoder is used, the feature information output by the encoder at different stages corresponds to multiple spatial attention modules input into the AOA decoder, and the spatial information in the feature information at different stages of the encoder is extracted; The spatial information in the feature information of different stages of the encoder is fused and input into the channel attention module of the AOA decoder to extract the important features in the spatial information of different stages; If an AIA decoder is used, the feature information output at different stages of the encoder corresponds to multiple spatial attention modules input into the AIA decoder respectively, and the spatial information in the feature information at different stages of the encoder is extracted. The feature information output at different stages of the encoder is fused and input into the channel attention module of the AIA decoder to obtain multiple weights; the weights are multiplied by the spatial information of the feature information at different stages, and the multiplication results are fused to determine the important features in different spatial feature information in the AIA decoder; The spatial attention module includes three input channels, two of which are used to obtain the spatial information of the feature, and an additional skip connection is set to increase the semantic information of the feature: X sa =soft max(reshape(conv 1×1 (X in ))) in, is the input feature map, represents the softmax output of the spatial attention module, represents the output of the spatial attention module, R C / 2×H×W represents the characteristic space of C / 2xHxW over the real number field, and the other two are the same; represents element-wise multiplication, represents concatenation; softmax represents the normalized exponential function, and reshape represents the transformation of the dimension of the tensor; In order to reduce the computational complexity, when calculating the spatial attention, a convolution kernel with a size of 1 is used. 1 × 1 X in The number of channels is reduced to C / 2, where C is the number of channels of the output feature map after fusion; The important features are output as semantic segmentation results.

2. The dual decoder based semantic segmentation method according to claim 1, characterized in that: The encoding method of the encoder is as follows: Setting the FaPN* module improves the FaPN module by replacing the feature selection module with a convolution module with a kernel size of 1: Among them, F l and F v are the inputs of FaPN* from the horizontal connection and vertical pathway, respectively; δ(·) represents 1x1 convolution, up(·) represents the upsampling function; f a (·) represents the feature alignment module, which aims to solve the problem of feature misalignment; represents the output after 1x1 convolution, F fam represents the output after feature alignment, F out Indicates the final splicing and F fam The subsequent output, concat represents the concatenation function.

3. The semantic segmentation method based on dual decoders as claimed in claim 1, characterized in that: The channel attention module uses global average pooling and softmax function to model the importance information of different channels of input features and assign different importance to feature maps of different channels.

4. A semantic segmentation system based on the method according to any one of claims 1 to 3, characterized in that: The invention comprises an encoder and an AOA decoder or an AIA decoder, wherein the output end of the encoder is connected to the input end of the AOA decoder or the input end of the AIA decoder.

Images