An adjustable multi-scale encoder enhancement method for image semantic segmentation

Abstract

The present invention discloses an adjustable multi-scale encoder enhancement method for semantic segmentation, which belongs to the field of semantic segmentation. The method includes performing a routing query on the current hierarchical structure and automatically selecting a specific hierarchical structure for channel input; adaptively extracting parameters of different numbers of channels according to the query index; inputting a specific scale feature map into a feature extraction module based on channel adaptive parameters, extracting multi-scale context information and performing cascade merging to obtain a semantic information feature map; performing weight learning on the feature map of the feature extraction module through 3x3 convolution, merging the output feature map with the main branch and inputting it to the next layer for feature extraction operation, and the decoder performs feature aggregation and decoding. The present invention adopts the above method for the Transformer semantic segmentation encoder, flexibly enhancing the Transformer block of any scale, being able to adapt to different needs and scenarios, and improving the segmentation effect of the network.

Description

Technical Field

[0001] The present invention relates to the technical field of semantic segmentation, and in particular to an adjustable multi-scale encoder enhancement method for image semantic segmentation. Background Art

[0002] In the field of semantic segmentation, ViT is gaining the same attention as fully convolutional networks as a benchmark model. It brings the Transformer model, originating from the natural language processing (NLP) field, to image tasks, achieving groundbreaking innovation. SegFormer, based on ViT, incorporates a multi-layered Transformer encoder that outputs multi-scale features and proposes an MLP decoder that aggregates information from different layers.

[0003] The Transformer structure has a strong dependence on the number of training data samples, which hinders the segmentation effect of some targets with a small number of samples. Summary of the Invention

[0004] The purpose of the present invention is to provide an adjustable multi-scale encoder enhancement method for image semantic segmentation to solve the problems existing in the background technology.

[0005] To achieve the above object, the present invention provides an adjustable multi-scale encoder enhancement method for image semantic segmentation, comprising the following steps:

[0006] S1, scale routing query, performs routing query on the current hierarchical structure and automatically selects a specific hierarchical structure for channel input;

[0007] S2, channel adaptive extraction, adaptively extracts the multi-scale feature parameters extracted by the attention mechanism with different numbers of channels according to the S1 query index;

[0008] S3, feature extraction, inputs the specific scale feature map into the feature extraction module according to the parameters of channel adaptive extraction, extracts multi-scale context information and performs concatenation and merging to obtain the semantic information feature map;

[0009] S4, feature fusion, the feature map of the feature extraction module is weighted through 3x3 convolution, the output feature map is merged with the main branch and input to the next layer for feature extraction, and then handed over to the decoder for feature aggregation and decoding.

[0010] Preferably, the S1 process is as follows:

[0011] By performing a cyclic routing query on the four-scale features output by the backbone network, the judgment of the four-scale input features is realized, and the four-scale features layer0-3 are passed as parameters through index variables so that the internal feature extraction module can select the appropriate parameters;

[0012] The number of layers obtained by enumerating the parameters is assigned to the index, and this index is passed from the cross-module parameter list of the outer module to the inner Transformer encoder layer to realize cross-layer transfer of the index.

[0013] Preferably, the index transferred across layers is extracted and embedded into the Transformer coding layer for judgment, and a decision is made as to whether to call the feature extraction module at a certain scale, and the network parameters corresponding to the number of layers are used to assign the feature enhancement module for use; in order to enable the module to extract features according to the number of channels of the adaptive network, the channel parameter table of the feature extraction module is preset in the backbone network, that is, the channel parameters from high channel scale to low channel scale are 256, 160, 64, and 32 respectively, and the scaling variables of the feature extraction module are provided in the Transformer coding layer, that is, corresponding to the channel parameter table as 1 / 4, 1 / 8, 1 / 16, and 1 / 32 respectively. These scaling variables are used to dynamically adjust the computational complexity and feature representation capability of the feature extraction module at different scales.

[0014] Preferably, the S3 process is as follows:

[0015] The input features are convolved with a four-step pooling kernel to generate four feature maps of successively lower resolution. These, combined with the original input, form five feature maps of different scales. These are then upsampled through a 1x1 convolution to produce feature maps f0 to f4 of the same resolution (referred to as f here). f0 is directly output as f0-1. f0 and f1 are concatenated and then subjected to a 3x3 convolution to produce f1-1. f1-1 and f2 are concatenated and then subjected to a 3x3 convolution to produce f2-1. This process continues, resulting in five different feature outputs, f0-1 to f4-1. These are then concatenated and then subjected to a 1x1 convolution to produce the main feature map. The main feature map is then concatenated with the feature map obtained by a separate 1x1 convolution with the original input to produce the final feature map. The multi-scale, multi-level cascade design enables the network to capture rich contextual information, facilitating feature extraction.

[0016] Preferably, the S4 process is as follows:

[0017] The feature map output by the feature extraction module is first subjected to a 3x3 convolution operation and then added to the original feedforward neural network. This combines the advantages of Transformer-like networks and convolutional neural networks, enabling stronger information extraction capabilities. This part can be combined with the channel adaptation part to enhance information in smaller network parts, effectively improving encoder performance without increasing the amount of computation and parameters.

[0018] Preferably, the weight learning in S4 can be simply expressed as Y=WX+b, where W is the weight learned in this part, b is the bias term, X is the input, and Y is the output.

[0019] Therefore, the present invention adopts the above-mentioned adjustable multi-scale encoder enhancement method for image semantic segmentation, which can flexibly enhance the multi-scale Transformer encoder and adaptively adjust the parameters of different scales to select the enhancement level that suits different scenarios and needs. Compared with other types of encoders, this encoder can enhance the encoder coding effect and is more flexible.

[0020] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the operation of an adjustable multi-scale encoder enhancement method for image semantic segmentation according to the present invention;

[0022] Figure 2 This is a structural diagram of an adjustable multi-scale encoder enhancement method for image semantic segmentation according to the present invention;

[0023] Figure 3 This is a structural schematic diagram of a feature extraction module of an adjustable multi-scale encoder enhancement method for image semantic segmentation according to the present invention. DETAILED DESCRIPTION

[0024] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.

[0025] See also Figure 1-3 ,A tunable multi-scale encoder enhancement method for image semantic segmentation,

[0026] S1. Scale routing query: perform routing query on the current hierarchical structure and automatically select a specific hierarchical structure for channel input.

[0027] By performing a cyclic routing query on the four-scale features output by the backbone network, accurate judgment of the four-scale input features is achieved. Therefore, in this step, this embodiment transfers the four-scale features (layer0-3) through index variables as parameters, so that the internal feature extraction module can select appropriate parameters.

[0028] Specifically, in the Transformer encoder of networks like Segformer, this is done by enumerating the parameter num_layers (the number of layers traversed in the code) and assigning the resulting number of layers to an index. This index can be passed from the outer module list to the inner Transformer encoder layer to achieve cross-layer index transfer.

[0029] S2, channel adaptive extraction, adaptively extracts the multi-scale feature parameters extracted by the attention mechanism with different numbers of channels based on the S1 query index.

[0030] By extracting the index transferred across layers and embedding it into the Transformer coding layer for judgment, it is decided whether to call the feature extraction module at a certain scale, and the network parameters corresponding to the number of layers are assigned to the feature enhancement module. In order to enable the module to extract features based on the number of channels of the adaptive network, this embodiment presets a channel parameter table for the feature extraction module in the backbone network, that is, the channel parameters from high channel scale to low channel scale are 256, 160, 64, and 32, respectively, and provides scaling variables for the feature extraction module in the Transformer coding layer, that is, corresponding to the channel parameter table in order of 1 / 4, 1 / 8, 1 / 16, and 1 / 32. These scaling variables are used to dynamically adjust the computational complexity and feature representation capabilities of the feature extraction module at different scales.

[0031] S3. Feature extraction: The specific scale feature map is input into the feature extraction module according to the parameters provided by channel adaptation, multi-scale context information is extracted and concatenated and merged to obtain the semantic information feature map.

[0032] The input features are convolved with a four-step pooling kernel to generate four feature maps of successively lower resolution. These, combined with the original input, form five feature maps of different scales. These are then upsampled through a 1x1 convolution to produce feature maps f0 to f4 of the same resolution (referred to as f here). f0 is directly output as f0-1. f0 and f1 are concatenated and then subjected to a 3x3 convolution to produce f1-1. f1-1 and f2 are concatenated and then subjected to a 3x3 convolution to produce f2-1. This process continues, resulting in five different feature outputs, f0-1 to f4-1. These are then concatenated and then subjected to a 1x1 convolution to produce the main feature map. The main feature map is then concatenated with the feature map obtained by a separate 1x1 convolution with the original input to produce the final feature map. The multi-scale, multi-level cascade design enables the network to capture rich contextual information, facilitating feature extraction.

[0033] S4, feature fusion, the feature map of the feature extraction module is weighted through 3x3 convolution, the output feature map is merged with the main branch and input to the next layer for feature extraction, and then handed over to the decoder for feature aggregation and decoding.

[0034] The feature map output by the feature extraction module is first subjected to a 3x3 convolution operation and then added to the original feedforward neural network. This combines the advantages of Transformer-like networks and convolutional neural networks, enabling stronger information extraction capabilities. This part can be combined with the channel adaptation part to enhance information in smaller network parts, effectively improving encoder performance without increasing the amount of computation and parameters.

[0035] Table 1 shows the training results of five models, A, B1, B4, from the 160kiter Cityscapes dataset, at a resolution of 2048x1024, using the mmcv1.x framework on an Nvidia RTX 4070 Ti SUPER. A represents the original Segformer encoder model, B1 is a model that uses the enhancement method of this embodiment only at the lowest level (1 / 32 scale), B2 is a model that uses the method of this embodiment at both scales (1 / 32 and 1 / 16), B3 is a model that uses the method of this embodiment at scales (1 / 32, 1 / 16, and 1 / 8), and B4 is a model that uses the enhancement method at all levels.

[0036] Table 1 Training results

[0037]

[0038] It can be seen that the B1 and B2 models have brought about an improvement in MloU without increasing the number of parameters and computational complexity, and have little impact on the number of running frames. The B3 and B4 models have also brought about a significant improvement in MloU, which is suitable for scenarios with low computational complexity and speed requirements. It provides a flexible and effective enhancement method for the Transformer encoder enhancement method in semantic segmentation and even other fields.

[0039] Therefore, the present invention adopts the above-mentioned adjustable multi-scale encoder enhancement method for image semantic segmentation for the Transformer semantic segmentation encoder, which can flexibly enhance the Transformer blocks of any scale, adapt to different needs and scenarios, and improve the segmentation effect of the network.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for enhancing image semantic segmentation using an adjustable multi-scale encoder, characterized in that: The following steps are involved: S1, scale routing query, performs routing query on the current hierarchical structure and automatically selects a specific hierarchical structure for channel input; S2, channel adaptive extraction, adaptively extracts the multi-scale feature parameters extracted by the attention mechanism with different numbers of channels according to the S1 query index; S3, feature extraction, inputs the specific scale feature map into the feature extraction module according to the parameters of channel adaptive extraction, extracts multi-scale context information and performs concatenation and merging to obtain the semantic information feature map; The S3 process is as follows: The input features are convolved through a four-level pooling kernel to generate four feature maps with successively lower resolutions, which together with the original input form five feature maps of different scales, and are all input into a 1x1 convolution for upsampling to obtain feature maps f0 to f4 of the same resolution. f0 is directly output as f0-1, f0 and f1 are spliced ​​and then subjected to a 3x3 convolution to output f1-1, f1-1 and f2 are spliced ​​and then subjected to a 3x3 convolution to output f2-1, and five different feature outputs from f0-1 to f4-1 are obtained in sequence. After the splicing operation, a 1x1 convolution operation is performed to output the main feature map, and the main feature map is spliced ​​with the feature map obtained by performing a 1x1 convolution operation on the original input to obtain the final output feature map; S4, feature fusion, the feature map of the feature extraction module is weighted through 3x3 convolution, the output feature map is merged with the main branch and input to the next layer for feature extraction, and the decoder performs feature aggregation and decoding; The S4 process is as follows: The feature map output by the feature extraction module is first subjected to a 3x3 convolution operation and then added to the original feedforward neural network; combined with channel adaptation, it is used to enhance the information of the small-scale network part.

2. The adjustable multi-scale encoder enhancement method for image semantic segmentation according to claim 1, characterized in that: The S1 process is as follows: By performing a cyclic routing query on the four-scale features output by the backbone network, the judgment of the four-scale input features is realized, and the four-scale features layer0-3 are passed as parameters through index variables; The number of layers obtained by enumerating the parameters is assigned to the index, and this index is passed from the cross-module parameter list of the outer module to the inner Transformer encoder layer to realize cross-layer transfer of the index.

3. The adjustable multi-scale encoder enhancement method for image semantic segmentation according to claim 2, characterized in that: The S2 process is as follows: By extracting the index transferred across layers and embedding it into the Transformer encoding layer for judgment, it is decided whether to call the feature extraction module at a certain scale, and the network parameters corresponding to the number of layers are given to the feature enhancement module for use; the channel parameter table of the feature extraction module is preset in the backbone network, and the scaling variables of the feature extraction module are provided in the Transformer encoding layer.

4. The adjustable multi-scale encoder enhancement method for image semantic segmentation according to claim 1, characterized in that: Weight learning in S4 is expressed as Y=WX+b, where W is the learned weight, b is the bias term, X is the input, and Y is the output.

Images