Feature focusing image semantic segmentation method based on visual Transform

Abstract

The invention discloses a visual Transform-based feature focusing image semantic segmentation method, which adopts a feature pyramid structure to generate four feature maps with different sizes, and uses an overlapping block embedding module in a feature map embedding process to ensure that local information is continuous to a certain extent while the feature maps are partitioned. A position code generation module is used for dynamically generating corresponding position codes according to local neighborhood information, translation invariance needed in computer vision is kept, and then a fusion module combining a focusing attention mechanism and convolution operation is adopted in the Transform coding process. A self-attention mechanism is utilized to grasp a dependency relationship between an overall structure and a long distance, and convolution attention is introduced to enable the model to maintain sensitivity to local information, so that the model can integrate global and local detail information at the same time. And finally, inputting the feature maps of different sizes generated in each stage of the model into a decoder for segmentation. According to the method, the segmentation performance of a target boundary and a complex scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and more specifically, to a feature-focused image semantic segmentation method based on visual Transformer. Background Art

[0002] Image semantic segmentation is a key task in computer vision. It is used for scene parsing and object category recognition, and can achieve pixel-level image understanding. The core of semantic segmentation is to segment the input image into some pixel regions (these regions have specific semantic meanings), identify the categories to which these regions belong, and finally generate a pixel-by-pixel annotated segmentation map. Image semantic segmentation uses the semantic information obtained from the image to segment the target objects in the image, and it can establish a mapping relationship between the low-level visual features and high-level semantic information of the image. Therefore, in recent years, researchers in deep learning and computer vision have become increasingly interested in the problem of image semantic segmentation, and more and more application scenarios require accurate and efficient segmentation technology. Compared with other computer vision tasks, semantic segmentation can make better use of the rich information in the image, thereby analyzing the image more deeply. It is widely used in many common scenes in life. For example, in the field of autonomous driving, the images of the surrounding scene are obtained through the front and rear cameras, sensors and lidar of the vehicle, and transmitted to the semantic segmentation network for accurate target segmentation, intelligent obstacle avoidance and safe driving; in the field of remote sensing, pictures and videos transmitted by satellite remote sensing can be used to accurately locate roads, fields, houses, rivers, etc. in the images or videos through semantic segmentation, providing basic analysis for geographic information. Such tasks can provide effective support in macro issues such as plateau terrain monitoring and road planning.

[0003] In recent years, inspired by the successful application of Transformer in NLP, researchers have tried to apply Transformer to the field of computer vision, and Visual Transformer (ViT) was born, achieving extremely high accuracy in image classification tasks. With the increasing number of studies based on Visual Transformer, algorithm models based on Visual Transformer have also achieved extremely high accuracy in semantic segmentation tasks, indicating the feasibility of using Transformer in downstream dense prediction tasks. Therefore, Visual Transformer is one of the most popular methods for image semantic segmentation today.

[0004] The present invention discloses a feature-focused image semantic segmentation method based on visual Transformer. The model structure adopts a feature pyramid structure to generate four feature maps of different sizes. In the process of embedding the feature map, overlapping blocks are used to embed to ensure the continuity of local information to a certain extent while ensuring the block segmentation. The position encoding generation module is used to dynamically generate the corresponding position encoding according to the local neighborhood information to maintain the translation invariance required in computer vision. Then, in the process of Transformer encoding, a fusion module combining a focused attention mechanism and a convolution operation is used. The focused attention mechanism belongs to linear attention, which can maintain the dependency on the overall structure and long distance with low computational complexity. The convolution attention fuses the shared weights and the context-aware weights to aggregate high-frequency local information. The combination of the two enables the model to integrate global and local detail information at the same time. Finally, the feature maps of different sizes generated at each stage of the model are input into the decoder for semantic segmentation. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a feature-focused image semantic segmentation method based on a visual Transformer to achieve semantic segmentation of the original image.

[0006] In order to achieve the above object, the embodiment of the present invention provides the following solution:

[0007] A feature-focused image semantic segmentation method based on visual Transformer, characterized by comprising the following five steps:

[0008] Step 1. Set up a multi-scale feature pyramid;

[0009] The backbone network of the model is divided into four stages. Each stage generates feature maps of different scales and the structure of each stage is similar. Each stage uses the feature map of the previous stage as input. The input passes through the embedding layer and the Transformer encoder to obtain four feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 times the size of the input image.

[0010] Step 2. Process the input image using the overlapping block embedding module;

[0011] The overlapping block embedding module makes the adjacent windows of the convolution overlap by half of their area and uses zero padding to ensure that the size of the feature map remains unchanged. This ensures a certain degree of local information continuity while dividing the feature map into blocks.

[0012] Step 3. Generate relative position code using position code generation module;

[0013] The input of the position encoding generation module is the block segmented by the feature map obtained in step 2. The size of the feature map changes with the size of the input image. The module can use local information to dynamically generate relative position encoding, which changes according to the size of the input, while ensuring the flexibility of Transformer and retaining the translation invariance of the image.

[0014] Step 4. Use the fusion module to integrate global and local information;

[0015] The input image passes through the block embedding and position encoding generation modules to obtain feature information as the input of the Transformer encoder. In the Transformer encoder, the fusion module uses the self-attention mechanism to grasp the overall structure and long-distance dependencies, and introduces convolutional attention to make the model sensitive to local information, integrating the global and local detail information of the model to help the model segmentation.

[0016] Step 5. Decode using the Feature Pyramid Network (FPN) decoder

[0017] The input image goes through four stages to generate four feature maps of different sizes. The four feature maps are upsampled to one-quarter the size of the original image through convolution. Then the upsampled high-level features are added element-by-element to the low-level features to form a new fused feature map. This process combines high-level semantic information with low-level detail information. Finally, the segmentation map of the same size as the input image is output through upsampling to ensure that each pixel has a corresponding category label.

[0018] Preferably, the number of images per batch during algorithm training is set to 8.

[0019] Preferably, the algorithm is iterated 800,000 times in total during training (ie, max_iteration is 800,000).

[0020] Preferably, the initial learning rate η0 during algorithm training is 4×10 -5 , the learning rate change strategy is the Poly learning rate strategy, where the power is 0.9.

[0021] The present invention discloses a feature-focused image semantic segmentation method based on visual Transformer. The model structure adopts a feature pyramid structure to generate four feature maps of different sizes. In the process of embedding the feature map, an overlapping block embedding module is used to ensure that the feature map is divided into blocks while ensuring a certain degree of continuity of local information. A position encoding generation module is used to dynamically generate corresponding position encodings according to local neighborhood information, which can maintain the translation invariance required in computer vision. Then, a fusion module combining a focused attention mechanism and a convolution operation is used in the Transformer encoding process. In the module, a self-attention mechanism is used to grasp the overall structure and long-distance dependencies, and convolution attention is introduced to make the model maintain sensitivity to local information, so that the model can integrate global and local detail information at the same time. Finally, the feature maps of different sizes generated at each stage of the model are input into the decoder for segmentation. The method disclosed in the present invention is improved on the basis of the visual Transformer, and the segmentation performance of target boundaries and complex scenes is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of a feature-focused image semantic segmentation method based on a visual Transformer provided by an example of the present invention;

[0024] Figure 2 A flowchart of a feature-focused image semantic segmentation method based on a visual Transformer provided in an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of an overlapping block embedding module provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a position code generation module provided in an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of a fusion module provided in an embodiment of the present invention;

[0028] Figure 6 A schematic diagram of a feature pyramid network (FPN) decoder provided in an embodiment of the present invention;

[0029] Figure 7 A training flow chart of a feature-focused image semantic segmentation method based on a visual Transformer provided in an embodiment of the present invention; DETAILED DESCRIPTION

[0030] In order to more clearly illustrate the present invention, the following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The model structure diagram of the feature-focused image semantic segmentation method based on visual Transformer proposed in this invention is as follows: Figure 1 As shown in the figure, the model adopts a feature pyramid structure. The model is mainly improved on the backbone network. The backbone network consists of four stages in total. Each stage generates feature maps of different scales and the structure of each stage is similar. Each stage uses the feature map of the previous stage as input. The input passes through the embedding layer and the Transformer encoder to obtain four feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 times the size of the input image. Feature maps of different sizes are sent to the FPN decoder as feature pyramids to generate segmentation results through convolution and upsampling.

[0032] The embodiment of the present invention discloses a feature-focused image semantic segmentation method based on visual Transformer to achieve semantic segmentation of images. Figure 2 , the above method at least comprises the following steps:

[0033] Step 1. Set up a multi-scale feature pyramid;

[0034] The algorithm of the present invention adopts a feature pyramid structure and divides the backbone network of the model into four stages. Each stage generates feature maps of different scales and the structure of each stage is similar. Each stage includes an embedding layer, a position encoding generation module and a Transformer encoder. Each stage uses the feature map of the previous stage as input. The input passes through the embedding layer and the Transformer encoder to obtain four feature maps with sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 times the size of the input image, wherein the number of channels of the four feature maps are 32, 64, 128 and 256 respectively.

[0035] Step 2. Process the input image using the overlapping block embedding module;

[0036] In the Visual Tranformer (ViT) model, the embedding layer uses a convolution kernel size equal to the step size to embed the input image, which results in the model being unable to model local continuous information. Therefore, the overlapping block embedding module is used in the algorithm of the present invention, such as Figure 3 As shown in the figure, the module makes the adjacent windows of the convolution overlap half of the area and uses zero padding to ensure that the size of the feature map remains unchanged. This ensures a certain degree of local information continuity while the feature map is divided into blocks.

[0037] Step 3. Generate relative position code using position code generation module;

[0038] In view of the fact that the absolute position encoding used in the traditional visual Transformer has the problems of insufficient model flexibility and lack of image translation invariance, the algorithm of the present invention uses relative position encoding and generates relative position encoding using the position encoding generation module, such as Figure 4 As shown in the figure, the input of the position encoding generation module is the block segmented by the feature map obtained in step 2. The size of the feature map changes with the size of the input image. The module can use local information to dynamically generate relative position encoding, which changes according to the size of the input, while ensuring the flexibility of Transformer and retaining the translation invariance of the image.

[0039] Step 4. Use the fusion module to integrate global and local information;

[0040] The input image passes through the block embedding and position encoding generation modules to obtain feature information as the input of the Transformer encoder. In the Transformer encoder, the fusion module uses the self-attention mechanism to grasp the overall structure and long-distance dependencies, and introduces convolutional attention to make the model sensitive to local information, integrating the global and local detail information of the model to help the model segmentation.

[0041] Fusion modules such as Figure 5As shown in the figure, the fusion module is divided into two branches, the focused attention branch that integrates global information and the convolutional attention branch that extracts local information. In the focused attention branch, the input passes through a linear layer to form query, key, and value vectors, where the query and key vectors adjust the directions of the query and key vectors through a focusing function, so that similar queries and key-value pairs are closer, allowing the focused attention to maintain a strong feature focusing ability while reducing the amount of computation. In addition, since d in the linear attention matrix is ​​generally much smaller than N, the rank of the attention matrix cannot reach full rank, resulting in a high similarity in the output features. Therefore, deep convolution (DWConv) is added to the focused attention to restore the attention matrix in the linear attention to full rank, maintaining the same feature diversity as the Softmax attention. Given an input In each head, the focused attention branch formula is expressed as follows:

[0042] Q = xW Q , K = xW k , V = xW v (1)

[0043]

[0044] Where W Q , W k , is the projection matrix, Sim(·) represents the similarity function, DWC(·) is the depthwise separable convolution, φ p is the focusing function, which is specifically expressed as follows:

[0045] Sim(Q,K)V=φ p (Q)φ p (K) T V(3)

[0046]

[0047] Where x **p Represents x raised to the power of p element by element, and the ReLU(·) function ensures the non-negativity of the input.

[0048] The focus function adjusts the direction of each query and key vector, making the linear attention distribution sharper and tending to the attention after Softmax processing. But in addition to the focusing ability, the rank of the attention matrix also limits the expression of linear attention. Under the feedforward of linear attention, the rank of the attention matrix in linear attention is limited by the number of elements N in each sequence and the channel dimension d. Since the channel dimension d is usually less than N, it can be seen from Formula 3-9 that the attention matrix of the linear attention mechanism is less than or equal to d, and the rank of the attention matrix is ​​limited to a lower range, so this leads to the expression of the attention matrix mapping being severely homogenized. Therefore, in Formula 3-6, a depthwise separable convolution is added to the attention matrix to ensure that the matrix maintains a high rank to ensure the diversity of features.

[0049] rank(φ(Q),φ(K)) T ≤min{rank(φ(Q)),rank(φ(Q)),}≤min{N,d} (4)

[0050] In the local branch, this paper introduces convolutional attention AttConv, which adopts the style of attention mechanism and effectively fuses shared weights and context-aware weights to aggregate high-frequency local information. Specifically, the module line uses deep separable convolution to extract local representations, where deep separable convolution has shared weights. Context-aware weights are then used to enhance local features. Different from the methods of generating context-aware weights such as Non-Local, a gating mechanism is used here to generate context-aware weights, introducing stronger nonlinearity than the commonly used attention mechanism. In addition, AttnConv applies the convolution operator to the query (Query) and (Key) to aggregate local information, and then calculates the element-by-element product of Q and K, and performs a series of linear or nonlinear transformations on the results to generate context-aware weights in the range of [-1,1]. It is worth noting that AttnConv inherits the translation equivariance of convolution because all its operations are based on convolution. The specific formula is as follows:

[0051] Q l =DWconv(Q),K l =DWconv(K),V l =DWconv(V) (5)

[0052] Attn t =FC(Swish(FC(Q l ⊙K l ))) (6)

[0053]

[0054] Attn t=FC(Swish(FC(Q l ⊙K l ))) (8)

[0055] X local =Attn⊙V l (9)

[0056] Where d represents the number of input channels, ⊙ represents the element-wise product, and FC represents the fully connected layer.

[0057] After the input passes through the global and layout branches respectively, the model connects the two outputs in the channel dimension, and then uses a fully connected layer to reduce the dimension of the output and input it into the multi-layer perceptron.

[0058] X t =Concat(X local ,X global ) (10)

[0059] X out =FC(X t ) (11)

[0060] Step 5. Decode using the Feature Pyramid Network (FPN) decoder

[0061] In order to be more suitable for downstream segmentation tasks, this paper adopts the traditional FPN feature pyramid decoding network in the decoding part, which is suitable for the network structure with Tranformer as the backbone. The FPN decoder uses the features of different levels extracted from the backbone network. The feature maps of each layer have different spatial resolutions and can capture information of different scales.

[0062] like Figure 6 As shown in the figure, the input image goes through four stages to generate four feature maps of different sizes. The four feature maps are upsampled to one-quarter the size of the original image through convolution. Then the upsampled high-level features are added element-by-element to the low-level features to form a new fused feature map. This process combines high-level semantic information with low-level detail information. Finally, the segmentation map of the same size as the input image is output through upsampling to ensure that each pixel has a corresponding category label.

[0063] During the model training phase, Figure 7 As shown in the figure, first follow the order of steps 1 to 6 to get the segmentation result, then use the loss function to calculate the loss function value of this iteration, and use the AdamW algorithm to update the network parameters. When the maximum number of iterations is reached or the iteration converges, the iteration is stopped and the network model result is saved.

[0064] The loss function of this method adopts the commonly used cross entropy loss function, and its calculation formula is as follows:

[0065]

[0066] Where M represents the amount of data per iteration, N represents the number of pixels in the image, and K represents the total number of pixel categories. express Tags, Representing model predictions The probability of belonging to the kth class.

[0067] Preferably, the number of images per batch during algorithm training is set to 8, the maximum number of iterations of the algorithm is 800,000 (i.e., max_iteration is 800,000), the optimizer uses AdamW, and the initial learning rate η0 is 4×10 -5 , the learning rate change strategy is the Poly learning rate strategy, where the power is 0.9, and the calculation process of the learning strategy is as follows:

[0068]

[0069] In the formula, iteration represents the number of iterations during the training process.

[0070] The disclosed method is verified on the ADE20K dataset, which contains 20,210 training images and 2,000 verification images. Since there are no public test images, the verification images are used in the experiment to evaluate the performance of the algorithm. The dataset contains 150 categories, covering a variety of objects and scene elements such as people, animals, vehicles, buildings, and natural landscapes. Each image is equipped with detailed pixel-level annotations, and the annotation information includes object categories and corresponding boundaries.

[0071] The present invention uses mean intersection over union (mIoU) and pixel accuracy (PA) as quantitative evaluation indicators. The mean intersection over union (IoU) is calculated by calculating the intersection over union (IoU) of each category and then averaging all categories; pixel accuracy is used to measure the prediction accuracy of the model at the pixel level. It provides a simple and intuitive performance evaluation by calculating the ratio of the number of correctly predicted pixels to the total number of pixels. The larger the value of mIoU and PA, the closer the segmentation result predicted by the network is to the annotated image, that is, the better the segmentation effect. Table 1 quantitatively compares the mIoU and PA of the method of the present invention and SegformerB0, PVT-T, PoolFormer, FlattenTransformer, PVTv2, VAN, CloFormer, EdgeViT, and PVT-S on the ADE20K dataset. It can be seen from Table 1 that the method of the present invention achieves 40.26% and 79.43% in mIoU and PA, respectively, which is better than other models in the table.

[0072] Table 1 Comparison of the method disclosed in the present invention with other methods in terms of mIoU and PA

[0073]

[0074] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the present invention may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present invention, but will conform to the widest scope consistent with the principles and novel features disclosed in the present invention.

Claims

1. A feature-focused image semantic segmentation method based on visual Transformer, characterized in that: The following steps are involved: Step 1. Set up a multi-scale feature pyramid; The backbone network of the model is divided into four stages. Each stage generates feature maps of different scales and the structure of each stage is similar. Each stage uses the feature map of the previous stage as input. The input passes through the embedding layer and the Transformer encoder to obtain four feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 times the size of the input image. Step 2. Process the input image using the overlapping block embedding module; The overlapping block embedding module makes the adjacent windows of the convolution overlap by half of their area and uses zero padding to ensure that the size of the feature map remains unchanged; this ensures a certain degree of local information continuity while dividing the feature map into blocks; Step 3. Generate relative position code using position code generation module; The input of the position encoding generation module is the block segmented by the feature map obtained in step 2. The size of the feature map changes with the size of the input image. This module can dynamically generate relative position encodings using local information, which changes according to the size of the input, while ensuring the flexibility of the Transformer and retaining the translation invariance of the image. Step 4. Use the fusion module to integrate global and local information; The input image passes through the block embedding and position encoding generation modules to obtain feature information as the input of the Transformer encoder. In the Transformer encoder, the fusion module uses the self-attention mechanism to grasp the overall structure and long-distance dependencies, and introduces convolutional attention to make the model sensitive to local information, integrating the global and local detail information of the model to help the model segmentation; Step 5. Decode using feature pyramid FPN decoder; The input image goes through four stages to generate four feature maps of different sizes. The four feature maps are upsampled to one-quarter the size of the original image through convolution. Then the upsampled high-level features are added element-by-element to the low-level features to form a new fused feature map. This process combines high-level semantic information with low-level detail information. Finally, the segmentation map of the same size as the input image is output through upsampling to ensure that each pixel has a corresponding category label.

2. The feature-focused image semantic segmentation method based on visual Transformer according to claim 1, characterized in that: The number of images per batch during training is set to 8.

3. The feature-focused image semantic segmentation method based on visual Transformer according to claim 1, characterized in that: A total of 800,000 iterations were performed during training.

4. The feature-focused image semantic segmentation method based on visual Transformer according to claim 1, characterized in that: The initial learning rate η0 during training is 4×10 -5 , the learning rate change strategy is the Poly learning rate strategy, where the power is 0.9.

Images