Image semantic segmentation method based on feature extraction and rfb context information optimization

Abstract

The application discloses an image semantic segmentation method based on feature extraction and RFB context information optimization, which comprises the following steps: firstly, inputting an image into a Swin Transformer feature extraction network to respectively output a feature map containing deep semantic information and a feature map containing shallow semantic information; then, inputting the feature map containing deep semantic information and the feature map containing shallow semantic information into RFB modules of different depths respectively to optimize context information, so as to obtain two feature maps containing more abundant context information; splicing and fusing the two feature maps after RFB module optimization; performing bilinear interpolation upsampling processing on the fused feature map; and then inputting the result into a decoding network to obtain a final image semantic segmentation result. The Swin Transformer module can extract deep and shallow semantic features of an image, and the RFB context information optimization module is combined to improve semantic segmentation precision.

Description

TECHNICAL FIELD

[0001] The present application relates to an image semantic segmentation method based on feature extraction and RFB context information optimization. BACKGROUND

[0002] Image semantic segmentation technology is one of the hot researches in the field of computer vision, and is a natural step in the process of image reasoning from coarse granularity to fine granularity. When the reasoning results obtained by image-level tasks similar to target detection cannot meet the actual requirements, semantic segmentation is a choice that can provide more detailed results. The essence of semantic segmentation is pixel-level image classification, that is, each pixel point is classified and judged to provide more rich image information. Semantic segmentation technology is widely used in automatic driving, medical image analysis, remote sensing image analysis, video AI face changing and many other fields.

[0003] At present, the existing semantic segmentation method is basically based on deep learning method, and the specific implementation precision has reached a high precision level. However, pixel-level classification means that resnet101 level deep network is often used to extract better features, and the existing CNN network is limited by the receptive field and has poor global feature extraction capability. Secondly, the existing method in the segmentation field still needs to be further improved for the receptive field problem.

[0004] In view of the existing problems, the Swin Transformer feature extraction and RFB context information image semantic segmentation method is introduced, aiming at improving the network feature extraction capability, and the Swin transformer and RFB are coupled to improve the context information, finally improving the segmentation precision and improving the system robustness. SUMMARY

[0005] Therefore, the present application mainly solves the problems of insufficient feature extraction and imperfect context information of the existing semantic segmentation technology, and proposes an image semantic segmentation method based on feature extraction and RFB context information optimization, which improves the feature extraction problem by using SwinTransformer, and improves the context information problem by using the mutual coupling of Swin Transformer and RFB, and finally improves the precision of image semantic segmentation.

[0006] The present application provides an image semantic segmentation method based on feature extraction and RFB context information optimization, comprising the following steps:

[0007] Step 1, inputting an RGB image with a size of HxW;

[0008] Step 2, inputting the image into the Swin Transformer feature extraction module to respectively output a feature map containing deep semantic information and a feature map containing shallow semantic information;

[0009] Step 3, the feature map containing deep semantic information and the feature map containing shallow semantic information are respectively input into RFB modules of different depths for context information optimization, and two feature maps containing more rich context information are obtained;

[0010] Step 4, the two feature maps optimized by the RFB module are spliced and fused, and then the result is input into a decoding network;

[0011] Step 5, the fused feature map is bilinearly interpolated and upsampled, and the final segmentation result is output after iterative optimization constrained by cross-entropy loss.

[0012] Further, the step 2 first performs window segmentation and block segmentation operations on the input image, and each image block is regarded as a to-be-processed object, then the image is input into a linear feature length encoding layer, the processed feature map is transmitted into a Swin Transformer module, and self-attention calculation is performed on each window.

[0013] Further, the step 3 extracts the feature map through the Swin Transformer module, and the finally obtained feature map and the 1 / 4 resolution feature map in the Swin Transformer are respectively input into different RFB context information modules, the 1 / 4 resolution feature map adopts a deeper RFB module, and the finally obtained feature map adopts a shallower RFB module.

[0014] The image semantic segmentation method based on feature extraction and RFB context information optimization of the application can extract deep and shallow semantic features of an image through the Swin Transformer module, and improve the semantic segmentation precision by combining the RFB context information optimization module. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of the application;

[0016] Figure 2 is the first image in the cityscapes dataset test set of the application;

[0017] Figure 3 is a Swin Transformer module structure diagram of the application;

[0018] Figure 4 is an RFB1 module network structure structure diagram of the application;

[0019] Figure 5 is an RFB2 module network structure structure diagram of the application;

[0020] Figure 6 The first image prediction result map of the cityscapes dataset test set of the present application. DETAILED DESCRIPTION

[0021] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided so that the disclosure of the present application can be more thoroughly and completely understood.

[0022] Referring to Figure 1 The present application provides an image semantic segmentation method based on Swin Transformer feature extraction and RFB context information, which uses a picture in a semantic segmentation dataset for experimental illustration:

[0023] Comprising the following steps:

[0024] Step 1, input an RGB image with a size of HxW; (such as Figure 2 )

[0025] Step 2, input the image into the Swin Transformer feature extraction module (such as Figure 3 ), respectively output the feature map containing deep semantic information and the feature map containing shallow semantic information;

[0026] First, the input image is windowed and block cut, the image is divided into M windows and N image blocks, each image block is treated as an object to be processed, the size is n x n, each window contains m x m image blocks, at this time the dimension of the feature of each image block is n x n x 3; then the image is input into a linear feature length encoding layer, and then the processed feature map is transmitted into the Swin Transformer module, the core of which is to perform self-attention calculation on each window, the specific calculation process is as follows:

[0027]

[0028] In formula (1), z l-1 is the output of the previous layer network, z l is the output after the first Swin transformer block, z l+1 is the output value after the continuous two Swin transformer modules; is the result of z l-1 once through LN and W-MSA calculation and added with its own residual error, is calculated by zl passing through LN and MLP in turn and adding the result to the residual of itself. In the formula, LN is a Layer Norm operation, MLP is a multi-layer perceptron, W-MSA is a window-based multi-head self-attention calculation, and SW-MSA is a multi-head self-attention calculation based on a sliding window. The above is the specific operation process after a feature map enters a SwinTransformer module;

[0029] wherein the specific calculation of MSA after expansion is:

[0030]

[0031] z i in formula (2) represents an input feature, z i passes through a conversion matrix Q k , k , k Qi, Ki, and Vi three vectors are obtained, and the second formula is the specific operation of self-attention, wherein K i T represents the transpose vector of the K i vector, represents the square root of the dimension of the K i vector, and softmax represents a normalized exponential function. i and K i T are multiplied and multiplied by a scaling factor , and then the softmax operation is performed to obtain the weight coefficient of self-attention, and then multiplied by the weight V i to obtain z i ; i ;

[0032] wherein the specific calculation of MLP after expansion is:

[0033] MLP(x) = Gelu(xW1 + b1)W2 + b2 (3)

[0034] Formula (3) MLP is essentially a multi-layer perceptron, wherein W1, W2, and b1, b2 represent linear mapping matrices and corresponding biases, and GELU represents a nonlinear activation function.

[0035] Step 3, constructing an RFB context information module, inputting the feature map containing deep semantic information and the feature map containing shallow semantic information into RFB modules of different depths respectively to optimize the context information, and obtaining two feature maps containing more rich context information.

[0036] The feature map extracted by the Swin Transformer module is sent to different RFB context information modules. Since the segmentation task requires more information about the upper and lower information, the final feature map and the 1 / 4 resolution feature map in the Swin Transformer are sent to different RFB upper and lower context information modules. The 1 / 4 resolution feature map contains more detailed information, so a deeper RFB module is used. The final feature map contains more semantic information, so a shallower RFB module is used. The specific implementation is as follows:

[0037] The RFB1 used for the feature map x' containing shallow information is as follows: Figure 4 ) as follows:

[0038]

[0039] y' = relu(conv(cat((x0, x1, x2, x3, x4), 1)) + x5) (5)

[0040] In formula (4), conv1 uses a 1x1 convolution kernel to compress the channel dimension. In x0 to x4, conv2 uses a 1x1 convolution kernel, a 1x3 convolution kernel, a 3x1 convolution kernel, a 3x3 convolution kernel, and a 5x5 convolution kernel. In x0 to x4, conv3 uses a 3x3 convolution kernel, but the hole rate is 1, 3, 3, 5, and 7, respectively. In formula (5), the features obtained in x0 to x4 in formula (4) are fused by channel concatenation, then a 1x1 convolution is used for channel compression, the result is added to the shortcut, and finally the fused result is subjected to nonlinear operation by relu;

[0041] The RFB2 used for the feature map x" containing deep information is as follows: Figure 5 ) as follows:

[0042]

[0043] y" = relu(conv(cat((x0, x1, x2), 1)) + x5) (7)

[0044] In formula (6), conv1 uniformly adopts a 1*1 convolution kernel, and the purpose is to compress the channel dimension, and conv2 adopts a 1*1 convolution kernel, a 3*3 convolution kernel and a 5*5 convolution kernel in x0-x4 respectively; conv3 uniformly adopts a 3*3 convolution kernel in x0-x4 of conv2, but the hole rate is 1, 3 and 5 in turn; in formula (7), the features obtained in x0-x3 of formula (6) are fused by channel splicing, and then a 1*1 convolution is used for compression, and then the result is added and fused with shortcut, and finally the fused result is subjected to nonlinear operation by relu; the shortcut in formula (5) and formula (7) is the result obtained after the feature map directly passes through a 1*1 convolution;

[0045] Step 4, constructing an RFB context information module, splicing and fusing the feature map containing deep semantic information and the feature map containing shallow semantic information;

[0046] y = conv (up (y' ') + y ') (8)

[0047] In formula (8), y' is a feature map rich in detailed information obtained after RFB1 processing, y'' is a feature map rich in semantic information obtained after RFB2 processing, up is a bilinear interpolation up-sampling, and conv is a 1*1 convolution.

[0048] Step 5, outputting the result after iterative optimization after up-sampling (such as Figure 6 )

[0049] The fused feature map is bilinearly interpolated and up-sampled, and the final segmentation result is output after iterative optimization constrained by cross-entropy loss.

[0050] The image semantic segmentation method based on feature extraction and RFB context information optimization of the application can extract deep and shallow semantic features of an image by using a SwinTransformer module, and can improve semantic segmentation precision by combining an RFB context information optimization module.

[0051] The above embodiments only express the embodiments of the application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent of the application. It should be noted that, for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which all belong to the protection scope of the application. Therefore, the protection scope of the patent of the application should be subject to the appended claims.

Claims

1. An image semantic segmentation method based on feature extraction and RFB context information optimization, characterized in that, The method comprises the following steps: Step 1, inputting an RGB image with a size of HxW; Step 2, inputting the image into a Swin Transformer feature extraction module to respectively output a feature map containing deep semantic information and a feature map containing shallow semantic information; Step 3, inputting the feature map containing deep semantic information and the feature map containing shallow semantic information into RFB modules with different depths respectively to optimize the context information, and obtaining two feature maps containing more rich context information; In the step 3, the feature map is extracted through the Swin Transformer module, the finally obtained feature map and the 1 / 4 resolution feature map in the Swin Transformer are respectively sent into different RFB context information modules, the 1 / 4 resolution feature map adopts a deeper RFB module, and the finally obtained feature map adopts a shallower RFB module; Step 4, splicing and fusing the two feature maps optimized by the RFB module, and then inputting the result into a decoding network; Step 5, performing bilinear interpolation upsampling on the fused feature map, and outputting a final segmentation result after iterative optimization by cross-entropy loss constraint.

2. The method of claim 1, wherein: In the step 2, the input image is firstly subjected to window segmentation and block segmentation operations, each image block is regarded as an object to be processed, then the image is input into a linear feature length encoding layer, the processed feature map is transmitted into a Swin Transformer module, and self-attention calculation is performed on each window.

Images