Fast semantic segmentation network and semantic segmentation method fusing local and global features

Abstract

The application provides a fast semantic segmentation network and a semantic segmentation method fusing local and global features, local information and global information of an input image are extracted through a double-branch structure, so that the local and global features are better fused to obtain segmentation features containing more information, the problem that the local and global features are difficult to interact is solved, and the functions of improving segmentation efficiency and overall performance are effectively realized. The application proposes a new multi-scale feature fusion module, which utilizes the context information of the Transformer and the local representation features of the convolution; the local and global feature information can be processed at the same time, and very excellent performance is shown in fast medical image segmentation. The application reduces the complexity of calculation while extracting global features and local features, reduces the time required for network training while ensuring the segmentation result, and realizes the function of improving the segmentation performance while ensuring the segmentation speed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and particularly relates to a fast semantic segmentation network and a semantic segmentation method fusing local and global features. BACKGROUND

[0002] Image segmentation plays an important role in medical image analysis, especially in clinical diagnosis, which is widely used for quantitative analysis of anatomical structures. With the development of deep learning technology, convolutional neural networks have made substantial progress in medical image segmentation, especially fully convolutional networks and their variants, such as UNet, DeepLab, which have become the actual choice. Based on these methods, many works have made great progress in medical applications, such as chest CT vessel segmentation, MRI heart segmentation, and lymph node segmentation, etc.

[0003] Early studies on object detection and image segmentation have shown the efficiency of multi-scale feature fusion. However, it is still not clear whether the segmentation performance can be further improved by integrating features from convolutional layers and Transformer layers. Therefore, we propose a multi-scale feature fusion module. For low-resolution feature maps, we use a linear bottleneck structure and interpolation operation, which can obtain feature maps with the same dimension and resolution compared to the previous output. For example, an input with a resolution of 1 / 32 of the original image will have the same size after upsampling operation in the linear bottleneck structure.

[0004] Although FCN-based methods have special representation capabilities, due to the local receptive ability of convolutional operations, their ability to capture global context information and long-range dependencies is limited. This limitation leads to suboptimal segmentation of deformable shapes and scale structures when capturing multi-scale context information. Previous studies have tried to alleviate this problem by using extended convolution in DeepLab, feature pyramid pooling in PSPNet, and self-attention mechanisms in UNet. However, in the medical image segmentation task, there is still no research that can fully extract global context features.

[0005] Transformer-based models are proposed in the field of NLP sequence-to-sequence modeling, and have achieved advanced results in various tasks. The self-attention mechanism in Transformer enables them to learn long-range dependencies and establish global relationships between sequences. Transformer has also achieved the most advanced performance in image classification tasks in computer vision. Later, many Transformer-based semantic segmentation works were proposed, such as SETR, Swin Transformer, TransUNet, Swin-UNet, DS-TransUNet, TransFuse, VOLO, etc. However, this kind of Transformer-based method has a great amount of calculation when modeling long-range dependencies, and the spatial structure is very complex. This greatly hinders real-time medical diagnosis in medical image processing, such as radiotherapy. SUMMARY

[0006] The technical problem solved by the present application is to provide a fast semantic segmentation network and a semantic segmentation method that fuse local and global features to improve the performance of segmented images.

[0007] The technical solution adopted by the present application to solve the above technical problems is: a fast semantic segmentation network that fuses local and global features, comprising a first branch, a second branch and an MSFFM multi-scale fusion module; the first branch is a CNN branch, comprising a plurality of convolutional layers, for extracting local feature information of an image; the second branch is a Transformer branch, comprising an LN layer, a plurality of Outlooker attention layers and a series of transformer MLPs, for extracting global feature information and context feature information of an image through downsampling and self-attention mechanism; the feature information output by the first branch and the second branch is fused in an interactive manner through a plurality of bilateral connections; the output of each calculation of the second branch interacts with the output of the first branch, to enable the first branch to better learn global features without increasing computational complexity; the output of each calculation of the second branch is connected to the MSFFM multi-scale fusion module, and the MSFFM multi-scale fusion module is used to fuse the feature information output by the first branch and the second branch after interactive fusion, and perform multi-scale fusion operation on the information.

[0008] According to the above scheme, in the first branch, X is the input feature image, Conv is the convolution operation, BN is the layer normalization operation, and ReLU is the activation operation; each convolutional layer includes a Conv operation, a BN operation and an activation operation ReLU function; the Conv operation is used to extract the features of the feature map; the BN operation is used to avoid the gradient explosion and gradient disappearance phenomenon in the process of gradient operation of the image; the ReLU function is used to control the output of each layer within a predetermined range and make the value of the negative number less than 0;

[0009] Therefore, the formula of the convolutional layer is:

[0010] X' = F.ReLU(BN(Conv(X))) (1).

[0011] Further, in the second branch, LN is the layer normalization; OutlookerAtt is the Outlooker attention layer, which is used for spatial coding to generate refined representation; MLP is used for channel information interaction and aggregation of global information to establish long-distance dependency relationship; each part is followed by a block embedding module for mapping the input to a specified shape; the input feature image X is sequentially subjected to the LN layer, the Outlooker module and the MLP to extract the global features of the image:

[0012] X' = OutlookerAtt(LN(X)) + X (2),

[0013] Z = MLP(LN(X')) + X' (3).

[0014] A fast semantic segmentation method fusing local features and global features, comprising the following steps:

[0015] S1: assuming that the input image to be subjected to semantic segmentation is A; performing a double-branch operation on the image A to extract the local features and global features of the image through CNN operation and Transformer operation respectively;

[0016] S2: inputting the outputs of the double-branch after interaction into the MSFFM multi-scale fusion module to fuse the local features extracted by the CNN operation and the global features extracted by the Transformer operation in different scales, and obtaining a feature map fused with global feature information and local feature information.

[0017] Further, in the step S1, the specific steps are as follows:

[0018] S11: the image A is subjected to twice convolution operation with downsampling to generate a feature map A1 with a resolution of 1 / 4 of the image A, and is input into the CNN branch and the Transformer branch respectively;

[0019] S12: The Transformer branch generates a feature map A2 with a resolution of 1 / 8 of the image A; the feature map A2 is input to the MSFFM multi-scale fusion module; at the same time, the feature map A2 is subjected to a convolution operation with upsampling and then added to the feature map A1, and input to the CNN branch;

[0020] S13: The new feature map obtained after the convolution operation of the CNN branch is subjected to a CNN operation with downsampling, and then added to the feature map A2;

[0021] S14: The Transformer branch uses a Transformer operation to fuse the added feature maps;

[0022] S15: Steps S12 to S14 are repeated three times, and the Transformer branch generates feature maps A2, A3 and A4 with resolutions of 1 / 8, 1 / 16 and 1 / 32 of the image A, respectively;

[0023] S16: The feature map A4 is subjected to a convolution operation with upsampling, and then added to the feature map output by the CNN branch in the previous step, and input to the CNN branch to output a feature map A5 after a convolution operation, with a resolution of 1 / 4 of the image A.

[0024] Further, in the step S2, the specific steps are as follows:

[0025] S21: The feature map A5 obtained by the CNN branch and the feature maps A2, A3 and A4 obtained by the Transformer branch are loaded into the fusion module;

[0026] S22: According to the resolution of the output feature map, the feature maps obtained by the Transformer branch are subjected to a convolution operation with upsampling and an interpolation operation of a linear bottleneck structure, respectively, to obtain feature maps with the same dimension and resolution; then, the feature maps obtained by the CNN branch are subjected to a concatenate operation to obtain the final output.

[0027] Further, in the step S22, the specific steps are as follows:

[0028] S221: The feature map A4 output by the Transformer branch is subjected to a CNN operation with upsampling to obtain a feature map A4' with a resolution twice that of the feature map A4, and the feature map A3 obtained by two Transformer operations is subjected to an interpolation operation of a linear bottleneck structure to obtain a feature map A3', and the feature maps A4' and A3' with the same resolution are added to obtain a feature A33;

[0029] S222: a CNN operation with upsampling is performed on the feature map A33 to obtain a feature map A22 with a resolution enlarged by two times, an interpolation operation of a linear bottleneck structure is performed on the feature map A2 obtained through one Transformer operation to obtain a feature map A2', and the feature maps A22 and A2' with the same resolution are added to obtain a feature A11.

[0030] S223: a CNN operation with upsampling is performed on A11 to obtain a feature map A00 with a resolution enlarged by two times, and then a concatenate operation is performed on the feature map A5 obtained through the CNN branch to obtain a final output.

[0031] A computer storage medium has a computer program stored therein, which can be executed by a computer processor, and the computer program executes a fast semantic segmentation method fusing local features and global features.

[0032] The present application has the following beneficial effects:

[0033] 1. The fast semantic segmentation network and method fusing local and global features of the present application extract local information and global information of an input image through a double-branch structure, so that the segmentation features containing more information can be better fused, the problem that local and global features are difficult to interact is solved, and the functions of improving segmentation efficiency and overall performance are effectively realized.

[0034] 2. The present application proposes a new multi-scale feature fusion module, which utilizes the context information of the Transformer and the local representation features of the convolution; the local and global feature information can be processed at the same time, and excellent performance is shown in fast medical image segmentation.

[0035] 3. The present application reduces the complexity of calculation while extracting global features and local features, reduces the time required for network training while ensuring segmentation results, and realizes the function of improving segmentation performance while ensuring segmentation speed. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of an embodiment of the present application.

[0037] Figure 2 is a flowchart of a fast semantic segmentation network fusing local and global features of an embodiment of the present application.

[0038] Figure 3 is a structure diagram of a fast semantic segmentation network fusing local and global features of an embodiment of the present application.

[0039] Figure 4 is a structure diagram of a new multi-scale feature fusion module of an embodiment of the present application. DETAILED DESCRIPTION

[0040] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0041] I. Fast semantic segmentation network fusing local and global features

[0042] Embodiments of the application include two branches, one of which is a convolution operation for extracting local feature information of an image, and the other branch is a Transformer operation, the main purpose of which is to extract global information. The information of the two branches is fused together in an interactive way, and finally all the feature information is completely fused together through a new multi-scale fusion module. Referring to Figure 3 , the first row is a CNN module, the second row is a Transformer module, and the third row is a new multi-scale fusion module connected to the CNN module.

[0043] S11, the segmentation network mainly considers the speed and performance of feature extraction in semantic segmentation, adopts a double-branch structure to extract local features and global features of the input image respectively, and then outputs them through a feature fusion module. For the feature extraction part, two different branches (convolution and Transformer) are used to extract features. One branch is composed of a series of convolution operations, which can learn the local features of the feature image, such as boundaries, shapes, etc. The other branch is composed of three Transformer modules, the main function of which is to perform down-sampling operation on the input features, and use the self-attention mechanism to extract rich global context information. For the feature fusion part, we propose a new multi-scale fusion module, which fuses the local feature information from the convolution block and the global feature information from the Transformer in an interactive way through multiple bilateral connections, which can effectively fuse the context information of the Transformer and the local detail information of the convolution.

[0044] S12, for the convolution branch of the feature extraction part, multiple groups of convolution layers are used to achieve the extraction of local detail features of the input image. Each group of convolution layers includes one Conv operation plus one BN operation, and uses the ReLU function to activate it, and the formula of the convolution layer is as follows:

[0045] X' = F.ReLU(BN(Conv(X))) (1)

[0046] Where X represents the input feature map, Cony represents the convolution layer, BN represents the layer normalization operation, and ReLU represents the activation operation. The purpose of the Conv operation is mainly to extract the features of the feature map, and the BN layer is mainly to avoid the gradient explosion and gradient disappearance phenomenon produced during the gradient operation of the image. The main purpose of the ReLU layer is to control the output of each layer within a range, so that the value is a negative number less than 0.

[0047] S13, the module based on the Transformer is mainly used to capture the global features and context features during image segmentation. Compared with the traditional convolution-based double-branch architecture, one branch of the semantic segmentation network is mainly composed of a variant of the Transformer, “Outlooker”, which is mainly used to establish long-distance dependencies. The module is composed of an Outlooker attention layer for spatial encoding and an MLP for inter-channel information interaction. The specific formula is as follows:

[0048] X' = OutlookerAtt(LN(X)) + X, (2)

[0049] Z = MLP(LN(X')) + X' (3)

[0050] Where X represents the input, and LN represents the layer normalization. The input feature image X is first subjected to the LN layer, and the LN layer has a similar operation to the BN operation in the convolution operation, which is used for standardization. Then it is subjected to the Outlooker module to extract the global features of the image. The Outlooker module can be regarded as a structure with two independent stages, the first part contains a bunch of OutLooker for generating fine-grained representation, and the second part deploys a series of converters to aggregate global information. Before each part, there is a block embedding module to map the input to a specified shape. The output of the Transformer branch each time interacts with the output of the CNN module in S12, which is to enable the CNN branch to better learn the global features without increasing the computational complexity too much.

[0051] At the same time, the output of the Transformer branch each time is also directly input to the MSFFM module, also known as the multi-scale fusion module. This module integrates the contents of the convolution branch and the Transformer branch together and performs a multi-scale fusion operation on the information.

[0052] Suppose there is an image that needs to participate in semantic segmentation, which is named A, and Figure 1 and Figure 2It can be seen that the image A first passes through two convolution operations with downsampling to generate feature map A1. The resolution of feature map A1 becomes 1 / 4 of the original image A, and feature map A1 is subjected to CNN operation and Transformer operation. The Transformer branch generates feature map A2 as the first input to the novel multi-scale fusion module, and the size of feature map A2 is 1 / 8 of the original image A. At the same time, feature map A2 is added to feature map A1, but since the resolutions of feature map A2 and feature map A1 do not match, feature map A2 needs to be subjected to convolution operation with upsampling so that feature map A2 can be added to feature map A1 and then subjected to CNN operation to obtain a new feature map. Similarly, after the convolution operation of the CNN branch, the new feature map is subjected to CNN operation with downsampling, and then added to feature map A2. Unlike the CNN branch, the Transformer branch uses Transformer operation to fuse the added feature maps.

[0053] The above operation is repeated 3 times, and 3 global feature maps are generated by the Transformer branch, which are named as feature maps A2, A3 and A4, respectively. Their sizes are 1 / 8, 1 / 16 and 1 / 32 of the original image, respectively. In Figure 2 , they are marked. The final output of the CNN branch is named as feature map A5, and its resolution size is 1 / 4 of the original image.

[0054] Then all the outputs are fused in the MSFFM multi-scale fusion module to obtain a feature map that fuses global feature information and local feature information.

[0055] In Figure 3 , it can be seen that each convolution operation in the CNN branch includes an upsampling or downsampling operation. The purpose of this is to make the sizes of the feature maps matchable when added.

[0056] II. Multi-scale feature fusion module (MSFFM)

[0057] S21, Figure 4 is the architecture diagram of the multi-scale feature fusion module. First, the convolution operation with upsampling is used to make the inputs of different sizes. A linear bottleneck structure and interpolation operation are used. Compared with the previous output, such operation can obtain feature maps with the same dimension and resolution. Thus, the feature fusion operation can be performed with the input of the CNN branch.

[0058] The outputs obtained by the two branches of the feature extraction part are loaded into the fusion module, specifically the feature map obtained by the convolution branch and the four feature maps obtained by the Transformer branch. First, the feature maps obtained by the Transformer branch are upsampled according to the size of the output; then they are fused with the branch obtained by the convolution branch. The specific steps are to concatenate the feature maps of the same size obtained by the upsampling operation, and finally obtain the final output.

[0059] The specific upsampling operation is as follows: the feature map F output by the Transformer is subjected to a CNN operation with upsampling to obtain a feature map F' that is twice as large. The feature map F obtained after two Transformers is subjected to a linear bottleneck operation to obtain F'1. The feature maps F' and F'1 of the same size are added to obtain feature A1. Then the above operation is repeated, and the feature map A1 is upsampled to the same size as the feature map F2 obtained after the first Transformer. After the linear bottleneck operation, F2 is fused to obtain the output A2. Finally, according to the result after upsampling, the size of the feature map obtained by the convolution operation is inconsistent, so A2 needs to be upsampled by a factor of two, and then concatenated with the feature map obtained by the convolution operation to obtain the final output of the fusion module.

[0060] S22. In the fusion of local and global features in the fast semantic segmentation network, many convolution blocks are designed, mainly to extract relatively high-resolution features. For example, the first convolution block uses a convolution kernel with a stride of 2 and a size of 3x3 to learn high-resolution features. This design is mainly a trade-off between accuracy and efficiency. Other convolution blocks are mainly used to extract features with a resolution of 1 / 4 of the original image size. Each image to be processed will first undergo two convolutions with a stride of 2 before entering the dual branch. The purpose of this is mainly to save time when performing the Transformer operation on the feature image. Moreover, doing only two downsampling operations can also ensure that the global feature information is not lost too much and thus affect the result.

[0061] S23, The main composition of the new multi-scale fusion module is completed by the convolution operation with upsampling and the interpolation operation with linear bottleneck structure. The resolution of the original feature map is reduced by half every time the Transform operation in S13 is used, so the resolution of the feature map input into the new multi-scale fusion module is not uniform every time. The resolution of the feature map input into the new multi-scale fusion module at the end of the CNN branch in S12 is 1 / 4 of the original image size. Therefore, upsampling operation is needed to make the global low-resolution information and the local high-resolution information of the image be fused. First, the global low-resolution information is fused together through upsampling and addition operation, and then the global feature information and the local feature information are fused together through concatenate operation after the three global low-resolution features are fused together.

[0062] The above examples are only used to illustrate the design idea and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and the protection scope of the present application is not limited to the above examples. Therefore, any equivalent changes or modifications made according to the principles and design ideas disclosed by the present application are within the protection scope of the present application.

Claims

1. A semantic segmentation method based on a fast semantic segmentation network that integrates local and global features, characterized in that: The fast semantic segmentation network that integrates local and global features includes a first branch, a second branch, and an MSFFM multi-scale fusion module; The first branch is the CNN branch, which includes multiple convolutional layers used to extract local feature information of the image; The second branch is the Transformer branch, which includes an LN layer, multiple Outlooker attention layers, and a series of Transformer MLPs, used to extract global and contextual feature information of the image through downsampling and self-attention mechanisms. The feature information output by the first branch and the second branch is fused interactively through multiple bilateral connections; each computation output of the second branch interacts with the output of the first branch, which is used to enable the first branch to learn global features better without increasing computational complexity. Each calculation output of the second branch is connected to the MSFFM multi-scale fusion module. The MSFFM multi-scale fusion module is used to fuse the feature information output by the first and second branches after interactive fusion, and to perform multi-scale fusion operation on the information. Includes the following steps: S1: Let the input image to be semantically segmented be A; perform a two-branch operation on image A, extracting local and global features of the image through CNN and Transformer operations respectively; the specific steps are as follows: S11: Image A undergoes two convolution operations with downsampling to produce a feature map A1 with a resolution of 1 / 4 of that of image A, which is then input into the CNN branch and the Transformer branch respectively. S12: The Transformer branch generates a feature map A2 with a resolution of 1 / 8 of that of image A; the feature map A2 is input into the MSFFM multi-scale fusion module; at the same time, the feature map A2 is subjected to a convolution operation with upsampling and then added to the feature map A1, and then input into the CNN branch; S13: Perform a CNN operation with downsampling on the new feature map obtained after the convolution operation of the CNN branch, and then add it to feature map A2; S14: The Transformer branch uses Transformer operations to fuse the summed feature maps; S15: Repeat steps S12 to S14 three times. The Transformer branch generates feature maps A2, A3 and A4 respectively, with resolutions of 1 / 8, 1 / 16 and 1 / 32 of image A. S16: Perform a convolution operation with upsampling on feature map A4, then add it to the feature map output from the previous step of the CNN branch, and input it into the CNN branch to output feature map A5 after convolution operation. Its resolution is 1 / 4 of that of image A. S2: The outputs of the two branches are interacted and then input into the MSFFM multi-scale fusion module to fuse the local features extracted by the CNN operation and the global features extracted by the Transformer operation at different scales, resulting in a feature map that fuses both global and local feature information; the specific steps are as follows: S21: Load the feature map A5 obtained from the CNN branch and the feature maps A2, A3 and A4 obtained from the Transformer branch into the fusion module; S22: Based on the resolution of the output feature map, perform convolution operations with upsampling and interpolation operations with linear bottleneck structures on the feature maps obtained from the Transformer branch to obtain feature maps of the same dimension and resolution; then perform a concatenation operation with the feature maps obtained from the CNN branch to obtain the final output; the specific steps are as follows: S221: Perform a CNN operation with upsampling on the feature map A4 output by the last branch of the Transformer to obtain a feature map A4′ with double the resolution. Perform a linear bottleneck interpolation operation on the feature map A3 obtained after two Transformer operations to obtain a feature map A3′. Add the feature maps A4′ and A3′ with the same resolution to obtain feature A33. S222: Perform a CNN operation with upsampling on feature map A33 to obtain feature map A22 with doubled resolution. Perform a linear bottleneck interpolation operation on feature map A2 obtained after one Transformer operation to obtain feature map A2′. Add feature maps A22 and A2′ with the same resolution to obtain feature A11. S223: Perform a CNN operation with upsampling on A11 to obtain a feature map A00 with doubled resolution, and then perform a concatenation operation with the feature map A5 obtained from the CNN branch to obtain the final output.

2. The semantic segmentation method based on a fast semantic segmentation network that fuses local and global features as described in claim 1, characterized in that: In the first branch, set up X Given the input feature image, Conv For convolution operations, BN For layer normalization operation, ReLU For activation operations; each convolutional layer includes one. Conv Operation, once BN Operation and activation operation ReLU The function performs this; Conv The operation is used to extract features from the feature map; BN This operation is used to avoid gradient explosion and gradient vanishing during gradient calculations. ReLU The function is used to control the output of each layer within a preset range and make its value a negative number less than 0; The formula for the convolutional layer is: (1)。 3. The semantic segmentation method based on a fast semantic segmentation network that integrates local and global features as described in claim 2, characterized in that: In the second branch, Let LN be the layer normalization layer; OutlookAtt be the Outlook attention layer, used for spatial encoding to generate a refined representation; MLP be used for inter-channel information interaction to aggregate global information and establish long-range dependencies; each part is preceded by a block embedding module to map the input to a specified shape; the input feature image X is sequentially passed through the LN layer, the Outlook module, and the MLP to extract the global features of the image: （2）， （3）。 4. A computer storage medium, characterized in that: It contains a computer program that can be executed by a computer processor, which executes a fast semantic segmentation network semantic segmentation method based on the fusion of local and global features as described in any one of claims 1 to 3.

Images