Medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion

Abstract

The invention relates to the technical field of image processing, in particular to a medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion, comprising: acquiring an original medical training data set, the original medical image training data including an original medical image and a corresponding segmentation map label; establishing a medical segmentation model, and training the medical segmentation model by using the medical image training data set; and inputting a medical image to be segmented into the trained medical segmentation model to obtain a segmentation image. According to the invention, a neighborhood self-attention NAF module enhances the contour and boundary of a target area; a semantic fusion module FMF based on depth separable convolution is adopted to fuse features of different levels so as to improve segmentation precision; meanwhile, the solid theoretical basis and scientific basis and reference are provided for detection of focus tissues in medical diagnosis.

Description

Technical Field

[0001] The present invention belongs to the field of medical image segmentation, and in particular relates to a medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion. Background Art

[0002] Cancer, as a global public health problem, poses a huge threat to human health. Among them, colorectal cancer has a particularly prominent impact. It is the third most common malignant tumor and the second most deadly cancer, causing millions of deaths each year. Among men, prostate cancer, lung and bronchial cancer, and colorectal cancer account for half of the cases. Among women, breast cancer, lung cancer, and colorectal cancer account for 52% of all newly diagnosed cancer cases. At the same time, according to statistics: among young male groups, colorectal cancer has surpassed brain tumors and is the main cause of cancer death in men aged 20-49.

[0003] Colorectal cancer often originates from intestinal polyps, which are masses on the inner wall of the intestine. If they can be detected and removed in time before they become cancerous, the occurrence of colorectal cancer can be effectively prevented. Colonoscopy is the main method for diagnosing and preventing colorectal cancer. Polyps are detected and located through an endoscope. Through colonoscopy, colorectal cancer can be detected and treated earlier, thereby reducing overall mortality and morbidity. However, studies have shown that 1 in 4 polyps is missed during colonoscopy. This problem occurs due to the complex and varied size and shape of polyps and the low contrast between polyps and normal tissues. Therefore, accurately segmenting polyps from colonoscopy images is a very challenging task. Traditional polyp segmentation methods rely on manually extracted features such as color, texture, and shape, but lack robustness in dealing with complex scenes and have limitations in capturing global context. With the development of deep learning technology, medical image segmentation algorithms based on this technology have made significant progress in accuracy and efficiency. Developing an automated polyp segmentation algorithm to assist doctors in diagnosis can not only reduce the workload of doctors and avoid misdiagnosis or missed diagnosis due to human factors, but also help speed up the diagnosis process and improve diagnostic efficiency. Therefore, it is of practical significance. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion, comprising:

[0005] S1: Acquire an original medical training data set, wherein the original medical image training data includes: an original medical image and a corresponding segmentation map label;

[0006] S2: Establish a medical segmentation model and train the medical segmentation model using a medical image training data set; the medical segmentation model includes: an MIT feature encoder, three NAF neighborhood attention feature extraction modules, and an improved FMF decoder;

[0007] S3: Input the medical image to be segmented into the trained medical segmentation model to obtain a segmentation map.

[0008] Beneficial effects of the present invention:

[0009] The present invention adopts the Transformer-based MIT (Mix Transformer) feature encoder as the backbone network to extract features. It combines four overlapping path merging modules and a self-attention mechanism for encoding in four different stages. These stages can provide both high-resolution coarse features and low-resolution detail features. The NAF (Neighborhood Attention Fusion) module is used to improve the model's neighborhood perception and feature expression capabilities for polyp pixels. The feature map processed by NAF is input into the FMF (Feature Map Fusion) decoder to achieve smooth and efficient multi-scale feature fusion, thereby obtaining better polyp segmentation results and improving the accuracy of segmentation detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a structural schematic diagram of the medical segmentation model of the present invention;

[0011] Figure 2 It is a structural schematic diagram of the FMF decoder of the present invention;

[0012] Figure 3 Schematic diagram of the structure of the NAF neighborhood self-attention module of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] See also Figure 1 The present invention provides a medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion, comprising:

[0015] S1: obtaining an original colorectal polyp image training data set, wherein the original colorectal polyp image training data includes: an original colorectal polyp image and a corresponding polyp segmentation map label;

[0016] In the specific implementation, the original colorectal polyp image training dataset used is Kvasir-SEG, which is a colorectal polyp segmentation dataset based on endoscopic images, with a total of 1000 groups of images, each group of images contains an endoscopic image containing polyps (i.e., sample image) and a corresponding true value label mask map, and the image resolution ranges from 332×487 to 1920×1072. In order to facilitate the experiment, the embodiment of the present invention adjusts the size of all sample images and true value label mask maps to 352×352 in the network training experiment, and uses 800 groups of images for model training, and the remaining 200 groups of images for testing.

[0017] S2: using the original colorectal polyp image training data set to train a colorectal polyp segmentation model, the colorectal polyp segmentation model includes: an MIT feature encoder, three NAF neighborhood attention feature extraction modules, and an FMF decoder;

[0018] S3: Input the colorectal polyp image to be segmented into the trained colorectal polyp segmentation model to obtain a polyp segmentation map.

[0019] Preferably, the training of the colorectal polyp segmentation model comprises:

[0020] S21: Input the original colorectal polyp image into the MIT feature encoder to extract global semantic features and obtain feature F 1 ~F 4 ;

[0021] In the implementation of the present invention, the MIT feature encoder is used as a backbone network to extract the global semantic features of the image. The MIT feature encoder combines four overlapping path merging modules and a self-attention mechanism in four different stages for encoding. These stages can provide both high-resolution coarse features and low-resolution detail features. These features are generally used to enhance the performance of semantic segmentation tasks. After an image is input into the MIT feature encoder, multiple multi-scale features of different levels can be obtained. The higher the level, the smaller the scale of the corresponding feature map, and the larger the dimension. In an embodiment of the present invention, the sample image is input into the MIT feature encoder for global semantic feature extraction to obtain low-level shallow features and multiple high-level deep features. The MIT feature encoder can output 4 levels of features. Let the features of these four levels be F from low level to high level. 1 、F 2 、F 3 and F 4 In the embodiment of the present invention, F 1 As a low-level shallow feature, F 2 、F 3 and F 4Considered as high-level deep features. Low-level shallow features contain rich detail information, such as texture, edge, etc., while high-level deep features contain abstract high-level semantic information.

[0022] See also Figure 2 In the embodiment of the present invention, the NAF module is used to extract context features and context information from high-level deep features, and significantly reduces the overall parameters of the model, achieving a balance between performance, model size and inference speed. Semantic interaction between high-level and low-level features helps to enhance the learning ability of the model. The attention mechanism allows the model to pay more attention to the areas that should be cared about during the polyp segmentation process and suppress noise. However, the self-attention mechanism calculates attention for all calculation points in the entire image, and its computational complexity is quadratic with the number of calculation points, which will bring about a large increase in computational overhead, which is not conducive to reducing model parameters and achieving lightweight. The neighborhood attention limits the calculation points that need to be calculated for attention each time to an area, which can greatly reduce the amount of calculation.

[0023] The neighborhood attention NAF module of the present invention includes two 1*1 convolution modules, two BatchNorm layers, and N window attention layers. First, the high-level feature H input to the NAF module is upsampled to obtain H′, and then the feature map obtained by concatenating H′ with the low-level feature L input to the NAF module using the Concat function is convolved by the 1*1 convolution module and then normalized by the BatchNorm layer to obtain the feature P. 1 ; Then convolve the feature map H′ through a 1*1 convolution module and then use the BatchNorm layer to normalize the feature map obtained by adding it to L to obtain the feature map C; then use C to obtain the Query matrix, Key matrix and Value matrix, perform matrix multiplication on the Query matrix and the Key matrix to obtain the Score attention score matrix, perform matrix multiplication on the score matrix and the Value matrix to obtain the result map after the attention calculation in a window, and after Slide sliding, continuously calculate the result map after the attention calculation in each window. Finally, the output P of the entire result map 2 With P 1 Add them together to get the output feature map of the NAF module.

[0024] S22: The original feature F 4 As a high-level feature H, F 3 As the low-level feature L, H is upsampled using bilinear interpolation, and then residually connected with L. The obtained feature map is input into the NAF module to obtain the low-level shallow feature T 1 ;

[0025] S23: The feature T 1 As the high-level feature H, the original feature F2 As the low-level feature L, H is upsampled using bilinear interpolation, and then residually connected with L. The obtained feature map is input into the NAF module to obtain the middle-level feature T 2 ;

[0026] S24: Feature T 2 As the high-level feature H, the original feature F 1 As the low-level feature L, H is upsampled using bilinear interpolation, and then residually connected with L. The obtained feature map is input into the NAF module to obtain the high-level feature T 3 .

[0027] See also Figure 3 In the FMF module of the embodiment of the present invention, after the initial polyp image is processed by the encoder and the multi-scale feature map is embedded in the high-dimensional semantic information through the NAF block, it is transmitted to the semantic fusion module for semantic fusion and decoding, and finally the segmentation result is obtained. In the image field, since the task objects account for different sizes in the image, receptive fields of different sizes are very necessary, and the pyramid-shaped feature intermediate map, from the low-dimensional features Figure 1 The design of gradually refining and compressing the feature map into a high-dimensional feature map to gradually increase the semantic information it contains has been chosen by many scholars and has been proven to be effective in practice. At the same time, in the field of polyp segmentation, multi-scale intermediate maps are usually involved in the final segmentation. The feature fusion capability of the decoder determines the final segmentation effect to a certain extent, and how to efficiently fuse these features of different scales is also a challenge.

[0028] Preferably, the improved FMF decoder comprises: 4 1*1 convolution modules, 4 BatchNorm layers, 3 hole convolution layers and 1 depth-separable convolution module; the 4 1*1 convolution modules in the FMF decoder respectively perform 1 ~F 4 The convolution operation is then normalized using the BatchNorm layer to obtain the feature F. 1 ′~f 4 ′, then the feature F 2 ′ is upsampled to obtain feature U 2 ; Use the Concat function to combine feature F 1 ′ and U 2 After splicing, the feature extraction is performed using the hole convolution to obtain the feature D 1 ; Then the feature F 3 ′ is upsampled to obtain feature L 3 ; Use the Concat function to combine feature F 2 ′ and L 3 After splicing, the dilated convolution is used to extract features, and then up-sampled to obtain feature D2 ; For feature L 3 Then upsample to get feature U 3 ; The feature F 4 ′ is upsampled to obtain feature L 4 ; Use the Concat function to combine feature F 3 ′ and L 4 After splicing, the dilated convolution is used to extract features, and then up-sampled to obtain feature D 3 ; For feature L 4 Then upsample to get feature U 4 ; The feature F 1 ′、D 1 , D 2 , D 3 , U 2 , U 3 , U 4 Add them together to get feature C; input feature C into the depth-wise separable convolutional layer to get the final segmentation map P.

[0029] S24: The intermediate feature T 1 ~T 3 and the original feature F 4 Input the FMF module for feature fusion and obtain the output segmentation map P.

[0030] S25: Construct a loss function of the colorectal polyp segmentation model based on the original polyp segmentation map P and the polyp segmentation map label corresponding to the original colon polyp image, update the parameters of the colorectal polyp segmentation model with the minimum loss function as the optimization goal, and complete the training of the colorectal polyp segmentation model.

[0031] Preferably, the loss function of the knot medical segmentation model includes:

[0032] L1=L BCE (P,G)+L IOU (P,G)

[0033] Wherein, L1 represents the loss function of the medical segmentation model, G represents the polyp segmentation map label corresponding to the original colon polyp image, and P represents the original polyp segmentation map of the original colon polyp image after segmentation by the medical segmentation model; L BCE () represents the binary cross entropy loss function; L IOU () represents the intersection-over-union loss function.

[0034] Preferably, this model is implemented using the Pytorch framework and trained on a Tesla A100 GPU with 40GB of video memory. Our model requires 300 epochs to complete training, and the batchsize is set to 8 for each training. We use the Adam optimizer and set the initial learning rate to 1e-4, while using poly learning rate as a strategy to adjust the learning rate during training. All input images are resized to 352×352 during training and testing. The data augmentation strategies used are random rotation, vertical flip, and horizontal flip.

[0035] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion, characterized in that: include: S1: Acquire an original medical training data set, wherein the original medical image training data includes: an original medical image and a corresponding segmentation map label; S2: Establish a medical segmentation model and train the medical segmentation model using a medical image training data set; the medical segmentation model includes: an MIT feature encoder, three NAF neighborhood attention feature extraction modules, and an improved FMF decoder; S3: Input the medical image to be segmented into the trained medical segmentation model to obtain a segmentation map.

2. The medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion according to claim 1, characterized in that: The training of the medical segmentation model comprises: S21: Input the original medical image into the MIT feature encoder to extract global semantic features and obtain features F1 to F4; S22: The original feature F4 is used as the high-level feature H, and F3 is used as the low-level feature L. H is upsampled using the bilinear interpolation method, and then residually connected with L. The obtained feature map is input into the NAF module to obtain the low-level shallow feature T1; S23: Take feature T1 as high-level feature H, take original feature F2 as low-level feature L, upsample H using bilinear interpolation, perform residual connection with L, and input the obtained feature map into NAF module to obtain middle-level feature T2; S24: Feature T2 is used as the high-level feature H, and the original feature F1 is used as the low-level feature L. H is upsampled using bilinear interpolation, and then residually connected with L. The obtained feature map is input into the NAF module to obtain the high-level feature T3; S25: Input the features T1-T3 and the original feature F4 into the improved FMF decoder for feature fusion, and output the original medical segmentation map P; S26: Construct a loss function of the medical segmentation model based on the original medical segmentation map P and the segmentation map label corresponding to the original medical image, update the parameters of the medical segmentation model with the minimum loss function as the optimization goal, and complete the training of the medical segmentation model.

3. The medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion according to claim 2, characterized in that: The improved FMF decoder comprises: 4 1*1 convolution modules, 4 BatchNorm layers, 3 hole convolution layers and 1 depth-separable convolution module; The four 1*1 convolution modules in the FMF decoder perform convolution operations on features T1~T3 and the original feature F4 respectively, and then use the BatchNorm layer to normalize them to obtain features F1′~F4′, and then upsample the feature F2′ to obtain feature U2; use the Concat function to concatenate features F1′ and U2, and then use the hole convolution to extract their features to obtain feature d1; then upsample the feature F3′ to obtain feature L3; use the Concat function to concatenate features F2′ and L3, and then use the hole convolution to extract their features Convolution is used to extract features, and then upsampling is performed to obtain feature d2; feature L3 is upsampled to obtain feature U3; feature F4′ is upsampled to obtain feature L4; feature F3′ and L4 are concatenated using the Concat function, and then feature extraction is performed using a hole convolution, and then upsampled to obtain feature D3; feature L4 is upsampled to obtain feature U4; features F1′, D1, D2, D3, U2, U3, and U4 are added together to obtain feature C; feature C is input into the depthwise separable convolution layer to obtain the final segmentation map P.

4. The medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion according to claim 2, characterized in that: The neighborhood attention NAF module includes: 2 1*1 convolution modules, 2 BatchNorm layers, and N window attention layers; The high-level feature H of the input NAF module is upsampled to obtain H′, and then H′ and the low-level feature L of the input NAF module are concatenated using the Concat function to obtain a feature map, which is then convolved by a 1*1 convolution module and then normalized by a BatchNorm layer to obtain feature P1; the feature map H′ is then convolved by a 1*1 convolution module and then normalized by a BatchNorm layer to obtain a feature map M obtained by adding it to L; M is then used to obtain the Query matrix, Key matrix and Value matrix, and the Query matrix and the Key matrix are matrix multiplied to obtain the Score attention score matrix, and the score matrix and the Value matrix are matrix multiplied to obtain the result map after attention calculation within a window. After Slide sliding, the result map after attention calculation within each window is continuously calculated; finally, the output P2 of the entire result map is added to P1 to obtain the output feature map of the NAF module.

5. The medical image segmentation method combining neighborhood self-attention and hierarchical feature fusion according to claim 2, characterized in that: The loss function of the medical segmentation model is constructed according to the original medical segmentation map P and the segmentation map label corresponding to the original medical image, including: L1=L BCE (P,G)+L IOU (P,G) Wherein, L1 represents the loss function of the medical segmentation model, G represents the polyp segmentation map label corresponding to the original colon polyp image, and P represents the original polyp segmentation map of the original colon polyp image after segmentation by the medical segmentation model; L BCE () represents the binary cross entropy loss function; L IOU () represents the intersection-over-union loss function.

Images