A medical image segmentation method based on local and global context information encoding

Abstract

The present invention discloses a medical image segmentation method based on local and global context information encoding. The method obtains a medical image to be segmented, performs preprocessing, and then inputs it into a constructed segmentation network model. The segmentation network model includes an encoding module, a feature extraction module, and a decoding module. In the encoding module, the preprocessed medical image to be segmented is sequentially subjected to four convolutions, a maximum pooling operation, and a connection operation. Then, multi-scale features are fused through depthwise separable convolution to output encoded features. In the feature extraction module, the encoded features are sequentially passed through a void attention module and a spatial grid attention module to extract a feature map containing global context information. In the decoding module, the feature map is subjected to a pixel reorganization operation, and the feature maps output by the pixel reorganization operation are spliced ​​and input into a convolution layer to obtain a segmentation result. The present invention effectively integrates local context information and global context information, and the segmentation result is more accurate.

Description

Technical Field

[0001] The present application belongs to the field of medical image segmentation technology, and in particular relates to a medical image segmentation method based on local and global context information encoding. Background Art

[0002] Image segmentation, a hot topic in the field of image understanding, is a classic problem in computer vision research. Medical image segmentation, a branch of image segmentation, can clarify anatomical or pathological structural changes in images, significantly improving diagnostic efficiency and accuracy. Currently, popular medical image segmentation processes primarily target images of various cells, tissues, and organs, which exhibit diverse size, shape, and texture.

[0003] With the continuous development of deep learning, deep neural networks have become the mainstream of medical image segmentation research. Convolutional neural networks (CNNs), such as fully convolutional networks (FCNs), U-Nets, and their variants, are widely used in medical image segmentation. However, in CNNs, most feature extraction is performed through convolution operations. Because the size of the convolution kernel is generally smaller than the size of the input image, the features extracted by CNNs tend to focus more on local information and ignore global information. In addition, medical images often suffer from noise, blurred boundaries, and low contrast, which means that CNN-based methods still have some limitations in the field of medical image segmentation.

[0004] Self-attention, one of the most widely used attention mechanisms, has achieved tremendous success in the field of Natural Language Processing (NLP). In recent years, with the continuous development of deep learning, more and more researchers have begun to apply self-attention to the field of computer vision (CV) to alleviate the limitations of CNNs in this field. The attention extracted by self-attention based on the feature map itself can capture long-range dependencies and global features between different elements in a sequence, thereby improving model performance. The computational complexity of the self-attention mechanism is high, especially when the input feature map has large dimensions, which leads to a waste of computing resources. Furthermore, the self-attention mechanism is a global feature extraction method and may ignore local information in the image, resulting in reduced performance in some tasks. Therefore, in practical applications, the use of the self-attention mechanism needs to be determined by weighing the computational complexity, training data, and task requirements. At the same time, it is also necessary to combine it with other feature extraction methods to achieve better performance.

[0005] Researchers have proposed numerous methods for combining local and global contextual information, typically through CNNs and self-attention mechanisms. Compared to traditional CNN models, the CMT model fuses CNNs and self-attention mechanisms, and uses a contextual conditioning module to segment objects of varying sizes at different resolutions. TransUnet, modeled after the Unet network structure, uses a self-attention-based Transformer at the bottom encoder layer to encode CNN feature maps into context sequences, combining the advantages of both CNNs and self-attention. Hybrid CNN-Attention methods have been proposed, primarily leveraging the locality of CNNs and the long-range dependencies of self-attention to encode both global and local features. However, these methods face several obstacles that hinder their ability to achieve higher performance: 1) The fusion of local and global information: Most methods combine local and global information by stacking a CNN and a Transformer in parallel, but this approach fails to fully leverage their respective strengths or fuse the information they extract. 2) The multi-scale information of layered encoders is underutilized: Layered encoders can extract features at different scales, but in some methods, this multi-scale information is underutilized. 3) High training complexity: Some methods have high training complexity, which leads to long training time and difficulty in applying to actual scenarios. Summary of the Invention

[0006] In view of the above problems, this application proposes an image segmentation method based on local and global context information encoding, which performs accurate medical image segmentation tasks by utilizing the local feature representation of CNN and the global long-range relationship of Self-Attention.

[0007] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0008] A medical image segmentation method based on local and global context information encoding, comprising:

[0009] Obtaining a medical image to be segmented, preprocessing it, and then inputting it into a constructed segmentation network model, wherein the segmentation network model includes an encoding module, a feature extraction module, and a decoding module;

[0010] In the encoding module, the preprocessed medical image to be segmented is subjected to four convolution and maximum pooling operations in sequence to obtain feature maps F1, F2, F3, and F4. The feature maps F1, F2, and F3 are adaptively average pooled to the same size as F4 and then concatenated with F4. Then, multi-scale features are fused through depthwise separable convolution to output the encoded features.

[0011] In the feature extraction module, the encoded features pass through the void attention module and the spatial grid attention module in sequence to extract the feature map containing local and global context information;

[0012] In the decoding module, the feature map containing local and global context information is combined with the feature maps F1, F2, F3, and F4 through a pixel reorganization operation. The feature maps output by the pixel reorganization operation are then spliced ​​and input into a convolutional layer to obtain the segmentation result.

[0013] Furthermore, the hole attention module performs the following operations:

[0014] Input the encoded feature into the first DAM branch of the hole attention module, which includes four cascade branches. The first cascade branch consists of a 3×3 ordinary convolution, the second cascade branch consists of a 3×3 hole convolution with a hole rate of 3 and a 1×1 ordinary convolution, the third cascade branch consists of a 3×3 ordinary convolution, a 3×3 convolution with a hole rate of 3 and a 1×1 ordinary convolution, and the fourth cascade branch consists of a 3×3 ordinary convolution, a 3×3 convolution with a hole rate of 3, a 3×3 convolution with a hole rate of 5 and a 1×1 ordinary convolution. The feature map obtained by each cascade branch is connected with the encoded feature to obtain the first feature map; after performing the SE operation on the first feature map, it is multiplied with the first feature map, and then subjected to 1×1 convolution to obtain the feature map output by the first DAM branch;

[0015] Input the encoded features into the second DAM branch of the hole attention module, and the second DAM branch performs a prospective attention operation to obtain a feature map output by the second DAM branch;

[0016] The features output by the first DAM branch are added to the feature map output by the second DAM branch to obtain the feature map output by the hole attention module.

[0017] Furthermore, the spatial grid attention module performs the following operations:

[0018] Input the feature map output by the hole attention module to the first SGAM branch of the spatial grid attention module, and the first SGAM branch performs a sparse global attention operation to obtain the feature map output by the first SGAM branch;

[0019] The feature map output by the hole attention module is input to the second SGAM branch of the spatial grid attention module, and the second SGAM branch performs a spatial attention operation to obtain a weight coefficient output by the second SGAM branch;

[0020] The feature map output by the first SGAM branch is multiplied by the weight coefficient output by the second SGAM branch to obtain a feature map containing global context information.

[0021] Furthermore, the loss function of the segmentation network model is as follows:

[0022]

[0023] Among them, G represents the real graph, S represents the segmentation result graph, |G| and |S| represent the number of elements of G and S respectively, and |G∩S| represents the intersection between G and S.

[0024] This application proposes an image segmentation method based on encoding local and global context information. It proposes a hole attention module (DAM) and a spatial grid attention module (SGAM). These modules effectively integrate local and global context information through convolution and attention mechanisms. In the decoding module, upsampling using pixel shuffle enables precise positioning of the segmentation target. This technical solution effectively integrates local and global context information, resulting in more accurate segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of the medical image segmentation method based on local and global context information encoding.

[0026] Figure 2 Schematic diagram of the overall framework of the segmentation network model for this application.

[0027] Figure 3 Schematic diagram of the DAM for this application.

[0028] Figure 4 Schematic diagram of SGAM in this application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0030] In one embodiment, Figure 1 As shown in the figure, a medical image segmentation method based on local and global context information encoding is proposed, including:

[0031] Step S1: Obtain a medical image to be segmented, perform preprocessing on it, and then input it into a constructed segmentation network model. The segmentation network model includes an encoding module, a feature extraction module, and a decoding module.

[0032] This application constructs a segmentation network model, trains the segmentation network model, and finally performs segmentation processing on the medical image to be segmented through the trained segmentation network model.

[0033] When training the segmentation network model, it is necessary to first obtain a training dataset. This embodiment selects appropriate images from a public dataset as a dataset. For example, the COVID-19 dataset, the MoNuSeg dataset, and the DRIVE dataset can be used. The COVID-19 dataset is derived from axial CT images of COVID-19 patients; the MoNuSeg dataset is derived from H&E stained tissue images magnified 40 times, and the DRIVE dataset is a dataset for retinal vascular segmentation.

[0034] The images in the dataset were preprocessed and augmented. Preprocessing included converting the image formats from nii, jpg, and png to npy, resizing the images to 512×512, and normalizing them. Data augmentation involved random cropping, flipping, and rotating the images to generate the training dataset. Random cropping involves cropping the original images to different sizes; flipping and rotation involve flipping the images horizontally, vertically, or diagonally, or rotating the images by varying angles.

[0035] Finally, the preprocessed and data-enhanced images are divided into training sets and validation sets according to the set ratio.

[0036] After the segmentation network model is trained, when using the trained segmentation network model to segment the medical image to be segmented, it is also necessary to perform certain preprocessing on the acquired medical image to be segmented, such as converting the image format from nii, jpg, png to npy, adjusting the image size to 512×512, normalizing the image, etc.

[0037] Segmentation network model such as Figure 2As shown in the figure, the proposed architecture comprises an encoding module, a feature extraction module, and a decoding module. In the encoding module, a series of convolution and pooling operations capture multi-scale features within a limited zoom range. In the feature extraction module, the Dilated Attention Module (DAM) and the Spatial-Grid Attention Module (SGAM) are proposed. These two modules effectively integrate local and global contextual information through convolution and attention mechanisms. The DAM is responsible for extracting fine local features. It uses dilated convolutions with different dilation rates to increase the receptive field and extract features at different scales. The SE module then captures the importance of each channel in the feature map. Furthermore, Outlook Attention captures spatial correlation between the channel information of the current region and the surrounding regions, obtaining spatially relevant local contextual information and compensating for the spatial independence of convolution. Grid Attention can capture long-range dependencies with minimal computational effort. This is because Grid Attention divides the feature map into multiple blocks using a uniformly sparse grid. Pixels within each block perform attention calculations on the pixels at the same location in other blocks. However, this approach ignores relevant spatial information within the same block. Therefore, this embodiment proposes SGAM, which uses Grid Attention to extract sparse global information while also using spatial attention. Attention weights are derived along the channel dimension. The attention weights are multiplied by the feature map obtained by Grid Attention to refine the features and capture detailed global context. In the decoding module, pixel shuffle upsampling achieves accurate positioning of the segmentation target.

[0038] Step S2: In the encoding module, the preprocessed medical image to be segmented is subjected to four convolution and maximum pooling operations in sequence to obtain feature maps F1, F2, F3, and F4. The feature maps F1, F2, and F3 are adaptively average pooled to the same size as F4 and then concatenated with F4. Then, multi-scale features are fused through depthwise separable convolution to output the encoded features.

[0039] The input size of the encoding module is C×H×W image (C, H and W are the number of channels, length and width of the image respectively). The input image undergoes continuous 3×3 convolution and maximum pooling operations to obtain feature maps F1, F2, F3, and F4. The sizes of these four feature maps are C1×H×W, and (C1 is the number of channels), F1, F2, and F3 are adaptively averaged pooled to the same size as F4, and the feature map after adaptive average pooling is concatenated with F4. At this time, the size of the feature map is Finally, the multi-scale features are fused through deep separable convolution to finally achieve the purpose of fusing high-level and low-level semantic information. The output encoding feature has a size of

[0040] Step S3: In the feature extraction module, the encoded features pass through the void attention module and the spatial grid attention module in sequence to extract the feature map containing local and global context information.

[0041] In this step, the feature extraction module includes the hole attention module (DAM) and the spatial grid attention module (SGAM). The operations performed by these two networks are described below.

[0042] The hole attention module (DAM) is as follows Figure 3 As shown, perform the following operations:

[0043] Input the encoded feature into the first DAM branch of the hole attention module, which includes four cascade branches. The first cascade branch consists of a 3×3 ordinary convolution, the second cascade branch consists of a 3×3 hole convolution with a hole rate of 3 and a 1×1 ordinary convolution, the third cascade branch consists of a 3×3 ordinary convolution, a 3×3 convolution with a hole rate of 3 and a 1×1 ordinary convolution, and the fourth cascade branch consists of a 3×3 ordinary convolution, a 3×3 convolution with a hole rate of 3, a 3×3 convolution with a hole rate of 5 and a 1×1 ordinary convolution. The feature map obtained by each cascade branch is connected with the encoded feature to obtain the first feature map; after performing the SE operation on the first feature map, it is multiplied with the first feature map, and then subjected to 1×1 convolution to obtain the feature map output by the first DAM branch;

[0044] Input the encoded features into the second DAM branch of the hole attention module, and the second DAM branch performs a prospective attention operation to obtain a feature map output by the second DAM branch;

[0045] The features output by the first DAM branch are added to the feature map output by the second DAM branch to obtain the feature map output by the hole attention module.

[0046] Specifically, such as Figure 3As shown in the figure, the encoded features are passed through two branches to extract local context information. In the first branch, the encoded features are passed through four cascade branches respectively. The first cascade branch consists of a 3×3 ordinary convolution, the second cascade branch consists of a 3×3 dilated convolution with a dilation rate of 3 and a 1×1 ordinary convolution, the third cascade branch consists of a 3×3 ordinary convolution, a 3×3 convolution with a dilation rate of 3 and a 1×1 ordinary convolution, and the fourth cascade branch consists of a 3×3 ordinary convolution, a 3×3 convolution with a dilation rate of 3, a 3×3 convolution with a dilation rate of 5 and a 1×1 ordinary convolution. As the dilation rate of the dilated convolution changes (from 1 to 1, 3 and 5), the receptive field of each branch also changes (to 3, 7, 9, 19). The feature map obtained by each cascade branch is connected to the original input encoded feature through Concatenate. The size of the feature map obtained after connection is The concatenated feature map is then input into the SE module (in this module, it is first pooled through an average pooling layer, and then the result is input into a fully connected layer. Next, it is activated using the ReLU activation function and input into another fully connected layer. Finally, the Sigmoid activation function is used for output) to obtain the attention weights between channels, enhance the channels of the feature map that are useful for the current task, and suppress the feature channels that are not very useful for the current task. The obtained attention weights are multiplied by the concatenated feature map to obtain a feature map (size Then a 1×1 convolution is performed to reduce the channel dimension (size is Get the feature map output by the first DAM branch.

[0047] In the second DAM branch, the Outlook Attention module is used to capture the spatial correlation information between the current area and the surrounding areas from the channel information of the current area, making up for the spatial irrelevance of the convolution. The feature maps obtained by the two branches are fused to complete the extraction of fine local context information and capture local context information with spatial correlation.

[0048] The operation process of Outlook attention is as follows:

[0049] The input feature map is of size C3×H×W, and for each spatial location, it is located at the center of a local window of size K×K ( Figure 3 The upper branch embeds it into (K 2 ×K 2 , 1, 1), and then reshape it into (1, K 2 , K 2), generates an attention matrix, and calculates attention weights using the softmax activation function. The following branch embeds the input features using a Linear function, mapping them to a new feature V. The Unfold operation then extracts a KxK region from the feature. The results from the two branches are matrix-multiplied, and the Fold function is used to restore the feature map to its original input size, resulting in the feature map output by the second DAM branch.

[0050] The feature maps output by the two branches are added together to obtain the output of the hole attention module (DAM).

[0051] The spatial grid attention module (SGAM) is as follows Figure 4 As shown, perform the following operations:

[0052] Input the feature map output by the hole attention module to the first SGAM branch of the spatial grid attention module, and the first SGAM branch performs a sparse global attention operation to obtain the feature map output by the first SGAM branch;

[0053] The feature map output by the hole attention module is input to the second SGAM branch of the spatial grid attention module, and the second SGAM branch performs a spatial attention operation to obtain a weight coefficient output by the second SGAM branch;

[0054] The feature map output by the first SGAM branch is multiplied by the weight coefficient output by the second SGAM branch to obtain a feature map containing global context information.

[0055] Specifically, the output of the hole attention module (DAM) is used as the input of the spatial grid attention module SGAM, and the input feature map is subjected to two branches for global context information extraction. In order to make up for the limitations of the local receptive field, this application completes the global context information feature extraction through two branches. One branch uses Grid Attention for global feature extraction. Since Grid Attention divides the feature map into multiple blocks through a uniform sparse grid, and performs self-attention calculation on the pixels at the same position in different blocks, spatial attention calculation is used in another branch to obtain the correlation within the same block.

[0056] In one branch, sparse global attention (Grid Attention) is used to extract global information. This method divides the input feature map F into multiple blocks using a uniform sparse grid. The pixels in each block perform attention calculations with the pixels at the same position in other blocks to obtain the feature map F'. In another branch, the input feature map F undergoes a spatial attention operation, and a 1×H×W channel description is obtained by average pooling of the channel dimension. Then, a 7×7 convolution layer and an FReLU activation function are used to obtain the weight coefficient. The feature map F' obtained in one branch is multiplied by the weight coefficient obtained in the other branch to complete the global context information extraction and obtain a feature map containing global context information. Figure 2 Indicated by F5.

[0057] Step S4: In the decoding module, the feature map containing global context information is subjected to a pixel reorganization operation with the feature maps F1, F2, F3, and F4. The feature maps output by the pixel reorganization operation are then spliced ​​and input into a convolutional layer to obtain the segmentation result.

[0058] This step uses pixel reorganization (PixelShuffle) to upsample the feature maps F1, F2, F3, and F4 obtained by the encoding module and the feature map containing global context information obtained by the feature extraction module to increase the image size and compensate for the information lost in the encoding layer. PixelShuffle uses convolution and multi-channel reorganization to generate high-resolution feature maps with the same target size as the input image.

[0059] The function implemented by PixelShuffle can be expressed by formula (4):

[0060] N×(C×r×r)×H×W →N×C′×(r×H′)×(r×W′) (4)

[0061] Where N represents the batch size, r represents the upsampling ratio, C, H, and W represent the number of channels, height, and width of the input feature map, respectively, and C′, H′, and W′ represent the number of channels, height, and width of the output feature map.

[0062] The sizes of feature maps F1, F2, F3, and F4 are C1×H×W, 2C1×H / 2×W / 2, 4C1×H / 4×W / 4, and 8C1×H / 8×W / 8, respectively (C1 is the number of channels). After pixel shuffling, the resulting feature maps F1', F2', F3', and F4' have sizes of C1×H×W, C1 / 2×H×W, C1 / 4×H×W, and C1 / 8×H×W, respectively. Feature map F1 remains unchanged. Feature map F5, which contains both local and global contextual information, has a size of 8C1×H / 8×W / 8. After pixel shuffling, the resulting feature map F5' has a size of C1 / 8×H×W.

[0063] like Figure 2 As shown in the figure, the feature maps output after pixel reorganization PixelShuffle are concatenated, and a 3×3 convolution is used to determine the location of the segmentation target to obtain the final segmentation result.

[0064] In a specific embodiment, when training the segmentation network model, the present application regards the segmentation loss as the total loss of the network (L seg ) is used for regression, and the formula is:

[0065]

[0066] Deep learning training is considered complete until the loss function of the network structure model no longer converges. Here, G represents the true graph, S represents the segmentation result graph, |G| and |S| represent the number of elements in G and S, respectively, and |G∩S| represents the intersection of G and S.

[0067] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A medical image segmentation method based on local and global context information encoding, characterized in that: The medical image segmentation method based on local and global context information encoding includes: Obtaining a medical image to be segmented, preprocessing it, and then inputting it into a constructed segmentation network model, wherein the segmentation network model includes an encoding module, a feature extraction module, and a decoding module; In the encoding module, the preprocessed medical image to be segmented is subjected to four convolution and maximum pooling operations in sequence to obtain feature maps F1, F2, F3, and F4. The feature maps F1, F2, and F3 are adaptively average pooled to the same size as F4 and then concatenated with F4. Then, multi-scale features are fused through depthwise separable convolution to output the encoded features. In the feature extraction module, the encoded features pass through the void attention module and the spatial grid attention module in sequence to extract the feature map containing local and global context information; In the decoding module, the feature map containing local and global context information is combined with the feature maps F1, F2, F3, and F4 through a pixel reassembly operation. The feature maps output by the pixel reassembly operation are then concatenated and input into a convolutional layer to obtain the segmentation result. The hole attention module performs the following operations: Input the encoded feature into the first DAM branch of the hole attention module, which includes four cascade branches. The first cascade branch consists of a 3×3 ordinary convolution, the second cascade branch consists of a 3×3 hole convolution with a hole rate of 3 and a 1×1 ordinary convolution, the third cascade branch consists of a 3×3 ordinary convolution, a 3×3 convolution with a hole rate of 3 and a 1×1 ordinary convolution, and the fourth cascade branch consists of a 3×3 ordinary convolution, a 3×3 convolution with a hole rate of 3, a 3×3 convolution with a hole rate of 5 and a 1×1 ordinary convolution. The feature map obtained by each cascade branch is connected with the encoded feature to obtain the first feature map; after performing the SE operation on the first feature map, it is multiplied with the first feature map, and then subjected to 1×1 convolution to obtain the feature map output by the first DAM branch; Input the encoded features into the second DAM branch of the hole attention module, and the second DAM branch performs a prospective attention operation to obtain a feature map output by the second DAM branch; Add the features output by the first DAM branch to the feature map output by the second DAM branch to obtain the feature map output by the hollow attention module; The spatial grid attention module performs the following operations: Input the feature map output by the hole attention module to the first SGAM branch of the spatial grid attention module, and the first SGAM branch performs a sparse global attention operation to obtain the feature map output by the first SGAM branch; The feature map output by the hole attention module is input to the second SGAM branch of the spatial grid attention module, and the second SGAM branch performs a spatial attention operation to obtain a weight coefficient output by the second SGAM branch; The feature map output by the first SGAM branch is multiplied by the weight coefficient output by the second SGAM branch to obtain a feature map containing global context information.

2. The medical image segmentation method based on local and global context information encoding according to claim 1, characterized in that The loss function of the segmentation network model is as follows: ； Among them, G represents the real graph, S represents the segmentation result graph, |G| and |S| represent the number of elements of G and S respectively, and |G∩S| represents the intersection between G and S.

Images