A pancreatic image segmentation method based on complementary attention

Abstract

The application belongs to the technical field of medical image processing, and particularly relates to a pancreas image segmentation method based on complementary attention; the method comprises the following steps: acquiring a pancreas CT image, pre-processing the pancreas CT image, extracting features of the pre-processed pancreas CT image by using a residual dense module to obtain a first feature map; processing the first feature map by using a progressive pyramid pooling module to obtain a second feature map; processing the second feature map by using a main branch decoder and an edge branch decoder respectively to obtain a region feature map and an edge feature map; processing the region feature map and the edge feature map respectively by using a complementary attention mechanism to obtain a pancreas result map and an edge result map; and the application can solve the problems of over-segmentation and under-segmentation, thereby improving the accuracy of pancreas segmentation.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a pancreatic image segmentation method based on complementary attention. Background Technology

[0002] Recently, the rapid development of deep neural networks has greatly advanced the automatic segmentation of organs in abdominal CT scans. However, pancreatic segmentation remains a challenging task, and neural networks encounter many difficulties when used for pancreatic segmentation, such as: 1. Low contrast between adjacent tissues in abdominal organs, resulting in similar Hu values. The ambiguous Hu values ​​between different classes can lead to oversegmentation. Neural networks may mistake surrounding tissues for pancreatic tissue because valuable information is lost during training after a series of convolutional pooling processes, and the network cannot effectively capture and accumulate feature information to address oversegmentation. 2. The pancreas occupies a small proportion of abdominal organs, and its shape, size, and location vary significantly among patients due to factors such as gender and age. This makes accurate pancreatic segmentation difficult, leading to undersegmentation. This problem arises because of significant intraclass variability in pancreatic tissue, making it difficult to extract complete edge features and accurately grasp the shape and edge information of the pancreas.

[0003] Because the pancreas is a small, soft organ located in the abdomen, existing technologies have proposed many pancreas segmentation methods based on two-stage models. However, the drawback of two-stage models is that the computational steps are more complex and cumbersome, making them difficult to generalize and apply in clinical medicine. Therefore, designing a better single-stage model is more suitable for practical use, but this is also a significant challenge. The significant anatomical differences of the pancreas make feature information difficult to extract. U-Net uses skip connections to repair the spatial information lost during upsampling. Existing technologies have incorporated gated attention into U-Net to better fuse encoded and decoded information, making the model more attentive to learning regions of interest. Although this model can identify feature information well, it does not pay attention to the differences between low-level and high-level features. At the same time, semantic relationships are ubiquitous, especially in segmentation tasks, and accurate pancreas segmentation depends on understanding the contextual semantic information. Since the boundaries between the pancreas and adjacent tissues are relatively blurred, edge information is also very important.

[0004] In summary, there is an urgent need for a pancreas segmentation method that can extract rich high-level semantic features and takes into account both semantic segmentation and edge detection tasks, in order to improve the accuracy of pancreas segmentation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a pancreatic image segmentation method based on complementary attention. The method includes: acquiring a pancreatic CT image; preprocessing the pancreatic CT image; inputting the preprocessed image into a trained pancreatic image segmentation network based on complementary attention to obtain the pancreatic image segmentation result. The pancreatic image segmentation network based on complementary attention includes an encoder and two decoders. The encoder includes a residual dense module and a progressive pyramid pooling module, and the two decoders are a main branch decoder and an edge branch decoder, respectively.

[0006] The training process of the pancreas image segmentation network based on complementary attention includes:

[0007] S1: Acquire training data of pancreatic CT images and preprocess them, then divide the preprocessed pancreatic CT images into training set and test set;

[0008] S2: The residual dense module is used to extract features from the pancreatic CT images in the training set to obtain the first feature map;

[0009] S3: The first feature map is processed using the progressive pyramid pooling module to obtain the second feature map;

[0010] S4: The main branch decoder and the edge branch decoder are used to extract features from the second feature map respectively, and the complementary attention mechanism is used to interact the feature information extracted by the two branches to obtain the pancreatic result and the edge result map.

[0011] S5: Calculate the segmentation loss based on the pancreas result image and the edge loss based on the edge result image; backpropagate based on the segmentation loss and edge loss and update the network parameters. When the sum of the segmentation loss and the edge loss is minimized, save the network parameters to obtain the trained pancreas image segmentation network based on complementary attention.

[0012] S6: Input the pancreatic CT images from the test set into the trained pancreatic image segmentation network based on complementary attention to obtain the pancreatic result image, i.e., the pancreatic image segmentation result.

[0013] Preferably, the preprocessing of pancreatic CT images includes: converting the original three-dimensional pancreatic CT image into NumPy format and slicing it to obtain a two-dimensional pancreatic CT image; calculating the Hu value of the two-dimensional pancreatic CT image; further processing the pancreatic CT image based on the Hu value to ensure that the Hu value is between -100 and 200; if the Hu value of the pancreatic CT image is less than -100, then processing the pancreatic CT image to make its Hu value equal to -100; if the Hu value of the pancreatic CT image is greater than 200, then processing the pancreatic CT image to make its Hu value equal to 200; and normalizing all processed pancreatic CT images to obtain a preprocessed pancreatic CT image.

[0014] Preferably, the process of extracting features from pancreatic CT images using a residual dense module includes: processing the pancreatic CT image sequentially using five consecutive residual convolutional blocks, performing downsampling after each convolution to obtain the first feature map; wherein the channel dimensions of the five residual convolutional blocks are 32, 64, 128, 256 and 512, respectively.

[0015] Preferably, the process of processing the first feature map using the progressive pyramid pooling module includes: performing pooling operations of different sizes on the first feature map to obtain multiple feature maps containing information of different sub-regions; performing upsampling processing on the multiple feature maps containing information of different sub-regions using bilinear interpolation to obtain multiple upsampled feature maps; concatenating the multiple upsampled feature maps and processing them using 1×1 convolution to obtain an intermediate feature map; and concatenating and fusing the first feature map and the intermediate feature map to obtain a second feature map.

[0016] Preferably, the process of extracting features from the second feature map using the main branch decoder includes: processing the second feature map using four main branch residual convolutional blocks, the output of each main branch residual convolutional block being the region feature map; wherein, the main branch residual convolutional block includes an upsampling layer and a convolutional layer.

[0017] Furthermore, the channel dimensions of the convolutional layers in the four main branch residual convolutional blocks are 512, 128, 64, and 32, respectively.

[0018] Preferably, the process of extracting features from the second feature map using an edge branch decoder includes: performing convolution processing on the second feature map using four edge branch residual convolution blocks, and the output of each edge branch residual convolution block is the edge feature map; wherein, the edge branch residual convolution block includes an upsampling layer and a convolution layer.

[0019] Furthermore, the channel dimensions of the convolutional layers in the four edge branch residual convolutional blocks are 512, 128, 64, and 32, respectively.

[0020] The beneficial effects of this invention are as follows: This invention designs an encoder and two decoders. In the encoding stage, this invention uses residual convolutional blocks and progressive pyramid pooling to enhance the ability to extract pancreatic features, and to some extent prevents the loss of feature information during downsampling. In the decoding stage, the main branch decoder is used to learn the overall segmentation of the pancreatic region, while the edge branch decoder is used to improve the ability to learn target edges and to converge useful feature information to the main branch for learning. Based on the above strategies, this invention can solve the problems of oversegmentation and undersegmentation, thereby improving the accuracy of image recognition results (pancreatic segmentation). Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the pancreas image segmentation network based on complementary attention in this invention;

[0022] Figure 2 This is a schematic diagram of the residual dense convolution block in this invention;

[0023] Figure 3 This is a schematic diagram of the pyramid pooling module in this invention;

[0024] Figure 4 This is a schematic diagram of the complementary attention module in this invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention proposes a pancreatic image segmentation method based on complementary attention, such as... Figure 1 As shown, the method includes the following: acquiring a pancreatic CT image to be identified, preprocessing the pancreatic CT image, inputting the preprocessed image into a trained pancreatic image segmentation network based on complementary attention, and obtaining a pancreatic image segmentation result; the pancreatic image segmentation network based on complementary attention includes an encoder and two decoders, wherein the encoder includes a residual dense module and a progressive pyramid pooling module, and the two decoders are a main branch decoder and an edge branch decoder, respectively;

[0027] The preprocessing of pancreatic CT images includes: converting the original three-dimensional pancreatic CT image into NumPy format and slicing it to obtain a two-dimensional pancreatic CT image; calculating the Hu value of the two-dimensional pancreatic CT image, and further processing the pancreatic CT image based on the Hu value to ensure that the Hu value is between -100 and 200. If the Hu value of the pancreatic CT image is less than -100, the pancreatic CT image is processed to make the Hu value equal to -100; if the Hu value of the pancreatic CT image is greater than 200, the pancreatic CT image is processed to make the Hu value equal to 200; and normalizing all processed pancreatic CT images to obtain a preprocessed pancreatic CT image.

[0028] The training process of the pancreas image segmentation network based on complementary attention includes:

[0029] S1: Acquire training data of pancreatic CT images and preprocess them, then divide the preprocessed pancreatic CT images into training set and test set.

[0030] S2: The residual dense module is used to extract features from the pancreatic CT images in the training set to obtain the first feature map.

[0031] Residual dense convolutional blocks can acquire the most important foreground information and accumulate features during feature extraction. This helps extract very low-level features, which are essential for segmentation tasks in different medical imaging modalities (such as pancreas segmentation). To avoid gradient instability and other problems caused by overly deep networks, the residual structure used in this invention can alleviate gradient vanishing to some extent, making information propagation smoother in both directions. Figure 2 As shown, the process of feature extraction from pancreatic CT images using a residual dense module includes: processing the pancreatic CT image sequentially using five consecutive residual convolutional blocks, and performing downsampling after each convolution to obtain the first feature map; wherein the channel dimensions of the five residual convolutional blocks are 32, 64, 128, 256 and 512, respectively.

[0032] The pancreatic CT image is first passed through a regular convolutional block to increase dimensionality and store richer semantic information. Then, it is passed through dense convolutional blocks to effectively accumulate feature information at different levels. Except for the first regular convolutional block which uses a 1×1 size, the remaining convolutional kernels are all 3×3 in size. Pooling is not used during feature extraction, which ensures that the size of the feature map does not change and avoids information loss, thereby enhancing the feature extraction capability.

[0033] S3: The first feature map is processed using the progressive pyramid pooling module to obtain the second feature map.

[0034] like Figure 3 As shown, the process of processing the first feature map using the progressive pyramid pooling module includes: performing pooling operations of different sizes on the first feature map to obtain multiple feature maps containing information of different sub-regions; preferably, the first feature map is subjected to three different pooling operations of 1*1, 3*3, and 5*5.

[0035] Bilinear interpolation is used to upsample multiple feature maps containing information from different sub-regions, resulting in multiple upsampled feature maps.

[0036] Multiple feature maps after upsampling are concatenated and processed using a 1×1 convolution to obtain an intermediate feature map; the first feature map and the intermediate feature map are then concatenated and fused to obtain a second feature map.

[0037] The entire process described above can reduce the semantic information gap between different scales, and ultimately fuse these feature maps into global prior information. To fuse appropriate global features, this invention introduces a progressive pyramid pooling module to gradually fuse features from different regions, avoiding the loss of contextual information representing the relationships between different sub-regions. By utilizing category clues from the global context through the above strategies, richer semantic information is integrated and the semantic gap between different scales is reduced, making the final segmentation result more reliable.

[0038] S4: The main branch decoder and the edge branch decoder are used to extract features from the second feature map respectively, and the complementary attention mechanism is used to interact the feature information extracted by the two branches to obtain the pancreatic result map and the edge result map.

[0039] The process of extracting features from the second feature map using the main branch decoder includes: processing the second feature map using four main branch residual convolutional blocks. The edge branch residual convolutional blocks include upsampling layers and convolutional layers. An upsampling operation is performed before each convolution. The output of each main branch residual convolutional block is the region feature map. The channel dimensions of the convolutional layers in the four main branch residual convolutional blocks are 512, 128, 64, and 32, respectively.

[0040] The process of processing the second feature map using the edge branch decoder includes: processing the second feature map using four edge branch residual convolutional blocks. Each edge branch residual convolutional block includes an upsampling layer and a convolutional layer. An upsampling operation is performed before each convolution. The output of each edge branch residual convolutional block is the edge feature map. The channel dimensions of the convolutional layers in the four edge branch residual convolutional blocks are 512, 128, 64, and 32, respectively.

[0041] Attention mechanisms include spatial attention and channel attention. Spatial attention emphasizes the main features of an image and allocates more weight to the feature map, while channel attention treats each channel as a discriminator, learning different types of features and allowing the network to autonomously choose which features to focus on in the current task. This invention utilizes two different complementary attention modules to establish the relationship between regions and boundary cues, further mining boundary cues. Due to this frequent complementary learning mechanism between regions and boundaries, during region learning, effective boundary cues can be drawn from the boundary learning branch to compensate for the lack of feature information, thereby enabling the model to correct some erroneous predictions.

[0042] Both the main branch decoder and the edge branch decoder contain three complementary attention modules. The output of each complementary attention module, namely the intermediate pancreatic result map (intermediate edge result map), is the input of the last three main branch residual convolutional blocks (edge ​​branch residual convolutional blocks). The fourth region feature map or edge result map is processed by a 1×1 convolutional layer to obtain the final output, namely the pancreatic result map or edge result map.

[0043] like Figure 4 As shown, the process of using complementary attention mechanism to interact with the feature information extracted from the two branches, i.e., processing the region feature map and edge feature map using complementary attention mechanism, includes:

[0044] Similar processing is performed in both branches: three complementary attention modules are used to process the feature maps output by the residual convolutional blocks (edge ​​branch convolutional blocks) of the first three main branches respectively. The first two complementary attention modules are complementary channel attention modules, and the last complementary attention module is a complementary spatial attention module.

[0045] In the complementary channel attention module, the processing of the region feature map and the edge feature map includes: performing average pooling on the region feature map in the spatial dimension to obtain a smooth feature map (size 1×1×c, where c represents the channel); processing the smooth feature map with a 1×1 convolution and then processing it with an activation function to obtain the first attention probability map; multiplying the first attention probability map and the region feature map, and interacting with the edge feature map based on the result of the multiplication and the first attention probability map to obtain the intermediate pancreatic result map;

[0046] In the complementary spatial attention module, the processing of region feature maps and edge feature maps includes: performing average pooling on the edge feature map in the channel dimension to obtain a channel feature map (size w×h×c, where w represents the horizontal dimension and h represents the vertical dimension); processing the channel feature map with a 1×1 convolution and then processing it with an activation function to obtain a second attention probability map; multiplying the second attention probability map and the edge feature map, and interacting with the region feature map based on the result of the multiplication and the second attention probability map to obtain the intermediate edge result map.

[0047] Preferably, in some embodiments, the network parameters of the present invention are shown in Table 1:

[0048] Table 1 Network Parameter Table

[0049]

[0050] S6: Calculate the segmentation loss based on the pancreas result image and the edge loss based on the edge result image; backpropagate based on the segmentation loss and edge loss and update the network parameters; when the sum of the segmentation loss and edge loss is minimized, save the network parameters to obtain the trained pancreas image segmentation network based on complementary attention.

[0051] S7: Input the pancreatic CT images from the test set into the trained pancreatic image segmentation network based on complementary attention to obtain the pancreatic result image, i.e., the pancreatic image segmentation result.

[0052] The acquired pancreatic CT images are preprocessed and then input into the pre-trained pancreatic image segmentation network based on complementary attention of this invention, which can yield pancreatic image segmentation results with high accuracy.

[0053] To evaluate this invention, the NIH pancreas dataset was used as input. The segmentation performance of this invention was compared with that of the comparative methods. The results are shown in Table 2.

[0054] Table 2 Comparison of segmentation results between the present invention and the comparative method

[0055]

[0056] As shown in Table 2, the proposed method outperforms classic networks (U-Net, Att-Unet, and SegNet) in terms of Dice segmentation results. It also outperforms other networks in other metrics (Jaccard similarity, Precision, Recall, Hausdorff distance, and ASSD mean surface distance), demonstrating that the proposed method is more effective for pancreas segmentation.

[0057] This invention proposes a U-shaped network pancreas segmentation network combining multi-scale dual-path and complementary attention mechanisms. It primarily addresses under-segmentation and discontinuous segmentation results caused by intra-class differences, as well as over-segmentation and missegmentation caused by intra-class ambiguity. Specifically, this invention extracts and accumulates feature information through residual dense blocks and fuses multi-scale information using progressive pyramid pooling. It also introduces a boundary branch and a complementary attention module to provide rich cues for region learning. Compared to U-shaped networks (U-Net), attention-based U-shaped networks (Att-Unet), and the deep fully convolutional neural network structure SegNet for semantic segmentation, this invention achieves better segmentation of the pancreas with higher accuracy. Furthermore, the segmentation results of this invention outperform the most advanced pancreas segmentation algorithms currently available.

[0058] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pancreas image segmentation method based on complementary attention, characterized in that, include: Pancreatic CT images are acquired, preprocessed, and then input into a trained pancreatic image segmentation network based on complementary attention to obtain pancreatic image segmentation results. The pancreatic image segmentation network based on complementary attention includes an encoder and two decoders. The encoder includes a residual dense module and a progressive pyramid pooling module, and the two decoders are a main branch decoder and an edge branch decoder, respectively. The training process of the pancreas image segmentation network based on complementary attention includes: S1: Acquire training data of pancreatic CT images and preprocess them, then divide the preprocessed pancreatic CT images into training set and test set; S2: The residual dense module is used to extract features from the pancreatic CT images in the training set to obtain the first feature map; S3: The first feature map is processed using the progressive pyramid pooling module to obtain the second feature map; S4: The main branch decoder and the edge branch decoder are used to extract features from the second feature map respectively, and the complementary attention mechanism is used to interact the feature information extracted by the two branches to obtain the pancreatic result map and the edge result map. The process of extracting features from the second feature map using the main branch decoder includes: processing the second feature map using four main branch residual convolutional blocks, with the output of each main branch residual convolutional block being the region feature map; wherein, the main branch residual convolutional block includes an upsampling layer and a convolutional layer; The process of extracting features from the second feature map using an edge branch decoder includes: performing convolution processing on the second feature map using four edge branch residual convolution blocks, and the output of each edge branch residual convolution block is the edge feature map; wherein, the edge branch residual convolution block includes an upsampling layer and a convolutional layer; The process of using complementary attention mechanism to interact with the feature information extracted from the two branches includes: Three complementary attention modules are used to process the feature maps output by the corresponding first three main branch residual convolutional blocks and edge branch convolutional blocks. The first two complementary attention modules are complementary channel attention modules, and the last complementary attention module is a complementary spatial attention module. The outputs of the complementary channel attention module and the complementary spatial attention module are fused with the intermediate processed features of the main branch decoder. The main branch decoder outputs the pancreatic result map, and the edge branch decoder outputs the edge result map. In the complementary channel attention module, the region feature map and the edge feature map are fused in the spatial dimension, and the middle region feature map and the middle edge feature map of the complementary channel attention module are output; in the complementary spatial attention module, the region feature map and the edge feature map are fused in the channel dimension, and the middle region feature map and the middle edge feature map of the complementary spatial attention module are output. S5: Calculate the segmentation loss based on the pancreas result image and the edge loss based on the edge result image; backpropagate based on the segmentation loss and edge loss and update the network parameters. When the sum of the segmentation loss and the edge loss is minimized, save the network parameters to obtain the trained pancreas image segmentation network based on complementary attention. S6: Input the pancreatic CT images from the test set into the trained pancreatic image segmentation network based on complementary attention to obtain the pancreatic result image, i.e., the pancreatic image segmentation result.

2. The pancreas image segmentation method based on complementary attention according to claim 1, characterized in that, The preprocessing of pancreatic CT images includes: converting the original three-dimensional pancreatic CT image into NumPy format and slicing it to obtain a two-dimensional pancreatic CT image; calculating the Hu value of the two-dimensional pancreatic CT image, and further processing the pancreatic CT image based on the Hu value to ensure that the Hu value is between -100 and 200. If the Hu value of the pancreatic CT image is less than -100, the pancreatic CT image is processed to make the Hu value equal to -100; if the Hu value of the pancreatic CT image is greater than 200, the pancreatic CT image is processed to make the Hu value equal to 200; and normalizing all processed pancreatic CT images to obtain a preprocessed pancreatic CT image.

3. The pancreas image segmentation method based on complementary attention according to claim 1, characterized in that, The process of feature extraction from pancreatic CT images using a residual dense module includes: processing the pancreatic CT image sequentially using five consecutive residual convolutional blocks, and performing downsampling after each convolution to obtain the first feature map; wherein the channel dimensions of the five residual convolutional blocks are 32, 64, 128, 256 and 512, respectively.

4. The pancreas image segmentation method based on complementary attention according to claim 1, characterized in that, The process of processing the first feature map using the progressive pyramid pooling module includes: performing pooling operations of different sizes on the first feature map to obtain multiple feature maps containing information of different sub-regions; upsampling the multiple feature maps containing information of different sub-regions using bilinear interpolation to obtain multiple upsampled feature maps; and concatenating the multiple upsampled feature maps and using... The convolution process is used to obtain an intermediate feature map; the first feature map and the intermediate feature map are then concatenated and fused to obtain the second feature map.

5. The pancreas image segmentation method based on complementary attention according to claim 1, characterized in that, The channel dimensions of the convolutional layers in the four main branch residual convolutional blocks are 512, 128, 64 and 32, respectively.

6. The pancreas image segmentation method based on complementary attention according to claim 1, characterized in that, The channel dimensions of the convolutional layers in the four edge branch residual convolutional blocks are 512, 128, 64 and 32, respectively.

Images