Medical image segmentation method based on multiple attention fusion

Abstract

The application relates to a medical image segmentation method based on multiple attention fusion. By improving and optimizing attention modules, the model pays more attention to the target area, and the segmentation accuracy of the model for the region of interest and the boundary is improved. The existing target detection method tends to segment and classify the instances of the target area, but the edge precision segmentation is not high. Experiments show that the application can better maintain the segmentation result of the backbone network, and the segmentation result of the lesion area is obviously better than that of the existing method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, in particular to a medical image segmentation method based on multiple attention fusion. BACKGROUND

[0002] The brain is the core of the human nervous system, and brain lesions can cause permanent brain function damage, resulting in disability or death. Precise detection and segmentation of brain lesions can help quantify various pathological indicators of brain lesions (such as total lesion volume, lesion location, and lesion block number). These quantitative indicators are closely related to brain aging and pathological changes, and can provide useful clues for patient prognosis, and can be further used to analyze the effect of drug intervention and guide the design of surgical intervention programs. Hemorrhagic transformation after cerebral infarction refers to the hemorrhage caused by the reperfusion of blood flow in the ischemic area after acute cerebral infarction. Hemorrhagic transformation after acute cerebral infarction is part of the natural course of cerebral infarction and is also the main adverse reaction of thrombolysis therapy. It is not only related to poor prognosis of cerebral infarction, but also an important reason for the lack of use of various blood flow improvement therapies. Rapid analysis and judgment of CT images of hemorrhagic transformation after cerebral infarction are related to whether doctors can make rapid and accurate diagnosis and treatment of patient conditions, so it is necessary to use scientific methods to quickly segment the hemorrhagic transformation area.

[0003] The use of deep learning automatic segmentation technology in the prior art for the segmentation and detection of brain lesions such as hemorrhagic transformation after cerebral infarction has important significance for further analysis of brain tissue and accurate diagnosis and positioning of brain diseases. However, for some tasks with high edge precision requirements, due to the dependence on the accuracy of the frame, the segmentation effect of some non-square objects is poor, especially for lesions which are not fixed in boundary and mostly irregularly shaped targets, the segmentation performance is often poor. SUMMARY

[0004] In order to overcome at least one deficiency in the prior art, the embodiments of the present application provide a medical image segmentation method based on multiple attention fusion.

[0005] In a first aspect, a medical image segmentation model based on multiple attention fusion is provided, comprising: a deep residual network, a dilated convolution spatial attention module, a pyramid expansion module, and a double-branch fusion module.

[0006] The deep residual network is used for feature extraction of the image to be segmented to obtain a plurality of first feature maps of different scales;

[0007] The dilated convolution spatial attention module is used for dilated convolution of the plurality of first feature maps of different scales to obtain a plurality of second feature maps of different scales;

[0008] The pyramid expansion module is configured to perform convolution and fusion operations on the plurality of second feature maps of different scales in descending order of scale to obtain a plurality of third feature maps of different scales; and perform convolution and fusion operations on the plurality of third feature maps of different scales in ascending order of scale to obtain a plurality of fourth feature maps of different scales.

[0009] The double-branch fusion module is configured to perform class mask prediction and foreground-background mask prediction on the plurality of fourth feature maps of different scales, and fuse the class mask prediction result and the foreground-background mask prediction result to obtain the image segmentation result.

[0010] In one embodiment, the spatial attention module includes an input layer, a max-pooling and average-pooling layer, and a dilated convolution layer connected in sequence.

[0011] In one embodiment, the double-branch fusion module includes a class mask prediction branch and a foreground-background mask prediction branch.

[0012] The class mask prediction branch includes four convolution layers, a deconvolution layer, and a convolution layer connected in sequence.

[0013] The foreground-background mask prediction branch includes two convolution layers and a fully connected layer connected in sequence.

[0014] The last convolution layer of the class mask prediction branch outputs a class mask prediction result, and the fully connected layer of the foreground-background mask prediction branch outputs a foreground-background mask prediction result.

[0015] In a second aspect, a medical image segmentation method based on multiple attention fusion is provided, including:

[0016] Inputting the image to be segmented into a medical image segmentation model based on multiple attention fusion to obtain an image segmentation result.

[0017] The medical image segmentation model based on multiple attention fusion is the medical image segmentation model based on multiple attention fusion described above.

[0018] In one embodiment, the method further includes training the medical image segmentation model based on multiple attention fusion to obtain a trained medical image segmentation model based on multiple attention fusion.

[0019] In one embodiment, training the medical image segmentation model based on multiple attention fusion includes:

[0020] Determining at least one target region for each fourth feature map obtained by the pyramid expansion module, and inputting the at least one target region into the double-branch fusion module.

[0021] In one embodiment, determining at least one target region for each fourth feature map includes:

[0022] Determine multiple predicted target region boxes in each fourth feature map;

[0023] Based on the actual target region bounding box and multiple predicted target region bounding boxes, calculate the intersection-union ratio (IUU) for each predicted target region bounding box;

[0024] Sort all predicted target region bounding boxes by intersection-union ratio (IU) from largest to smallest, and select the predicted target region bounding box with the highest IU in the sorted order as at least one target region.

[0025] In one embodiment, the intersection-union ratio (IoU) for each predicted target region is calculated based on the true target region bounding box and multiple predicted target region bounding boxes. new The following formula is used:

[0026]

[0027] Where S1 is the predicted target region bounding box, S2 is the true target region bounding box, and λ is the penalty factor.

[0028] In one embodiment, training a medical image segmentation model based on multi-attention fusion includes:

[0029] The image slices containing the lesion area were truncated using the HU index and augmented to obtain training data.

[0030] In one embodiment, during model training, the deep residual network in the medical image segmentation model based on multi-attention fusion is a pre-trained deep residual network.

[0031] Compared with the prior art, this application has the following beneficial effects:

[0032] This application improves and optimizes modules such as attention to make the model focus more on the target region, thereby improving the segmentation accuracy of the model for the region of interest and its boundaries. It solves the problem that existing object detection methods often tend to segment and classify instances of the target region, but often have low edge segmentation accuracy. Experiments show that this application maintains the segmentation results of the backbone network well, and is significantly better than existing methods in the segmentation results of lesion areas. Attached Figure Description

[0033] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings:

[0034] Figure 1 A schematic diagram of a medical image segmentation model based on multiple attention fusion according to an embodiment of this application is shown;

[0035] Figure 2 A schematic diagram of a dilated convolutional spatial attention module according to an embodiment of this application is shown;

[0036] Figure 3 A schematic diagram of a pyramid expansion module according to an embodiment of this application is shown;

[0037] Figure 4 A schematic diagram of a dual-branch fusion module according to an embodiment of this application is shown;

[0038] Figure 5 A comparison diagram of the image segmentation experimental results of this application and existing technologies is shown. Detailed Implementation

[0039] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0040] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution according to this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0041] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0042] This application provides a medical image segmentation method based on multi-attention fusion, comprising: inputting the image to be segmented into a medical image segmentation model based on multi-attention fusion to obtain the image segmentation result.

[0043] The following details the specific structure of the medical image segmentation model based on multi-attention fusion. Figure 1 A schematic diagram of a medical image segmentation model based on multiple attention fusion according to an embodiment of this application is shown. The model includes: a deep residual network, a dilated convolutional spatial attention (DCSA) module, a pyramid expansion module, and a dual branch fusion module (DBF). The specific implementation functions of each module are described in detail below.

[0044] Deep residual networks are used to extract features from images to be segmented, resulting in multiple first feature maps at different scales;

[0045] The dilated convolution spatial attention module is used to perform dilated convolution on multiple first feature maps of different scales to obtain multiple second feature maps of different scales.

[0046] The pyramid expansion module is used to perform convolution and fusion operations on multiple second feature maps of different scales in descending order of scale to obtain multiple third feature maps of different scales; and to perform convolution and fusion operations on multiple third feature maps of different scales in ascending order of scale to obtain multiple fourth feature maps of different scales.

[0047] The dual-branch fusion module is used to perform class mask prediction and foreground / background mask prediction on multiple fourth feature maps at different scales, and to fuse the class mask prediction results and foreground / background mask prediction results to obtain the image segmentation result.

[0048] In this embodiment, the deep residual network ResNet has 50 layers. The image to be segmented is input into the deep residual network, and feature extraction is performed on the image to be segmented to obtain multiple first feature maps [C2, C3, C4, C5] of different scales (128×128×256, 64×64×256, 32×32×256, 16×16×256). Experiments show that as the number of ResNet network layers increases, its performance tends to saturate, and 50 network layers are sufficient.

[0049] In one embodiment, the dilated convolutional spatial attention module includes an input layer, a max pooling layer, an average pooling layer, and a dilated convolutional layer connected in sequence.

[0050] To enable the network to automatically notice pixel-rich areas in an image during learning with only a minimal increase in computational cost, a simple yet effective dilated convolutional spatial attention module, similar to that used in feedforward convolutional neural networks, was creatively incorporated into the model's feature extraction part. Figure 2 A schematic diagram of a dilated convolutional spatial attention module provided according to an embodiment of this application is shown. DCSA is an improved SAM spatial attention module, specifically replacing the ordinary convolution part in the SAM spatial attention module with dilated convolution.

[0051] In this embodiment, dilated convolution offers two main advantages. First, it helps expand the receptive field while reducing computational cost. A larger receptive field allows for the detection and segmentation of large targets, while higher resolution enables precise target localization. Second, it captures multi-scale contextual information. Dilated convolution has a parameter that allows setting the dilation rate; therefore, different dilation rates result in different receptive fields, thus acquiring multi-scale information. Multi-scale information is crucial in visual tasks. Therefore, this embodiment combines traditional SAM spatial attention with dilated convolution. Experiments show that a dilation rate of 2 significantly improves the model's ability to detect and segment brain lesion regions. After passing through the dilated convolution spatial attention module, the first feature map yields multiple second feature maps [D2, D3, D4, D5] at different scales.

[0052] Specifically, Figure 3 A schematic diagram of a pyramid expansion module according to an embodiment of this application is shown. The pyramid expansion module performs convolution and fusion operations on multiple second feature maps [D2, D3, D4, D5] of different scales in descending order of scale to obtain multiple third feature maps [P2, P3, P4, P5] of different scales. Specifically, D5 is first subjected to a 1×1 convolution operation to obtain the third feature map P5; then D4 is subjected to a 1×1 convolution operation, and the result of the convolution operation is fused with the third feature map P5 to obtain the third feature map P4; then D3 is subjected to a 1×1 convolution operation, and the result of the convolution operation is fused with the third feature map P4 to obtain the third feature map P3; finally, D2 is subjected to a 1×1 convolution operation, and the result of the convolution operation is fused with the third feature map P3 to obtain the third feature map P2.

[0053] Multiple third feature maps [P2, P3, P4, P5] of different scales are convolved and fused in ascending order of scale to obtain multiple fourth feature maps [N2, N3, N4, N5] of different scales; that is, first, a 3×3 convolution operation is performed on P2 to obtain the fourth feature map N2; then a 3×3 convolution operation is performed on P3, and the result of the convolution operation is fused with the fourth feature map N2 to obtain the fourth feature map N3; then a 3×3 convolution operation is performed on P4, and the result of the convolution operation is fused with the fourth feature map N3 to obtain the fourth feature map N4; finally, a 3×3 convolution operation is performed on P5, and the result of the convolution operation is fused with the fourth feature map N4 to obtain the fourth feature map N5.

[0054] In one embodiment, Figure 4A schematic diagram of a dual-branch fusion module according to an embodiment of this application is shown. The dual-branch fusion module includes a category mask prediction branch and a foreground / background mask prediction branch. The category mask prediction branch includes four convolutional layers, a deconvolutional layer, and a convolutional layer connected in sequence. The foreground / background mask prediction branch includes two convolutional layers and a fully connected layer connected in sequence. The last convolutional layer of the category mask prediction branch outputs the category mask prediction result, and the fully connected layer of the foreground / background mask prediction branch outputs the foreground / background mask prediction result.

[0055] In this embodiment, the category mask prediction branch is the main path, which is a small FCN network consisting of four consecutive convolutional layers and one deconvolutional layer. It is an end-to-end network, and its main modules include convolution and deconvolution. First, multiple convolutions are performed on the image to extract deep information; then, deconvolution is performed, which is an interpolation operation to continuously increase the feature map. Finally, each pixel value is classified to achieve accurate segmentation of the input image. Each convolutional layer in the main path consists of a 3×3 convolutional kernel and a deconvolutional layer that upsamples the features by a factor of 2, independently predicting a binary pixel mask for each class to obtain the category mask prediction result, thus achieving segmentation and classification.

[0056] A short path, the foreground / background mask prediction branch, is added after the third convolutional layer (conv3) of the main path. This branch consists of two 3×3 convolutional layers and one fully connected layer. The fully connected layer predicts class-independent foreground / background masks; it is not only efficient but also allows the parameters in the fully connected layer to be trained with more samples, thus achieving better generalization. The main path connects a deconvolution and a convolution operation after the third convolution to adjust the feature map dimensions. To obtain the final mask prediction, the features of each class from the main path are fused with the foreground / background predictions from the fully connected short path. Using only one fully connected layer in the short path prediction, instead of multiple fully connected layers, prevents the hidden spatial feature maps from being folded into a short feature vector, thus avoiding the loss of spatial information. Ablation experiments show that adding the short path branch starting from conv3 and then fusing it at the end yields the best results.

[0057] Furthermore, the medical image segmentation method based on multi-attention fusion in this application embodiment further includes: training the medical image segmentation model based on multi-attention fusion to obtain the trained medical image segmentation model based on multi-attention fusion.

[0058] Specifically, training a medical image segmentation model based on multi-attention fusion includes:

[0059] For each fourth feature map obtained by the pyramid expansion module, at least one target region is determined, and the at least one target region is input into the dual-branch fusion module.

[0060] In this embodiment, by determining the target region, candidate regions (Regions of Interest, RoIs) with low classification scores can be filtered out to alleviate the class imbalance problem and reduce the subsequent calculation of unnecessary information.

[0061] Specifically, for each fourth feature map, at least one target region is determined, including:

[0062] Multiple predicted target region boxes are determined in each fourth feature map; here, the Pyramid RoI Align module in the prior art can be used to determine multiple predicted target region boxes.

[0063] Then, based on the true target region bounding box and multiple predicted target region bounding boxes, the intersection-union ratio (IUR) for each predicted target region bounding box is calculated; specifically, the IUR can be determined using the following formula:

[0064]

[0065] Where S1 is the predicted target region bounding box, S2 is the true target region bounding box, and λ is the penalty factor.

[0066] Finally, the intersection-union ratios (IU) of all predicted target region boxes are sorted from largest to smallest, and the predicted target region box with the highest IU is selected as at least one target region.

[0067] Furthermore, the medical image segmentation model based on multi-attention fusion is trained, including HU index truncation and data augmentation of image slices containing lesion areas to obtain training data. Here, data augmentation methods can include cropping / filling, horizontal flipping, vertical flipping, affine transformation, and other methods that do not alter the original pixel values ​​of the CT image, thus preserving the original image features to the maximum extent. Since medical image data is inherently scarce, and manually labeled data by doctors is even scarcer, data augmentation increases the number and diversity of training samples, creating as much value as possible from limited data. HU index truncation is a standard operation in medical image processing and can help observe post-stroke hemorrhage transformation data, effectively improving the lesion detection rate.

[0068] Furthermore, during model training, the deep residual network in the medical image segmentation model based on multi-attention fusion is a pre-trained deep residual network. Using a pre-trained deep residual network accelerates model convergence and facilitates the rapid acquisition of a better-performing model. Experiments demonstrate that this method can effectively improve lesion segmentation accuracy.

[0069] This application innovatively introduces a dilated convolutional spatial attention module and a two-branch fusion module on the basis of the backbone network. The evaluation metric used is AP (Average Precision), a common evaluation metric for object detection, which is the average AP for a given class. For example, a fixed set of confidence thresholds is set (0.7 is chosen in the experiment), and then TP (True Positive, representing correctly detected positive samples), FP (False Positive, representing incorrectly detected positive samples), and FN (False Negative, representing incorrectly detected negative samples) are calculated. Then, the predicted values ​​precision = TP / (TP+FP) and recall = TP / (TP+FN) are calculated for each confidence threshold, resulting in a set of predicted values ​​precision and return values ​​recall. After calculating precision and recall, the thresholds can be adjusted to obtain a series of (precision, recall) points, and a PR curve can be plotted to further calculate AP. A higher AP value indicates better object detection results. This application has been compared with several state-of-the-art models, and the results are shown below:

[0070] Table 1 Comparison Results

[0071]

[0072] In Table 1, AP50 represents the measured AP value when the IoU (Intersection over Union) threshold is 0.5, and AP75 represents the measured value when the IoU threshold is 0.75. S Represents a pixel area of ​​less than 32 2 The target bounding box is the AP measurement value, AP M Represents a pixel area of ​​32 2 -96 2 Measurements of the target bounding box between the two points, AP L Represents a pixel area greater than 96 2 The AP measurement value of the target box.

[0073] The first row shows the segmentation results of the backbone network model using Mask R-CNN. The second, third, and fourth rows show the segmentation results of existing methods such as Mask transfiner, DCT-Mask, and Refine mask, respectively. The fifth row shows the segmentation results of the method proposed in this application. As can be seen from Table 1, this application achieves better image segmentation results.

[0074] Figure 5The image segmentation experimental results of this application and prior art are shown in a comparison chart. The first column is the original CT slice image, the second column is the label data manually annotated by the doctor, the third column is the image segmentation result of the backbone network of the prior art, and the fourth column is the image segmentation result of this application. Figure 5 As can be seen, compared with the backbone network, this application enhances the propagation of semantic information through several simple and effective components, achieving better results in segmentation and detection accuracy. The results of this application are closer to the real value label data manually annotated by doctors, proving the feasibility of this application. Furthermore, this application can also capture the low-level feature information in lesion slices that cannot be observed by the naked eye of doctors, and the segmentation effect of these regions is better than the results of manual annotation by doctors. In fact, this application can effectively assist doctors in analyzing lesion regions, which is conducive to doctors making rapid and accurate diagnoses and treatments.

[0075] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A medical image segmentation model based on multi-attention fusion, characterized in that, include: Deep residual network, dilated convolutional spatial attention module, pyramid augmentation module, and dual-branch fusion module; The deep residual network is used to extract features from the image to be segmented, resulting in multiple first feature maps at different scales; The dilated convolutional spatial attention module is used to perform dilated convolution on multiple first feature maps of different scales to obtain multiple second feature maps of different scales. The pyramid expansion module is used to perform convolution and fusion operations on multiple second feature maps of different scales in descending order of scale to obtain multiple third feature maps of different scales. Multiple third feature maps of different scales are convolved and fused in order of increasing scale to obtain multiple fourth feature maps of different scales; The dual-branch fusion module is used to perform class mask prediction and foreground / background mask prediction on multiple fourth feature maps of different scales, and to fuse the class mask prediction results and foreground / background mask prediction results to obtain image segmentation results. The dilated convolutional spatial attention module is an improved SAM spatial attention module, which replaces the ordinary convolution part in the SAM spatial attention module with dilated convolution. The pyramid expansion module processes multiple second feature maps at different scales. By performing convolution and fusion operations in descending order of scale, multiple third feature maps of different scales are obtained. That is, first to Perform 1 1. Convolution operation to obtain the third feature map ; Again Perform 1 1. Convolution operation, the result of the convolution operation and the third feature map The third feature map is obtained by fusion. ; then Perform 1 1. Convolution operation, the result of the convolution operation and the third feature map The third feature map is obtained by fusion. ; then Perform 1 1. Convolution operation, the result of the convolution operation and the third feature map The third feature map is obtained by fusion. ; Multiple third feature maps at different scales Convolution and fusion operations are performed according to the scale from small to large to obtain multiple fourth feature maps of different scales. That is, first to Perform 3 The third convolution operation yields the fourth feature map. ; then Perform 3 3. Convolution operation, the result of the convolution operation and the fourth feature map The fourth feature map is obtained by fusion. ; then Perform 3 3. Convolution operation, the result of the convolution operation and the fourth feature map The fourth feature map is obtained by fusion. ; then Perform 3 3. Convolution operation, the result of the convolution operation and the fourth feature map The fourth feature map is obtained by fusion. ; The dual-branch fusion module includes a category mask prediction branch and a foreground / background mask prediction branch; The category mask prediction branch consists of four convolutional layers, a deconvolutional layer, and a convolutional layer connected in sequence; the foreground / background mask prediction branch consists of two convolutional layers and a fully connected layer connected in sequence; the last convolutional layer of the category mask prediction branch outputs the category mask prediction result, and the fully connected layer of the foreground / background mask prediction branch outputs the foreground / background mask prediction result.

2. The model as described in claim 1, characterized in that, The dilated convolutional spatial attention module includes an input layer, a max pooling layer, an average pooling layer, and a dilated convolutional layer connected in sequence.

3. A medical image segmentation method based on multi-attention fusion, characterized in that, include: The image to be segmented is input into a medical image segmentation model based on multi-attention fusion to obtain the image segmentation result; The medical image segmentation model based on multi-attention fusion is the medical image segmentation model based on multi-attention fusion as described in any one of claims 1-2.

4. The method as described in claim 3, characterized in that, The method further includes training the medical image segmentation model based on multi-attention fusion to obtain the trained medical image segmentation model based on multi-attention fusion.

5. The method as described in claim 4, characterized in that, Training the aforementioned medical image segmentation model based on multi-attention fusion includes: For each fourth feature map obtained by the pyramid expansion module, at least one target region is determined, and the at least one target region is input into the dual-branch fusion module.

6. The method as described in claim 5, characterized in that, For each fourth feature map, at least one target region is determined, including: Determine multiple predicted target region boxes in each fourth feature map; Based on the actual target region bounding box and the plurality of predicted target region bounding boxes, calculate the intersection-union ratio (IUGR) for each predicted target region bounding box; The intersection-union ratios (IURs) of all predicted target region boxes are sorted from largest to smallest, and the predicted target region box with the highest IUR is selected as the at least one target region.

7. The method as described in claim 6, characterized in that, in, Based on the actual target region bounding box and the plurality of predicted target region bounding boxes, calculate the Intersection over Union (IoU) for each predicted target region bounding box. new The following formula is used: Where S1 is the predicted target region bounding box, S2 is the true target region bounding box, and λ is the penalty factor.

8. The method as described in claim 4, characterized in that, Training the aforementioned medical image segmentation model based on multi-attention fusion includes: The image slices containing the lesion area were truncated using the HU index and augmented to obtain training data.

9. The method as described in claim 4, characterized in that, During model training, the deep residual network in the medical image segmentation model based on multi-attention fusion is a pre-trained deep residual network.

Images