Multi-scale large-kernel attention decoder and decoding method for medical image segmentation

Abstract

The invention relates to the technical field of medical image segmentation, in particular to a multi-scale large-kernel attention decoder and decoding method for medical image segmentation, comprising an encoder, a connection structure and a decoder, the connection structure comprises a large-kernel attention module and a variable self-attention module, the encoder edits input picture data to obtain multi-scale feature data; the large kernel attention module replaces deep convolution with deep separable convolution with an expansion characteristic based on a deep point-by-point large kernel attention mechanism, the variable self-attention module extracts information in the multi-scale characteristic data to obtain extracted information, the decoder performs medical image segmentation in cooperation with different scale characteristic tensor sizes of the encoder, and the medical image segmentation result is obtained. The multi-scale large-kernel attention decoder is based on the idea of multi-scale feature fusion, uses numerous cross-layer residual links, realizes efficient integration of information on different scales, extracts and fuses key features of pictures from two dimensions of channel and space, and improves expressive force and accuracy of the whole network.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a multi-scale large-core attention decoder and a decoding method for medical image segmentation. Background Art

[0002] In medical practice, various medical images play an inestimable role in doctors' diagnosis. However, medical resources are not unlimited. At this time, computer-aided diagnosis plays an important role. It can help medical professionals analyze complex medical images, so as to make diagnoses faster and more accurately and reduce the pressure on the medical system. The core of this intelligent medical system is the automatic segmentation method of medical images. Different from the research based on traditional methods such as edge detection, since 2012, CNN has gradually become the most concerned technical means in the field of vision due to the great success of AlexNet, and the field of medical images is no exception. Based on this technology, classic and effective models such as U-net, Unet++, and DeepLab were born. Because of the excellent characteristics of CNN, they can automatically extract complex feature representations from pixel scales that are beyond the reach of the human eye. But nothing is absolute. Due to the inherent locality of convolution operations, even if the encoder-decoder architecture is used to connect internal features, they are difficult to extract long-range features, which is a huge defect for medical images with target structures that often have large differences in shape and size.

[0003] Based on this, TransUNet and other works introduced the attention mechanism into the field of medical images. Although these methods combining CNN and attention can often achieve better performance in recent years, most of them find it difficult to balance the local features brought by convolution and the global dependencies brought by attention, and they also lack multi-scale performance. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-scale large-core attention decoder and decoding method for medical image segmentation, aiming to solve the problem.

[0005] To achieve the above-mentioned object, in a first aspect, the present invention provides a multi-scale large-core attention decoder for medical image segmentation, comprising an encoder, a connection structure and a decoder, wherein the encoder, the connection structure and the decoder are connected in sequence, the connection structure comprises a large-core attention module and a variable self-attention module, and the large-core attention module is connected to the variable self-attention module;

[0006] The encoder is used to edit the input image data to obtain multi-scale feature data;

[0007] The large core attention module replaces the deep convolution with the deep separable convolution with dilation characteristics based on the deep point-by-point large core attention mechanism;

[0008] The variable self-attention module is used to extract information from the multi-scale feature data to obtain extracted information;

[0009] The decoder is used to perform medical image segmentation in cooperation with the different scale feature tensor sizes of the encoder.

[0010] In a second aspect, the present invention further provides a multi-scale large core attention decoder decoding method for medical image segmentation, which is applied to the multi-scale large core attention decoder decoding method for medical image segmentation as described in the first aspect above, and is characterized in that it includes the following steps:

[0011] The feature map is input into the encoder backbone network, and outputted once in four layers respectively, and the resolution of the feature map is reduced respectively to obtain a first output tensor, a second output tensor, a third output tensor and a third output tensor;

[0012] The fourth output tensor is directly input into the variable self-attention module, and the first output tensor, the second output tensor, and the third output tensor are first summed with the previous layer output tensor that has been reduced in dimension and interpolated to keep the same size, and then input into the variable self-attention module;

[0013] The fourth layer output tensor is directly input into the large core attention module and reshaped to match the tensor size of the previous layer. The first output tensor, the second output tensor and the third output tensor are input into the large core attention module together with the reshaped output tensor of the previous layer and reshaped to match the tensor size of the previous layer.

[0014] The output of the second layer of the third layer and the output of the first layer of the second layer are linked respectively to obtain the final output of the network.

[0015] Among them, the feature map format is H*W*3, the first output tensor is H / 4*W / 4, the second output tensor is H / 8*W / 8, the second output tensor is H / 16*W / 16, and the fourth output tensor is H / 32*W / 32.

[0016] Among them, the variable self-attention module is composed of variable convolution, multi-head self-attention and fully connected layers and activation functions.

[0017] The activation function includes two loss functions: BCE and IoU.

[0018] The multi-scale large-core attention decoder for medical image segmentation of the present invention, the encoder is used to edit input image data to obtain multi-scale feature data; the large-core attention module replaces the deep convolution with the deep separable convolution with expansion characteristics based on the deep point-by-point large-core attention mechanism; the variable self-attention module is used to extract information in the multi-scale feature data to obtain extracted information; the decoder cooperates with the different scale feature tensor sizes of the encoder to perform medical image segmentation so as to learn multi-scale and multi-resolution image features, and has superior performance. The multi-scale large-core attention decoder is based on the idea of ​​multi-scale feature fusion, and cleverly uses many cross-layer residual links to achieve efficient integration of information at different scales. It can accurately extract and fuse the key features of the image from the two dimensions of channel and space, thereby improving the expressiveness and accuracy of the entire network. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a connection diagram of a multi-scale large-core attention decoder for medical image segmentation provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the overall architecture of the multi-scale large-core attention decoder for medical image segmentation provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the specific architecture of the VSA module.

[0023] Figure 4 It is a schematic diagram of the DPLA module, where the left side is the DPLA attention mechanism and the right side is the DPLA module.

[0024] Figure 5 It is a schematic diagram of PCA structure.

[0025] Figure 6 From top to bottom, they are the changing trends of the three loss functions in the Synapse dataset, the changing trends of the loss function and the DSC score of the test set in the ACDC dataset, and the changing trends of the loss function and the scores of the two internal datasets in the polyp dataset.

[0026] Figure 7From top to bottom, they are the overall intuitive performance of all models on the Synapse dataset and the ACDC dataset, respectively, and then the overall intuitive performance of all models on the polyp dataset.

[0027] Figure 8 This is a visualization diagram of the model segmentation results on five polyp datasets, where from top to bottom are the original polyp image, the ground truth image, the VDM segmentation image, the PraNet segmentation image, and the PolypPVT segmentation image. From left to right, they represent images from different polyp datasets.

[0028] Fig. 9 This is a visualization diagram of the segmentation results of the model on the Synapse dataset, where, from top to bottom, they are the ground truth image, TransUNet segmentation image, 2D D-LKA-Net segmentation image, and VDM segmentation image.

[0029] Fig.10 The following is a detailed structural diagram of DPLA, LKA, and two advanced variants of LKA. DW-Conv1 has a convolution kernel size of 55, DW-Conv2 has a convolution kernel size of 77, and a dilation factor of 3. In LSKA, 1D convolution kernels are used, with specific sizes of 15, 51, 17, and 71. These sizes can capture spatial information in different directions.

[0030] Fig.11 This is a flow chart of the decoding method of the multi-scale large-core attention decoder for medical image segmentation provided by the present invention.

[0031] In the figure: 1-encoder, 2-connection structure, 3-decoder, 4-large core attention module, 5-variable self-attention module. DETAILED DESCRIPTION

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0033] See also Figure 1 to Figure 2 In a first aspect, the present invention provides a multi-scale large core attention decoder for medical image segmentation, comprising an encoder 1, a connection structure 2 and a decoder 3, wherein the encoder 1, the connection structure 2 and the decoder 3 are connected in sequence, the connection structure 2 comprises a large core attention module 4 and a variable self-attention module 5, and the large core attention module 4 and the variable self-attention module 5 are connected;

[0034] The encoder 1 is used to edit the input image data to obtain multi-scale feature data;

[0035] The large core attention module 4 is used to extract information from the multi-scale feature data to obtain extracted information;

[0036] The variable self-attention module 5 replaces the deep convolution with the deep separable convolution with dilation characteristics based on the deep point-by-point large core attention mechanism;

[0037] The decoder 3 is used to perform medical image segmentation in cooperation with the different scale feature tensor sizes of the encoder.

[0038] In an embodiment of the present invention, the encoder 1 is used to edit input image data to obtain multi-scale feature data; the large core attention module 4 extracts information from the multi-scale feature data to obtain extracted information; the variable self-attention module 5 replaces the deep convolution with the deep separable convolution with expansion characteristics based on the deep point-by-point large core attention mechanism; the decoder 3 cooperates with the different scale feature tensor sizes of the encoder to perform medical image segmentation so as to learn multi-scale and multi-resolution image features, and has superior performance. The multi-scale large core attention decoder 3 is based on the idea of ​​multi-scale feature fusion and cleverly uses many cross-layer residual links to achieve efficient integration of information at different scales. It can accurately extract and fuse the key features of the image from the two dimensions of channel and space, thereby improving the expressiveness and accuracy of the entire network.

[0039] See also Figures 3 to 11 In a second aspect, the present invention further provides a decoding method of a multi-scale large core attention decoder 3 for medical image segmentation, which is applied to the decoding method of the multi-scale large core attention decoder 3 for medical image segmentation as described in the first aspect above, and is characterized in that it comprises the following steps:

[0040] The S1 feature map is input into the encoder 1 backbone network, and outputted once in four layers respectively, and the resolution of the feature map is reduced respectively to obtain a first output tensor, a second output tensor, a third output tensor and a third output tensor;

[0041] In an embodiment of the present invention, the feature map format is H*W*3, the first output tensor is H / 4*W / 4, the second output tensor is H / 8*W / 8, the second output tensor is H / 16*W / 16, and the fourth output tensor is H / 32*W / 32. At the same time, the channels are expanded to 96, 192, 384, and 768 (if it is a PVT backbone network, it is 64, 128, 320, and 512. MaxViT is taken as the backbone network as an example here).

[0042] S2: the fourth output tensor is directly input into the variable self-attention module 5; the first output tensor, the second output tensor and the third output tensor are first summed with the previous layer output tensor which has been reduced in dimension and interpolated to keep the same size, and then input into the variable self-attention module 5;

[0043] In an embodiment of the present invention, the variable self-attention module 5 (VSA) is intended to further extract features without changing the module size.

[0044] S3: the fourth layer output tensor is directly input into the big core attention module 4, and is reshaped to match the tensor size of the previous layer; the first output tensor, the second output tensor, and the third output tensor are input into the big core attention module 4 together with the reshaped output tensor of the previous layer, and are reshaped to match the tensor size of the previous layer;

[0045] S4 links the output of the second layer of the third layer and the output of the first layer of the second layer respectively to obtain the final output of the network.

[0046] In the embodiment of the present invention, the final output format is (H*W*C), where C is the classification status of the classification task, the Synapse dataset is 9, and the ACDC dataset is 4, which can gradually capture feature information from fine-grained to coarse-grained.

[0047] It is worth mentioning that the decoder does not adopt the common strategy of adjusting resolution and depth such as Upsampling, but uses an output structure similar to MLP, such as Figure 2 As shown in the figure, we first convert the tensor from BCHW to BNC format, then use linear expansion to expand its channels, and finally use Rearrange to reshape the tensor into the target shape. This operation is more efficient and avoids the interpolation process and possible problems of the traditional network structure during upsampling.

[0048] In medical practice, different human organs and the surrounding background composed of human tissues are highly similar, which poses a huge challenge to image segmentation. In order to improve the accuracy of recognition, a linker that can integrate multi-scale features is essential. Figure 3 As shown in FIG. 5 , VSA (variable self-attention module 5) is composed of variable convolution, multi-head self-attention and fully connected layers and some activation functions. The original intention of designing this module is to extract more useful information from the multi-scale feature data output by the encoder 1. The multi-head self-attention mechanism can be used to learn global features well, and its formula is as follows: i Q , W i K , W i VThere are three learnable weight matrices, which are used to generate query, key and value representations, W O is a learnable weight matrix

[0049]

[0050] MultiHead(X)=Concat(head1,head2,…,head h )W O

[0051] The input data comes from different layers and is relatively more complex, while the deformable convolution has a learnable offset and strong dynamic adjustment capability. Therefore, we use 3*3 deformable convolution to perform detailed preliminary learning of features of different scales before attention. Next, finally, we use the fully connected layer to integrate the previously obtained information and combine the residual idea to enhance the model's expressiveness and generalization capabilities. The formula of the entire model is as follows:

[0052] X out =MLP(X+SelfAttention(DeformConv(X)))

[0053] The MLP consists of two connected Linear layers and an activation function. The VSA module will be connected to the same layer and adjacent layers of the backbone network to extract the multi-scale features contained in them, forming a powerful information extractor to act on the decoder 3 network described later.

[0054] like Figure 4 (Left) shows that the deep point-by-point large kernel attention mechanism is used as the core of the module. In conventional LKA, the K*K large kernel is simulated as a deep convolution (DW-Conv) and a deep dilated convolution with a dilation rate of d (DW-D-Conv). The specific size of the convolution kernel is as follows:

[0055] DWConv = (2d-1)*(2d-1)

[0056] DWDConv=(K / d,K / d)

[0057] Finally, it uses 1*1 convolution to capture the relationship in the channel dimension, and combined with the attention mechanism, it introduces the adaptability of the channel dimension. Inspired by this, we believe that 1*1 point-by-point convolution may play a more important role in the model. We follow each deep convolution or deep dilated convolution with a 1*1 point-by-point convolution. In this way, without destroying the original large-core simulation capability, the deep convolution replaces the deep separable convolution (DS-Conv and DS-D-Conv) with dilation characteristics, in order to better capture the relationship information of the channel dimension. And add the classic 3*3 convolution of visual tasks to promote the extraction and learning of features at different levels. At the same time, LKA mentioned in the experiment that they tried to add the Sigmoid function but found that it had no effect. This may be due to its nonlinear characteristics that destroyed some features. Therefore, we used the linear activation function ReLU to enable the global receptive field, which can better improve the computational efficiency. Therefore, the formula for DPLA attention is:

[0058]

[0059] In addition, to match DPLA attention and make it better integrated into the overall network structure, such as Figure 4 As shown in (right), this paper uses its simple visual attention module, in which the MLP consists of two 1*1 convolutions used for the deepening channel and the restoration channel respectively, and a classic 3*3 convolution in the middle. The overall formula of the module is as follows:

[0060] Output=MLP(Conv(DPLA(Conv(Norm(input))))+Norm(input))

[0061] In addition, as introduced in Section 3.1, the network of this paper is divided into 4 layers. In the deepest layer, the number of channels C is relatively large, while the image size is reduced by H / 32, W / 32 times. The simulated 21*21 convolution size is even larger than the feature map size. The large core attention is not suitable in this case. Therefore, we designed another simple attention mechanism PCA (parallelChannel attention parallel spatial attention), which is composed of a simple channel attention composed of average pooling, 1*1 convolution and sigmoid activation function, and a common 3*3 convolution in parallel. Figure 5 As shown, it is designed to help the model handle feature extraction and learning in deep and small sizes. The formula is:

[0062]

[0063] In order to achieve better training results, this paper considers the strategy of using the two common loss functions BCE and IoU in parallel in the current field, and adds them according to their respective weights. The specific formula is as follows. Where y represents the true label, and p represents the probability of the model predicting the positive class

[0064]

[0065]

[0066] Loss = L IoU *0.4+L BCE *0.6

[0067] The multi-scale large-core attention decoder adopts two types of experimental methods from TransUNet and PraNet, and conducts comparative experiments on a total of seven public datasets, further demonstrating the powerful versatility of the multi-scale large-core attention decoder.

[0068] Synapse dataset and ACDC dataset

[0069] According to the conventional scheme used in TransUNet, etc.: the Synapse dataset and ACDC challenge are used to demonstrate the performance of the model.

[0070] The Synapse multi-organ dataset contains 50 cases and a total of 3779 axial abdominal clinical CT images. Each CT volume contains 85 to 198 slices, and the size of each slice is 512×512 pixels. The spatial resolution varies in the range of ([0.54~0.54]×[0.98~0.98]×[2.5~5.0])mm3. Similar to TransUNet, we randomly divided the dataset into 18 scans for training (2212 axial slices) and 12 scans for validation.

[0071] The ACDC dataset contains 100 cardiac MRI scans, each consisting of three organs: right ventricle (RV), myocardium (Myo), and left ventricle (LV). Following TransUNet, we use 70 cases (1930 axial slices) for training, 10 for validation, and 20 for testing.

[0072] Random flipping and rotation were used as training strategies for these two datasets, and the average DSC metric and average Hausdorff distance (HD95) were mainly used as evaluation criteria. Their formulas are as follows, where Y and ^Y are the true value and predicted segmentation map, respectively.

[0073]

[0074] Polyp Dataset

[0075] According to the conventional scheme used in PraNet, the training set is trained with the combined dataset between the ClinicDB dataset and the Kvasir-SEG dataset: 900 training samples selected from Kvasir-SEG and 550 training samples selected from CVC-ClinicDB, a total of 1450 samples, are used as the training set, supplemented by a multi-scale training strategy such as [0.75, 1, 1.25]. The remaining 62 and 100 images of the ClinicDB dataset and the Kvasir-SEG dataset, respectively, plus the ColonDB dataset, part of the EndoScene and ETIS datasets are used as the validation set [30-34], so that the model of this article can be examined from both the learning ability and generalization ability.

[0076] Dice and IoU are similarity metrics at the regional level, focusing on the internal consistency of the segmented object. They are also the most widely concerned indicators in the field of polyp research and the most important indicators in this paper. The formula is as follows:

[0077]

[0078] The variables are: True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN). MAE is a pixel-by-pixel comparison indicator, which represents the average absolute error between the predicted value and the true value, where N is the number of samples. Yi and Xi are the actual value and predicted value of the i-th sample, and the formula is as follows.

[0079]

[0080] All experiments are implemented using the Pytorch 2.2.0 framework and use the NVIDIA RTX4070 SUPER GPU for accelerated computing. The specific experimental steps are as follows:

[0081] 1. Create a command line parameter parser and preset various parameters such as image address and storage address;

[0082] 2. Group the data set according to the grouping method in 4.1, and then read the images of the training set and the test set respectively;

[0083] 3. Process and enhance the data of the read images, that is, first convert their sizes to the specified sizes (352 and 224), and then perform data enhancement on these images. The specific operations include random rotation of 90 degrees, random vertical flipping, random horizontal flipping, scaling, conversion to tensor, and standardization;

[0084] 4. Create a multi-scale large kernel attention decoder model (VDMNET) instance and optimizer;

[0085] 5. Start the training process, input the processed images into the model, obtain the feature map output by the model, and calculate the loss function.

[0086] 6. Perform back propagation and loop the preset rounds to obtain the weight of model training and obtain the accuracy of the model in the test set;

[0087] 7. Test the obtained weights on the validation set to obtain the final visualization conclusion and model accuracy.

[0088] In order to conduct experiments better, for the Synapse dataset and the ACDC dataset, this paper uses MaxViT as the backbone network and outputs it at the end of the decoder 3; the polyp-related experiments use PVT as the backbone network and output it once at each layer. The overall network architecture is the same. On both backbone networks, this paper uses the pre-trained weights published by the backbone network authors as the basis.

[0089] Table 1 Experimental details

[0090]

[0091] The two types of datasets have their own mainstream experimental methods, which lead to differences in some experimental parameters or settings for different datasets, as shown in Table 1. In order to better train network parameters, we selected the AdamW optimizer for all experiments and used the cosine annealing strategy for learning rate control. At the same time, in order to balance the effect and computational overhead, we trained 100 rounds on the Synapse dataset and 200 rounds on the ACDC dataset and polyp dataset. Figure 6 (Top) shows the changing trends of the three loss functions in the Synapse dataset. Figure 6 (Middle) shows the changing trend of loss function and test set DSC score in ACDC dataset. Figure 6 (Bottom) shows the changing trends of the loss function in the polyp dataset and the scores of the two internal datasets.

[0092] Experimental results on Synapse dataset and ACDC dataset

[0093] We selected V-Net, TransUNet, Swin-UNet, ScaleFormer, Missform, ParaTransCNN, Trans-CASCADE, and 2D D-LKA-Ne for comparison on the ACDC and Synapse datasets, respectively. These works are specifically conducted on these two datasets, and except for the early V-Net, all follow the research methods of TransUNet. To be fair, the comparison data are all from the conclusions of the papers corresponding to the models (2D D-LKA-Net only did the Synapse dataset, so the conclusions of ACDC were reproduced by us, and the early V-Net came from TransUNet). Experiments have shown that on two different tasks, the networks proposed in this paper achieved the best scores. The overall performance of all networks on the two datasets is as follows. Figure 7 (above) as shown.

[0094] Experimental results on the Synapse dataset

[0095] As shown in Table 2, the network composed of VDM achieved excellent results in the Synapse dataset, and the average DSC and HD95 indicators all achieved the best values ​​among all models. Specifically, the average DSC of the network in this paper is 85.26, which is 1.17% higher than the second-ranked 84.27 and 10.04% higher than the classic TransUNet. The hd95 is only 13.37, which is 17.0% higher than the best 15.86, and it is more than 100% higher than TransUNet. Specifically for each organ, six of the eight organs achieved the best value, and the remaining Liver and Gallbladdr also ranked fourth and third among all models respectively. Taking all the conditions into consideration, the model in this paper undoubtedly took the lead.

[0096] Table 2 Performance of the compared models on the Synapse dataset

[0097]

[0098] Experimental results on ACDC dataset

[0099] As shown in Table 3, the specific situation in the ACDC dataset is listed. The network composed of VDM still achieved the best results in ACDC, showing good versatility. Specifically, the average DSC score of 92.47 achieved by the network in this challenge is 0.7% higher than the second-ranked 91.80 and 3.08% higher than the classic TransUNet. Specifically, in the three subdivided categories of RV, Myo and LV, except for RV, which ranks second only to ParaTransCNN, Myo and LV both achieve the best values. In summary, the comprehensive level of the model in this paper is better than other comparison models.

[0100] Combining the performance of the two datasets, we can see that in the relatively more difficult Synapse task, our network has achieved a greater lead. This is because the score of the ACDC dataset has reached a level above 92, which is more difficult and less obvious to improve.

[0101] Table 3 Performance of the compared models on the ACDC dataset

[0102]

[0103] Experimental results on the polyp dataset

[0104] In order to further illustrate the general capabilities of the VDM decoder, a second round of experiments was conducted using a completely different experimental method with an unfamiliar backbone network. We selected UNet++, PraNet, UACANet-L, PolypPVT, BDG-Net, SSform-L, Cara-Net, MEGANet, and MLFF-Net, which are models that use polyp datasets as research objects, to conduct comparative experiments on five polyp datasets. In addition, except for UNet++ and SSForm-L, whose experimental methods are different from PraNet and the results are reproduced by us, the comparison data of the remaining models are all from the conclusions of the corresponding papers. Experiments have shown that the model proposed in this article also achieved the best results in the tasks of the polyp dataset. The three evaluation criteria are 1.56%, 1.82% and 9.1% higher than the PolypPVT proposed by the PVT backbone author team. The intuitive performance of all models is as follows Figure 7 (Below), the weighted average Dice, IoU and MAE scores on all datasets, as well as the specific data of the weighted average Dice scores of the internal and external datasets are shown in Table 4. Table 4 The weighted average scores of the main parameters of each model on the polyp dataset, as well as the weighted average Dice scores of the internal and external datasets.

[0105]

[0106] Experimental results on internal datasets

[0107] This paper uses the validation set of ClinicDB and Kvasir-SEG datasets to evaluate the learning ability of the proposed model. Our training set is constructed from a part of Kvasir-SEG and CVC-ClinicDB. Such related test samples are more inclined to verify the learning ability of the model. The detailed experimental results of all models on the two internal datasets are shown in Table 5. The experiments show that our model exhibits good learning ability, and the weighted average Dice, IoU and MAE scores of the internal datasets have achieved the highest values ​​among all models. Specifically, the highest Dice score (0.927) was achieved on Kvasir, which is higher than the second-ranked MLFF-Net (0.919), while the performance on CVC-ClinicDB (0.935) is slightly lower than MLFF-Net (0.943), Polyp-PVT (0.937) and Cara-Net (0.936). Overall, the learning ability of our model is better than all other models involved in the comparison, but because the internal dataset is relatively simple, the overall lead has declined to a certain extent.

[0108] Table 5. Full parameter scores of each model on the internal polyp dataset. From left to right are ClinicDB and Kvasir-SEG.

[0109]

[0110]

[0111] Experimental results on external datasets

[0112] The generalization ability of the model was verified using three external datasets: ETIS, ColonDB, and EndoScene. ETIS and ColonDB are completely unfamiliar datasets, while EndoScene has some of the same content as the training set CVC-ClinicDB. Although PraNet's experimental method only selected different parts of EndoScene as external datasets, it may still be related to ClinicDB to some extent, and PraNet's scores on all models are significantly higher than those on the other two datasets, so the results on this dataset need to be viewed with caution.

[0113] The detailed experimental results are shown in Table 6. Compared with the learning ability, the generalization ability of our model is more outstanding. Its average Dice, IoU and MAE on the external dataset are 0.825, 0.747 and 0.021 respectively, which are the highest values ​​of all models, and 1.85%, 2.19% and 9.7% higher than Polyp-PVT proposed by the author of the PVT framework. Specifically, on the test set CVC-ClinicDB with the largest number of images, our model achieved a Dice score of 0.826. On the most difficult ETIS dataset, our model achieved a Dice score of 0.801, which is the highest value among all models. The experimental results fully prove that the network proposed in this paper has strong generalization performance.

[0114] Table 6. The complete parameter scores of each model on the external polyp dataset. From left to right are ColonDB, ETIS, and EndoScene.

[0115]

[0116] Visual analysis

[0117] Figure 8 and Fig. 9 Some of its final segmentation renderings on the Synapse dataset and polyp dataset are shown respectively. At the same time, TransUNet and PraNet, which pioneered the experimental method, as well as 2D D-LKA-Net, which also uses LKA, and PolypPVT, the renderings of the PVT backbone proposed by the team, are selected for comparison, which fully reveals the outstanding capabilities of the model in this paper in multi-organ segmentation tasks and polyp segmentation tasks. From these two pictures, we can clearly see that the network composed of VDMs shows excellent segmentation effects, whether in complex organ structures or subtle polyp morphology. In most areas, the model can achieve accurate segmentation of organs.

[0118] Ablation studies

[0119] In order to verify the role of the cross-layer link design, VSA blocks and DPLA blocks in our experiment, we conducted an ablation experiment. In order to more accurately reflect the performance improvement brought by the network, we used the network of the complete VDM decoder combined with the MaxViT encoder as the benchmark, and compared the performance of the MaxViT encoder with the simplest upsampling decoder U-connected network (each layer only includes a 1*1 dimensionality reduction convolution and an upsampling module). We performed ablation studies on the Synapse dataset to remove the VSA block and DPLA block. The specific operations and results are shown in Table 7.

[0120] Table 7 Ablation study of modules on the Synapse dataset

[0121]

[0122] Experiments show that eliminating the VSA module will reduce the overall performance by 1.26%, while eliminating the DPLA will reduce the model performance by 1.57%. In short, after eliminating any module, the performance of all aspects of the model has dropped significantly, which further proves the effectiveness of the overall network structure, VSA block and DPLA block designed by us. These modules have greatly improved the performance of our model and performed segmentation tasks more effectively. At the same time, by comparing the number of parameters, we can see that compared with the backbone network, our parameter quantity has only increased by 42.4%, which shows that most of the parameters of the VDM network come from the backbone network. These modules designed in this paper are concise and effective.

[0123] At the same time, in order to further demonstrate the powerful performance of DPLA, we conducted further replacement experiments on the ACDC dataset with smaller fluctuations. Without changing any other parameter settings, network and module structures, we replaced DPLA with the original LKA, D-LKA, and LSKA respectively and conducted experiments. The specific architectures of these modules are as follows: Fig.10 The final results of the replacement experiment are shown in Table 8. The experiment proves that our module has significant advantages over these other variant modules without affecting the parameters and computational complexity.

[0124] Table 8 Alternative experiments using other advanced variants of LKA

[0125]

[0126]

[0127] What is disclosed above is only a preferred embodiment of the multi-scale large-core attention decoder and decoding method for medical image segmentation of the present invention. Of course, this cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments are implemented, and equivalent changes made according to the claims of the present invention are still within the scope of the invention.

Claims

1. A multi-scale large-kernel attention decoder for medical image segmentation, characterized by: It includes a connection structure and a decoder, the encoder, the connection structure and the decoder are connected in sequence, the connection structure includes a large core attention module and a variable self-attention module, and the large core attention module is connected to the variable self-attention module; The encoder is used to edit the input image data to obtain multi-scale feature data; The large core attention module replaces the deep convolution with the deep separable convolution with dilation characteristics based on the deep point-by-point large core attention mechanism; The variable self-attention module is used to extract information from the multi-scale feature data to obtain extracted information; The decoder is used to perform medical image segmentation in cooperation with the different scale feature tensor sizes of the encoder.

2. A multi-scale large core attention decoder decoding method for medical image segmentation, applied to the multi-scale large core attention decoder decoding method for medical image segmentation as claimed in claim 1, characterized in that: The following steps are involved: The feature map is input into the encoder backbone network, and outputted once in four layers respectively, and the resolution of the feature map is reduced respectively to obtain a first output tensor, a second output tensor, a third output tensor and a third output tensor; The fourth output tensor is directly input into the variable self-attention module, and the first output tensor, the second output tensor, and the third output tensor are first summed with the previous layer output tensor that has been reduced in dimension and interpolated to keep the same size, and then input into the variable self-attention module; The fourth layer output tensor is directly input into the large core attention module and reshaped to match the tensor size of the previous layer. The first output tensor, the second output tensor and the third output tensor are input into the large core attention module together with the reshaped output tensor of the previous layer and reshaped to match the tensor size of the previous layer. The output of the second layer of the third layer and the output of the first layer of the second layer are linked respectively to obtain the final output of the network.

3. The multi-scale large-core attention decoder decoding method for medical image segmentation according to claim 2, characterized in that: The feature map format is H*W*3, the first output tensor is H / 4*W / 4, the second output tensor is H / 8*W / 8, the second output tensor is H / 16*W / 16, the fourth output tensor is H / 32*W / 32, and the final output is H*W*C, where C is the number of categories.

4. The multi-scale large-core attention decoder decoding method for medical image segmentation according to claim 2, characterized in that: The variable self-attention module is composed of variable convolution, multi-head self-attention and fully connected layers and activation functions.

5. The multi-scale large-core attention decoder decoding method for medical image segmentation according to claim 4, characterized in that: The activation function includes two loss functions: BCE and IoU.

Images