Liver and tumor segmentation method based on mixed attention

Abstract

The invention relates to the field of image segmentation, and discloses a liver and tumor segmentation method based on mixed attention, and the method is realized by adopting a liver and tumor segmentation network based on mixed attention, and comprises two branches: a local feature extraction branch proposes a local deep splitting module to extract local information, extracts context information in a multi-scale manner, and extracts a local feature; the method has the advantages that the influence of useless noise is inhibited, fine-grained feature representation is integrated, the segmentation precision and remote information dependence are improved, CNNform remote dependence branches are constructed based on a Transform model, global context information is concerned, global features are captured, category information is refined, the parameter quantity is reduced, and the segmentation efficiency is improved. An interactive fusion module is proposed based on cross attention to aggregate feature mappings obtained from two branches to further improve network segmentation performance, and finally, a deep loss supervision mechanism is integrated between different levels of a network decoder to provide additional feedback from a deeper intermediate layer, which restricts network weight in a target area to improve network segmentation performance. And optimizing the segmentation result.

Description

Technical Field

[0001] The invention relates to the field of image segmentation, in particular to a liver and tumor segmentation method based on mixed attention. Background Art

[0002] The liver is one of the most important metabolic organs in the human body, with important functions such as digestion and excretion. Liver lesions are often accompanied by liver tumors, which cause liver cancer. Liver cancer is the most common disease in the world. Many people die from liver cancer every year. Early detection, diagnosis and treatment of liver cancer are crucial to improving the survival rate and quality of life of patients. With the development of computer technology, many medical imaging technologies have emerged. Among them, computed tomography (CT) is a widely used imaging tool. It is widely used to detect liver shape, texture and focal lesions. Therefore, quickly and accurately identifying, locating and segmenting lesion areas from CT is an important prerequisite for doctors to formulate cancer diagnosis and treatment plans for patients. Traditional CT image segmentation methods require radiologists to check CT slices one by one to accurately segment the tumor site, which is easy to miss or misdiagnose due to fatigue or subjective factors. At the same time, due to the different shapes of tumors, fuzzy boundaries and low contrast with other organs, doctors need to have rich clinical experience to accurately segment tumors. Therefore, an automatic and accurate liver tumor segmentation method is of great significance to the diagnosis and medical research of liver tumors.

[0003] The segmentation process of early machine learning-based methods is divided into two steps: feature extraction and classifier training. According to these features, these methods can be divided into threshold methods, region-based segmentation methods, clustering methods, model-based segmentation methods, and edge detection-based segmentation methods. These methods require experts to manually extract image features, rely heavily on the experience of researchers, and are difficult to expand. With the development of GPU hardware and the open source availability of a large number of medical data sets, deep learning has achieved unprecedented development. Methods based on convolutional neural networks have an overwhelming position in the field of liver tumor segmentation. Convolutional neural networks can automatically learn the most useful features from large data sets. Among them, the most famous are FCN and U-Net. The emergence of U-Net networks has proved the role of U-shaped networks in the field of medical image segmentation. Its success is largely attributed to their jump connections, which combine high-level and low-level semantic information and reduce the semantic gap between encoders and decoders. In order to combine high-level features and low-level features of refined boundaries, ResNet and SENet modules are introduced in the U-Net model to enhance the receptive field of the feature extraction stage. In order to avoid the repeated processing of low-level features by jump connections, the receptive field of the feature extraction stage is increased, and the receptive field of the feature extraction stage is increased. In order to avoid the repeated processing of low-level features by jump connections, the receptive field of the feature extraction stage is increased, and the receptive field of the feature extraction stage is increased. In addition ... The resolution feature information is lost, which causes the small target area to disappear. mU-Net achieves better liver and liver tumor segmentation by adding a residual path with deconvolution and activation operations in the jump connection of U-Net. Some scholars have introduced ResNet and SENet modules in the U-Net model to enhance the receptive field of the feature extraction stage. U-Net++ uses dense connections instead of jump connections to achieve multi-scale fusion of high- and low-level semantics. In addition, various attention perception mechanisms are introduced into the network to optimize feature representation. Based on these operations, the improved model can break the scale consistency limitation and further integrate high-level abstract information and low-level pixel information. In recent years, Transformer has used the characteristics of self-attention mechanism to obtain global features and has achieved certain results in natural language processing (NLP) and computer vision (CV). The most famous of these is the visual Transformer, which applies Transformer to CV and uses self-attention heads to effectively extract global information features in images. Many Transformer-based methods are widely used in liver and tumor segmentation. Swin was introduced in U-Net. Transformer enhances the globality of the network. In order to enhance the network's perception and global understanding of the image path, Transformer is used instead of downsampling, and RDCTransU-Net is proposed for liver segmentation. In order to learn the characteristics of different tumors, a dynamic hierarchical Transformer network is proposed using hierarchical operations with different receptive field sizes. In addition, the self-attention mechanism is the core of Transformer. Introducing this dot product attention into the model can also capture long-term contextual information and improve the accuracy of liver tumor segmentation.

[0004] Although the above method has achieved good segmentation effect, it still cannot make full use of the rich local and long-range dependency information contained in medical images. On the one hand, due to the sparse interaction characteristics of the convolution operation, the receptive field based on the convolution method can only extract local semantic information of the image, resulting in the background area being identified as the lesion area. On the other hand, Transformer uses the dispense convolution operator and the attention mechanism to map a series of image block sequences into an abstract continuous representation, thereby obtaining a global dependency on the entire image, effectively avoiding the erroneous segmentation results of the background in the medical image. However, the Transformer structure ignores local information, and over-segmentation or under-segmentation of different lesion areas will occur. Therefore, how to simultaneously extract local features and global information to improve the network model of the segmentation accuracy of the liver and tumor is urgently needed. Based on this, the present invention proposes a liver and tumor segmentation method based on mixed attention to solve the above problems. Summary of the invention

[0005] The object of the present invention is to provide a liver and tumor segmentation method based on mixed attention to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a liver and tumor segmentation method based on mixed attention, the method is implemented by a liver and tumor segmentation network based on mixed attention, comprising:

[0007] For the input image I∈R w×h×c , respectively using the local feature extraction branch based on the CNN structure and the long-range dependency feature extraction branch based on the Transformer structure to extract pixel-level detail features and object-level global features. Among them, c i ∈{64,128,256,512}, i∈{1,2,3,4};

[0008] Subtract the global features extracted from the remote dependency branch from the local features extracted from the local feature branch in the corresponding stage to obtain high-resolution edge contour features, and then perform a maximum pooling downsampling operation;

[0009] The local features and long-range dependency features finally extracted by the two branches are respectively input into the dual-branch interaction module to aggregate the local features and long-range dependency features;

[0010] The aggregated feature vector containing local features and global features is input into the decoder. At the same time, four skip connections are used to fuse the features of the encoding path and the decoder. The first and second skip connections obtain shallow information containing high-resolution texture features from the local feature branch and fuse them into the decoder. The third and fourth skip connections pass the deep edge information in the global dependency branch to the decoder, so that the feature vector has high-resolution texture and deep edge information, and the preliminary liver and tumor segmentation results are obtained.

[0011] Based on each decoding block, the corresponding classifier is added to generate multi-scale segmentation maps at different stages to calculate the loss function and obtain the final liver and tumor segmentation results.

[0012] Preferably: the local feature extraction branch based on the CNN structure is composed of a deep split module, and a fusion attention mechanism is integrated on different branches to capture the interactive information of cross-features. Specifically, given an input feature I∈R H×W×C , where H, W, C are the height, width and number of channels respectively. The feature I is evenly divided into K feature vectors along the channel dimension. Subsequently, each feature vector is divided into R individuals, that is, the feature map I is divided into G = KR groups in total. The K feature vectors are input into K parallel basic branches, each of which is composed of R split branches with fused attention. Each split branch consists of two convolutional layers, followed by BN and a ReLU activation function, and outputs a feature map with a height of H / 4 and a width of W / 4;

[0013] The fused attention is integrated into the split branch to connect the R feature maps, denoted as {U1,U2,...,U R}, firstly, the R feature maps are fused by element-wise summation operation, and a global pooling operation is applied to obtain the global context information s:

[0014]

[0015] After the global pooling layer, two consecutive fully connected layers with nonlinear activation functions are applied to predict the R attention weights {a1,a2,...,a R}, and then use each attention weight pair {U1,U2,...,U R} Perform weighted fusion and output a basic feature map V:

[0016]

[0017] Accordingly, we get K basic feature maps, {V1,V2,...,V K} indicates that the K basic feature maps are then connected piece by piece along the channel dimension, and a 1×1 convolution layer is used to produce the final split convolution block output C. The network pays more attention to the correlation between the adjacent channels of the two variables.

[0018]

[0019] where f 1×1 is a convolutional layer with a kernel size of 1×1, n is the number of channels of V, is the nth cut of k feature maps, with hyperparameters K and R being 2.

[0020] Preferably: the Transformer-based long-range dependency feature extraction branch is composed of CNNformer, and the CNNformer is composed of a CNN structure composed of Conv_Block as a token mixer, which is used to optimize the MLP into a convolutional structure that is more suitable for image processing. The specific mathematical expression is:

[0021] cf(x)=mlp(cnn(x))

[0022] cnn(x)=Conv 3×3 (Conv 3×3 (x))+x

[0023] mlp(x cnn )=Conv 1×1 (ReLU(Conv 1×1 (x cnn )))+x cnn

[0024] Among them, Conv 3×3 and Conv 1×1 represents the convolutional layers with kernels of 3×3 and 1×1, cnn(·) and mlp(·) represent CNN and MLP, ReLU(·) is the ReLU activation function, and x cnn is the output of cnn(·).

[0025] Preferably: the specific process of the dual-branch interaction module is as follows: first, the local feature extraction branch C i Use spatial attention as a spatial filter to remove feature noise and enhance local details to get the output For remote dependency branch T i Perform channel attention mechanism separately to promote the output of global information At the same time, for C i and T i Interactive feature extraction i, Secondly, the position encoding of the same dimension is embedded into two branches, and two cross-interaction attention modules are used to perform attention interaction fusion by receiving the corresponding branch feature map and the other branch feature map to generate two outputs c f and t f Finally, the participating feature c f , t f Connect with the interactive features and pass through the residual block to output the result feature fi. The specific formula is as follows:

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] f i =Resideual(c f , t f , b i )

[0032] Where |⊙| is the dot product and Conv is a 3x3 convolutional layer.

[0033] Preferably: During the training of the network, the loss function is the cross entropy loss function L BCE and dice loss L Dice Composition, among which R gt and R seg Represent the true label value and the predicted result respectively, L BCE and L Dice They are defined as,

[0034] L BCE (R gt ,R seg )=-(1-R gt ) og (1-R seg )-R gt logR seg

[0035]

[0036] In the above formula, ε is a constant. BCE and L Dice The loss function in the experiment can be obtained.

[0037] L seg(R gt ,R seg )=λL BCE +L Dice

[0038] Among them, λ is 0.5, and the network model applies L to each decoder in the second stage decoding process. seg , establish a deep supervision loss function, where M GT Represents the true segmentation result map, so the loss of the i-th decoder is can be defined as,

[0039]

[0040] In the above formula, is defined as

[0041]

[0042] in, The factor is 2 i-1 Downsampling, in addition, L seg It is also applied to the output of the first stage decoding process. The loss of the first stage is defined as,

[0043] L c (M GT ,M c )=L seg (M GT ,M c )

[0044] The overall loss function during the training process of the model is defined as,

[0045]

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The local feature extraction branch of the present invention proposes a local deep splitting module to extract local information, extracts context information at multiple scales, suppresses the influence of useless noise, integrates fine-grained feature representation, improves segmentation accuracy, and remote information dependency. A CNNformer remote dependency branch is constructed based on the Transformer model, which focuses on global context information, captures global features, refines category information, and reduces the amount of parameters. An interactive fusion module is proposed based on cross-attention to aggregate feature maps obtained from the two branches, further improving the network segmentation performance. Finally, a deep loss supervision mechanism is integrated between different levels of the network decoder to provide additional feedback from deeper intermediate layers, which limits the network weights to the target area and optimizes the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural diagram of MANet of the present invention;

[0049] Figure 2 Figure 2 shows the deep splitting module (a) and fusion attention (b) of the present invention.

[0050] Figure 3 It is a comparison diagram of the Transformer structure and the CNNformer structure of the present invention;

[0051] Figure 4 This is a diagram of the dual-branch interactive attention structure of the present invention;

[0052] Figure 5 This is a visualization diagram of the segmentation results of the present invention on the LiTS dataset;

[0053] Figure 6 This is a visualization diagram of the segmentation result of the present invention on 3DIRCADb;

[0054] Figure 7 Visualization of the segmentation results of the ablation experiment of the present invention;

[0055] Figure 8 This is the confusion matrix diagram of LiTs and 3DIRCADB of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Embodiment 1

[0058] See also Figure 1 , the structure diagram of the liver and tumor segmentation method based on mixed attention is shown in the figure, and the method is implemented by using a liver and tumor segmentation network based on mixed attention, including:

[0059] For the input image I∈R w×h×c , respectively using the local feature extraction branch based on the CNN structure and the long-range dependency feature extraction branch based on the Transformer structure to extract pixel-level detail features and object-level global features. Among them, c i ∈{64,128,256,512}, i∈{1,2,3,4};

[0060] Subtract the global features extracted from the remote dependency branch from the local features extracted from the local feature branch in the corresponding stage to obtain high-resolution edge contour features, and then perform a maximum pooling downsampling operation;

[0061] The local features and long-range dependency features finally extracted by the two branches are respectively input into the dual-branch interaction module to aggregate the local features and long-range dependency features;

[0062] The aggregated feature vector containing local features and global features is input into the decoder. At the same time, four skip connections are used to fuse the features of the encoding path and the decoder. The first and second skip connections obtain shallow information containing high-resolution texture features from the local feature branch and fuse them into the decoder. The third and fourth skip connections pass the deep edge information in the global dependency branch to the decoder, so that the feature vector has high-resolution texture and deep edge information, and the preliminary liver and tumor segmentation results are obtained.

[0063] Based on each decoding block, the corresponding classifier is added to generate multi-scale segmentation maps at different stages to calculate the loss function and obtain the final liver and tumor segmentation results.

[0064] In this embodiment, the hybrid attention liver and tumor segmentation network, namely MANet, is based on the classic medical image segmentation network U-Net as a benchmark, and adopts an asymmetric dual-stream encoder-decoder architecture. Specifically, the network is composed of two parallel branch feature extractors, namely a local feature extraction branch based on a CNN structure and a long-range dependency feature extraction branch based on a Transformer structure. The CNN branch is composed of deep split modules, which can represent multi-scale features in a fine-grained manner, increase the local feature extraction capability of the network, and refine edge detail texture features. For the long-range dependency branch, a CNNformer is adopted. Since a convolutional layer is used instead of complex self-attention, it has a stronger feature extraction capability and reduces resource consumption.

[0065] Furthermore, the local feature extraction branch based on the CNN structure consists of a deep split module, and a fusion attention mechanism is integrated on different branches to capture the interactive information of cross features. The CNN encoder includes three encoder layers, such as Figure 2 As shown, given an input feature I∈R H×W×C , where H, W, and C are the height, width, and number of channels, respectively. The feature I is evenly divided into K feature vectors along the channel dimension. Subsequently, each feature vector is divided into R individuals, that is, the feature map I is divided into G = KR groups in total. The K feature vectors are input into K parallel basic branches, and each parallel basic branch consists of R split branches with fused attention, such as Figure 2As shown in (a), each split branch consists of two convolutional layers, followed by BN and a ReLU activation function, and outputs a feature map with a height of H / 4 and a width of W / 4; the fused attention is integrated into the split branch to connect the R feature maps, denoted as {U1,U2,...,U R},like Figure 2 As shown in (b), the module first fuses R feature maps through an element-wise summation operation and applies a global pooling operation to obtain the global context information s:

[0066]

[0067] After the global pooling layer, two consecutive fully connected layers with nonlinear activation functions are applied to predict the R attention weights {a1,a2,...,a R}, and then use each attention weight pair {U1,U2,...,U R} Perform weighted fusion and output a basic feature map V:

[0068]

[0069] Accordingly, we get K basic feature maps, {V1,V2,...,V K} indicates that the K basic feature maps are then connected piece by piece along the channel dimension, and a 1×1 convolution layer is used to produce the final split convolution block output C. The network pays more attention to the correlation between the adjacent channels of the two variables.

[0070]

[0071] where f 1×1 is a convolutional layer with a kernel size of 1×1, n is the number of channels of V, is the nth cut of k feature maps, and the hyperparameters K and R are set to 2 in the present invention.

[0072] The Transformer-based long-range dependency feature extraction branch is composed of CNNformer. According to the self-attention structure of Transformer, the proposed Cnnformer structure is used for the long-range dependency branch. The specific Cnnformer structure is as follows Figure 3 As shown in the figure, CNNformer is a CNN structure composed of Conv_Block as a token mixer, which is used to optimize MLP into a convolution structure more suitable for image processing. The specific mathematical expression is:

[0073] cf(x)=mlp(cnn(x))

[0074] cnn(x)=Conv 3×3 (Conv3×3 (x))+x

[0075] mlp(x cnn )=Conv 1×1 (ReLU(Conv 1×1 (x cnn )))+x cnn

[0076] Among them, Conv 3×3 and Conv 1×1 represents the convolutional layers with kernels of 3×3 and 1×1, cnn(·) and mlp(·) represent CNN and MLP, ReLU(·) is the ReLU activation function, and x cnn is the output of cnn(·).

[0077] In this embodiment, dual-branch fine fusion is crucial to the success of MANet, which can capture features from two different perspectives of the same image. In order to capture this interactive fusion feature, the present invention proposes a dual-branch interactive fusion module based on cross attention, namely DIM. Figure 4 Specifically, this module aims to fuse the feature information between the local feature extraction branch and the long-range dependency extraction branch, and interactively propagate information in a dynamic and learnable manner. Compared with simply concatenating features from different perspectives, the DIM module promotes the adaptive integration between the two feature representations, thereby allowing the representation of more informative features. The specific process of the dual-branch interaction module is as follows: First, the local feature extraction branch C i Use spatial attention as a spatial filter to remove feature noise and enhance local details to get the output For remote dependency branch T i Perform channel attention mechanism separately to promote the output of global information At the same time, for C i and T i Interactive feature extraction i , Secondly, the position encoding of the same dimension is embedded into two branches, and two cross-interaction attention modules are used to perform attention interaction fusion by receiving the corresponding branch feature map and the other branch feature map to generate two outputs c f and t f Finally, the participating feature c f , t f Connect with the interactive features and pass through the residual block to output the result feature fi. The specific formula is as follows:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] f i =Resideual(c f , t f , b i )

[0084] Where |⊙| is the dot product and Conv is a 3x3 convolutional layer. In DIM, the attention mechanism is the basic operation for designing feature fusion interaction modules. N×C For example, the general attention function (Att) mainly performs the dot product operation after scaling, and the formula is as follows:

[0085]

[0086] However, the implementation of soft max logit and attention map requires O(n 2 ) and O(n 2 c) Time complexity, inspired by the study of self-attention linearization, this paper uses two functions φ(·), The attention map is factorized and roughly estimated by calculating the matrix multiplication of keys and values ​​as follows:

[0087]

[0088] Factorization reduces the space complexity and time complexity to O(NC) and O(NC) respectively. 2 ), they are all linear functions with a sequence length of N. is the scale factor 1 / √C, For softmax:

[0089]

[0090] Although this attention decomposition is not an unbiased approximation of the scaled dot product attention, it can still be considered as a generalized attention mechanism, which uses Q, K, and V to establish model feature interactions and factorize the attention module, which can proportionally reduce the computational burden of the dot product attention. In addition, since the present invention first calculates S = softmax((K) T V)∈R C×C, then S can be considered as a data-dependent global linear transformation for each feature vector queryQ mapping, which indicates that there may be two equivalent query vectors q1 and q2 in Q. Therefore, the relevant self-attention output can theoretically be defined as

[0091]

[0092] However, this characteristic will lead to bad results for feature extraction. Specifically, based on this operation, the output will only depend on the structure of the input token, and will not notice the differences in local features. That is to say, this characteristic is harmful to image tasks, especially for intensive tasks such as segmentation. In order to enhance the relative relationship between model tokens, the present invention adds a position encoding with a window size of M To capture the relative attention map λQ∈R N×C

[0093]

[0094]

[0095] in, It is the Hadamard Product, each element λQ is the relative attention feature map of the relationship (q, v), which represents the relationship from qi to vi, and aggregates all related value vectors to qi separately. The process is as follows Figure 3 As shown in (b), in comparison, the space complexity and time complexity of λQ are O(NC) and O(NCM2) respectively, which is more efficient.

[0096] During the training of the network, the loss function is the cross entropy loss function L BCE and dice loss L Dice Composition, among which R gt and R seg Represent the true label value and the predicted result respectively, L BCE and L Dice They are defined as,

[0097] L BCE (R gt ,R seg )=-(1-R gt )log(1-R seg )-R gt logR seg

[0098]

[0099] In the above formula, ε is a constant. BCE and L DiceThe loss function in the experiment can be obtained.

[0100] L seg (R gt ,R seg )=λL BCE +L Dice

[0101] Among them, λ is 0.5, and the network model applies L to each decoder in the second stage decoding process. seg , establish a deep supervision loss function, where M GT Represents the true segmentation result map, so the loss of the i-th decoder is can be defined as,

[0102]

[0103] In the above formula, is defined as

[0104]

[0105] in, The factor is 2 i-1 Downsampling, in addition, L seg It is also applied to the output of the first stage decoding process. The loss of the first stage is defined as,

[0106] L c (M GT ,M c )=L seg (M GT ,M c ), the overall loss function of the model during training is defined as,

[0107]

[0108] The method of the present invention was used to perform segmentation experiments on abdominal CT. All models were based on the Pytorch framework and python 3.8. All experiments were performed on a deep learning workstation equipped with an Intel(R) Core i7-13900K. In addition, it also had 32GB of DDR5 RAM and a 24GB NVIDIA GeForce RTX 4090 graphics processing unit (GPU). In addition, the hyperparameters of all models were set to the same, with an initial learning rate of 0.0003, a batch size of 16, an Epoch of 200, and an optimizer of Adam.

[0109] The LiTS dataset is a public dataset of the liver tumor segmentation challenge held by ISBI 2017 and MICCAI 2017. It is currently the most commonly used dataset in liver and tumor segmentation research. The LiTS dataset consists of a training set of 131 CT scans, and the number of CT slices contained in each scan ranges from 42 to 1026. The cross-sectional spacing is 0.45-6.0 mm. The present invention uses 90 patients (43219 axial slices in total) and 10 patients (1500 axial slices in total) to construct the training set and validation set, respectively, and then the remaining 30 patients (15419 axial slices in total) are used as the test set.

[0110] 3DIRCADb is a small dataset containing 22 3D data with image size of 512×512, slice thickness of 1-4 mm, pixel spacing of 0.56-0.86 mm, and slice number of 184-260. It is divided into 17 patients for training and 10 patients for testing.

[0111] In order to enhance the contrast between the liver, tumor and other tissues, the HU value of the CT image is set to [-200, 250]. Next, the noise points existing in the image are subjected to local histogram equalization for each pixel in the CT image data according to the grayscale distribution of its adjacent pixels to further improve the visual quality and diagnostic accuracy of the medical image. In addition, random horizontal flipping, random vertical flipping and random scale rotation shift are used as data enhancement methods.

[0112] In order to quantitatively analyze the segmentation results, five evaluation indicators are used, including Dice coefficient (Dice), intersection over union (IOU), relative volume difference (RVD), average symmetric surface distance (ASSD), and maximum symmetric surface distance (MSD). gt ,R seg For label values ​​and predicted segmentation results, the mathematical formulas of these indicators are as follows:

[0113] DIC coefficient (DIC): It specifically represents the ratio of the intersection area of ​​two objects to the total area. Its value range is [0,1], and the ideal separation value is 1.

[0114]

[0115] Intersection-over-Union (IOU): The overlap ratio between the segmentation result and the actual result.

[0116]

[0117] Relative Volume Error (RVD): It is used to indicate the volume difference between the actual segmentation result and the predicted segmentation result. The closer the value is to zero, the higher the segmentation accuracy.

[0118]

[0119] Average Symmetric Surface Distance (ASSD): Segmentation result R seg The surface to label R gt The average distance between a and b is d(a,b), where d(a,b) represents the distance between a and b.

[0120]

[0121] Maximum symmetric surface distance (MSD): segmentation result R seg With label R gt The maximum surface distance between the two samples. The lower the MSD value, the higher the matching degree between the two samples.

[0122]

[0123] The segmentation results of the method of the present invention are compared with those of existing models. These models can be divided into two categories, namely, CNN-based methods and CNN-Transformer-based methods. Among them, the CNN-based methods include DeepLabv3+, U-Net, AttentionU-Net, ResU-Net, U-Net++, Double UNet, and the CNN-Transformer-based methods include Swim-UNet, TransUNet and Hiformer.

[0124] For quantitative analysis and comparison, the network of the present invention, namely MANet, and the existing models are firstly quantitatively analyzed on the LiTS2017 dataset. The evaluation indicators of the experimental results include DIC, IOU, RAVD, ASSD and MSD, as shown in Table 1. As can be seen from the table, MANet outperforms the existing methods, such as HiFormer, TransUNet, Swin-UNet and Double in almost all indicators. UNet, in particular, MANet achieves 92.5% and 91.34% DIC and IOU respectively in liver segmentation, which are 0.93% and 0.47 higher than HiFormer. In the tumor segmentation task, MANet’s DIC and IOU reach 55.51% and 42.16% respectively, which are 0.08% and 1.52% higher than HiFormer. Functionally, the CNN-based method and MANet’s GCA have similar purposes, which are to extract local semantic information and make the segmentation more refined. At the same time, the Transformer-based method has the same function as MANet’s dual-branch interaction mechanism, which is to expand the network’s receptive field and extract rich global features. Figure 5 A visual comparison of the segmentation results is given in Figure 2. Obviously, the qualitative results of MANet's liver and tumor segmentation also reach the SOTA performance. Compared with other methods, MANet can accurately identify the liver and tumor areas, accurately segment the edge areas of the liver, and distinguish tumors with blurred image edges, especially those small areas of the liver and tumors similar to the color and structure of liver tissue, which are often missed in liver CT readings. In general, the predicted masks generated by MANet have almost the same boundaries and shapes as the true labels.

[0125] In addition, compared with the baseline model, it can be observed that two variants of U-Net, including Double UNet and Swim-UNet, achieved 0.02% and 0.03% improvement on DIC over U-Net, respectively, demonstrating the positive impact of the carefully designed architecture. In fact, some CNN-based models, such as U-Net++ and Double U-Net, achieve better performance than some Transformer-based models, such as Swim-UNet. This phenomenon indicates that both CNN and Transformer can extract intrinsic features in the dataset to a certain extent, which also indicates the feasibility of improving the model performance by combining the two branches proposed in the present invention. Specifically, MANet has a DIC of 95.82%, an IOU of 91.34%, a RAVD of 0.005, an ASSD of 1.82, and an MSD of 8.18 for liver segmentation on the LiTS dataset. The DIC of tumor segmentation is 55.59%, the IOU is 42.16%, the RAVD is 1.26, the ASSD is 9.01, and the MSD is 30.89.

[0126] Table 1 Evaluation indicators of the comparison methods in the LiTS dataset

[0127]

[0128]

[0129] In order to further evaluate the excellent segmentation performance, generalizability and robustness of MANet proposed in this article, experiments were continued on 3DIRCADb. Table 2 lists the segmentation results of advanced state-of-the-art models and MANet, which are also measured by DIC, IOU, RAVD, ASSD and MSD. It can be seen from the table that MANet of the present invention outperforms the existing CNN method and Transformer-based method in almost all evaluation indicators in both liver segmentation and tumor segmentation tasks. Specifically, in the liver segmentation task, DIC of MANet is Compared with the second-ranked hiformer, MANet's DIC is improved by 0.91% and IOU is improved by 0.15%. In addition, other indicators are also ranked at the top. In the tumor segmentation task, MANet's DIC is 59.85% and IOU is 44.27%, which is 0.3% higher than DIC and 0.24% higher than IOU. These data also prove that the combination of local convolution operation and long-range attention dependence is positive for liver and tumor segmentation. MANet has excellent segmentation ability and can accurately segment the liver and tumor areas of abdominal CT.

[0130] also, Figure 6 The figure shows a visual comparison of the liver and tumor segmentation results of MANet and other models. The first and second columns represent the original abdominal CT sections and the true labels, respectively. The qualitative segmentation results of the proposed MANet network and the comparison network are explained as follows. It is obvious that the predicted mask generated by MANet is very similar to the boundary and shape of the true label value. The segmentation result of MANet is finer than that of the CNN and Transformer based methods, and there is no situation where the background is identified as the target area. This is due to the extraction of detailed texture information by the Gabor attention convolution proposed in the present invention, and the dual encoding branch, which extracts rich local and global feature information.

[0131] Table 23 Evaluation index results of the comparison network in DIRCADb

[0132]

[0133]

[0134] The MANet network proposed in this paper consists of a dual-branch encoder, so various combinations of encoders and decoders are designed, and experiments are carried out on the LiTS dataset. In addition, in the ablation study, each comparison model is run in the same data enhancement and computing environment for fair comparison. This paper uses the U-Net model as the benchmark model to gradually add modules. The experimental results are shown in Table 3. For simplicity, U is the benchmark model U-Net, S is the proposed deep split module (DSM), T is the encoding path of Transfomer, D is the dual-branch interaction module proposed in this paper, L represents the deep loss supervision mechanism, MA represents the MANet proposed in this paper, and - represents removal.

[0135] In order to analyze the effectiveness of DSM, the standard convolution in U-Net is replaced by DSM. From the comparison of the experimental data results of U and U+S in Table 3, it can be seen that the performance of U-Net on the LiTS dataset is obviously inferior to that of U+G. In terms of liver segmentation, the DIC and IOU of U+G are improved by 0.05% and 0.04% respectively. In terms of tumor segmentation, the IOU is improved by 0.24%. In addition, Figure 7 The U+G method has a more refined edge segmentation and clearer texture.

[0136] In order to analyze the impact of the dual-branch CNNformer, the single encoder of U-Net is improved into a dual encoding branch, one branch uses DSM and the other branch uses CNNformer, and element-level addition is directly performed at the end of encoding. In addition, the S branch is jump-connected to the decoder. From the experimental data results of U+S and U+S+T in Table 3, it can be seen that U+S+T has significantly higher performance than U+S in both the liver segmentation task and the tumor segmentation task. In addition, it can be clearly seen from the figure that U+S is prone to mispredict the normal liver area as the tumor area, and is also prone to miss the recognition of the tumor.

[0137] In order to verify the rationality of the dual-branch interaction module, the single encoder of U-Net is improved into a dual encoding branch, one branch uses DSM and the other branch uses CNNformer, and the dual-branch interaction module is used at the end of the encoding stage. In addition, the S branch is jump-connected with the decoder. The experimental results are shown in U+S+T and U+S+T+D in Table 3. In the liver segmentation task, U+S+T lags behind U+S+T+D by 0.12% and 0.07% in DIC and IOU, respectively. In the tumor segmentation task, DIC and IOU lag behind by 0.28% and 0.29%, respectively. At the same time, it can be seen from the figure that the lack of the DIM module will lead to incorrect segmentation of the liver area, which is consistent with the numerical results of the evaluation indicators.

[0138] In order to analyze the effectiveness of the deep loss supervision mechanism, the present invention removes the deep loss supervision mechanism in MANet and only calculates the loss function for the true value of the segmentation result. The experimental results are shown in MA-L and MA in Table 3. Whether in segmenting the liver or the tumor, MA is better than MA-L in all evaluation indicators. Figure 6 It can also be seen that the visualization of the segmentation results of MA is slightly better than that of MA-L, which also verifies that the deep loss supervision mechanism has a guiding role in the segmentation results.

[0139] In addition, the above models, which are built to verify the effectiveness of the dual-branch interaction module and the deep loss supervision mechanism, can prove the influence of the jump connection between the decoder and the encoder of the model of the present invention. From the U+G+T+D and MA-L data results in Table 3, it can be seen that the result of MA-L is better than U+S+T+D, which also shows that the jump connection of MANet enables the feature vector in the decoding stage to have high-resolution texture and deep edge information, thereby improving the segmentation accuracy.

[0140] Table 3 Evaluation index data of ablation experiment

[0141]

[0142] The computational complexity of the model is an important aspect of evaluating the performance of the model, and the computational complexity also affects the effect of the model in actual clinical diagnosis. The present invention also conducts experiments on LiTS2017, using the well-known floating-point operations (GFLOPs), inference speed and number of parameters to quantify the computational complexity. The specific experimental results are shown in Table 4. It can be seen from the table that the MANet model achieves excellent segmentation results at the expense of the efficiency of the model. The need for a large number of parameters is a major disadvantage of MANet, and the main reason is that there are two branches in the encoding path of the network. Specifically, whether it is the number of parameters, Flops and inference time, MANet is better than DeepLabv3+ and TransUNet, but not better than HiFormer, which ranks second in segmentation accuracy. In terms of parameters, compared with other models, the improvement of MANet is significant, the segmentation accuracy is greatly improved, the parameters of the model are less or equivalent, and the inference speed is improved.

[0143] Table 4 Complexity analysis results

[0144]

[0145] In order to more intuitively demonstrate that the proposed MANet has excellent learning feature capabilities, the 3D confusion matrix of the two data sets is used to show the segmentation effect. The diagonal bar graph shows the classification accuracy of the corresponding category of each pixel, and the others are the proportion of misclassification. Figure 8 It is not difficult to see that MANet segmentation can well segment the liver and tumor in liver CT. This is due to the dual-branch deconstruction proposed in the present invention. The local branch extracts rich detail information in liver CT well, and classifies the pixels corresponding to each category as correctly as possible; the global dependency branch extracts global features, grasps the global category, and avoids the background from being mis-segmented as liver and tumor. At the same time, the dual-branch interaction module fully integrates local features and context-dependent features to avoid the ambiguity of semantic features and improve the accuracy of segmentation.

[0146] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0147] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A liver and tumor segmentation method based on mixed attention, wherein the method is implemented by a liver and tumor segmentation network based on mixed attention, and is characterized in that: include: For the input image I∈R w×h×c , respectively using the local feature extraction branch based on the CNN structure and the long-range dependency feature extraction branch based on the Transformer structure to extract pixel-level detail features and object-level global features. Among them, c i ∈{64,128,256,512}, i∈{1,2,3,4}; Subtract the global features extracted from the remote dependency branch from the local features extracted from the local feature branch in the corresponding stage to obtain high-resolution edge contour features, and then perform a maximum pooling downsampling operation; The local features and long-range dependency features finally extracted by the two branches are respectively input into the dual-branch interaction module to aggregate the local features and long-range dependency features; The aggregated feature vector containing local features and global features is input into the decoder. At the same time, four skip connections are used to fuse the features of the encoding path and the decoder. The first and second skip connections obtain shallow information containing high-resolution texture features from the local feature branch and fuse them into the decoder. The third and fourth skip connections pass the deep edge information in the global dependency branch to the decoder, so that the feature vector has high-resolution texture and deep edge information, and the preliminary liver and tumor segmentation results are obtained. Based on each decoding block, the corresponding classifier is added to generate multi-scale segmentation maps at different stages to calculate the loss function and obtain the final liver and tumor segmentation results.

2. The liver and tumor segmentation method based on mixed attention according to claim 1, characterized in that: The local feature extraction branch based on the CNN structure consists of a deep split module and integrates a fusion attention mechanism on different branches to capture the interactive information of cross-features. Specifically, given an input feature I∈R H×W×C , where H, W, C are the height, width and number of channels respectively. The feature I is evenly divided into K feature vectors along the channel dimension. Subsequently, each feature vector is divided into R individuals, that is, the feature map I is divided into G = KR groups in total. The K feature vectors are input into K parallel basic branches, each of which is composed of R split branches with fused attention. Each split branch consists of two convolutional layers, followed by BN and a ReLU activation function, and outputs a feature map with a height of H / 4 and a width of W / 4; The fused attention is integrated into the split branch to connect the R feature maps, denoted as {U1,U2,...,U R }, firstly, the R feature maps are fused by element-wise summation operation, and a global pooling operation is applied to obtain the global context information s: After the global pooling layer, two consecutive fully connected layers with nonlinear activation functions are applied to predict the R attention weights {a1,a2,...,a R }, and then use each attention weight pair {U1,U2,...,U R } Perform weighted fusion and output a basic feature map V: Accordingly, we get K basic feature maps, {V1,V2,...,V K } indicates that the K basic feature maps are then connected piece by piece along the channel dimension, and a 1×1 convolution layer is used to produce the final split convolution block output C. The network pays more attention to the correlation between the adjacent channels of the two variables. where f 1×1 is a convolutional layer with a kernel size of 1×1, n is the number of channels of V, is the nth cut of k feature maps, with hyperparameters K and R being 2.

3. The liver and tumor segmentation method based on mixed attention according to claim 1, characterized in that: The Transformer-based long-range dependency feature extraction branch is composed of CNNformer, and the CNN structure composed of Conv_Block in CNNformer is used as a token mixer to optimize MLP into a convolution structure that is more suitable for image processing. The specific mathematical expression is: cf(x)=mlp(cnn(x)) cnn(x)=Conv 3×3 (Conv 3×3 (x))+x mlp(x cnn )=Conv 1×1 (ReLU(Conv 1×1 (x cnn )))+x cnn Among them, Conv 3×3 and Conv 1×1 represents the convolutional layers with kernels of 3×3 and 1×1, cnn(·) and mlp(·) represent CNN and MLP, ReLU(·) is the ReLU activation function, and x cnn is the output of cnn(·).

4. The liver and tumor segmentation method based on mixed attention according to claim 1, characterized in that: The specific process of the dual-branch interaction module is as follows: first, the local feature extraction branch C i Use spatial attention as a spatial filter to remove feature noise and enhance local details to get the output For remote dependency branch T i Perform channel attention mechanism separately to promote the output of global information At the same time, for C i and T i Interactive feature extraction i , Secondly, the position encoding of the same dimension is embedded into two branches, and two cross-interaction attention modules are used to perform attention interaction fusion by receiving the corresponding branch feature map and the other branch feature map to generate two outputs c f and t f Finally, the participating feature c f , t f Connect with the interactive features and pass through the residual block to output the result feature fi. The specific formula is as follows: f i =Resideual(c f ,t f ,b i ) Where |⊙| is the dot product and Conv is a 3x3 convolutional layer.

5. The liver and tumor segmentation method based on mixed attention according to claim 1, characterized in that: During the training of the network, the loss function is the cross entropy loss function L BCE and dice loss L Dice Composition, among which R gt and R seg Represent the true label value and the predicted result respectively, L BCE and L Dice They are defined as, L BCE (R gt ,R seg )=-(1-R gt )log(1-R seg )-R gt logR seg In the above formula, ε is a constant. BCE and L Dice The loss function in the experiment can be obtained, L seg (R gt ,R seg )=λL BCE +L Dice Among them, λ is 0.5, and the network model applies L to each decoder in the second stage decoding process. seg , establish a deep supervision loss function, where M GT Represents the true segmentation result map, so the loss of the i-th decoder is can be defined as, In the above formula, is defined as in, The factor is 2 i-1 Downsampling, in addition, L seg It is also applied to the output of the first stage decoding process. The loss of the first stage is defined as, L c (M GT ,M c )=L seg (M GT ,M c ) The overall loss function during the training process of the model is defined as,

Images