Multi-scale feature fusion triple branch network method for multi-organ segmentation

Abstract

The invention discloses a multi-scale feature fusion triple branch network method for multi-organ segmentation, and the method comprises the following steps: extracting local features through a CNN, and extracting global features through Transform; the features extracted by the CNN and the Transfomer are processed by using space attention and channel attention respectively, the processed features are fused, and output features are obtained through a feature processing module; the method comprises the following steps: setting a mask mechanism Smart Mask, improving a fixed mask mode in a Swin Transform, and capturing a long-distance dependency relationship; according to the method, global features in a medical image are learned through parallel Vision Mama, features obtained after CNN and Transform fusion are guided, and a segmented image is obtained. The CNN, Transform and Mamba encoders are combined, so that efficient medical image segmentation is realized. And the introduced parallel Vision Mama layer increases a small number of parameters and enhances the global context capture capability at the same time, so that the performance and efficiency of the model are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a method of a multi-scale feature fusion triple branch network for multi-organ segmentation. Background Art

[0002] Health issues are now one of the most concerned topics in the world. Disease diagnosis is usually time-consuming and laborious, requiring doctors to invest a lot of time and energy to analyze and distinguish pathological images. However, manual methods cannot guarantee the accuracy and efficiency of segmentation. Therefore, the technology that can automatically and accurately classify medical images at the pixel level to locate lesions can not only save manpower and time, but also has great significance for disease diagnosis and clinical medicine. Especially in the clinical applications of multi-organ segmentation and heart segmentation, medical image segmentation technology has become an important auxiliary diagnostic tool. There are great differences between each organ or tissue in the multi-organ, and the heart is an organ that is constantly working, and its shape will also change during movement. Timely and accurate segmentation is of vital help to diagnosis and provides efficient and reliable technical support for medical staff.

[0003] The application of deep learning technology in the field of medical image segmentation research has been quite extensive. Among them, FCN was successfully applied to the field of image segmentation for the first time. After convolutional neural networks (CNN) were widely used in the field of medical image segmentation. Although CNN effectively improves the ability to express local features in medical image segmentation, it still has limitations and is difficult to capture long-distance dependencies and global context information. Transformers were originally designed for inter-sequence prediction, mainly using the multi-head self-attention (MSA) mechanism to effectively establish long-range dependencies and capture global context information. However, as the reasoning context becomes longer, the memory usage of models such as Transformer based on the self-attention mechanism increases, and the computational complexity brought about significantly increases the number of model parameters, resulting in huge computing power consumption, slowing down the reasoning speed, and prolonging the training time. To solve this problem, SSM is integrated into the end neural network, and Mamba is proposed to perform well in reasoning speed, with more efficient throughput than Transformer, and linear scaling in sequence length. Vision Mamba and VMamba first applied Mamba to computer vision, selectively scanning image blocks in two and four directions, respectively, achieving performance comparable to Swin Transformer, while significantly reducing time complexity.

[0004] Although Mamba addresses the limitations of the Transformer, without paying attention to the entire context, the output quality of Mamba is poor, especially on recall-related tasks. Summary of the invention

[0005] The purpose of the present invention is to provide a method of multi-scale feature fusion triple branch network for multi-organ segmentation in response to the above-mentioned problems, and propose a three-branch architecture to effectively save computing resources without losing effectiveness.

[0006] The technical solution of the present invention is as follows:

[0007] A method for multi-scale feature fusion triple branch network for multi-organ segmentation, comprising the following steps:

[0008] Extract local features through CNN and global features through Transformer;

[0009] Use spatial and channel attention to process the features extracted by CNN and Transformer respectively, fuse the processed features, and obtain the output features through the feature processing module;

[0010] Set up the mask mechanism Smart Mask to improve the fixed mask mode in Swin Transformer to capture long-distance dependencies;

[0011] By learning the global features in medical images in parallel with Vision Mamba, the features after fusion of CNN and Transformer are guided to obtain the segmented image.

[0012] Furthermore, the extracting of local features by CNN includes the following steps:

[0013] Reshape the channels of the original image and keep the feature information in the local feature list;

[0014] Two convolutional layers with a kernel size of 3x3, batch normalization, and ReLU activation function are set at each level to store feature information in a local feature list;

[0015] Downsampling is performed through a 2x2 maximum pooling layer;

[0016] The local features extracted by at least four layers of CNN encoders are fused with the features extracted by Transformer. Specifically, the local features extracted by Transformer are fused using the features from the first block. The second block The third block and the fourth block The output of is fused with the feature map from the Transformer encoder branch.

[0017] Furthermore, the CNN encoder includes the following calculation formula:

[0018] F = Conv(Bn(ReLU(L i-1 ))),

[0019] F′=Conv(Bn(ReLU(F))),

[0020] L i =MAP(F′),

[0021] Among them, Conv represents the convolution operation, Bn represents batch normalization, ReLU represents the activation function, and MAP represents the maximum pooling operation.

[0022] Furthermore, the Swin Transformer module includes window-based multi-head self-attention and shifted window-based multi-head self-attention. The window-based multi-head self-attention module is used to capture the interaction between pixels within the window, focusing on local information; the shifted window-based multi-head self-attention ensures cross-window connections by shifting windows.

[0023] Furthermore, the mask mechanism Smart Mask acts on the self-attention matrix; the calculation formula of the self-attention module is:

[0024]

[0025] Among them, Q, K and represents the query, key, and value matrix from input X, M 2 represents the number of patches in the window, d is the dimension of Q or K, It has the same size as the self-attention matrix and generates different masks in the channels.

[0026] Furthermore, the Swin Transformer module includes:

[0027] The input in the i-th layer Block is represented as Z i-1 , the intermediate output of Block is expressed as and The output of the i-th layer is represented as Z i+1 , the mathematical formula of this module is as follows:

[0028]

[0029]

[0030]

[0031]

[0032] Among them, LN is a regularization of LayerNorm, W-MSA represents the window self-attention mechanism, SSW-MSA represents the smart shift window self-attention mechanism, and MLP is the multi-layer perceptron layer.

[0033] Furthermore, the features extracted by the CNN and Transformer are fused through a multi-scale feature fusion module. The multi-scale feature fusion module includes a spatial attention mechanism, a channel attention mechanism, and a reverse residual multi-layer perceptron. The multi-scale feature fusion module includes the following calculation formula:

[0034]

[0035]

[0036] F″ i =Avgpool(Conv 1×1 (F i-1 )),

[0037] F′ i =Conv 1×1 (Concat[G i , L i , F″ i ]),

[0038] F i =IRMLP(LN(Concat[G′ i , L′ i , F′ i ])),

[0039] in, represents element-wise multiplication, G′ i Generated by the combination of channel attention, L′ i Generated by the combination of spatial attention, F″ i Generated by the previous downsampling of the MSFF block, F′ i It is the result of the fusion of global and local features and the previous stage. Finally, F′ i , G′ i , L′ i The eigenvector, eigenvector and eigenvector are connected through the IRMLP module to generate the eigenvector F i .

[0040] Furthermore, the multi-scale feature fusion module includes the following steps:

[0041] The global features are fed into the channel attention mechanism, and the interdependencies between channel mappings are exploited to improve the feature representation of specific semantics:

[0042] GA(x)=σ(MLP(AvgPool(x))+MLP(MaxPool(x)))

[0043] Where σ is the Sigmoid function;

[0044] The local features are fed into the spatial attention mechanism to selectively enhance important regions and suppress irrelevant information, improving the preservation of important local details:

[0045] LA(x)=σ(f 7×7 (Concat[AvgPool(x),MaxPool(x)]))

[0046] Where σ is the Sigmoid function, f 7×7 It is a convolution operation with a convolution kernel size of 7×7;

[0047] The inverse residual multilayer perceptron consists of a 3×3 depthwise convolution with residual and 2 linear transformation layers, with nonlinear transformations performed by GELU in the expanded channel dimension:

[0048] IRMLP(x)=Conv 1×1 (Conv 1×1 (Conv depth3×3 (x)+x)).

[0049] Furthermore, the parallel Vision Mamba divides the features into four sub-channels for processing, and finally merges them to keep the total number of channels unchanged; specifically, the following steps are included:

[0050] The feature X with the number of channels C is divided into channels with the number of channels C / 4 after passing through the LayerNorm layer. Each feature is fed into the VSS block, and the output is residual cascaded and the adjustment factor is adjusted to optimize the long-range spatial information acquisition ability. Finally, the four features are combined into the feature X of channel number C through the concat operation. out , respectively output through LayerNorm and Projection operations; the calculation formula is:

[0051]

[0052] Out=Pro[LN(X out )],

[0053] Where LN is LayerNorm, Sp is the Split operation, VSS is the VSS Block operation, θ is the adjustment factor of the residual connection, Cat is the concat operation, and Pro is the Projection operation.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. A method of multi-scale feature fusion triple branch network for multi-organ segmentation. For the two branches extracting global and local features, an effective feature fusion method (MSFF) is proposed by cross-fusion of CNN and improved Transformer transform domain features. First, the features extracted by CNN and Transformer are processed using spatial and channel attention respectively to enhance their representation ability. Secondly, the features of the two are fused, and finally the output features are obtained through the feature processing module. For global features, a novel mask mechanism Smart Mask is introduced to improve the fixed mask mode in Swin Transformer to better capture long-distance dependencies.

[0056] 2. A method of multi-scale feature fusion triple branch network for multi-organ segmentation, setting up a parallel VisionMamba method to learn global features in medical images with a small number of parameters, thereby guiding the features after CNN and Transformer fusion to further improve the segmentation ability;

[0057] 3. A method of multi-scale feature fusion triple branch network for multi-organ segmentation, testing its feasibility and effectiveness in two publicly available datasets (Synapse and ACDC). Extensive experimental results show that the TB-Net of this application has a stronger competitive performance advantage than other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is the overall structure diagram of the TB-Net network of this application.

[0059] Figure 2 This is a schematic diagram of the CNN module of this application.

[0060] Figure 3 This is a schematic diagram of the structure of the Smart Transformer of this application.

[0061] Figure 4 This is the PVMamba layer architecture diagram of this application.

[0062] Figure 5 This is the architecture diagram of the multi-scale feature fusion module (MSFF) of this application.

[0063] Figure 6 This is the IRMLP module architecture diagram of this application.

[0064] Figure 7 Comparison of average DSC (%) based on different methods on the Synapse dataset.

[0065] Figure 8 Visual comparison of different methods on the Synapse dataset.

[0066] Fig. 9 Visual comparison of different methods on the ACDC dataset. DETAILED DESCRIPTION

[0067] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0068] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0069] See also Figure 1-9 , a method of multi-scale feature fusion triple branch network for multi-organ segmentation, e.g. Figure 1 As shown, the following steps are included:

[0070] Extract local features through CNN and global features through Transformer;

[0071] CNN modules such as Figure 2 As shown. Extracting local features through CNN includes the following steps:

[0072] Reshape the channels of the original image and keep the feature information in the local feature list;

[0073] Two convolutional layers with a kernel size of 3x3, batch normalization, and ReLU activation function are set at each level to store feature information in a local feature list;

[0074] Downsampling is performed through a 2x2 maximum pooling layer;

[0075] The local features extracted by at least four layers of CNN encoders are fused with the features extracted by Transformer. Specifically, the local features extracted by Transformer are fused using the features from the first block. The second block The third block and the fourth block The output of is fused with the feature map from the Transformer encoder branch.

[0076] The CNN encoder includes the following calculation formula:

[0077] F = Conv(Bn(ReLU(L i-1 ))),

[0078] F′=Conv(Bn(ReLU(F))),

[0079] L i =MAP(F′),

[0080] Among them, Conv represents the convolution operation, Bn represents batch normalization, ReLU represents the activation function, and MAP represents the maximum pooling operation.

[0081] Use spatial and channel attention to process the features extracted by CNN and Transformer respectively, fuse the processed features, and obtain the output features through the feature processing module;

[0082] The features extracted by CNN and Transformer are fused through a multi-scale feature fusion module. The multi-scale feature fusion module includes a spatial attention mechanism, a channel attention mechanism, and a reverse residual multi-layer perceptron. The multi-scale feature fusion module includes the following calculation formula:

[0083]

[0084]

[0085] F″ i =Avgpool(Conv 1×1 (F i-1 )),

[0086] F′ i =Conv 1×1 (Concat[G i , L i , F″ i ]),

[0087] F i =IRMLP(LN(Concat[G′ i , L′ i , F′ i ])),

[0088] in, represents element-wise multiplication, G′ i Generated by the combination of channel attention, L′i Generated by the combination of spatial attention, F″ i Generated by the previous downsampling of the MSFF block, F′ i It is the result of the fusion of global and local features and the previous stage. Finally, F′ i , G′ i , L′ i The eigenvector, eigenvector and eigenvector are connected through the IRMLP module to generate the eigenvector F i .

[0089] like Figure 5 As shown, the multi-scale feature fusion module includes the following steps:

[0090] The global features are fed into the channel attention mechanism, and the interdependencies between channel mappings are exploited to improve the feature representation of specific semantics:

[0091] GA(x)=σ(MLP(AvgPool(x))+MLP(MaxPool(x))),

[0092] Where σ is the Sigmoid function;

[0093] The local features are fed into the spatial attention mechanism to selectively enhance important regions and suppress irrelevant information, improving the preservation of important local details:

[0094] LA(x)=σ(f 7×7 (Concat[AvgPool(x),MaxPool(x)]))

[0095] Where σ is the Sigmoid function, f 7×7 It is a convolution operation with a convolution kernel size of 7×7;

[0096] like Figure 6 As shown in Figure 1, the inverse residual multilayer perceptron consists of a 3×3 depthwise convolution with residual and 2 linear transformation layers, with nonlinear transformation performed by GELU in the expanded channel dimension:

[0097] IRMLP(x)=Conv 1×1 (Conv 1×1 (Conv depth3×3 (x)+x)).

[0098] Set up the mask mechanism Smart Mask to improve the fixed mask mode in Swin Transformer to capture long-distance dependencies;

[0099] The Swin Transformer module includes window-based multi-head self-attention and shift-window-based multi-head self-attention. The window-based multi-head self-attention module is used to capture the interaction between pixels within the window, focusing on local information; the shift-window-based multi-head self-attention ensures cross-window connections by shifting windows.

[0100] The masking mechanism Smart Mask acts on the self-attention matrix; the calculation formula of the self-attention module is:

[0101]

[0102] Among them, Q, K and represents the query, key, and value matrix from input X, M 2 represents the number of patches in the window, d is the dimension of Q or K, With the same size as the self-attention matrix, different masks are generated in channels. Compared with the fixed mask in Swin Transformer, the smart mask module creates different masks in different channels to fully exploit the advantages of channel features. By maintaining accurate contextual information and eliminating meaningless interactions, the ability of SSW-MSA to refine long-range dependencies is improved. Figure 3 The content in the bottom black box represents the details of SSM-MSA. In addition, based on the local and global information provided by W-MSA and SSW-MSA, the proposed Smart SwinTransformer is able to focus on channel-level feature linking to achieve effective fusion of multi-scale information, rather than SwinTransformer. Figure 3 The top shows the structure of the Smart Swin Transformer Block.

[0103] The Swin Transformer module includes:

[0104] The input in the i-th layer Block is represented as Z i-1 , the intermediate output of Block is expressed as and The output of the i-th layer is represented as Z i+1 , the mathematical formula of this module is as follows:

[0105]

[0106]

[0107]

[0108]

[0109] Among them, LN is a regularization of LayerNorm, W-MSA represents the window self-attention mechanism, SSW-MSA represents the smart shift window self-attention mechanism, and MLP is the multi-layer perceptron layer.

[0110] By learning the global features in medical images in parallel with Vision Mamba, the features after fusion of CNN and Transformer are guided to obtain the segmented image.

[0111] Parallel Vision Mamba divides the features into four sub-channels for processing and then merges them to keep the total number of channels unchanged; the specific steps include:

[0112] The feature X with the number of channels C is divided into channels with the number of channels C / 4 after passing through the LayerNorm layer. Each feature is fed into the VSS block, and the output is subjected to residual concatenation and adjustment factors to optimize the ability to obtain long-range spatial information. Finally, the four features are combined into the feature X of channel number C through the conca operation. out , respectively output through LayerNorm and Projection operations; the calculation formula is:

[0113]

[0114] Out=Pro[LN(X out )],

[0115] Where LN is LayerNorm, Sp is the Split operation, VSS is the VSS Block operation, θ is the adjustment factor of the residual connection, Cat is the concat operation, and Pro is the Projection operation.

[0116] The processing characteristics of parallel Vision Mamba are adopted and fully utilized, while ensuring that the total number of channels processed remains constant, minimizing parameters while establishing global contextual relationships. By introducing parallel Vision Mamba operations, the final features are further fused with the previously extracted and fused CNN and Transformer features, so that the model performance is further improved, while only a small amount of parameters are added, which improves the competitiveness of the model performance. Figure 4 shown.

[0117] Experiment and analysis:

[0118] Dataset:

[0119] The Synapse multi-organ segmentation dataset includes 30 clinical abdominal CT scans processed from MICCAI 2015. The dataset contains a total of 3779 CT images, and the labels of each image contain information about eight different organs, such as aorta, gallbladder, left kidney, right kidney, liver, pancreas, spleen, and stomach. According to the division used in TransUNet and SwinUNet, the training and test sets are divided based on 18 training cases (including 2212 axial slices) and 12 test cases. The final training set consists of 2212 slice images from 18 groups of cases, while the test set consists of the remaining 12 groups of case images.

[0120] The Automatic Cardiac Diagnosis Challenge (ACDC) dataset collects MRI scans from 100 different patients. For each case, a series of electromagnetic resonance MRI images, including short-axis sections of the heart, were obtained during apnea. Similar to the Synapse dataset, each case contains multiple slices, and after data processing, the total number of slices is 1902. The label of each slice image includes information about three organs: the left ventricle (LV), the right ventricle (RV), and the myocardium (MYO). This application divides the dataset into 80 training samples and 20 test samples.

[0121] The dataset is provided from a public website in an unprocessed compressed nii.gz format, which cannot be directly read by the network. This application requires processing the dataset and converting it into the '.npz' format required for the training set and the '.h5' format required for the test set. In the '.npz' format, the case images need to be traversed by the number of channels and saved in two dimensions: image height (H) and image width (W). In the '.h5' format, they need to be saved in three dimensions: number of channels (C), image height (H), and image width (W). It is worth noting that in this article, the image size is required to be 224x224 in order to compare with the benchmark experimental results.

[0122] Experimental details:

[0123] This application was trained on a device equipped with an NVIDIA RTX 3090 GPU and 24GB of video memory, and experiments were conducted using the PyTorch 1.13.O deep learning framework. Data augmentation methods include random flipping, affine transformation, etc. In different medical image segmentation tasks, the hyperparameter settings during training, such as learning rate (lr), learning rate strategy (lr-scheduler), batch size, optimizer selection, and loss function definition, are as follows:

[0124] Synapse: lr=0.0001; lr-scheduler=CosineAnnealing; optimizer=Adam-W (weight_decay=0.0001); Loss=0.4×Focal loss (Cross-Entropy Loss)+0.6×DICE (Dice Loss), warm-up=10, batchsize=16;

[0125] ACDC: lr=0.0001; lr-scheduler=CosineAnnealing; optimizer=AdamW(weight_decay=0.0001); Loss=0.4×CE(Cross-Entropy Loss)+0.6×DICE(DiceLoss), warm-up=10, batchsize=16;

[0126] Loss function:

[0127] In medical image analysis tasks, the background area usually occupies most of the image area. Simply misclassifying foreground information as background may lead to a higher accuracy score, but this method cannot truly reflect the actual effect of segmentation. Therefore, a single loss function cannot fully evaluate the performance of medical image segmentation models. In order to meet this challenge, multiple loss functions are usually combined to improve the performance of the model, especially when dealing with class imbalance and multi-scale features.

[0128] Cross-Entropy Loss (CE Loss) is a common loss function that is mainly used to measure the difference between two probability distributions, especially in multi-category classification problems. The goal of cross-entropy loss is to minimize the difference between the true label distribution and the model predicted probability distribution. Its formula is as follows:

[0129] CE(P, Q)=-∑i P(i)log(Q(i)),

[0130] Among them, P(i) represents the probability that the true label belongs to the i-th class, and Q(i) represents the probability predicted by the model. The smaller the cross entropy loss, the closer the predicted distribution of the model is to the true distribution. This loss function is very effective in dealing with multi-category medical image segmentation, but for pixels with class imbalance or difficult classification, using cross entropy loss alone may not be enough to capture the complex relationship between pixels.

[0131] To solve this problem, the Dice loss function was introduced. It is a loss function based on the Dice coefficient, which is used to measure the overlap between the predicted results and the true values. The higher the Dice coefficient, the higher the similarity between the predicted results and the true segmentation. The calculation formula of Dice loss is as follows:

[0132]

[0133] Where P and Q represent the predicted value and the true value, respectively. Dice loss directly optimizes the overlapped part of the foreground region and can better handle the class imbalance problem. However, it ignores the imbalance between the foreground and the background and only focuses on the overlap ratio between the predicted and true values.

[0134] In order to make up for the shortcomings of Dice loss, Focal Loss was introduced and was initially used in the field of target detection. Focal loss focuses on processing pixels that are difficult to classify, especially those in boundary areas or difficult to classify pixels, by adding an exponential weight term to the cross entropy loss. The formula for Focal loss is:

[0135] Focal_Loss(p t )=-(1-p t )γlog(p t ),

[0136] Among them, pt is the predicted probability of the current pixel, and γ is a hyperparameter that controls the penalty intensity for difficult-to-classify pixels, which is usually set to 2. Focal loss can optimize the imbalance of positive and negative samples, reduce the bias of the model for simple samples, and strengthen the learning of difficult samples.

[0137] In the experiment, in order to cope with the characteristics of different data sets, this application uses different combinations of loss functions. For example, on the Synapse data set, a combination of Dice loss and Focal loss (Dice+Focal Loss) is used. This combination can better capture pixel features at boundaries or multi-scales. On the ACDC data set, a combination of Dice loss and cross entropy loss (Dice+CE) is used because cross entropy performs better on this data set. In these two combinations, the weight parameters of the loss function are set to 0.6 and 0.4, respectively. The experimental results show that this setting can achieve the best segmentation effect.

[0138] In general, the loss function design in medical image segmentation tasks needs to be adjusted according to the characteristics of the specific data set. A single loss function cannot fully evaluate the performance of the model. The combination of Dice, cross entropy and Focal loss can effectively deal with challenges such as class imbalance, complex pixel classification and multi-scale feature recognition. Through this combination of multiple loss functions, the model can more accurately capture different pixel features, thereby improving the performance of medical image segmentation.

[0139] Evaluation Metrics:

[0140] Dice similarity coefficient, DSC (%). Dice similarity coefficient is a set similarity measure, usually used to calculate the similarity of two samples, with a value range of 0 to 1. When the segmentation result is the best, the value is 1, and when the segmentation result is the worst, the value is 0. DSC (%) is used to measure the similarity between the true value and the prediction of two sample sets. The calculation of DSC (%) is shown in formula (1).

[0141]

[0142] Where P and Q represent the group truth and prediction, respectively. 95% Hausdorff distance, HD95(mm). Hausdorff distance is defined as the distance between any two sets in the metric space. This paper uses HD95(mm) as the evaluation. The HD95(mm) metric is defined as the 95% quantized value of the distance between the ground truth and the prediction, which is calculated as the maximum surface distance. The calculation method of HD95(mm) is as follows.

[0143] HD95=max k95％ [d(P,Q),d(Q,P)],

[0144] Where P and Q represent the group truth and prediction respectively.

[0145] Comparison method:

[0146] This application summarizes the baseline comparisons in medical image segmentation experiments based on CNN, Transformer, and Mamba.

[0147] · CNN-based structures. (1) UNet. (2) UNet++ optimizes the skip connection of UNet. (3) The skip structure of Att-UNet is the same as that of UNet. On this basis, an Attention Gate is added to suppress irrelevant areas in the input image and highlight the significant features of specific local areas.

[0148] Transformer-based structure. (1) SwinUNet combines the global feature learning capability of Swin Transformer and the excellent segmentation performance of U-Net. (2) DAE-Former designs a new Transformer architecture and redesigns the skip connection path to maintain model efficiency while improving performance.

[0149] (3) MISSFormer redesigns the feedforward network ReMixFFN and proposes a ReMixedTransformer context bridge to extract the global correlation and local context correlation in the multi-scale features generated by the Transformer encoder. (4) SSTrans-Net improves the fixed mask in Swin Transformer and proposes a new mask mechanism Smart Mask to obtain rich channel feature representation.

[0150] · Cascaded hybrid structure of CNN and Transformer. (1) TransUNet encodes tokenized image patches from convolutional neural network (CNN) feature maps as input sequences to extract global context. The decoder upsamples the encoded features and then combines them with high-resolution CNN feature maps to achieve accurate localization. (2) GPA-UNet designs a new attention mechanism method based on TransUNet: GPA. And uses it and Transformer as a joint encoder. (3) E-TUNet designs an enhanced Transformer as an encoder by introducing EMSA and DynaMixerMLP. The G-LMLP block with global-local spatial interaction capability is introduced to effectively combine CNN and MLP to capture local and global correlations. (4) BiTransNet introduces the BiFusion module in the decoding stage to achieve effective global and local feature fusion by supporting feature integration from various modules. In addition, a multi-layer loss (ML) strategy is introduced to supervise the learning process of each decoding layer and optimize the context features using global and local fusion at different scales.

[0151] Hierarchical hybrid structure of CNN and Transformer. (1) H2Former integrates the local information of CNN, multi-scale channel attention features and long-range features of Transformer into a unified block, combining their advantages and enhancing the feature representation ability of the model. At the same time, a lightweight multi-scale channel attention (MSCA) branch is proposed to target segmentation tasks of different sizes and shapes.

[0152] (2) TransCASCADE / PVTCASCADE proposes a new hierarchical cascaded attention-based decoder (CASCADE) for two-dimensional medical image segmentation, which fully utilizes the multi-level feature representation of visual transformers while learning multi-scale and multi-resolution spatial representations. (3) PFD-UNet uses the Transformer based on PVTv2 as the main encoder to capture global information, the auxiliary encoder (CNN encoder) to capture local information, and further enhances the local and global feature representations through the fast Fourier convolution residual (FFCR) module. (4) CTC-Net designs a dual encoding path, namely CNN and Transformer encoders, to produce complementary features, and proposes an effective feature complementary module (FCM) to effectively combine the functions of CNN and Transformer.

[0153] ·Mamba-based structure. (1) VM-UNet is the first medical image segmentation model built on a pure SSM model. (2) MedMamba introduces a new hybrid basic block SS-Conv-SSM, which integrates convolutional layers for extracting local features and the ability of SSM to capture long-range dependencies, aiming to effectively model medical images of different image modalities. (3) HC-Mamba proposes the HC-Mamba module. By combining dilated convolution and depthwise separable convolution and introducing them into the Mamba structure, HC-Mamba is able to process large-scale medical image data at a much lower computational cost while maintaining a high level of performance.

[0154] The model of this application has three branches: CNN, Transformer and Mamba, which extract local and global features respectively. For global features, the mask mechanism Smart Mask is introduced to enhance the ability to obtain collocation information. Secondly, an effective feature fusion method (MSFF) is used to cross-fuse CNN and improve the features of the Transformer transform domain. Finally, a lightweight parallel Vision Mamba method is introduced to supplement the information lost in the fusion process of CNN and Transformer.

[0155] Experimental results:

[0156] Tables 1 and 2 show the experimental results obtained from the Synapse and ACDC datasets. The comparative experimental results on the Synapse dataset are shown in Table 1. From the overall situation, the Transformer-based method is currently in the leading position. The TB-Net of this application was compared with 20 methods, including pure Transformer-based models, pure CNN-based models, and models combining CNN and Transformer. The TB-UNet of this application is significantly better than other methods, including CNN-based UNet, TransUNet based on a mixture of CNN and Transformer, SwinUNet based on pure Transformer, etc. The average DSC (%) and HD95 (mm) of TB-UNet reached 84.85% and 13.11mm on the Synapse dataset, obtaining the current optimal indicators. TB-UNet obtained the best segmentation morphology on the pancreas. As shown in Figure 7 , Figure 8 and Fig. 9 As shown, TB-Net performs well in edge information segmentation and organ integrity segmentation. As for Gallbladder, Kidney (R), Kidney (L), Spleen and Stomach, the TB-Net of this application obtained the second highest DSC among the twenty methods. This is because the Smart Transformer and PVM introduced in this application improve the ability of global information modeling, and at the same time realize the effective fusion of global and local features in the MSFF module, which improves the expression ability of information. The experimental results show that the TB-Net of this application achieves accurate segmentation of large organs, long and narrow, such as kidneys and pancreas. The long-range dependency of Transformer and Mamba enables the network of this application to segment large organs such as kidneys and liver well. The local details of CNN also enable the network of this application to improve the segmentation accuracy of small organs, such as gallbladder. In terms of HD metric, the model of this application also won the second place, far ahead of most models. Synapse is a dataset of multi-organ CT images, while ACDC is a dataset of cardiac MR images. In order to illustrate that the model of this application can successfully process images with different modalities and targets, this application selected 16 advanced methods for comparison on the ACDC dataset. The comparative test is shown in Table 2. The average DSC (%) of TB-UNet reaches 91.80%, which is higher than the currently available advanced models. The visualization results of the comparative test are shown in the figure. It can be seen that TB-UNet exhibits a more sensitive segmentation performance, and the segmentation effect of the left ventricle is also better than other methods. From the above experimental data and visualization, it can be seen that TB-UNet has greater potential than other competing methods in multi-organ segmentation tasks.

[0157] Table 1 Synapse experiment (average dice similarity coefficient, unit is %)

[0158]

[0159]

[0160]

[0161] Table 2 ACDC experiment (average dice similarity coefficient, unit is %)

[0162]

[0163] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for multi-scale feature fusion triple branch network for multi-organ segmentation, characterized in that: The following steps are involved: Extract local features through CNN and global features through Transformer; Use spatial and channel attention to process the features extracted by CNN and Transformer respectively, fuse the processed features, and obtain the output features through the feature processing module; Set up the mask mechanism Smart Mask to improve the fixed mask mode in Swin Transformer to capture long-distance dependencies; By learning the global features in medical images in parallel with Vision Mamba, the features after fusion of CNN and Transformer are guided to obtain the segmented image.

2. The method of a multi-scale feature fusion triple branch network for multi-organ segmentation according to claim 1, characterized in that: The extraction of local features by CNN comprises the following steps: Reshape the channels of the original image and keep the feature information in the local feature list; Two convolutional layers with a kernel size of 3x3, batch normalization, and ReLU activation function are set at each level to store feature information in a local feature list; Downsampling is performed through a 2x2 maximum pooling layer; The local features extracted by at least four layers of CNN encoders are fused with the features extracted by Transformer. Specifically, the local features extracted by Transformer are fused using the features from the first block. The second block The third block and the fourth block The output of is fused with the feature map from the Transformer encoder branch.

3. The method of a multi-scale feature fusion triple branch network for multi-organ segmentation according to claim 2, characterized in that: The CNN encoder includes the following calculation formula: F=Conv(Bn(ReLU(L i-1 ))), F′=Conv(Bn(ReLU(F))), L i =MAP(F′), Among them, Conv represents the convolution operation, Bn represents batch normalization, ReLU represents the activation function, and MAP represents the maximum pooling operation.

4. The method of a multi-scale feature fusion triple branch network for multi-organ segmentation according to claim 1, characterized in that: The Swin Transformer module includes window-based multi-head self-attention and shifted window-based multi-head self-attention. The window-based multi-head self-attention module is used to capture the interaction between pixels in the window, focusing on local information; the shifted window-based multi-head self-attention ensures cross-window connections by shifting windows.

5. The method of a multi-scale feature fusion triple branch network for multi-organ segmentation according to claim 1 or 4, characterized in that: The mask mechanism Smart Mask acts on the self-attention matrix; the calculation formula of the self-attention module is: Among them, Q, K and represents the query, key, and value matrix from input X, M 2 represents the number of patches in the window, d is the dimension of Q or K, It has the same size as the self-attention matrix and generates different masks in the channels.

6. The method of a multi-scale feature fusion triple branch network for multi-organ segmentation according to claim 5, characterized in that: The Swin Transformer module includes: The input in the i-th layer Block is represented as Z i-1 , the intermediate output of Block is expressed as and The output of the i-th layer is represented as Z i+1 , the mathematical formula of this module is as follows: Among them, LN is a regularization of LayerNorm, W-MSA represents the window self-attention mechanism, SSW-MSA represents the smart shift window self-attention mechanism, and MLP is the multi-layer perceptron layer.

7. The method of claim 1 for multi-organ segmentation using a multi-scale feature fusion triple branch network, characterized in that: The features extracted by the CNN and Transformer are fused through a multi-scale feature fusion module. The multi-scale feature fusion module includes a spatial attention mechanism, a channel attention mechanism, and a reverse residual multi-layer perceptron. The multi-scale feature fusion module includes the following calculation formula: F″ i =Avgpool(Conv 1×1 (F i-1 )), F′ i =Conv 1×1 (Concat[G i ,L i ,F″ i ]), F i =IRMLP(LN(Concat[G′ i ,L′ i ,F′ i ])), in, represents element-wise multiplication, G′ i Generated by the combination of channel attention, L′ i Generated by the combination of spatial attention, F″ i Generated by the previous downsampling of the MSFF block, F′ i It is the result of the fusion of global and local features and the previous stage. Finally, F′ i , G′ i , L′ i The eigenvector, eigenvector and eigenvector are connected through the IRMLP module to generate the eigenvector F i .

8. The method of a multi-scale feature fusion triple branch network for multi-organ segmentation according to claim 7, characterized in that: The multi-scale feature fusion module comprises the following steps: The global features are fed into the channel attention mechanism, and the interdependence between channel mappings is exploited to improve the semantic feature representation: GA(x)=σ(MLP(AvgPool(x))+MLP(MaxPool(x))) Where σ is the Sigmoid function; The local features are fed into the spatial attention mechanism to selectively enhance important regions and suppress irrelevant information, improving the preservation of important local details: LA(x)=σ(f 7×7 (Concat[AvgPool(x),MaxPool(x)])) Where σ is the Sigmoid function, f 7×7 It is a convolution operation with a convolution kernel size of 7×7; The inverse residual multilayer perceptron consists of a 3×3 depthwise convolution with residual and 2 linear transformation layers, with nonlinear transformations performed by GELU in the expanded channel dimension: IRMLP(x)=COnv 1×1 (COnv 1×1 (COnv depth3×3 (x)+x))。 9. The method of a multi-scale feature fusion triple branch network for multi-organ segmentation according to claim 1, characterized in that: The parallel Vision Mamba divides the features into four sub-channels for processing, and finally merges them to keep the total number of channels unchanged; specifically, the following steps are included: The feature X with the number of channels C is divided into channels with the number of channels C / 4 after passing through the LayerNorm layer. Each feature is fed into the VSS block, and the output is residual cascaded and the adjustment factor is adjusted to optimize the long-range spatial information acquisition ability. Finally, the four features are combined into the feature X of channel number C through the concat operation. out , respectively output through LayerNorm and Projection operations; the calculation formula is: Out=Pro[LN(X out )], Where LN is LayerNorm, Sp is the Split operation, VSS is the VSS Block operation, θ is the adjustment factor of the residual connection, Cat is the concat operation, and Pro is the Projection operation.

Images