A method of multi-scale feature fusion triple branch network for multi-organ segmentation

Abstract

The application discloses a kind of multi-scale feature fusion triple branch network for multi-organ segmentation, comprising the following steps: extracting local features by CNN, extracting global features by Transformer;Respectively using spatial and channel attention to process the features extracted by CNN and Transfomer, fuse the processed features, obtain the output features by feature processing module;Set mask mechanism Smart Mask, improve the fixed mask mode in Swin Transformer, capture long-distance dependence;Learn global features in medical images through parallel Vision Mamba, guide the features after the fusion of CNN and Transformer, and obtain a segmented image.Combined with CNN, Transformer and Mamba encoder to realize efficient medical image segmentation.The introduced parallel Vision Mamba layer increases a small number of parameters while enhancing the global context capture ability, significantly improving the performance and efficiency of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation, and in particular to a method of a multi-scale feature fusion triple branch network for multi-organ segmentation. BACKGROUND

[0002] Health problems are one of the most important issues of global concern today. Disease diagnosis is usually time-consuming and labor-intensive, requiring doctors to invest a lot of time and effort to analyze and distinguish pathological images. However, the manual method cannot guarantee the accuracy and efficiency of segmentation. Therefore, a technology that can automatically and accurately classify medical images at the pixel level to locate lesions not only saves manpower and time, but also has great significance for disease diagnosis and clinical medicine. Especially in the clinical application of multi-organ segmentation and heart segmentation, medical image segmentation technology has become an important auxiliary diagnostic tool. There are great differences between each organ or tissue in multi-organ, and the heart is an organ that never stops working, and its shape will change during movement. Timely and accurate segmentation is of great help to diagnosis and provides efficient and reliable technical support for medical personnel.

[0003] Deep learning technology has been widely used in the field of medical image segmentation research. Among them, FCN was first successfully applied to the field of image segmentation. After the wide application of convolutional neural network (CNN) in the field of medical image segmentation. Although CNN effectively improves the expression ability of local features in medical image segmentation, there are still limitations, making it difficult to capture long-distance dependencies and global context information. Transformers were originally designed for inter-sequence prediction, mainly using multi-head self-attention (MSA) mechanism to effectively establish long-distance dependencies and capture global context information. However, as the reasoning context becomes longer, the memory occupancy of the Transformer model based on the self-attention mechanism increases significantly, bringing a significant increase in model parameters, resulting in huge computational power consumption, slowing down the reasoning speed and prolonging the training time. In order to solve this problem, SSM is integrated into the end-to-end neural network, and Mamba is proposed, which performs well in reasoning speed and has higher throughput than Transformer, and performs linear scaling on sequence length. Vision Mamba and VMamba first apply Mamba to computer vision, respectively in two and four directions to selectively scan image blocks, achieving performance comparable to Swin Transformer while significantly reducing time complexity.

[0004] Although Mamba solves the limitations of Transformer, the output quality of Mamba is poor without focusing on the entire context, especially in recall-related tasks. SUMMARY

[0005] The present application aims at the above-mentioned problems existing at present, and provides a multi-scale feature fusion triple branch network method for multi-organ segmentation.

[0006] The technical scheme of the present application is as follows:

[0007] A multi-scale feature fusion triple branch network method for multi-organ segmentation comprises the following steps:

[0008] Local features are extracted by CNN, and global features are extracted by Transformer;

[0009] The features extracted by CNN and Transformer are processed respectively using spatial and channel attention, the processed features are fused, and the output features are obtained through a feature processing module;

[0010] A mask mechanism Smart Mask is set to improve the fixed mask mode in Swin Transformer and capture long-distance dependency relationships;

[0011] Global features in medical images are learned through parallel Vision Mamba, the features after the fusion of CNN and Transformer are guided, and a segmented image is obtained.

[0012] Further, the extraction of local features by CNN comprises the following steps:

[0013] The channels of the original image are reshaped, and the feature information is retained in the local feature list;

[0014] Two convolution layers with a kernel size of 3x3, batch normalization and ReLU activation function are set at each level, and the feature information is stored in the local feature list;

[0015] Downsampling operation is performed through a 2x2 max-pooling layer;

[0016] The local features extracted by the CNN encoder with at least four layers are fused with the features extracted by the Transformer, specifically: the outputs from the first block The second block The third block And the fourth block are fused with the feature maps from the Transformer encoder branch.

[0017] Further, the CNN encoder comprises the following calculation formula:

[0018] F = Conv(Bn(ReLU(L i-1 ))),

[0019] F' = Conv(Bn(ReLU(F))),

[0020] L i = MAP(F' ),

[0021] where Conv denotes convolution operation, Bn denotes batch normalization, ReLU denotes activation function, and MAP denotes maximum pooling operation.

[0022] Further, the Swin Transformer module includes a window-based multi-head self-attention and a shift window-based multi-head self-attention, the window-based multi-head self-attention module is used to capture the interaction between pixels within the window, and focuses on local information; the shift window-based multi-head self-attention ensures the connection across windows by means of shift window.

[0023] Further, the Smart Mask mechanism acts on the self-attention matrix; the calculation formula of the self-attention module is as follows:

[0024]

[0025] where Q, K and represent the query, key and value matrices from the input X, M 2 represents the number of patches within the window, d is the dimension of Q or K, has the same size as the self-attention matrix, and generates different masks in the channel.

[0026] Further, the Swin Transformer module includes:

[0027] The input in the i-th layer Block is represented as Z i-1 , the intermediate output of the Block is represented as and The output of the i-th layer is represented as Z i+1 , and the mathematical formula of the module is as follows:

[0028]

[0029]

[0030]

[0031]

[0032] where LN is a LayerNorm kind of regularization, W-MSA denotes a window self-attention mechanism, SSW-MSA denotes a smart shift window self-attention mechanism, and MLP is a multi-layer perceptron layer.

[0033] Further, the features extracted by the CNN and the Transfomer are fused by a multi-scale feature fusion module, the multi-scale feature fusion module comprising a spatial attention mechanism, a channel attention mechanism and a reverse residual multi-layer perceptron, the multi-scale feature fusion module comprising the following calculation formula:

[0034]

[0035]

[0036] F″ i =Avgpool(Conv 1×1 (F i-1 )),

[0037] F′ i =Conv 1×1 (Concat[G i ,L i ,F″ i ]),

[0038] F i =IRMLP(LN(Concat[G′ i ,L′ i ,F′ i ])),

[0039] wherein, denotes an element-wise multiplication, G′ i is generated by the combination of channel attention, L′ i is generated by the combination of spatial attention, F″ i is generated by the down-sampling of the previous stage of the MSFF block, F′ i is the result of global and local features and the fusion of the previous stage, finally, F′ i , G′ i , L′ i are connected by the IRMLP module to generate a feature vector F i .

[0040] Further, the multi-scale feature fusion module comprises the following steps:

[0041] The global features are input into the channel attention mechanism to improve the feature representation of specific semantics by utilizing the mutual dependence between the channel mappings:

[0042] GA(x) = σ(MLP(AvgPool(x)) + MLP(MaxPool(x)))

[0043] where σ is the Sigmoid function;

[0044] The local features are input into the spatial attention mechanism to selectively enhance important regions and suppress irrelevant information, improving the preservation of important local details:

[0045] LA(x) = σ(f 7×7 (Concat[AvgPool(x), MaxPool(x)]))

[0046] where σ is the Sigmoid function, f 7×7 is a convolution operation with a kernel size of 7x7;

[0047] The inverse residual multi-layer perceptron includes a 3x3 deep convolution with a residual and two linear transformation layers, which perform nonlinear transformation through GELU in the expanded channel dimension:

[0048] IRMLP(x) = Conv 1×1 (Conv 1×1 (Conv depth3×3 (x) + x)).

[0049] Further, the parallel Vision Mamba divides the features into four sub-channels for processing, and then combines them at the end to maintain the total number of channels; specifically including the following steps:

[0050] The feature X with channel number C is divided into four features with channel number C / 4 after passing through the LayeNorm layer. Each feature is fed into the VSS block, and the output is residual concatenated and adjusted to optimize the ability to obtain remote spatial information, and finally the four features are combined into a feature X out with channel number C through the concat operation, and are output through LayerNorm and Projection operations respectively; the calculation formula is:

[0051]

[0052] Out = Pro[LN(X out )],

[0053] where LN is LayerNorm, Sp is Split operation, VSS is VSS Block operation, θ is the adjustment factor of residual connection, Cat is concat operation, and Pro is Projection operation.

[0054] The beneficial effects of the present application compared with the existing technology are:

[0055] 1. A method of a multi-scale feature fusion triple branch network for multi-organ segmentation, for the two branches of extracting global and local features, an effective feature fusion method (MSFF) is proposed by cross-fusing the features of CNN and improved Transformer transformation domain, first, the features extracted by CNN and Transfomer are processed using spatial and channel attention respectively to enhance their representation ability, second, the features of the two are fused, and finally the output features are obtained through a feature processing module, for global features, a novel mask mechanism Smart Mask is introduced to improve the fixed mask pattern in Swin Transformer, so as to better capture long-distance dependencies;

[0056] 2. A method of a multi-scale feature fusion triple branch network for multi-organ segmentation, a parallel VisionMamba method is set up to learn global features in medical images through a small number of parameters, thereby guiding the features after the fusion of CNN and Transformer, and further improving the segmentation capability;

[0057] 3. A method of a multi-scale feature fusion triple branch network for multi-organ segmentation, the feasibility and effectiveness of which are tested in two publicly available datasets (Synapse and ACDC). A large number of experimental results show that the TB-Net of the present application has a stronger competitive performance advantage than other methods. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 It is a general structure diagram of the TB-Net network of the present application.

[0059] Figure 2 It is a schematic diagram of the CNN module of the present application.

[0060] Figure 3 It is a structural schematic diagram of the Smart Transformer of the present application.

[0061] Figure 4 It is a PVMamba layer architecture diagram of the present application.

[0062] Figure 5 It is an architecture diagram of the multi-scale feature fusion module (MSFF) of the present application.

[0063] Figure 6 It is an IRMLP module architecture diagram of the present application.

[0064] Figure 7 It is a comparison of average DSC (%) based on different methods on the Synapse dataset.

[0065] Figure 8 Visualization comparison of different methods on the Synapse dataset.

[0066] Figure 9 Visualization comparison of different methods on the ACDC dataset. DETAILED DESCRIPTION

[0067] It should be noted that the relational terms herein, such as first and second, and the like, are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0068] The features and nature of the present application will become more apparent from the detailed description set forth below, taken in conjunction with the accompanying drawings.

[0069] Referring to Figures 1-9 , a method for multi-organ segmentation of a multi-scale feature fusion triple branch network, as shown in Figure 1 , comprises the following steps:

[0070] Local features are extracted by a CNN, and global features are extracted by a Transformer;

[0071] The CNN module is as shown in Figure 2 . The extraction of local features by the CNN comprises the following steps:

[0072] The channels of the original image are reshaped, and the feature information is retained in the local feature list;

[0073] Two convolutional layers with a kernel size of 3x3, batch normalization and ReLU activation functions are set at each level, and the feature information is stored in the local feature list;

[0074] Downsampling is performed by a 2x2 max-pooling layer;

[0075] The local features extracted by the CNN encoder with at least four layers are fused with the features extracted by the Transformer, specifically: the first block The second block The third block and the fourth block The output is fused with the feature map from the Transformer encoder branch.

[0076] The CNN encoder includes the following calculation formula:

[0077] F = Conv(Bn(ReLU(L i-1 ))),

[0078] F' = Conv(Bn(ReLU(F))),

[0079] L i = MAP(F'),

[0080] where Conv represents convolution operation, Bn represents batch normalization, ReLU represents activation function, and MAP represents maximum pooling operation.

[0081] The features extracted by the CNN and the Transformer are processed using spatial and channel attention respectively, and the processed features are fused to obtain the output features through the feature processing module;

[0082] The features extracted by the CNN and the Transformer are fused through a multi-scale feature fusion module, which includes spatial attention mechanism, channel attention mechanism and reverse residual multilayer perceptron, and the multi-scale feature fusion module includes the following calculation formula:

[0083]

[0084]

[0085] F" i = Avgpool(Conv 1×1 (F i-1 )),

[0086] F' i = Conv 1×1 (Concat[G i , L i , F" i ]),

[0087] F i = IRMLP(LN(Concat[G' i , L' i , F' i ])),

[0088] where, represents element-wise multiplication, G' i is generated by the combination of channel attention, and L'i generated by the combination of spatial attention, F" i generated by the down-sampling of the previous stage of MSFF block, F' i is the result of global and local features and the fusion of the previous stage, finally, F' i , G' i , L' i Connect the feature vectors, feature vectors and feature vectors through the IRMLP module to generate the feature vector F i .

[0089] As shown in Figure 5 , the multi-scale feature fusion module includes the following steps:

[0090] Input the global feature into the channel attention mechanism to improve the feature representation of specific semantics by utilizing the mutual dependence between channel mappings:

[0091] GA(x)=σ(MLP(AvgPool(x))+MLP(MaxPool(x))),

[0092] where σ is the Sigmoid function;

[0093] Input the local feature into the spatial attention mechanism to selectively enhance important regions and suppress irrelevant information, and improve the preservation of important local details:

[0094] LA(x)=σ(f 7×7 (Concat[AvgPool(x),MaxPool(x)]))

[0095] where σ is the Sigmoid function, and f 7×7 is a convolution operation with a convolution kernel size of 7×7;

[0096] As shown in Figure 6 , the reverse residual multilayer perceptron includes a 3×3 deep convolution with a residual and two linear transformation layers, which performs nonlinear transformation through GELU in the expanded channel dimension:

[0097] IRMLP(x)=Conv 1×1 (Conv 1×1 (Conv depth3×3 (x)+x))).

[0098] Set the mask mechanism Smart Mask to improve the fixed mask pattern in Swin Transformer to capture long-range dependencies;

[0099] The Swin Transformer module includes a window-based multi-head self-attention and a shift window-based multi-head self-attention, the window-based multi-head self-attention module is used to capture the interaction between pixels within a window, and focuses on local information; the shift window-based multi-head self-attention ensures the connection across windows by means of a shift window.

[0100] The mask mechanism Smart Mask acts on the self-attention matrix; the calculation formula of the self-attention module is:

[0101]

[0102] Where Q, K and represent the query, key and value matrices from the input X, M 2 represents the number of patches within a window, d is the dimension of Q or K, has the same size as the self-attention matrix, and different masks are generated in the channel. Compared with the fixed mask in Swin Transformer, the smart mask module creates different masks in different channels to take full advantage of the channel function. By maintaining accurate contextual information and eliminating meaningless interactions, the ability of SSW-MSA to refine long-range dependencies is improved. Figure 3 The content in the bottom black frame represents the details of SSM-MSA. In addition, based on the local and global information provided by W-MSA and SSW-MSA, the proposed Smart Swin Transformer can focus on channel-level feature linking to achieve effective fusion of multi-scale information, rather than Swin Transformer. Figure 3 The top shows the structure of the Smart Swin Transformer Block.

[0103] The Swin Transformer module includes:

[0104] The input in the i-th layer Block is represented as Z i-1 , the intermediate output of the Block is represented as and The output of the i-th layer is represented as Z i+1 The mathematical formula of the module is as follows:

[0105]

[0106]

[0107]

[0108]

[0109] Where LN is LayerNorm, W-MSA represents window self-attention mechanism, SSW-MSA represents smart shift window self-attention mechanism, and MLP is a multi-layer perceptron layer.

[0110] The global features in the medical image are learned by parallel Vision Mamba, and the features after the fusion of CNN and Transformer are guided to obtain a segmentation image.

[0111] Parallel Vision Mamba divides the features into four sub-channels for processing, and then combines them to keep the total number of channels unchanged; the specific steps include the following:

[0112] The features X with a channel number of C are divided into four features with a channel number of C / 4 by the LayerNorm layer. Each feature is fed into the VSS block, and the output is residual concatenated and adjusted to optimize the ability of long-range spatial information acquisition, and finally the four features are combined into a feature X with a channel number of C by the conca operation. out The output is output by LayerNorm and Projection operation respectively; the calculation formula is:

[0113]

[0114] Out=Pro[LN(X out )],

[0115] Where LN is LayerNorm, Sp is Split operation, VSS is VSS Block operation, θ is the adjustment factor of residual connection, Cat is concat operation, and Pro is Projection operation.

[0116] The processing characteristics of parallel Vision Mamba are fully utilized, and the total number of processing channels is kept constant, which establishes the global context relationship while minimizing the parameters. Through the introduced parallel Vision Mamba operation, the finally obtained features are further fused with the previously extracted and fused CNN and Transformer features, which further improves the model performance while only paying a small increase in parameters, improves the competitiveness of model performance. PVMamba is shown as follows. Figure 4

[0117] Experiment and analysis:

[0118] Dataset:

[0119] ​Synapse multi-organ segmentation dataset includes 30 clinical abdominal CT scans processed on MICCAI 2015. The dataset contains a total of 3779 CT images, and the label of each image contains information of eight different organs, such as aorta, gallbladder, left kidney, right kidney, liver, pancreas, spleen and stomach. According to the division used in TransUNet and SwinUNet, the division of training set and test set is based on 18 training cases (including 2212 axial slices) and 12 test cases. The final training set consists of 2212 slice images from 18 cases, and the test set consists of the remaining 12 case images.

[0120] The automatic cardiac diagnosis challenge dataset (ACDC) collects MRI scans from 100 different patients, each case obtains a series of electromagnetic resonance MRI images during apnea, including short-axis sections of the heart. Similar to the Synapse dataset, each case contains multiple slices, and after data processing, the total number of slices is 1902. The label of each slice image includes information of three organs, such as left ventricle (LV), right ventricle (RV) and myocardium (MYO). The dataset is divided into 80 training samples and 20 test samples in this application.

[0121] The dataset is provided in the uncompressed nii.gz format from the public website, which cannot be directly read by the network. This application needs to process the dataset and convert it into the '.npz' format required for the training set and the '.h5' format required for the test set. In the '.npz' format, the case image needs to be traversed by channel number, and saved in two dimensions of image height (H) and image width (W). In the '.h5' format, it needs to be saved in three dimensions of channel number (C), image height (H) and image width (W). It is worth noting that in this paper, the image size is required to be 224x224 in order to compare with the benchmark experimental results.

[0122] Experimental details:

[0123] This application is trained on a device equipped with NVIDIA RTX 3090 GPU with 24 GB video memory, and experiments are conducted using PyTorch 1.13.0 deep learning framework. Data augmentation methods include random flipping, affine transformation, etc. In different medical image segmentation tasks, the hyperparameters during training, such as learning rate (lr), learning rate strategy (lr-scheduler), batch size, optimizer selection and loss function definition, are as follows:

[0124] Synapse: lr = 0.0001; lr-scheduler = CosineAnnealing; optimizer = Adam-W (weight_decay = 0.0001); Loss = 0.4 x Focal loss (Cross-Entropy Loss) + 0.6 x DICE (Dice Loss), warm-up = 10, batchsize = 16;

[0125] ACDC: lr = 0.0001; lr-scheduler = CosineAnnealing; optimizer = AdamW (weight_decay = 0.0001); Loss = 0.4 x CE (Cross-Entropy Loss) + 0.6 x DICE (Dice Loss), warm-up = 10, batchsize = 16;

[0126] Loss function:

[0127] In medical image analysis tasks, the background region usually occupies a large part of the image area. Simply misclassifying foreground information as background can lead to high accuracy scores, but this approach cannot truly reflect the actual segmentation effect. Therefore, a single loss function cannot comprehensively evaluate the performance of a medical image segmentation model. To address this challenge, multiple loss functions are often combined to improve model performance, especially when dealing with class imbalance and multi-scale features.

[0128] Cross-Entropy Loss (CE Loss) is a common loss function used to measure the difference between two probability distributions, especially in multi-class classification problems. The goal of cross-entropy loss is to minimize the difference between the true label distribution and the model's predicted probability distribution. Its formula is as follows:

[0129] CE(P, Q) = -∑i P(i)log(Q(i)),

[0130] where P(i) represents the probability of the true label belonging to the ith class, and Q(i) represents the model's predicted probability. The smaller the cross-entropy loss, the closer the model's predicted distribution is to the true distribution. This loss function is very effective in handling multi-class medical image segmentation, but for class imbalance or difficult-to-classify pixels, using cross-entropy loss alone may not be sufficient to capture complex pixel-to-pixel relationships.

[0131] To address this issue, the Dice loss function is introduced, which is a loss function based on the Dice coefficient to measure the overlap between the predicted results and the true values. The higher the Dice coefficient, the higher the similarity between the predicted results and the true segmentation. The calculation formula of the Dice loss is as follows:

[0132]

[0133] where P and Q represent the predicted value and the true value respectively. The Dice loss directly optimizes the overlapping part of the foreground region, and can better handle the class imbalance problem. However, it ignores the imbalance between the foreground and the background, and only focuses on the overlap ratio of the predicted and true values.

[0134] To make up for the shortcomings of the Dice loss, the Focal loss function is introduced, which is originally used in the field of object detection. The Focal loss adds an exponential weight term to the cross-entropy loss, focusing on processing pixels with greater classification difficulty, especially boundary regions or difficult-to-classify pixels. The formula of the Focal loss is as follows:

[0135] Focal_Loss(p t )=-(1-p t )γlog(p t ),

[0136] where pt is the predicted probability of the current pixel, and γ is a hyperparameter that controls the punishment intensity of difficult-to-classify pixels, usually set to 2. The Focal loss can optimize the situation of positive and negative sample imbalance, reduce the bias of the model to simple samples, and strengthen the learning of difficult samples.

[0137] In the experiment, in order to cope with the characteristics of different data sets, different combinations of loss functions are used. For example, on the Synapse data set, the combination of Dice loss and Focal loss (Dice+Focal Loss) is used, which can better capture the pixel features at the boundary or multi-scale. While on the ACDC data set, the combination of Dice loss and cross-entropy loss (Dice+CE) is used, because the cross-entropy performs better on this data set. In these two combinations, the weight parameters of the loss functions are set to 0.6 and 0.4 respectively, and the experimental results show that this setting can achieve the best segmentation effect.

[0138] Overall, the loss function design in medical image segmentation tasks needs to be adjusted according to the characteristics of the specific data set. A single loss function cannot comprehensively evaluate the performance of the model. The combination of Dice, cross-entropy and Focal loss can effectively deal with class imbalance, complex pixel classification and multi-scale feature recognition challenges. Through the combination of such multi-loss functions, the model can more accurately capture different pixel features, thereby improving the performance of medical image segmentation.

[0139] Evaluation indicators:

[0140] Dice Similarity Coefficient, DSC (%). Dice Similarity Coefficient is a set similarity measure, commonly used to calculate the similarity of two samples, with a value range of 0-1, and a value of 1 when the segmentation result is best, and a value of 0 when the segmentation result is worst. DSC (%) is used to measure the similarity between the ground truth and the prediction of two sample sets. The calculation of DSC (%) is shown in equation (1).

[0141]

[0142] where P and Q represent the ground truth and prediction, respectively. 95% Hausdorff Distance, HD95 (mm). Hausdorff distance is defined as the distance between any two sets in a metric space. In this paper, HD95 (mm) is used as the evaluation. HD95 (mm) is defined as the 95% quantization value of the distance between the ground truth and the prediction, and the distance is calculated as the maximum surface distance. The calculation method of HD95 (mm) is shown below.

[0143] HD95 = max k95％ [d(P, Q), d(Q, P)],

[0144] where P and Q represent the ground truth and prediction, respectively.

[0145] Comparison method:

[0146] This application summarizes the baseline comparison in the medical image segmentation experiment based on CNN, Transformer and Mamba.

[0147] • CNN-based structure. (1) UNet. (2) UNet++ optimizes the skip connection of UNet. (3) The skip structure of Att-UNet is the same as UNet, and on this basis, an Attention Gate is added to suppress irrelevant areas in the input image, while highlighting the salient features of specific local areas.

[0148] • Transformer-based structure. (1) SwinUNet combines the global feature learning ability of Swin Transformer and the excellent segmentation performance of U-Net. (2) DAE-Former designs a new Transformer architecture and redesigns the skip connection path to maintain model efficiency while improving performance.

[0149] (3) MISSFormer redesigns the feedforward network ReMixFFN and proposes a ReMixed Transformer context bridge to extract the global correlation and local context correlation in the multi-scale features generated by the Transformer encoder. (4) SSTrans-Net improves the fixed mask in Swin Transformer and proposes a new mask mechanism Smart Mask to obtain rich channel feature representation.

[0150] • Cascade hybrid structure of CNN and Transformer. (1) TransUNet encodes labeled image patches from convolutional neural network (CNN) feature maps into input sequences to extract global context. The decoder upsamples the encoded features and then combines them with high-resolution CNN feature maps to achieve precise positioning. (2) GPA-UNet designs a new attention mechanism method: GPA based on TransUNet and uses it and Transformer as a joint encoder. (3) E-TUNet designs an enhanced Transformer as an encoder by introducing EMSA and DynaMixerMLP. It introduces a G-LMLP block with global-local spatial interaction capability to effectively combine CNN and MLP to capture local and global correlations. (4) BiTransNet introduces a BiFusion module in the decoding stage to achieve effective global and local feature fusion by supporting feature integration from each module. In addition, it introduces a multi-level loss (ML) strategy to supervise the learning process of each decoding layer and optimize the use of global and local fused context features at different scales.

[0151] • Hierarchical hybrid structure of CNN and Transformer. (1) H2Former integrates the local information of CNN, multi-scale channel attention features, and long-range features of Transformer into a unified block, while combining their advantages to enhance the model's feature representation capability. It also proposes a lightweight multi-scale channel attention (MSCA) branch to target segmentation tasks of different sizes and shapes.

[0152] (2)TransCASCADE / PVTCASCADE proposes a new hierarchical cascaded attention-based decoder (CASCADE) that learns multi-scale and multi-resolution spatial representations while fully exploiting the multi-level feature representation of the vision transformer.(3)PFD-UNet uses PVTv2-based Transformer as the main encoder to capture global information, auxiliary encoders (CNN encoders) to capture local information, and further enhances local and global feature representations through a fast Fourier convolution residual (FFCR) module.(4)CTC-Net designs a dual-encoding path, i.e., CNN and Transformer encoders, to produce complementary features, and proposes an effective feature complementarity module (FCM) to effectively combine the functions of CNN and Transformer.

[0153] • Mamba-based structure.(1) VM-UNet is the first medical image segmentation model based on a pure SSM model.(2) MedMamba introduces a new hybrid basic block SS-Conv-SSM that integrates convolution layers for extracting local features and the ability of SSM to capture long-range dependencies, aiming to effectively model medical images of different image modalities.(3) HC-Mamba proposes the HC-Mamba module, which combines dilated convolution and depthwise separable convolution and introduces it into the Mamba structure. HC-Mamba can process large-scale medical image data at much lower computational cost while maintaining high performance.

[0154] • The model of the present application has three branches of CNN, Transformer and Mamba, which extract local and global features respectively. For global features, the Smart Mask mechanism is introduced to enhance the ability to obtain the same collocation information. Secondly, an effective feature fusion method (MSFF) is used to cross-fuse the features of CNN and the improved Transformer transformation domain. Finally, a lightweight parallel Vision Mamba method is introduced to supplement the information lost in the fusion process of CNN and Transformer.

[0155] Experimental results:

[0156] Tables 1 and 2 present the experimental results obtained from the Synapse and ACDC datasets. The comparative experimental results on the Synapse dataset are shown in Table 1. Overall, Transformer-based methods are currently in the leading position. The TB-Net of this application was compared with 20 methods, including pure Transformer-based models, pure CNN-based models, and models combining CNN and Transformer. The TB-UNet of this application significantly outperforms other methods, including CNN-based UNet, TransUNet (a hybrid of CNN and Transformer), and SwinUNet (a pure Transformer-based model). TB-UNet achieves an average DSC (%) and HD95 (mm) of 84.85% and 13.11mm, respectively, on the Synapse dataset, achieving state-of-the-art metrics. TB-UNet achieved the best segmentation morphology on the pancreas. Figure 7 , Figure 8 and Figure 9 As shown, TB-Net performs well in edge information segmentation and organ integrity segmentation. Compared to Gallbladder, Kidney(R), Kidney(L), Spleen, and Stomach, our TB-Net achieves the second-highest DSC among the twenty methods. This is because the Smart Transformer and PVM introduced in our application improve the ability to model global information, while the effective fusion of global and local features in the MSFF module enhances information representation. Experimental results show that our TB-Net accurately segments large organs, both long and narrow, such as the kidneys and pancreas. The long-range dependencies of Transformer and Mamba enable our network to segment large organs well, such as the kidneys and liver. The local details of CNNs also improve the segmentation accuracy of our network for small organs, such as the gallbladder. In terms of HD metrics, our model also achieves second place, far surpassing most other models. Synapse is a dataset of multi-organ CT images, while ACDC is a dataset of cardiac MR images. To demonstrate that the proposed model can successfully handle images with different modalities and targets, 16 state-of-the-art methods were compared on the ACDC dataset. The comparison results are shown in Table 2. TB-UNet achieved an average DSC (%) of 91.80%, higher than currently available state-of-the-art models. The visualization results of the comparison tests are shown in the figure. It can be seen that TB-UNet exhibits more sensitive segmentation performance, and its segmentation effect on the left ventricle is also superior to other methods. In summary, the experimental data and visualizations show that TB-UNet has greater potential than other competing methods in multi-organ segmentation tasks.

[0157] Table 1 Synapse experiment (average dice similarity coefficient in %)

[0158]

[0159]

[0160]

[0161] Table 2 ACDC experiment (average dice similarity coefficient in %)

[0162]

[0163] The above embodiments only express the specific implementation of the present application, which is described in detail, but it cannot be understood as the limitation of the protection scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the technical scheme concept of the present application, a number of modifications and improvements can be made, which belong to the protection scope of the present application.

Claims

1. A method of a multi-scale feature fusion triple branch network for multi-organ segmentation, characterized in that, The method comprises the following steps: Local features are extracted by CNN, and global features are extracted by Transformer; The features extracted by CNN and Transformer are processed respectively using spatial and channel attention, the processed features are fused, and the output features are obtained through a feature processing module; A mask mechanism Smart Mask is set to improve the fixed mask mode in Swin Transformer and capture long-distance dependencies; Global features in medical images are learned through parallel Vision Mamba, and the features fused by CNN and Transformer are guided to obtain a segmented image; The Swin Transformer module comprises a window-based multi-head self-attention and a shift window-based multi-head self-attention, the window-based multi-head self-attention module is used to capture the interaction between pixels within a window and focuses on local information, and the shift window-based multi-head self-attention ensures the connection across windows through the shift window mode; The mask mechanism Smart Mask acts on the self-attention matrix; the calculation formula of the self-attention module is: ， wherein , and denote the query, key and value matrices from the input , denotes the number of patches within the window, is the dimension of or , has the same size as the self-attention matrix, generating different masks in the channels.

2. The method of claim 1, wherein, The local features extracted by CNN comprise the following steps: The channels of the original image are reshaped, and the feature information is stored in the local feature list; Two convolutional layers with a kernel size of 3x3, batch normalization and ReLU activation functions are set at each level, and the feature information is stored in the local feature list; A 2x2 maximum pooling layer is used for downsampling operation; The local features extracted by the CNN encoder of at least four layers are fused with the features extracted by the Transfomer, specifically: the outputs from the first block , the second block , the third block and the fourth block are fused with the feature maps from the Transformer encoder branch.

3. The method of claim 2, wherein, The CNN encoder comprises the following calculation formula: ， ， ， Wherein Conv represents convolution operation, Bn represents batch normalization, ReLU represents activation function, and MAP represents maximum pooling operation.

4. The method of claim 1, wherein, The Swin Transformer module comprises: No. The input in the layer block is represented as The intermediate output of a Block is represented as and , No. The output of the layer is represented as The mathematical formula for this module is as follows: ， ， ， ， wherein, is a LayerNorm kind of regularization, denotes a window self-attention mechanism, denotes a smart-shift window self-attention mechanism, is a multi-layer perceptron layer.

5. The method of claim 1, wherein, The features extracted by CNN and Transformer are fused through a multi-scale feature fusion module, the multi-scale feature fusion module comprises a spatial attention mechanism, a channel attention mechanism and a reverse residual multilayer perceptron, and the multi-scale feature fusion module comprises the following calculation formula: ， ， ， ， ， wherein, denotes element-wise multiplication, generated by the combination of channel attention, generated by the combination of spatial attention, generated by the downsampling of the previous stage of MSFF block, is the result of global and local features and the fusion of the previous stage, finally, , , The feature vector, the feature vector and the feature vector are connected through the IRMLP module to generate the feature vector .

6. The method of claim 5, wherein, The multi-scale feature fusion module comprises the following steps: The global features are input into the channel attention mechanism to improve the feature representation of semantics by utilizing the mutual dependence between channel mappings: wherein is a Sigmoid function; The local features are input into the spatial attention mechanism to selectively enhance important regions and suppress irrelevant information and improve the preservation of important local details: wherein is a Sigmoid function, is a convolution operation with a kernel size of 7x7; The reverse residual multilayer perceptron comprises a 3x3 deep convolution with a residual and two linear transformation layers, and the nonlinear transformation is performed on the expanded channel dimension through GELU: 。 7. The method of claim 1, wherein, The parallel Vision Mamba processes the features into four sub-channels, and then combines them to keep the total number of channels unchanged; specifically comprising the following steps: The feature X with the number of channels C is divided into the feature with the channel number C / 4 by the LayeNorm layer , , , The feature is fed into the VSS block, the output is residually concatenated and adjusted by a factor to optimize the remote spatial information acquisition capability, and finally the four features are combined into a feature with the channel number C by the concat operation , and are output by the LayerNorm and Projection operations respectively; the calculation formula is: ， ， ， ， where LN is LayerNorm, Sp is Split operation, VSS is VSS Block operation, is the adjustment factor for the residual connection, Cat is concat operation, Pro is Projection operation.

Images