A medical image segmentation method based on two-branch mixture model

Abstract

The present invention discloses a medical image segmentation method based on a dual-branch hybrid model, which belongs to the field of medical image segmentation and comprises the following steps: step 1, obtaining an existing data set from a public website, preprocessing the existing data set into an npz format file, and dividing the npz format file into a training set and a test set in proportion; step 2, constructing a dual-branch hybrid model; the dual-branch hybrid model comprises an encoder, a dual-branch cross-fusion module, and a decoder; step 3, training the model with a training set, and testing the model with a test set; step 4, obtaining a current image to be segmented, and performing image segmentation based on the trained dual-branch hybrid model. The present invention realizes high-precision segmentation of medical images.

Description

Technical Field

[0001] The invention belongs to the field of medical image segmentation, and in particular relates to a medical image segmentation method based on a dual-branch hybrid model. Background Art

[0002] Medical image segmentation is a key task in the field of computer vision and medical image processing. It is the process of dividing an image into several regions based on the similarities or differences between regions, so as to more accurately locate and analyze the anatomical structure or lesion area of ​​interest. This task is crucial for doctors to make accurate diagnoses, formulate effective treatment plans, and monitor diseases. However, in clinical practice, medical images contain noise and artifacts due to patient movement, aging of equipment, and blurred boundaries of organs and bones, making segmentation accuracy still face huge challenges.

[0003] Image segmentation methods can be roughly divided into two categories: traditional image segmentation methods and deep learning-based image segmentation methods. Traditional segmentation methods have advantages in computational efficiency and simplicity, but they are highly dependent on the expertise of doctors and radiologists when processing complex images and cannot meet the actual application needs of medical image segmentation. Compared with traditional segmentation methods, deep learning methods have the advantages of high accuracy, strong adaptability and strong scalability.

[0004] At present, there are two mainstream models of image segmentation methods based on deep learning: the segmentation model based on convolutional neural network (CNN) and the segmentation model based on Transformer, but both have their own limitations, as follows.

[0005] The Unet model with an encoder-decoder structure is a CNN-based segmentation model. The encoder uses a series of convolutional pooling operations to downsample and extract deep features of the image; the decoder performs upsampling to restore the lost detail information, and uses jump connections for feature fusion. This makes Unet one of the most popular segmentation methods. Subsequently, variants of the Unet network, such as Unet++, Unet3+, and Res-UNet, have emerged, all of which have improved the precision and accuracy of medical image segmentation. Although CNN-based methods have achieved good performance in medical image segmentation tasks, the convolution operation lacks the perception of global information due to the fixed convolution kernel size and limited receptive field, which has limitations in dense task prediction.

[0006] Transformer is a model that uses an attention mechanism. Transformer-based segmentation model: Influenced by Transformer in natural language processing tasks, Transformer is applied to computer vision tasks.

[0007] SwinUnet is the first pure Transformer model. This model uses the Swin Transformer module instead of the two-dimensional convolutional block as the basic unit for feature representation and interactive learning of long-range semantic information. It has good segmentation accuracy on abdominal multi-organ and heart segmentation datasets. The C2Former network redesigns the cross-convolution self-attention mechanism algorithm to model long-range and short-range dependencies and improve the understanding of semantic features. However, the Transformer-based segmentation model pays more attention to long-range dependencies and ignores local information. It is not as good as the specially designed CNN model in small organ segmentation and precise edge segmentation. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes a medical image segmentation method based on a dual-branch hybrid model. The dual-branch hybrid model HCMAformer is designed based on CNN and Transformer. The model consists of an encoder, a dual-branch cross-fusion module, and a decoder. The encoder consists of a multi-layer CNN and a multi-attention Transformer with two branches. The CNN branch is used to extract local information, and the multi-attention Transformer branch extracts global information while focusing on the association between different samples; the dual-branch cross-fusion module fuses the feature maps of different resolutions generated by the dual branches, thereby establishing an effective combination of coarse-grained features and fine-grained features; the decoder consists of CNN to generate the final segmentation map. This model achieves high-precision segmentation of medical images, surpassing the current SOTA model.

[0009] The technical solution of the present invention is as follows:

[0010] A medical image segmentation method based on a dual-branch mixture model comprises the following steps:

[0011] Step 1: Obtain an existing dataset from a public website, preprocess the existing dataset into an npz format file, and divide the npz format file into a training set and a test set in proportion;

[0012] Step 2: construct a dual-branch hybrid model; the dual-branch hybrid model includes an encoder, a dual-branch cross-fusion module, and a decoder;

[0013] Step 3: Use the training set to train the model, and use the test set to test the model;

[0014] Step 4: Obtain the current image to be segmented and perform image segmentation based on the trained dual-branch hybrid model.

[0015] Furthermore, in step 1, the existing data sets include an abdominal data set and a heart data set, and the existing data sets are all 3D nii format files. Each slice of the nii format file is traversed, and the existing data sets are saved as npz format files.

[0016] Furthermore, the specific process of step 2 is as follows:

[0017] Step 2.1, construct an encoder consisting of a convolutional neural network branch and a multi-attention Transformer branch; the convolutional neural network branch includes three levels of convolution modules; the multi-attention Transformer branch includes three levels of multi-attention Transformer modules; the output of the convolution module is input to the multi-attention Transformer module through a skip connection;

[0018] Step 2.2, construct a dual-branch cross fusion module to fuse images of different resolutions from two branches;

[0019] Step 2.3. Construct a decoder consisting of a retaining convolution layer, a first upsampling convolution layer, a second upsampling convolution layer, and a segmentation head.

[0020] Furthermore, the specific process of step 2.1 is as follows:

[0021] Step 2.1.1, build a convolutional neural network branch;

[0022] According to the existing convolutional neural network model, ResNet50 is selected as the backbone of the convolutional neural network model; the convolutional module branches of the three levels are composed of various stages of RestNet50;

[0023] Step 2.1.2, build a jump connection structure;

[0024] The jump connection structure includes 1×1 convolution, 3×3 deep convolution layer, SE attention mechanism, and 1×1 convolution connected in sequence; the output of the convolution module passes through 1×1 convolution, 3×3 deep convolution layer, SE attention mechanism, and 1×1 convolution in sequence, and then enters the multi-attention Transformer module;

[0025] Step 2.1.3, introduce the cross-covariance attention mechanism and external attention mechanism on the basis of the SwinTransformer module to construct a multi-attention Transformer module; use the multi-level multi-attention Transformer module as the Transformer branch;

[0026] Each multi-attention Transformer module contains two cascaded sub-modules, namely the first sub-module and the second sub-module; when constructing a three-level multi-attention Transformer module, each level contains 2, 2, and 6 multi-attention Transformer modules respectively, and each level is connected through a patch merging layer as a Transformer branch; the first sub-module and the second sub-module have the same normalization layer and multi-layer perceptron layer, and the output of the first sub-module is the input of the second sub-module.

[0027] Furthermore, in step 2.1.1, the working process of the convolutional neural network branch is as follows: the input original image passes through three convolution modules in sequence to obtain three feature maps with different resolutions, and the resolutions are , , ,in , are the width and height of the original image respectively; the resolution is and The feature maps are represented as and .

[0028] Furthermore, in step 2.1.3, the working process of each multi-attention Transformer module is:

[0029] In the first submodule, the feature map is first normalized by the first normalization layer, and then the cross-covariance attention mechanism is used to focus on the channel of the feature map. After residual connection, the second normalization layer is used to perform the second normalization. The result of the second normalization is input to the multi-layer perceptron layer for multi-layer perceptron operation, and finally the output of the first submodule is obtained through the residual connection layer. The working process of the first submodule is formulated as:

[0030] (1);

[0031] (2);

[0032] in, is the input of the first submodule, is the input of the second normalization layer in the first submodule is the output of the first submodule; Normalize the layer; It is the cross-covariance attention mechanism; is a multi-layer perceptron;

[0033] The output of the first submodule is passed as input to the second submodule. The third normalization layer of the second submodule performs the third layer normalization processing, and then uses external attention to focus on the association between different samples. After residual connection, the fourth layer normalization processing is performed through the fourth normalization layer. The result after the fourth layer normalization is input to the multi-layer perceptron layer for multi-layer perceptron operation, and finally the output of the second submodule is obtained through the residual connection layer. The working process of the second submodule is formulated as:

[0034] (3);

[0035] (4);

[0036] in, is the input of the fourth normalization layer in the second submodule; is the output of the second submodule; It is an external attention mechanism;

[0037] The outputs of the three layers of the convolutional neural network are connected to the three layers of the multi-attention Transformer module in a skip connection manner, and three feature maps with different resolutions are obtained. , , .

[0038] Furthermore, in step 2.2, the working process of the dual-branch cross-fusion module is as follows:

[0039] Will and As input, we first fuse a class token, then perform position encoding, and then process them through linear projection and multiple ViT modules to get the output of the two branches. The output results of the two branches are fused through the cross attention mechanism, and finally the convolutional feedforward neural network module is used to replace the traditional feedforward neural network to aggregate information. Finally, we get two feature maps with different resolutions. The resolutions are and ; The convolutional feedforward neural network module is formulated as:

[0040] (5);

[0041] in, It is the input of the convolutional neural network module; It is a convolutional feed-forward neural network module; It is a 3×3 convolution; is the depth convolution; is the activation function;

[0042] for In this branch, the cross attention mechanism is formulated as:

[0043] (6);

[0044] in, is a class token, for Number of channels; and Respectively Dimension-aligned projection and dimension-anti-aligned projection of branches; for The dimensions of the branches are aligned with the input of the projection, for The output of the branched criss-cross attention mechanism;

[0045] for In this branch, the cross attention mechanism is formulated as:

[0046] (7);

[0047] in, is a class token; for Number of channels; and Respectively Dimension-aligned projection and dimension-anti-aligned projection of branches; for The dimensions of the branches are aligned against the input of the projection; for The output of the branched criss-cross attention mechanism.

[0048] Furthermore, in step 2.3, the upsampling convolution layer structures used in the first upsampling convolution layer and the second upsampling convolution layer are exactly the same;

[0049] Keep the feature map resolution of the convolutional layer input to , the convolution layer is used to keep the feature map resolution unchanged; the convolution layer includes a 3×3 convolution, a group normalization layer and a ReLU activation function;

[0050] The resolution of the feature map input to the first upsampling convolutional layer is , through the first upsampling convolutional layer, the resolution of the feature map becomes ;

[0051] The input of the second upsampling convolutional layer is the result of adding the output of the convolutional layer and the output of the first upsampling convolutional layer.

[0052] Each upsampling convolution layer contains a 3×3 convolution, bilinear interpolation upsampling, group normalization layer and ReLU activation function; the upsampling convolution layer is formulated as:

[0053] (8);

[0054] in, is the upsampling convolution layer; Normalize for the group; Upsampling for bilinear interpolation; is the ReLU activation function;

[0055] The resolution of the feature map input to the segmentation head is , by using a 3×3 convolution, the number of channels of the input feature map is adjusted to the required number while keeping the resolution of the feature map unchanged to complete the segmentation task;

[0056] The working process of the decoder is as follows: the dual-branch cross-fusion module outputs two feature maps with different resolutions, where the resolution is The feature map of is processed by the convolutional layer to keep its resolution unchanged; and the resolution is The feature map of is passed through two first upsampling convolutional layers, so that the feature map resolution is increased to ; then set the resolution to The two feature maps are added together and then input into two second upsampling convolutional layers to further increase the resolution to ; Finally, the resolution of the adjusted channel number is The feature map is input into the segmentation head to complete the final segmentation task and obtain the segmented image.

[0057] Beneficial technical effects brought by the present invention: The present invention proposes a dual-branch hybrid model based on CNN and Transformer, which makes full use of the advantages of CNN and Transformer to improve the accuracy and effect of medical image segmentation tasks. The present invention designs a multi-attention Transformer module to improve the problem of quadratic computational complexity and inability to pay attention to the association between different samples in the traditional self-attention mechanism. The present invention also designs a dual-branch cross-fusion module, which can well maintain the compatibility and consistency between features when fusing features of the CNN layer and the Transformer layer. At the same time, the feature maps of the two resolutions are effectively fused to generate rich image features. The DSC index and HD95 index of the present invention on the Synapse dataset are 81.31% and 14.02mm, respectively, surpassing the current SOTA model. The present invention can be used in clinical applications to help doctors more accurately identify and locate lesion areas, thereby improving the accuracy of diagnosis and early detection of diseases; in medical image processing and analysis, it can reduce dependence on manual intervention and improve work efficiency; it can help patients customize personalized treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a structural diagram of the double-branch hybrid model of the present invention.

[0059] Figure 2 This is a structural diagram of the jump connection of the present invention.

[0060] Figure 3 This is a structural diagram of the multi-attention Transformer module of the present invention.

[0061] Figure 4 It is a structural diagram of the dual-branch cross-fusion module of the present invention.

[0062] Figure 5 This is a visualization of the segmentation results of the Synapse dataset of the present invention.

[0063] Figure 6 This is a visualization of the segmentation results of the ACDC dataset of the present invention. DETAILED DESCRIPTION

[0064] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0065] The present invention takes abdominal multi-organ segmentation and heart segmentation as research objects, and takes improving the accuracy of medical image segmentation as the core goal. By designing a multi-attention Transformer module and a dual-branch cross-fusion module, a dual-branch hybrid model HCMAformer is built. The present invention gives full play to the respective advantages of CNN and Transformer, and can effectively fuse the feature maps output by the CNN layer and the Transformer layer. The present invention can achieve accurate segmentation of abdominal multi-organs and the heart.

[0066] like Figure 1 As shown, the present invention proposes a medical image segmentation method based on a dual-branch hybrid model, and the dual-branch hybrid model is composed of an encoder, a dual-branch cross-fusion module, and a decoder. The specific process of the medical image segmentation method is as follows:

[0067] Step 1: Obtain the existing abdominal dataset (Synapse dataset) and cardiac dataset (ACDC dataset) from the public website, preprocess the existing dataset into npz format files, and divide the npz format files into training sets and test sets in proportion. Both the abdominal dataset (Synapse dataset) and cardiac dataset (ACDC dataset) are 3D nii format files, and model training requires 2D format data input, so traverse each slice of the nii format file and save it as an npz format file.

[0068] Step 2: Construct a two-branch hybrid model HCMAformer; the two-branch hybrid model HCMAformer includes an encoder, a two-branch cross-fusion module, and a decoder. The specific process is:

[0069] Step 2.1: Construct an encoder consisting of a CNN (convolutional neural network) branch and a multi-attention Transformer branch; the CNN branch includes three levels of convolution modules (CNN modules), and the multi-attention Transformer branch includes three levels of multi-attention Transformer modules. The output of the convolution module is input to the multi-attention Transformer module through a skip connection.

[0070] Step 2.1.1, build CNN branch;

[0071] According to the existing CNN model, ResNet50 is selected as the backbone of the CNN module branch. The three-level convolutional module branches are composed of various stages of RestNet50. Compared with ResNet34 and ResNet101, ResNet50 has achieved a good balance between depth and computational cost, and can capture rich features while maintaining low computational cost.

[0072] The working process of the CNN branch is as follows: the input original image passes through three convolution modules in sequence to obtain three feature maps with different resolutions. , , ,in , are the width and height of the original image respectively. and The feature maps are represented as and .

[0073] Step 2.1.2: Build a skip connection structure. The skip connection structure is as follows: Figure 2 As shown in the figure, it includes 1×1 convolution, 3×3 deep convolution layer, SE attention mechanism (the main idea of ​​SE attention mechanism is to improve the performance of the model by compressing and exciting the input features), and 1×1 convolution. The output of the convolution module passes through 1×1 convolution, 3×3 deep convolution layer, SE attention mechanism, 1×1 convolution in sequence, and then enters the multi-attention Transformer module;

[0074] Step 2.1.3, introduce the cross-covariance attention mechanism and external attention mechanism on the basis of the SwinTransformer (Swin Transformer is a deep learning model based on Transformer) module, build a multi-attention Transformer module, and use the multi-level multi-attention Transformer module as the Transformer branch;

[0075] Each multi-attention Transformer module contains two cascaded sub-modules, namely the first sub-module and the second sub-module. When constructing a three-level multi-attention Transformer module, each level contains 2, 2, and 6 multi-attention Transformer modules, respectively, and each level is connected as a Transformer branch through a patch merging layer. The first sub-module and the second sub-module have the same LayerNorm layer (normalization layer) and multi-layer perceptron layer, and the output of the first sub-module is the input of the second sub-module.

[0076] The working process of each multi-attention Transformer module is:

[0077] like Figure 3As shown in the figure, the feature map in the first submodule first passes through the first LayerNorm layer for the first layer normalization processing, and then uses the cross-covariance attention mechanism to focus on the channel of the feature map (i.e., perform channel attention operation), and after residual connection, passes through the second LayerNorm layer for the second layer normalization processing, and the result after the second layer normalization is input into the multi-layer perceptron layer for multi-layer perceptron operation, and finally passes through the residual connection layer to obtain the output of the first submodule. The working process of the first submodule can be formulated as:

[0078] (1);

[0079] (2);

[0080] in, is the input of the first submodule, is the input of the second normalization layer in the first submodule is the output of the first submodule; Normalize the layer; It is the cross-covariance attention mechanism; It is a multi-layer perceptron.

[0081] The output of the first submodule is passed as input to the second submodule. The third LayNorm layer of the second submodule performs the third layer normalization, and then uses external attention to focus on the association between different samples. After residual connection, the fourth layer normalization is performed through the fourth LayerNorm layer. The result after the fourth layer normalization is input to the multi-layer perceptron layer for multi-layer perceptron operation, and finally the output of the second submodule is obtained through the residual connection layer. The working process of the second submodule can be formulated as:

[0082] (3);

[0083] (4);

[0084] in, is the input of the fourth normalization layer in the second submodule; is the output of the second submodule; is the external attention mechanism.

[0085] The outputs of the three layers of the CNN branch are connected to the three layers of the multi-attention Transformer module in a skip connection manner, and three feature maps with different resolutions are obtained. , , .

[0086] Step 2.2, construct a dual-branch cross fusion module to fuse images of different resolutions from two branches;

[0087] like Figure 4 As shown in the figure, the working process of the dual-branch cross-fusion module is as follows:

[0088] The two branches are generated and As input, a class token is fused respectively, and then position encoding is performed respectively. Then, linear projection (linear projection includes size-aligned projection and size-anti-aligned projection) and multiple ViT modules (ViT stands for Vision Transformer, which processes images by dividing patches and using position encoding methods. In the forward process, VIT cuts the image into patches and processes them through multiple layers of TransformerBlock) are used to obtain the output of the two branches. The output results of the two branches are fused through the cross-attention mechanism, and finally the ConvFFN (convolutional feedforward neural network) module is used instead of the traditional FFN (feedforward neural network) to aggregate information. Finally, two feature maps with different resolutions are obtained. The resolutions are and The ConvFFN module can be formulated as:

[0089] (5);

[0090] in, It is the input of the convolutional neural network module; It is a convolutional feed-forward neural network module; It is a 3×3 convolution; is the depth convolution; is the activation function.

[0091] In the present invention, In this branch, the number of ViT modules is for In this branch, the number of ViT modules is indivual;

[0092] for In this branch, the cross attention mechanism is formulated as:

[0093] (6);

[0094] in, is a class token, for Number of channels; and Respectively Dimension-aligned projection and dimension-anti-aligned projection of branches; for The dimensions of the branches are aligned with the input of the projection, for The output of the branch's cross-attention mechanism; for In this branch, the cross attention mechanism is formulated as:

[0095] (7);

[0096] in, is a class token; for Number of channels; and Respectively Dimension-aligned projection and dimension-anti-aligned projection of branches; for The dimensions of the branches are aligned against the input of the projection; for The output of the branched criss-cross attention mechanism.

[0097] Step 2.3, construct a decoder consisting of a retaining convolution layer, a first upsampling convolution layer, a second upsampling convolution layer and a segmentation head; the upsampling convolution layer structures used in the first upsampling convolution layer and the second upsampling convolution layer are exactly the same;

[0098] Keep the feature map resolution of the convolutional layer input to , keeping its resolution unchanged, keeping the convolution layer containing a 3×3 convolution, a group normalization layer GroupNorm and a ReLU activation function.

[0099] The resolution of the feature map input to the first upsampling convolutional layer is , through the first upsampling convolutional layer, the resolution of the feature map becomes ;

[0100] The input of the second upsampling convolutional layer is the result of adding the output of the convolutional layer and the output of the first upsampling convolutional layer.

[0101] Each upsampling convolution layer contains a 3×3 convolution, bilinear interpolation upsampling, group normalization layer GroupNorm and ReLU activation function. Each upsampling convolution layer can be formulated as:

[0102] (8);

[0103] in, is the upsampling convolution layer; Normalize for the group; Upsampling for bilinear interpolation; is the ReLU activation function.

[0104] The resolution of the feature map input to the segmentation head is , by using a 3×3 convolution, the number of channels of the input feature map is adjusted to the required number while keeping the resolution of the feature map unchanged to complete the segmentation task.

[0105] The working process of the decoder is as follows: the dual-branch cross-fusion module outputs two feature maps with different resolutions, where the resolution is The feature map of is processed by the convolutional layer to keep its resolution unchanged; and the resolution is The feature map of is passed through two first upsampling convolutional layers to increase its resolution to . Then set the resolution to The two feature maps are added together and then input into two second upsampling convolutional layers to further increase the resolution to Finally, the resolution of the adjusted channel number is The feature map is input into the segmentation head to complete the final segmentation task and obtain the segmented image.

[0106] Step 3: Use the training set to train the model; use the test set to test the model, and compare the test results with other models for qualitative and quantitative analysis. Use the test set to test and get the segmented image and two indicators, DSC and HD95. The DSC indicator is used to measure the similarity between the predicted segmentation result and the true label. The DSC value is between 0 and 1. The closer the DSC value is to 1, the more similar it is to the true label, and the higher the prediction accuracy. The HD95 indicator is used to measure the difference between the boundary predicted by the model and the true boundary. The closer the HD95 value is to 0, the better the performance.

[0107] Step 4: Obtain the current image to be segmented and perform image segmentation based on the trained dual-branch hybrid model.

[0108] The segmentation index of the Synapse dataset is compared between the present invention and seven models including Unet, AttnUnet, LeVit-Unet, TransUnet, Swin-Unet, HiFormer-L and MTUnet, and the comparison results are shown in Table 1. Among them, the CNN-based methods include Unet and AttnUnet; the Transformer-based methods include Swin-Unet and MTUnet; and the hybrid-based methods include LeVit-Unet, TransUnet and HiFormer-L.

[0109] Unet uses data augmentation and a network structure with a symmetric shrink-expand path to achieve accurate image segmentation with a small number of training samples while maintaining efficient use of contextual information and accurate localization capabilities. AttnUnet introduces an attention gate to automatically learn to focus on the target structure in medical images, which improves the model's sensitivity to images and prediction accuracy by highlighting useful salient features and suppressing irrelevant features. LeVit-Unet integrates LeViT (an efficient network structure that mixes Transformer and convolutional blocks) as an encoder into the U-Net architecture, while using multi-scale feature maps to effectively reuse the spatial information of feature maps through jump connections. TransUnet combines the global self-attention mechanism of Transformer and the high-resolution feature fusion of U-Net, and achieves global context capture and accurate detail localization in medical image segmentation tasks through a hybrid CNN-Transformer architecture and cascaded upsamplers. Swin-Unet is a pure Transformer architecture built on SwinTransformer. It learns local and global semantic features through a U-shaped encoder-decoder network and jump connections to achieve accurate edge prediction and better global context capture in medical image segmentation tasks. HiFormer-L is a novel hybrid CNN-Transformer framework that effectively fuses global and local features in the jump connection of the encoder-decoder structure by designing a two-layer fusion module to achieve accurate medical image segmentation. MTUnet combines local-global Gaussian weighted self-attention and external attention through a hybrid Transformer module to simultaneously learn internal and external associations between data samples to improve the accuracy and efficiency of medical image segmentation.

[0110] As can be seen from Table 1, the DSC index of the present invention reaches 81.31%, and the HD95 index is 14.02mm, which shows obvious advantages compared with networks such as UNet based on CNN. Compared with TransUnet and MTUnet, the DSC value of the present invention is increased by 3.83% and 2.72%, respectively, and the HD95 value is reduced by 17.67mm and 12.57mm, respectively. The present invention shows an overwhelming advantage in the HD95 index, and is also superior to other methods in the segmentation effect of five organs, including the left kidney, right kidney, liver, pancreas, and spleen. In particular, there is a significant improvement in the DSC index of the left kidney, which proves that the present invention can achieve accurate segmentation.

[0111] Table 1 Comparison results of segmentation indicators of each model for Synapse dataset

[0112] .

[0113] The visualization of the segmentation results of the Synapse dataset is as follows Figure 5 As shown, the present invention only shows the visualization results of 3 cases in the Synapse dataset.

[0114] In Case 1 and Case 2, it can be seen that the present invention pays more attention to the boundaries of organs, the boundary segmentation is clearer, and it shows obvious advantages in the segmentation of pancreas and spleen, which is consistent with the results in Table 1. In Case 3, the method of the present invention shows strong robustness and significantly reduces the false detection phenomenon.

[0115] The comparison of different methods and the visualization of the results fully prove that the method of the present invention can give full play to the respective advantages of CNN and Transformer, use the dual-branch cross-fusion module to fuse rich multi-scale information, improve the robustness of the model, achieve high-precision segmentation, and obtain better segmentation results.

[0116] The segmentation indicators of the ACDC dataset are compared using five models, namely, Unet, AttnUnet, TransUnet, Swin-Unet and MTUnet. The comparison results are shown in Table 2.

[0117] Table 2 Comparison results of segmentation indicators of each model on ACDC dataset

[0118] .

[0119] As can be seen from Table 2, the DSC index of the present invention for the segmentation of the heart dataset reaches 91.44%, and the present invention is also superior to other methods in the DSC index of the right ventricle and myocardium. This proves that the method of the present invention has good generalization and robustness and can be applied to multiple datasets.

[0120] The segmentation visualization results of the ACDC dataset are as follows Figure 6 As shown in the figure, the present invention only shows the visualization results of 3 cases in the ACDC dataset. As can be seen from Case 1, the image predicted by the present invention is closer to the true label. In Case 2 and Case 3, for the right ventricle, Swin-Unet and MTUnet both have serious missed detections, while the method of the present invention can make good predictions, further verifying its generalization and robustness.

[0121] In summary, the present invention improves the segmentation accuracy of medical images and provides technical support for medical image segmentation.

[0122] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A medical image segmentation method based on a dual-branch mixture model, characterized in that: The steps include: Step 1: Obtain an existing dataset from a public website, preprocess the existing dataset into an npz format file, and divide the npz format file into a training set and a test set in proportion; Step 2: Construct a dual-branch hybrid model; the dual-branch hybrid model includes an encoder, a dual-branch cross-fusion module, and a decoder; the specific process is: Step 2.1: Construct an encoder consisting of a convolutional neural network branch and a multi-attention Transformer branch; the convolutional neural network branch includes three levels of convolutional modules; the multi-attention Transformer branch includes three levels of multi-attention Transformer modules; the output of the convolutional module is input to the multi-attention Transformer module through a jump connection; the specific process is: Step 2.1.1, build a convolutional neural network branch; According to the existing convolutional neural network model, ResNet50 is selected as the backbone of the convolutional neural network model; the convolutional module branches of the three levels are composed of various stages of RestNet50; Step 2.1.2, build a jump connection structure; The jump connection structure includes 1×1 convolution, 3×3 deep convolution layer, SE attention mechanism, and 1×1 convolution connected in sequence; the output of the convolution module passes through 1×1 convolution, 3×3 deep convolution layer, SE attention mechanism, and 1×1 convolution in sequence, and then enters the multi-attention Transformer module; Step 2.1.3, introduce the cross-covariance attention mechanism and external attention mechanism on the basis of the SwinTransformer module to construct a multi-attention Transformer module; use the multi-level multi-attention Transformer module as the Transformer branch; Each multi-attention Transformer module contains two cascaded sub-modules, namely the first sub-module and the second sub-module. When constructing a three-level multi-attention Transformer module, each level contains 2, 2, and 6 multi-attention Transformer modules, respectively, and each level is connected through a patch merging layer as a Transformer branch. The first submodule and the second submodule have the same normalization layer and multi-layer perceptron layer, and the output of the first submodule is the input of the second submodule; Step 2.2: Construct a dual-branch cross fusion module to fuse images of two branches with different resolutions; the working process of the dual-branch cross fusion module is as follows: x large and x small As input, first fuse a class token respectively, then perform position encoding respectively, and then process them through linear projection and multiple ViT modules to obtain the output of two branches. The output results of the two branches are fused through the cross attention mechanism, and finally the convolutional feedforward neural network module is used to replace the traditional feedforward neural network to aggregate information. Finally, two feature maps with different resolutions are obtained, which are (H / 4, W / 4) and (H / 16, W / 16) respectively. Step 2.3, construct a decoder consisting of a retaining convolution layer, a first upsampling convolution layer, a second upsampling convolution layer and a segmentation head; Step 3: Use the training set to train the model, and use the test set to test the model; Step 4: Obtain the current image to be segmented and perform image segmentation based on the trained dual-branch hybrid model.

2. The medical image segmentation method based on the dual-branch mixture model according to claim 1, characterized in that: In the step 1, the existing data sets include an abdominal data set and a heart data set, and the existing data sets are all 3D nii format files. Each slice of the nii format file is traversed, and the existing data sets are saved as npz format files.

3. The medical image segmentation method based on the dual-branch mixture model according to claim 1, characterized in that: In step 2.1.1, the working process of the convolutional neural network branch is as follows: the input original image passes through three convolution modules in sequence to obtain three feature maps with different resolutions, the resolutions are (H / 4, W / 4), (H / 8, W / 8), (H / 16, W / 16), where H and W are the width and height of the original image respectively; the feature maps with resolutions of (H / 4, W / 4) and (H / 16, W / 16) are represented as x respectively. large and x small .

4. The medical image segmentation method based on the dual-branch mixture model according to claim 3, characterized in that: In step 2.1.3, the working process of each multi-attention Transformer module is: In the first submodule, the feature map is first normalized by the first normalization layer, and then the cross-covariance attention mechanism is used to focus on the channel of the feature map. After residual connection, the second normalization layer is used to perform the second normalization. The result of the second normalization is input to the multi-layer perceptron layer for multi-layer perceptron operation, and finally the output of the first submodule is obtained through the residual connection layer. The working process of the first submodule is formulated as: Among them, z l-1 is the input of the first submodule, is the input of the second normalization layer in the first submodule, z l is the output of the first submodule; LN(·) is layer normalization; CA(·) is the cross-covariance attention mechanism; MLP(·) is a multi-layer perceptron; The output of the first submodule is passed as input to the second submodule. The third normalization layer of the second submodule performs the third layer normalization processing, and then uses external attention to focus on the association between different samples. After residual connection, the fourth layer normalization processing is performed through the fourth normalization layer. The result after the fourth layer normalization is input to the multi-layer perceptron layer for multi-layer perceptron operation, and finally the output of the second submodule is obtained through the residual connection layer. The working process of the second submodule is formulated as: in, is the input of the fourth normalization layer in the second submodule; z l+1 is the output of the second submodule; EA(·) is the external attention mechanism; The outputs of the three layers of the convolutional neural network are input into the three-layer multi-attention Transformer module in a skip connection manner, and three feature maps with different resolutions are obtained, namely (H / 4, W / 4), (H / 8, W / 8), and (H / 16, W / 16).

5. The medical image segmentation method based on the dual-branch mixture model according to claim 4, characterized in that: In step 2.2, the convolutional feedforward neural network module is formulated as: ConvFFN(x)=Conv(DWConv(GELU(Conv(x)))) (5); Where x is the input of the convolutional feedforward neural network module; ConvFFN(·) is the convolutional feedforward neural network module; Conv(·) is a 3×3 convolution; DWConv(·) is a depthwise convolution; GELU(·) is the activation function; For x large In this branch, the cross attention mechanism is formulated as: in, is the class token, C1 is x large Number of channels; f large (·) and g large (·) respectively represent x large The size-aligned projection and size-anti-aligned projection of the branch; y large For x large The size of the branch is anti-aligned with the input of the projection, z large For x large The output of the branched criss-cross attention mechanism; For x small In this branch, the cross attention mechanism is formulated as: in, is a class token; C2 is x small Number of channels; f small (·) and g small (·) represent x small The size-aligned projection and size-anti-aligned projection of the branch; y small For x small The size of the branch is anti-aligned with the input of the projection; z small For x small The output of the branched criss-cross attention mechanism.

6. The medical image segmentation method based on the dual-branch mixture model according to claim 5, characterized in that: In step 2.3, the upsampling convolution layer structures used in the first upsampling convolution layer and the second upsampling convolution layer are exactly the same; The feature map resolution of the convolution layer input is kept as (H / 4, W / 4), and the convolution layer is used to keep the feature map resolution unchanged; the convolution layer includes a 3×3 convolution, a group normalization layer, and a ReLU activation function; The resolution of the feature map input by the first upsampling convolution layer is (H / 16, W / 16), and the resolution of the feature map becomes (H / 4, W / 4) through the first upsampling convolution layer; The input of the second upsampling convolutional layer is the result of adding the output of the convolutional layer and the output of the first upsampling convolutional layer. Each upsampling convolution layer contains a 3×3 convolution, bilinear interpolation upsampling, group normalization layer and ReLU activation function; The upsampling convolutional layer is formulated as: UpConv(x)=Up(ReLU(GN(Conv(x)))) (8); Among them, UpConv(·) is the upsampling convolution layer; GN(·) is group normalization; Up(·) is bilinear interpolation upsampling; ReLU(·) is the ReLU activation function; The resolution of the feature map input to the segmentation head is (H, W). By using a 3×3 convolution, the number of channels of the input feature map is adjusted to the required number while keeping the resolution of the feature map unchanged to complete the segmentation task; The working process of the decoder is as follows: the dual-branch cross-fusion module outputs two feature maps with different resolutions, where the feature map with a resolution of (H / 4, W / 4) is processed by the retention convolution layer to keep its resolution unchanged; and the feature map with a resolution of (H / 16, W / 16) is processed by two first upsampling convolution layers to increase the feature map resolution to (H / 4, W / 4); then the two feature maps with a resolution of (H / 4, W / 4) are added together and then input into two second upsampling convolution layers to further increase the resolution to (H, W); finally, the feature map with a resolution of (H, W) after adjusting the number of channels is input into the segmentation head to complete the final segmentation task and obtain the segmented image.

Images