Medical image segmentation model training method and medical image segmentation method

Abstract

The invention relates to a training method of a medical image segmentation model and a medical image segmentation method, and the training method comprises the steps: carrying out the supervision training of an image segmentation model through a sample medical image; wherein the medical image segmentation model comprises a Transform network and a CNN (Convolutional Neural Network); the coding part of the Transform network comprises a coarse-grained coding branch, a fine-grained coding branch and a feature information fusion branch, feature output of the coarse-grained coding branch and feature output of the fine-grained coding branch are subjected to feature fusion through the feature information fusion branch, and feature output of the feature information fusion branch serves as feature input of the decoding part of the Transform network; the feature output of the feature information fusion branch also serves as the feature input of the coding part of the CNN network, and the feature input of the decoding part of the CNN network comprises the feature output of the coding part of the CNN network. The Transform network and the CNN network are effectively combined, and image features extracted by the two networks can be fully fused, so that the advantages and disadvantages of the two networks are complementary.

Description

Technical Field

[0001] The present application relates to the field of medical image processing, and in particular to a training method for a medical image segmentation model and a medical image segmentation method. Background Art

[0002] The development of deep learning technology has brought new breakthroughs in medical image segmentation. Convolutional neural networks (CNN) and their variants have become the mainstream methods in the field of medical image segmentation. However, due to certain limitations of the CNN model itself, it is insufficient in capturing long-range dependencies, which restricts the further improvement of the segmentation model performance. In order to solve this problem, researchers applied the Transformer model applied to the field of NLP to the field of medical image segmentation. The Transformer model is well known for its excellent performance in processing sequence data, especially in long-range dependency modeling and parallel training, which can well complement the shortcomings of the CNN model. However, the Transformer model cannot effectively obtain local information, which also restricts the further improvement of the segmentation model performance.

[0003] The current training method of medical image segmentation models fails to effectively combine the Transformer model and the CNN model, and is unable to train a medical image segmentation model with better segmentation performance, thereby limiting the segmentation accuracy of the medical image segmentation method. Summary of the invention

[0004] The present invention provides a training method for a medical image segmentation model and a medical image segmentation method to solve the problem that the current training method for a medical image segmentation model fails to effectively combine the Transformer model and the CNN model, and fails to train a medical image segmentation model with better segmentation performance.

[0005] In a first aspect, the present invention provides a training method for a medical image segmentation model, comprising:

[0006] Supervised training of image segmentation models using sample medical images;

[0007] Wherein, the medical image segmentation model includes a Transformer network and a CNN network;

[0008] The encoding part of the Transformer network includes a coarse-grained encoding branch, a fine-grained encoding branch and a feature information fusion branch, the feature outputs of the coarse-grained encoding branch and the fine-grained encoding branch are feature fused through the feature information fusion branch, and the feature output of the feature information fusion branch is used as a feature input of the decoding part of the Transformer network;

[0009] The feature output of the feature information fusion branch is also used as the feature input of the encoding part of the CNN network, and the feature input of the decoding part of the CNN network includes the feature output of the encoding part of the CNN network;

[0010] The feature outputs of the decoding part of the Transformer network and the decoding part of the CNN network are fused through a target fusion module and then decoded through a target decoder to obtain a segmentation result of the sample medical image;

[0011] The sample medical image is simultaneously used as the image input of the coarse-grained coding branch, the fine-grained coding branch and the coding part of the CNN network, and the image input size of the coarse-grained coding branch is larger than the image input size of the fine-grained coding branch.

[0012] In a second aspect, the present invention provides a medical image segmentation method, comprising:

[0013] Medical image segmentation is performed using the medical image segmentation model obtained by the medical image segmentation model training method described in the first aspect.

[0014] Compared with the related art, the medical image segmentation model adopted by the present invention effectively combines the Transformer network and the CNN network, and performs feature fusion not only in the encoding stage but also in the decoding stage, and can fully fuse the image features extracted by the two networks, so that the advantages and disadvantages of the two networks complement each other. Therefore, through the training method of the medical image segmentation model provided by the present invention, a medical image segmentation model with better segmentation performance can be obtained, which solves the problem that the current training method of the medical image segmentation model fails to effectively combine the Transformer model and the CNN model, and fails to train a medical image segmentation model with better segmentation performance.

[0015] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a structural diagram of a medical image segmentation model in some embodiments of the present invention;

[0017] Figure 2 is a schematic diagram of a feature processing flow of a feature information fusion module in some embodiments of the present invention;

[0018] Figure 3 is a schematic diagram of a feature processing flow of an edge information fusion module in some embodiments of the present invention;

[0019] Figure 4 1 is a schematic diagram of the feature processing flow of the target fusion module and the four intermediate CNN encoders in some embodiments of the present invention. DETAILED DESCRIPTION

[0020] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0021] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] In an embodiment of the present invention, a method for training a medical image segmentation model is provided, and the process includes the following steps:

[0023] Step S110, performing supervised training on the image segmentation model using sample medical images.

[0024] Before using sample medical images to input the image segmentation model for training, the medical images need to be processed to obtain sample medical images. These processes mainly include data renaming, data preprocessing, data set division, etc. For example, data renaming is to rename the medical images according to the folder category of the obtained data set; data preprocessing is to use the processed medical images as model samples and perform data enhancement on the medical images, such as random flipping, mirror flipping, etc.; data set division is to randomly divide the enhanced medical images into training sets, validation sets, and test sets in a ratio of 8:1:1. The medical images in the training set are sample medical images, which are used for supervised training of the image segmentation model. The trained image segmentation model can be verified and tested for segmentation performance through the validation set and the test set, respectively.

[0025] Reference Figure 1 , where the medical image segmentation model includes Transformer network and CNN network.

[0026] The encoding part of the Transformer network includes a coarse-grained encoding branch and a fine-grained encoding branch (both encoding branches are composed of multiple Transformer encoders. Figure 1 The A module is the Transformer encoder) and the feature information fusion branch (composed of multiple feature information fusion modules, Figure 1 The B module in the figure is a feature information fusion module). The feature outputs of the coarse-grained encoding branch and the fine-grained encoding branch are fused through the feature information fusion branch. The feature output of the feature information fusion branch is used as the decoding part of the Transformer network (composed of multiple Transformer decoders, Figure 1 The C module is the Transformer decoder) feature input; the feature output of the feature information fusion branch is also used as the encoding part of the CNN network (composed of multiple CNN encoders, Figure 1 The D module is the feature input of the CNN encoder), the decoding part of the CNN network (composed of multiple CNN decoders, Figure 1 The feature input of the E module is the CNN decoder) includes the feature output of the encoding part of the CNN network; the feature output of the decoding part of the Transformer network and the decoding part of the CNN network are combined through the target fusion module ( Figure 1 The G module is the target fusion module) and then passes through the target decoder ( Figure 1 The H module in the example is the target decoder) and the sample medical image is decoded to obtain the segmentation result of the sample medical image; the sample medical image is simultaneously used as the image input of the coarse-grained coding branch, the fine-grained coding branch and the coding part of the CNN network, and the image input size of the coarse-grained coding branch is larger than the image input size of the fine-grained coding branch.

[0027] In the above technical solution, the Transformer network and the CNN network are effectively combined to form a new medical image segmentation model. Among them, the encoding part of the Transformer network encodes the medical image through two encoding branches respectively. The medical images input to the two encoding branches are the same. The difference is that the two encoding branches use different Patch blocks to divide the medical image, that is, the image input size is different. The image input size (Patch block size) of the coarse-grained encoding branch is larger than the image input size of the fine-grained encoding branch. Exemplarily, the image input size of the coarse-grained encoding branch is set to 8*8, and the image input size of the fine-grained encoding branch is set to 4*4. The two encoding branches of different sizes independently extract the feature information of the medical image, and effectively interact and fuse the feature information in the feature information fusion branch, and provide the fused feature information to the encoding part of the CNN network. The encoding part of the CNN network will also extract the feature information of the medical image separately, and decode it after fusing the feature information provided by the Transformer network. At the same time, the Transformer network and the CNN network perform feature decoding respectively and fuse them again, and then decode the fused image features to obtain the image segmentation result.

[0028] In summary, the medical image segmentation model adopted by the present invention effectively combines the Transformer network and the CNN network, and performs feature fusion not only in the encoding stage, but also in the decoding stage, and can fully fuse the image features extracted by the two networks, so that the advantages and disadvantages of the two networks complement each other. Therefore, through the training method of the medical image segmentation model provided by the present invention, a medical image segmentation model with better segmentation performance can be obtained, which solves the problem that the current training method of the medical image segmentation model fails to effectively combine the Transformer model and the CNN model, and fails to train a medical image segmentation model with better segmentation performance.

[0029] In some preferred embodiments, the medical image segmentation model further includes an edge information fusion branch (composed of multiple edge information fusion modules, Figure 1 The F module in the figure is the edge information fusion module); the input of the edge information fusion branch includes the mask output of the decoding part of the Transformer network, the mask output of the decoding part of the CNN network, and the feature output of the encoding part of the CNN network. The feature input of the decoding part of the CNN network also includes the feature output of the edge information fusion branch.

[0030] In some embodiments, the medical image segmentation model also uses an edge information fusion branch to fuse edge information in the model, and the edge information mainly includes mask information output by the decoding part of the Transformer network and the CNN network. The edge information fusion branch integrates the edge information into the feature information extracted by the encoding part of the CNN network, and provides the fused feature information to the decoding part of the CNN network for decoding. Through the edge information fusion branch, the information in the Transformer network and the CNN network can be better integrated, further improving the image segmentation performance of the medical image segmentation model.

[0031] Further, in some embodiments thereof:

[0032] For the Transformer network part, the coarse-grained encoding branch and the fine-grained encoding branch both include four Transformer encoders connected in sequence, and the feature information fusion branch includes four feature information fusion modules. The four feature information fusion modules correspond to the four Transformer encoders in sequence, and each feature information fusion module is used to fuse the feature outputs of the corresponding Transformer encoders in the coarse-grained encoding branch and the fine-grained encoding branch.

[0033] The four transformer encoders connected in sequence can be defined as the first transformer encoder, the second transformer encoder, the third transformer encoder and the fourth transformer encoder in the order of connection, and the output of the previous transformer encoder is used as the input of the next transformer encoder. The four feature information fusion modules can be defined as the first feature information fusion module, the second feature information fusion module, the third feature information fusion module and the fourth feature information fusion module. The first feature information fusion module is used to fuse the feature outputs of the two first transformer encoders (the first transformer encoders in the coarse-grained coding branch and the fine-grained coding branch, respectively). Similarly, the fourth feature information fusion module is used to fuse the feature outputs of the two fourth transformer encoders (the fourth transformer encoders in the coarse-grained coding branch and the fine-grained coding branch, respectively).

[0034] The decoding part of the Transformer network includes four Transformer decoders connected in sequence. The four Transformer decoders correspond to the four feature information fusion modules in reverse order. The feature input of each Transformer decoder includes the feature output of the corresponding feature information fusion module.

[0035] The four transformer decoders connected in sequence can be defined as the first transformer decoder, the second transformer decoder, the third transformer decoder and the fourth transformer decoder in the order of connection. The input of the fourth transformer decoder includes the output of the first feature information fusion module, the input of the third transformer decoder includes the output of the second feature information fusion module, the input of the second transformer decoder includes the output of the third feature information fusion module, and the input of the first transformer decoder includes the output of the fourth feature information fusion module. In addition, the input of the next transformer decoder also includes the output of the previous transformer decoder.

[0036] For the CNN network part, the encoding part of the CNN network includes six CNN encoders connected in sequence, the middle four CNN encoders correspond to the four feature information fusion modules in sequence, and the feature input of each CNN encoder in the middle four CNN encoders includes the feature output of the corresponding feature information fusion module.

[0037] The six CNN encoders connected in sequence can be defined as the first CNN encoder, the second CNN encoder, the third CNN encoder, the fourth CNN encoder, the fifth CNN encoder and the sixth CNN encoder in the order of connection. Among them, the input of the second CNN encoder includes the output of the first feature information fusion module, the input of the third CNN encoder includes the output of the second feature information fusion module, the input of the fourth CNN encoder includes the output of the third feature information fusion module, and the input of the fifth CNN encoder includes the output of the fourth feature information fusion module. In addition, the input of the next CNN encoder also includes the output of the previous CNN encoder.

[0038] The edge information fusion branch includes five edge information fusion modules connected in sequence, and the five edge information fusion modules correspond to the first five CNN encoders in reverse order. The first four edge information fusion modules also correspond to the four Transformer decoders in sequence. The decoding part of the CNN network includes five CNN decoders connected in sequence, and the last four CNN decoders correspond to the first four edge information fusion modules in sequence. The input of each edge information fusion module includes the feature output of the corresponding CNN encoder, and the input of each edge information fusion module in the first four edge information fusion modules also includes the mask output (Tmask) of the corresponding CNN decoder and the mask output (Cmask) of the corresponding Transformer decoder. The first edge information fusion module also includes the feature output of the last CNN decoder. The input of each CNN decoder in the last four CNN decoders includes the feature output of the corresponding edge information fusion module, and the feature input of the first CNN decoder is the feature output of the last CNN decoder.

[0039] The five edge information fusion modules connected in sequence can be positioned as the first edge information fusion module, the second edge information fusion module, the third edge information fusion module, the fourth edge information fusion module and the fifth edge information fusion module according to the connection order. The input of the first edge information fusion module includes the feature output of the fifth CNN encoder, the feature output of the sixth CNN encoder, the mask output of the first Transformer decoder and the mask output of the second CNN decoder, the input of the second edge information fusion module includes the feature output of the fourth CNN encoder, the feature output of the first edge information fusion module, the mask output of the second Transformer decoder and the mask output of the third CNN decoder, the input of the third edge information fusion module includes the feature output of the third CNN encoder, the feature output of the second edge information fusion module, the mask output of the third Transformer decoder and the mask output of the fourth CNN decoder, the input of the fourth edge information fusion module includes the output of the second CNN encoder, the feature output of the third edge information fusion module, the mask output of the fourth Transformer decoder and the mask output of the fifth CNN decoder, and the input of the fifth edge information fusion module includes the feature output of the first CNN encoder and the feature output of the fourth edge information fusion module. And, the input of the first CNN decoder includes the feature output of the sixth CNN encoder, the input of the second CNN decoder includes the feature output of the first edge information fusion module, the input of the third CNN decoder includes the feature output of the second edge information fusion module, the input of the fourth CNN decoder includes the feature output of the third edge information fusion module, and the input of the fifth CNN decoder includes the feature output of the fourth edge information fusion module. In addition, the input of the next CNN decoder also includes the output of the previous CNN decoder.

[0040] Furthermore, the target decoder is a CNN decoder; the feature input of the target decoder also includes the feature output of the last edge information fusion module, and the input of the last edge information fusion module also includes the mask output of the target decoder. The target decoder can be defined as the seventh CNN decoder, and the input of the sixth edge information fusion module also includes the output of the seventh CNN decoder. The input of the seventh CNN decoder includes the output of the sixth edge information fusion module in addition to the output of the target fusion module.

[0041] How, in some embodiments, the module composition of each branch of the medical image segmentation model and the connection relationship outside the module are specifically explained.

[0042] CNN encoder and CNN decoder can adopt the conventional encoding and decoding structure in the existing CNN network, and Transformer encoder and Transformer decoder can also adopt the conventional encoding and decoding structure in the existing Transformer network, and then the present invention does not specifically describe these encoders and decoders. Exemplarily, the Transformer encoder can adopt a multi-head self-attention module, and the Transformer decoder can adopt a Swin Transformer decoder module. The focus of the present invention is to combine the CNN network and the Transformer network, and secondly, in order to achieve good fusion, some new information fusion modules are provided. For example, feature information fusion module, edge information fusion module and target fusion module. As follows, these modules are described in detail.

[0043] Reference Figure 2 In some of the embodiments, the information processing steps of each feature information fusion module include:

[0044] The size of the fine-grained feature map is reshaped to the same size as the coarse-grained feature map, the fine-grained feature map is the output of the corresponding Transformer encoder in the fine-grained encoding branch, and the coarse-grained feature map is the output of the corresponding Transformer encoder in the coarse-grained encoding branch; the resized fine-grained feature map is subjected to one-dimensional average pooling and flattening to obtain a first intermediate feature map; the coarse-grained feature map is subjected to one-dimensional average pooling and flattening to obtain a second intermediate feature map; the first intermediate feature map is concatenated with the coarse-grained feature map and then subjected to a self-attention operation to obtain a third intermediate feature map; the second intermediate feature map is concatenated with the resized fine-grained feature map and then subjected to a self-attention operation to obtain a fourth intermediate feature map; the third intermediate feature map and the fourth intermediate feature map are concatenated and then subjected to a 1*1 convolution to obtain the feature output of the feature information fusion module.

[0045] And, the information processing steps of each CNN encoder in the middle four CNN encoders include:

[0046] The feature output of the previous CNN encoder is convolved with the feature output of the corresponding feature information fusion module to obtain the fifth intermediate feature map; the fifth intermediate feature map is sequentially convolved with 1*1, batch normalized and linearly rectified to obtain the feature output of the CNN encoder.

[0047] Reference Figure 3 In some of the embodiments, the information processing steps of each edge information fusion module include:

[0048] The size of the low-level features is reshaped to be the same as that of the high-level features. The low-level features are the feature outputs of the corresponding CNN encoders, and the high-level features are the feature outputs of the upper-level modules. It should be noted that for the first edge information fusion module, its upper-level module is the last CNN encoder, and for the last four edge information fusion modules, their upper-level modules are all their own upper-level edge information fusion modules, as well as the deep separable convolution (DW Conv) and bilinear interpolation are used to adjust the size of high-level features; the high-level features and the resized low-level features are divided into four corresponding feature maps along the channel dimension (the low-level features are divided into y1, y2, y3, y4, and the high-level features are divided into x1, x2, x3, x4), and the corresponding feature maps are spliced ​​to obtain four sixth intermediate feature maps; each sixth intermediate feature map is spliced ​​with the mask output of the corresponding CNN decoder and the mask output of the corresponding Transformer decoder to obtain four seventh intermediate feature maps; the four seventh intermediate feature maps are respectively subjected to 3*3 convolution and then spliced ​​to obtain the eighth intermediate feature map, and the four 3*3 convolutions respectively use different expansion rates. For example, the expansion rates of the four 3*3 convolutions are 1 / 2 / 5 / 7 respectively; the eighth intermediate feature map is subjected to 1*1 convolution (for restoring the feature map size) to obtain the feature output of the edge information fusion module.

[0049] Reference Figure 4 In some embodiments, the information processing step of the target fusion module includes:

[0050] The feature outputs of the last three Transformer decoders are upsampled to the same dimension and then concatenated to obtain the first Transformer feature map; the feature outputs of the last three CNN decoders are upsampled to the same dimension and then concatenated to obtain the first CNN feature map; the first Transformer feature map and the first CNN feature map are respectively subjected to attention calculation processing to obtain the second Transformer feature map and the second CNN feature map. The formula for attention calculation processing is:

[0051]

[0052]

[0053] X=F+M S (F)

[0054] in, Represents the feature output of the attention calculation process, Conv 1×1 represents 1*1 convolution, M C represents the channel attention calculation, M Srepresents spatial attention calculation, F represents the feature input of channel attention calculation, MLP represents multi-layer perceptron, AvgPool and MaxPool represent global average pooling and maximum pooling respectively, σ represents batch normalization, W1 and W2 represent hidden neurons, and They represent the global average pooling result and the maximum pooling result of F, respectively. X represents the feature input of spatial attention calculation. Conv 7×7 represents 7*7 convolution, and Represent the global average pooling result and maximum pooling result of X respectively.

[0055] The feature aggregation layer has a great influence on the quality of the segmentation result. The target fusion module provided in this embodiment is used to perform feature aggregation, which can improve the quality of the image segmentation result.

[0056] Reference Figure 1 In some embodiments, the loss function L used in the training of the image segmentation model is total for:

[0057] L total =αL final (G,S final )+βL t (G,S t )+γL c (G,S c )

[0058]

[0059] Among them, L final (G,S final ) represents the final prediction loss, L t (G,S t ) represents the Transformer branch loss, L c (G,S c ) represents the CNN branch loss, α, β and γ all represent loss weights, G represents the label, S final Represents the model prediction result, S t represents the mask output of the decoding part of the Transformer network, represents the mask output of the i-th Transformer decoder, S c represents the mask output of the decoding part of the CNN network, represents the mask output of the i-th CNN decoder, L IOU represents the IOU loss function, L DICE represents the DICE loss function, L BCE represents the BCE loss function, λi represents the weight of the i-th Transformer decoder, η i represents the weight of the i-th CNN decoder.

[0060] Specifically, this embodiment adopts a fully supervised training method, so each training set image corresponds to a corresponding mask. During the training process, the batchsize can be set to 16, and the optimizer uses the adam optimizer. At the same time, the initial learning rate can be set to 0.0001, and the decay coefficient can be set to 0.00001. The training process of our framework includes multiple epochs, each epoch contains many iterations, and a verification is performed at the end of all iterations of each epoch. The total number of epochs for a training can be set to 200. After the above settings are completed, the loss function in this embodiment can be used for model training. For example, α, β and γ can be set to 0.6 / 0.2 / 0.2, respectively, λ i can be 0.5 / 0.4 / 0.3 / 0.2 / 0.1 respectively (corresponding to the embodiment using 5 Transformer decoders), η i They can be 0.5 / 0.3 / 0.2 / 0.1 respectively (corresponding to an embodiment using 4 CNN decoders).

[0061] As described above, at least one embodiment of each part of the training method for a medical image segmentation model provided by the present invention has been described respectively. It should be noted that the various parts of the training method for a medical image segmentation model can be combined using different embodiments to form different embodiments of the training method for a medical image segmentation model.

[0062] Reference Figures 1 to 4 In a specific overall embodiment, the training method of the medical image segmentation model includes:

[0063] 1. Rename the data.

[0064] This example uses ISIC2018, ISIC2017, DBS2018, Kvasir, CVC-ClinicDB, CVC-ColonDB, ETIS, CVC-300, CHAOS, ACDC, and CAMUS datasets as tests, covering modalities such as CT, MRI, colonoscopy, cell microscopy, and ultrasound. All images are renamed uniformly, and the naming rules are as follows: dataset name, number / dataset name, patient number, number. In this example, all images are uniformly scaled to an image size of 512*512.

[0065] 2. Data preprocessing.

[0066] Due to the limited number of data and images, this embodiment performs data enhancement methods such as random horizontal / vertical flipping on the images to expand the data set, which helps to improve the generalization ability of the model. Enhancement operations such as random cropping, rotation, scaling and color transformation can help the model learn features at different angles and scales, which helps to improve segmentation accuracy.

[0067] 3. Data set division.

[0068] In this embodiment, each data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. Since the data is preprocessed, the same image and its different augmented images must belong to the same data set (both training sets or test sets) to ensure that there is no data image overlap between the test set and the validation set.

[0069] 4. Model construction.

[0070] The medical image segmentation model in this embodiment follows the encoder-decoder structure as a whole, consisting of a CNN branch (CNN network) and two Transformer branches of different scales (constituting the Transformer network), and also includes several special modules, a dual-scale information fusion module (feature information fusion module B), a dual-branch information interaction module (four CNN encoders in the middle), a multi-branch edge information fusion module (edge ​​information fusion module) and a multi-level dual-branch feature information fusion module (target fusion module).

[0071] The specific description of the model is as follows:

[0072] (1) Encoder.

[0073] In this embodiment, the model encoder stage consists of three branches in total, wherein the Transformer branch contains two independent branches of different scales, both branches use SwinTransformer blocks (this embodiment uses SwinTransformer blocks as Transformer encoders), the difference is that the Patch block size set for the coarse-grained branch is 8*8, and the Patch block size set for the fine-grained branch is 4*4, the two branches of different sizes independently extract image information, and effectively interact and fuse information in the DTF module (dual-scale information fusion module). In the CNN branch, the encoder stage is divided into six layers, and the number of channels in each layer is 8, 16, 32, 64, 128, and 256, respectively. Each layer uses the same 3*3 convolution, maximum pooling, and ReLu operations. At the same time, the four middle layers of CNN receive the information fused from the DTF module of the Transformer branch, and use the characteristics of Transformer that can effectively capture long-range dependencies to provide information supplement for the CNN branch, thereby optimizing the training effect of the CNN branch.

[0074] (2) Dual-branch information exchange module (CTFB).

[0075] In the Transformer branch, a coarse-grained and a fine-grained SwinTransformer branch (corresponding to the coarse-grained encoding branch and the fine-grained encoding branch, respectively) are used to extract feature information respectively. After obtaining the output features of the dual-branch encoder, this embodiment uses the MSA mechanism to form efficient interaction and fusion between multi-scale feature information, and generates a token sequence of a specified size based on the feature map of the coarse-grained branch. For the fine-grained branch, we reshape it into the same size as the coarse-grained branch feature map to generate a token sequence (h+w). At the same time, one-dimensional average pooling is performed on both branches, and they are flattened and spliced ​​into the token sequence of the other branch (h+w+1). Then, the Transformer Self-Attention operation is performed on the two branches respectively, and the feature maps of the two branches are concat and then restored to the original coarse-grained branch feature map size through 1*1conv, so that the fine-grained branch can also obtain coarse-grained information from the large-scale branch, which can bring effective multi-scale branch feature fusion. After that, the feature information fused by the feature information fusion module is input into the CNN branch (the decoding part of the CNN network), and effectively interacts and fuses with the feature information of the CNN branch. The Transformer branch effectively captures the characteristics of long-range dependencies to provide information supplement for the CNN branch, so as to improve the performance of the model and optimize the segmentation accuracy of the model. The module structure diagram is shown in the figure below: Figure 2 As shown, the overall formula is as follows:

[0076] For the same stage i (i=1, 2, 3, 4) of the two branches of Transformer:

[0077]

[0078] The fine-grained feature information is expressed as:

[0079]

[0080] Perform average pooling on the fine-grained branches:

[0081]

[0082] The Transformer self-attention operation is performed on the fine-grained sequence after average pooling and the coarse-grained branch after serialization, and the new coarse-grained branch information is expressed as:

[0083]

[0084] The final coarse-grained branch output is:

[0085]

[0086] Correspondingly, the calculation steps of fine-grained branch output are as follows to obtain the final fine-grained branch output:

[0087]

[0088] (3) Multi-branch edge information fusion module (GIFB).

[0089] In order to better obtain edge information, this embodiment outputs masks as a supplement to edge information through deep supervision in the decoder stage of the Transformer branch and the CNN branch. The masks output by the two branches are used as input information in the multi-branch edge information fusion module. There are four input information in this module: low-level features, high-level features, masks output by the Transformer branch, and masks output by the CNN branch. First, we adjust the size of high-level features to match the size of low-level features by using depthwise separable convolution (DWConv) and bilinear interpolation. Secondly, we divide the two feature maps into four groups along the channel dimension (C) and connect the low-level features of each group with the high-level features to obtain four groups of fused features. At the same time, the masks from the two branches are connected to the four groups of fused features. Next, a 3*3 convolution kernel is used with different expansion rates {1, 2, 5, 7} to each group to obtain feature information of different scales. Finally, the four groups of fused features are connected and the size is restored using 1*1 convolution to achieve the interaction of feature information of different scales and the extraction and fusion of edge information of different branches. The module structure diagram is shown in the figure. Figure 3 shown.

[0090] (4) Multi-level feature information fusion module (MLFB).

[0091] The feature aggregation layer has a great impact on the quality of the segmentation result, so this embodiment adopts a multi-level feature information fusion module (MLFB) to achieve more effective output feature fusion in the decoder. Figure 4 As shown, for the last three stages of the two-branch decoder, in each branch, for a given output feature map Upsample the feature maps to the same dimension, where i∈{1, 2, 3}, and then fully connect the three feature maps to extract a large feature map F∈R C×H×W Next, the channel attention mechanism is used to focus on the relationship between features. The channel attention weight matrix can be expressed as M C ∈R C×1×1, the channel attention calculation formula is expressed as:

[0092]

[0093] Next, we use the spatial attention mechanism to focus on the relationship within the feature map space to generate a spatial attention feature map. The spatial attention weight matrix can be expressed as: S (X)∈R H,W , two pooling methods are used in the channel dimension to produce 2D feature maps: Therefore, the calculation formula of spatial attention can be expressed as:

[0094]

[0095] X=F+M S (F)

[0096] Finally, the transformed feature output is obtained:

[0097]

[0098] The outputs of the two branches are effectively interactively fused and output to the final stage of the CNN branch decoder to obtain the final prediction result.

[0099] 5. Model training.

[0100] The model in this embodiment belongs to the fully supervised method, so each training set image corresponds to the corresponding mask. During the training process, the batch size is set to 16, and the optimizer uses the adam optimizer. At the same time, we set the initial learning rate to 0.0001 and the decay coefficient to 0.00001. The training process of the model framework includes multiple epochs, each epoch contains many iterations, and a verification is performed at the end of all iterations of each epoch. The total number of epochs for one training is set to 200.

[0101] Reference Figure 1 , Figure 1 GT represents the label. The loss function used by the model in supervision is a combination of the IOU loss function, the DICE loss function, and the BCE loss function. The final loss function can be expressed as:

[0102] L total =αL final (G,S final )+βL t (G,S t )+γL c (G,S c )

[0103]

[0104] Among them, L final (G,S final ) represents the final prediction loss, L t (G,S t ) represents the Transformer branch loss, L c (G,S c ) represents the CNN branch loss, α, β, and γ represent the loss weights, which are set to 0.6, 0.2, and 0.2 respectively, G represents the label, S final Represents the model prediction result, S t represents the mask output of the decoding part of the Transformer network, represents the mask output of the i-th Transformer decoder, S c represents the mask output of the decoding part of the CNN network, represents the mask output of the i-th CNN decoder, L IOU represents the IOU loss function, L DICE represents the DICE loss function, L BCE represents the BCE loss function, λ i represents the weight of the i-th Transformer decoder, which is set to 0.5, 0.4, 0.3, 0.2, and 0.1 respectively, and η i represents the weights of the i-th CNN decoder, which are set to 0.5, 0.3, 0.2, and 0.1 respectively.

[0105] Compared with the existing models, the medical image segmentation model in the above specific embodiment has the following advantages:

[0106] 1. Multi-scale semantic information fusion: This model adopts a scheme that combines a coarse-grained branch and a fine-grained branch in the Transformer branch, and effectively interacts and fuses the information extracted by the two branches through self-attention, and finally inputs it into the CNN branch for fusion interaction, which effectively alleviates the insufficient and missed information extraction at the same scale, can effectively solve the problem of large differences in the size of segmented targets in medical images, and can effectively improve the generalization ability of the model.

[0107] 2. Efficient fusion of CNN branch and Transformer branch: This model effectively combines the CNN method and the Transformer method. Unlike the imbalance in the importance of the two methods in previous combination methods, the CNN branch and the Transformer branch in this method are independent branches. The original image is processed separately by the two independent branches and then effectively interactively fused, so that this model can effectively combine the respective advantages of the two methods, solving the problem that the CNN method cannot effectively capture long-range dependencies and the deficiency of the Transformer method in extracting local information.

[0108] 3. Dual-branch edge information extraction: In the decoder stage of the Transformer branch and the CNN branch, this model supplements the edge information by outputting the output information of each decoder stage as a mask. The masks output by the two branches are used as input information in the multi-branch edge information fusion module, and are effectively combined with the low-level feature information and high-level feature information of the decoder stage through group connection. This realizes the interaction of feature information of different scales and the extraction and fusion of edge information of different branches, which can effectively refine the edge of the segmentation target and help doctors obtain more accurate diagnosis.

[0109] 4. Multi-level feature information fusion: In the final stage of decoder feature information output, this model effectively fuses the feature information of the last three stages of the CNN branch and the Transformer branch decoder, and adds channel attention and spatial attention mechanisms, so that the model can effectively improve the segmentation performance through a multi-level feature fusion strategy, in order to obtain more accurate segmentation results.

[0110] In an embodiment of the present invention, a medical image segmentation method is also provided, comprising:

[0111] Medical image segmentation is performed using a medical image segmentation model obtained by the training method of the medical image segmentation model provided by the present invention.

[0112] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0113] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

Claims

1. A training method for a medical image segmentation model, characterized in that: include: Supervised training of image segmentation models using sample medical images; Wherein, the medical image segmentation model includes a Transformer network and a CNN network; The encoding part of the Transformer network includes a coarse-grained encoding branch, a fine-grained encoding branch and a feature information fusion branch, the feature outputs of the coarse-grained encoding branch and the fine-grained encoding branch are feature fused through the feature information fusion branch, and the feature output of the feature information fusion branch is used as a feature input of the decoding part of the Transformer network; The feature output of the feature information fusion branch is also used as the feature input of the encoding part of the CNN network, and the feature input of the decoding part of the CNN network includes the feature output of the encoding part of the CNN network; The feature outputs of the decoding part of the Transformer network and the decoding part of the CNN network are fused through a target fusion module and then decoded through a target decoder to obtain a segmentation result of the sample medical image; The sample medical image is simultaneously used as the image input of the coarse-grained coding branch, the fine-grained coding branch and the coding part of the CNN network, and the image input size of the coarse-grained coding branch is larger than the image input size of the fine-grained coding branch.

2. The training method of the medical image segmentation model according to claim 1, characterized in that: The medical image segmentation model also includes an edge information fusion branch; The input of the edge information fusion branch includes the mask output of the decoding part of the Transformer network, the mask output of the decoding part of the CNN network, and the feature output of the encoding part of the CNN network, and the feature input of the decoding part of the CNN network also includes the feature output of the edge information fusion branch.

3. The training method of the medical image segmentation model according to claim 2, characterized in that: The coarse-grained coding branch and the fine-grained coding branch each include four Transformer encoders connected in sequence, the feature information fusion branch includes four feature information fusion modules, the four feature information fusion modules correspond to the four Transformer encoders in sequence, and each feature information fusion module is used to fuse feature outputs of the corresponding Transformer encoders in the coarse-grained coding branch and the fine-grained coding branch; The decoding part of the Transformer network includes four Transformer decoders connected in sequence, the four Transformer decoders correspond to the four feature information fusion modules in reverse order, and the feature input of each Transformer decoder includes the feature output of the corresponding feature information fusion module; The feature input of the target fusion module includes the feature outputs of the last three Transformer decoders.

4. The training method of the medical image segmentation model according to claim 3, characterized in that: The encoding part of the CNN network includes six CNN encoders connected in sequence, the middle four CNN encoders correspond to the four feature information fusion modules in sequence, and the feature input of each of the four CNN encoders in the middle includes the feature output of the corresponding feature information fusion module; The edge information fusion branch includes five edge information fusion modules connected in sequence, the five edge information fusion modules correspond to the first five CNN encoders in reverse order, and the first four edge information fusion modules also correspond to the four Transformer decoders in sequence; The decoding part of the CNN network includes five CNN decoders connected in sequence, and the last four CNN decoders correspond to the first four edge information fusion modules in sequence; The input of each edge information fusion module includes the feature output of the corresponding CNN encoder, the input of each edge information fusion module in the first four edge information fusion modules also includes the mask output of the corresponding CNN decoder and the mask output of the corresponding Transformer decoder, the first edge information fusion module also includes the feature output of the last CNN decoder, the input of each CNN decoder in the last four CNN decoders includes the feature output of the corresponding edge information fusion module, and the feature input of the first CNN decoder is the feature output of the last CNN decoder; The feature input of the target fusion module includes the feature outputs of the last three CNN decoders.

5. The training method of the medical image segmentation model according to claim 4, characterized in that: The target decoder is a CNN decoder; The feature input of the target decoder also includes the feature output of the last edge information fusion module, and the input of the last edge information fusion module also includes the mask output of the target decoder.

6. The training method of the medical image segmentation model according to claim 4, characterized in that: The information processing steps of each feature information fusion module include: Reshape the size of the fine-grained feature map to be the same as the size of the coarse-grained feature map, the fine-grained feature map is the output of the corresponding Transformer encoder in the fine-grained encoding branch, and the coarse-grained feature map is the output of the corresponding Transformer encoder in the coarse-grained encoding branch; Performing one-dimensional average pooling and flattening processing on the resized fine-grained feature map to obtain a first intermediate feature map; Performing one-dimensional average pooling and flattening processing on the coarse-grained feature map to obtain a second intermediate feature map; The first intermediate feature map is concatenated with the coarse-grained feature map, and then a self-attention operation is performed to obtain a third intermediate feature map; The second intermediate feature map is concatenated with the resized fine-grained feature map, and then a self-attention operation is performed to obtain a fourth intermediate feature map; The third intermediate feature map and the fourth intermediate feature map are concatenated and then 1*1 convolved to obtain the feature output of the feature information fusion module; The information processing steps of each of the four CNN encoders in the middle include: The feature output of the CNN encoder is convolved with the feature output of the corresponding feature information fusion module to obtain a fifth intermediate feature map; The fifth intermediate feature map is sequentially subjected to 1*1 convolution, batch normalization and linear rectification to obtain the feature output of the CNN encoder.

7. The training method of the medical image segmentation model according to claim 4, characterized in that: The information processing steps of each edge information fusion module include: Reshape the size of low-level features to the same size as high-level features, where the low-level features are feature outputs of the corresponding CNN encoders and the high-level features are feature outputs of the upper-level modules; Respectively dividing the high-level features and the resized low-level features into four one-to-one corresponding feature maps along the channel dimension, and splicing the corresponding feature maps to obtain four sixth intermediate feature maps; Concatenate each of the sixth intermediate feature maps with the mask output of the corresponding CNN decoder and the mask output of the corresponding Transformer decoder to obtain four seventh intermediate feature maps; Performing 3*3 convolution on the four seventh intermediate feature maps respectively and then splicing them to obtain an eighth intermediate feature map, wherein the four 3*3 convolutions respectively use different expansion rates; Performing a 1*1 convolution on the eighth intermediate feature map obtains the feature output of the edge information fusion module.

8. The training method of the medical image segmentation model according to claim 4, characterized in that: The information processing steps of the target fusion module include: The feature outputs of the last three Transformer decoders are upsampled to the same dimension and then concatenated to obtain a first Transformer feature map; The feature outputs of the last three CNN decoders are upsampled to the same dimension and then concatenated to obtain a first CNN feature map; The first Transformer feature map and the first CNN feature map are respectively subjected to attention calculation processing to obtain a second Transformer feature map and a second CNN feature map. The formula for the attention calculation processing is: X=F+M S (F) in, Represents the feature output of the attention calculation process, Conv 1×1 represents 1*1 convolution, M C represents the channel attention calculation, M S represents spatial attention calculation, F represents the feature input of channel attention calculation, MLP represents multi-layer perceptron, AvgPool and MaxPool represent global average pooling and maximum pooling respectively, σ represents batch normalization, W1 and W2 represent hidden neurons, and They represent the global average pooling result and the maximum pooling result of F, respectively. X represents the feature input of spatial attention calculation. Conv 7×7 represents 7*7 convolution, and Represent the global average pooling result and maximum pooling result of X respectively.

9. The training method of the medical image segmentation model according to claim 4, characterized in that: The loss function L used in the training of the image segmentation model is total for: L total =αL final (G,S final )+βL t (G,S t )+γL c (G,S c ) Among them, L fimal (G,S final ) represents the final prediction loss, L t (G,S t ) represents the Transformer branch loss, L c (G,S c ) represents the CNN branch loss, α, β and γ all represent loss weights, G represents the label, S final Represents the model prediction result, S t represents the mask output of the decoding part of the Transformer network, represents the mask output of the i-th Transformer decoder, S c represents the mask output of the decoding part of the CNN network, represents the mask output of the i-th CNN decoder, L IOU represents the IOU loss function, L DICE represents the DICE loss function, L BCE represents the BCE loss function, λi represents the weight of the i-th Transformer decoder, and ηi represents the weight of the i-th CNN decoder.

10. A medical image segmentation method, characterized in that: include: Medical image segmentation is performed using a medical image segmentation model obtained by the training method for a medical image segmentation model described in any one of claims 1 to 9.

Images