A multi-modal medical image segmentation method based on convolutional transformer

Abstract

The application discloses a multi-modal medical image segmentation method based on convolutional Transformer, and belongs to the technical field of medical image segmentation.The multi-modal data is used, so that the medical image can be fully utilized; compared with the traditional Transformer module based on the self-attention mechanism, the convolutional Transformer module can obtain the global feature of the neural network while reducing the operation amount; the feature pyramid post-fusion mode is used, so that the final segmentation result can retain the high-level semantic information of the convolution while containing the low-level detail information; the model iteration direction is jointly constrained by the Dice Loss and the Focal Loss, the adverse influence of the reverse propagation of the Dice Loss is solved, and the training is more stable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation, and in particular to a multi-modal medical image segmentation method based on convolutional Transformer. BACKGROUND

[0002] Medical images play an important role in doctors understanding the causes of patients, with the development of science and technology, the number of medical images grows exponentially, since different modal medical images often play a complementary role in the diagnosis of patients, so how to effectively use multi-modal medical images is an important problem.

[0003] With the development of machine learning and deep learning, computer-aided diagnosis plays a vital role in clinical medicine. It is also a common task in computer vision to help doctors diagnose by segmenting medical images. Convolutional neural networks are commonly used to extract features, but this convolution can only extract local features, and the receptive field is very limited. Even through different degrees of down-sampling or hole convolution, the effect is not particularly obvious.

[0004] The emergence of Transformer (a method to improve model accuracy) solves the limitations of convolutional receptive field. Transformer obtains global information through multi-head self-attention mechanism, flattens the image tensor to calculate the weight matrix, establishes global dependency, and extracts features based on global dependency. However, Transformer makes the parameter calculation very large, and more computing resources are often needed for larger medical images, which limits the development of Transformer in medical image segmentation. The above problems need to be solved, therefore, a multi-modal medical image segmentation method based on convolutional Transformer is proposed. SUMMARY

[0005] The technical problem to be solved by the present application is how to propose a multi-modal medical image segmentation model that can effectively reduce the Transformer operation amount without affecting the accuracy. A multi-modal medical image segmentation method based on convolutional Transformer is provided, which uses double-path parallel Dense convolution combined with Transformer, uses convolution self-attention mechanism instead of multi-head self-attention mechanism, greatly reduces the parameter operation, and through the construction of pyramid feature fusion method, the high-level semantic information and low-level detail information of convolution are fused, realizing high-precision medical image segmentation, and providing more accurate image basis for medical diagnosis.

[0006] The present application solves the above technical problems by the following technical solutions, the present application comprises the following steps:

[0007] S1: dataset division

[0008] Obtain a multi-modal medical image dataset, randomly divide a training set and a test set;

[0009] S2: image preprocessing

[0010] After reading the original image in the dataset, the original image is preprocessed;

[0011] S3: model construction

[0012] Construct a multi-modal medical image segmentation model based on convolutional Transformer;

[0013] S4: model training

[0014] Use a combined loss function to calculate the model loss, train the model constructed in step S3, and obtain an optimal model;

[0015] S5: image segmentation

[0016] Input the image to be segmented into the optimal model obtained in step S4, and output the segmentation result.

[0017] Further, the step S2 includes the following processing process:

[0018] S21: After reading the original image, the original image is normalized;

[0019] S22: The image background is cropped by center cropping.

[0020] Further, in the step S3, the multi-modal medical image segmentation model based on convolutional Transformer adopts the encoder-decoder symmetric structure of the U-Net network, including an encoder, a decoder, and a feature fusion module, wherein the encoder and the decoder are both five layers, each layer is a Double Block module, the corresponding layers of the encoder and the decoder are connected by jumping, the feature fusion module is connected with each Double Block module in the decoder, and the feature fusion module uses the feature post-fusion method of the feature pyramid.

[0021] Further, the Double Block module adopts a double parallel structure, including a dense convolution module and a ConvTF Block module, after the dense convolution module and the ConvTF Block module respectively obtain the tensors entering the Double Block module, they respectively perform convolution operation and obtain attention mechanism weight operation, finally the tensors of the ConvTF Block module and the dense convolution module are spliced and then pass through a 1x1 convolution for channel number restoration.

[0022] Further, the processing procedure of the dense convolution module is as follows:

[0023] X0=Conv kernel＝3×3 , padding＝1 (input)

[0024] X i =Conv kernel＝3×3 , padding＝1 (Concat(X0,X1…X i-1 ))

[0025] Wherein, input is the tensor entering the dense convolution module, X0,X1…X i is the output result through each convolution operation, Conv is the convolution operation, kernel is the convolution kernel size, padding is the convolution padding length, and Concat represents the tensor splicing operation.

[0026] Further, the ConvTF Block module uses convolution operations with different convolution kernel sizes of 5x5, 11x11 and 21x21 to replace the operation of obtaining global weights in the self-attention mechanism, and the specific operation process is as follows:

[0027] Q=Conv kernel＝5×5 (Conv kernel＝1×1 (input))

[0028] K=Conv kernel＝11×11 (Conv kernel＝1×1 (input))

[0029] V=Conv kernel＝21×21 (Conv kernel＝1×1 (input))

[0030] Attention=Softmax(QxK) 1 / 2 xV

[0031] Wherein, input is the tensor entering the ConvTF Block module, Conv is the convolution operation, kernel is the convolution kernel size, padding is the convolution padding length, Concat represents the tensor splicing operation, Q, K and V are respectively the query vector, index vector and content vector, and Softmax is the activation function.

[0032] Further, each layer in the decoder obtains a feature map, the feature fusion module first up-samples all the feature maps of each layer to the size of the final output segmentation result, then performs tensor splicing to obtain a feature pyramid shape, and finally obtains the output segmentation result through tensor reduction by a 1x1 convolution kernel:

[0033] F = Conv kernel＝1×1 (Concat(F0, F1…F5))

[0034] Wherein, F is the final segmentation result, F0…F5 is the feature map obtained by each layer of the decoder, Concat represents the tensor splicing operation, Conv is the convolution operation, and kernel is the convolution kernel size.

[0035] Further, in the step S4, the loss function part uses the combined loss of Dice Loss and Focal Loss to constrain the model iteration direction, and the specific implementation process is as follows:

[0036]

[0037]

[0038]

[0039]

[0040] Wherein, FP P (c), FN P (c), TP P (c) respectively represent true positive, false negative, and false positive for class c; g n (c) is the doctor's labeled real image when the voxel N is of class c; p n (c) is the probability predicted by the model for c; τ, α, β are weight parameters.

[0041] Further, in the step S5, the pre-processed multi-modal medical images in the test set are also used to test the optimal model obtained in the step S4.

[0042] Compared with the prior art, the multi-modal medical image segmentation method based on convolutional Transformer has the following advantages: the multi-modal medical image segmentation method based on convolutional Transformer makes full use of medical images by using multi-modal data; compared with the traditional Transformer module based on self-attention mechanism, the convolutional Transformer module reduces the computational complexity while obtaining global features; the feature pyramid post-fusion method makes the final segmentation result retain the high-level semantic information of convolution while containing the low-level detailed information; the DiceLoss and Focal Loss jointly constrain the iteration direction of the model, solve the adverse effect of Dice Loss in back propagation, and make the training more stable. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a structure diagram of the multi-modal medical image segmentation model based on convolutional Transformer in the embodiment one of the present application.

[0044] Figure 2 is a structure diagram of the Double Block module in the embodiment one of the present application.

[0045] Figure 3 is a structure diagram of the ConvTF Block module in the embodiment one of the present application.

[0046] Figure 4 is an implementation process diagram of the multi-modal medical image segmentation method based on convolutional Transformer in the embodiment two of the present application.

[0047] Figure 5 is an output result diagram in the embodiment two of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application will be described in detail below, and the embodiments are implemented on the premise of the technical scheme of the present application, and detailed implementation modes and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0049] Embodiment one

[0050] The embodiment provides a technical scheme: a multi-modal medical image segmentation method based on convolutional Transformer, comprising the following steps:

[0051] I. Original image preprocessing

[0052] 1) After reading the original image, the original image is normalized;

[0053] 2) The medical image background accounts for a large proportion of the whole image, and the background is of no help for segmentation. Since the region to be segmented is in the middle of the image, the data is balanced by centering the data and removing the black background that has no effect on segmentation;

[0054] 3) For multi-modal data, by combining the slices of each modality or merging the slices into multiple channels, the preprocessed data is finally saved in the form of an array.

[0055] II. Constructing a multi-modal medical image segmentation model based on convolutional Transformer

[0056] 1) The multi-modal medical image segmentation model adopts the encoder-decoder symmetric structure of the U-Net network, as shown in Figure 1 , wherein the encoder and the decoder are 5 layers, each layer is a Double Block module (double parallel feature extraction module), and the decoder can obtain more features of the encoder through the jump connection. Finally, the feature pyramid feature post-fusion method is used to make the final segmentation result have a lot of feature information.

[0057] 2) The Double Block module uses a double parallel structure, as shown in Figure 2 , the local feature extraction uses a dense convolution module. Since the dense convolution module does not need to re-learn redundant feature maps, it has fewer parameters compared to traditional convolutional networks. The specific processing process of the dense convolution module is as follows:

[0058] X0=Conv kernel＝3×3 , padding＝1 (input)

[0059] X i =Conv kernel＝3×3 , padding＝1 (Concat(X0,X1…X i-1 ))

[0060] Wherein, input is the tensor entering the dense convolution module, X0,X1…X i is the output result through each convolution operation, Conv is the convolution operation, kernel is the convolution kernel size, padding is the convolution padding length, and Concat represents the tensor splicing operation.

[0061] Using the dense convolution module, each convolution layer of the network is connected with the previous convolution layer, realizing feature reuse, while reducing the number of parameters to a certain extent, improving the back propagation of the gradient, and making the feature extraction more comprehensive.

[0062] 3) The structure of the ConvTF Block module (convolutional self-attention mechanism module) is shown in Figure 3 To avoid excessive parameter calculation caused by the multi-head self-attention mechanism, the ConvTF Block module uses convolution operations with different kernel sizes of 5x5, 11x11, and 21x21 to replace the operation of obtaining global weights in the self-attention mechanism. The specific operation process is as follows:

[0063] Q = Conv kernel＝5×5 (Conv kernel＝1×1 (input))

[0064] K = Conv kernel＝11×11 (Conv kernel＝1×1 (input))

[0065] V = Conv kernel＝21×21 (Conv kernel＝1×1 (input))

[0066] Attention = Softmax(Q x K) 1 / 2 x V

[0067] Where input is the tensor entering the ConvTF Block module, Conv is the convolution operation, kernel is the convolution kernel size, padding is the convolution padding length, Concat represents the tensor concatenation operation, Q, K, and V are the query vector, index vector, and content vector, respectively, and Softmax is the activation function.

[0068] Finally, the ConvTF Block module is concatenated with the tensor of the dense convolution module, and then a 1x1 convolution is performed to restore the channel number.

[0069] The feature fusion module, each layer of the decoder obtains a feature map, by first upsampling all the feature maps of each layer to the same size as the final output result, then performing tensor concatenation to obtain a feature pyramid shape, and then performing tensor restoration through a 1x1 convolution kernel, finally obtaining the output segmentation result:

[0070] F = Conv kernel＝1×1 (Concat(F0, F1…F5))

[0071] Where F is the final segmentation result, F0…F5 is the feature map obtained by each layer of the decoder, Concat represents the tensor concatenation operation, Conv is the convolution operation, and kernel is the convolution kernel size.

[0072] III. The combination of Dice Loss and Focal Loss is used to constrain the iteration direction of the model

[0073] The loss function part uses a combination loss of Dice Loss+Focal Loss to constrain the iteration direction of the model, and the specific implementation process is as follows:

[0074]

[0075]

[0076]

[0077]

[0078] Where, FP P (c), FN P (c), TP P (c) respectively represent true positive, false negative, false positive for class c; g n (c) is the doctor's labeled real image when the voxel N is the class c; p n (c) is the probability predicted by the model that it is c; τ, α, β are weight parameters.

[0079] Embodiment two

[0080] The application will be further described below mainly in combination with the drawings and specific embodiments.

[0081] This embodiment selects brain multi-modal MRI images for analysis, selects Flair, T1, T1ce, T2 four modal medical images to illustrate the corresponding results after implementation of the application as shown in Figure 5 The specific implementation steps are as follows (see Figure 4 ) :

[0082] A. Obtain a multi-modal medical image dataset, and randomly divide it into a training set and a test set;

[0083] B. The computer reads the original image of the multi-modal MRI image. First, pre-process the data: first, normalize each modal data respectively; then, through the center cropping method, crop the medical image background to reduce the network operation amount;

[0084] C. Construct a multi-modal medical image segmentation model based on convolution Transformer, use the double-path parallel structure of Double Block to replace the original convolution module on the basis of the original U-Net network; at the same time, add the post-fusion method of the feature pyramid;

[0085] D. Use the combination loss function of Dice Loss and Focal Loss to calculate the model loss, train the model, and obtain the optimal model;

[0086] E. Use the test set in A to predict the optimal model in D, and evaluate the optimal model using the segmentation evaluation index.

[0087] After implementing the above steps, the final output result is as shown in Figure 5 , and the network prediction result is the prediction result of the optimal model. Figure 5

[0088] In summary, the multi-modal medical image segmentation method based on convolutional Transformer of the above-mentioned embodiments improves the medical image segmentation model based on U-Net network. On the basis of traditional addition of Transformer to obtain global features, the convolutional Transformer is replaced to combine dense convolution and convolutional Transformer to obtain DoubleBlock as the encoder and decoder of the network, and the post-fusion method of the feature pyramid is used, so that the final segmentation result contains high-level semantic information and also contains low-level detailed information. The model iterative direction is jointly constrained by using Dice Loss and Focal Loss, so that the model gradient convergence direction is better, the model convergence effect is better, and the final medical image segmentation result is more accurate.

[0089] Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.​

Claims

1. A multimodal medical image segmentation method based on convolutional Transformer, characterized in that, Includes the following steps: S1: Dataset Partitioning Obtain a multimodal medical image dataset and randomly divide it into training and test sets; S2: Image Preprocessing After reading the original images from the dataset, the original images are preprocessed. S3: Building the Model Construct a multimodal medical image segmentation model based on convolutional Transformer; S4: Model Training The model loss is calculated using the combined loss function, and the model constructed in step S3 is trained to obtain the optimal model. S5: Image Segmentation Input the image to be segmented into the optimal model obtained in step S4, and output the segmentation result; In step S3, the multimodal medical image segmentation model based on convolutional Transformer adopts the encoder-decoder symmetric structure of U-Net network, including encoder, decoder, and feature fusion module. The encoder and decoder are both five layers, and each layer is a Double Block module. Corresponding layers of encoder and decoder are skipped. The fusion module is connected to each Double Block module in the decoder. The feature fusion module uses the feature pyramid feature post-fusion method. The Double Block module adopts a dual-path parallel structure, including a dense convolution module and a ConvTF Block module. After the dense convolution module and the ConvTF Block module obtain the tensors entering the Double Block module, they respectively perform convolution operations and obtain attention mechanism weight operations. Finally, the tensors of the ConvTF Block module and the dense convolution module are concatenated and then restored to the channel number through a 1×1 convolution.

2. The multimodal medical image segmentation method based on convolutional Transformer according to claim 1, characterized in that: Step S2 includes the following processing procedures: S21: After reading the original image, regularize the original image; S22: Then, crop the image background using the center cropping method.

3. The multimodal medical image segmentation method based on convolutional Transformer according to claim 1, characterized in that: The processing procedure of the dense convolution module is as follows: X0=Conv kernel=3×3，padding=1 (input) X i =conv kernel＝3×3，padding＝1 (Concat(X0,X1…X i-1 )) Where input is the tensor entering the dense convolution module, X0, X1…X i The output of each convolution operation is represented by Conv, where Conv is the convolution operation, kernel is the kernel size, padding is the padding length, and Concat indicates a tensor concatenation operation.

4. The multimodal medical image segmentation method based on convolutional Transformer according to claim 3, characterized in that: The ConvTF Block module replaces the operation of obtaining global weights in the attention mechanism with convolution operations of different kernel sizes of 5×5, 11×11, and 21×21. The specific operation process is as follows: Q=Conv kernel＝5×5 (Conv kernel＝1×1 (input)) K=Conv kernel＝11×11 (Conv kernel＝1×1 (input)) V=Conv kernel＝21×21 (Conv kernel＝1×1 (input)) Attention=Softmax(Q×K) 1 / 2 ×V Wherein, input is the tensor entering the ConvTF Block module, Conv is the convolution operation, kernel is the convolution kernel size, padding is the convolution padding length, Concat indicates the tensor concatenation operation, Q, K, and V are the query vector, index vector, and content vector, respectively, and Softmax is the activation function.

5. The multimodal medical image segmentation method based on convolutional Transformer according to claim 4, characterized in that: Each layer in the decoder obtains a feature map. The feature fusion module first upsamples all feature maps from each layer until the final output segmentation result has the same size, then performs tensor concatenation to obtain the shape of a feature pyramid. A 1×1 convolution kernel is then used to reconstruct the tensor, finally obtaining the output segmentation result. F=Conv kernel=1×1 (Concat(F0,F1…F5)) Where F represents the final segmentation result, F0…F5 represents the feature maps obtained by each layer of the decoder, Concat represents the tensor concatenation operation, Conv represents the convolution operation, and kernel represents the convolution kernel size.

6. The multimodal medical image segmentation method based on convolutional Transformer according to claim 5, characterized in that: In step S4, the loss function uses a combination of Dice Loss and Focal Loss to constrain the direction of model iteration. The specific implementation process is as follows: Among them, FP P (c) FN P (c) TP P (c) represents true positives, false negatives, and false positives for category c, respectively; g n (c) is the doctor-annotated real image when voxel N is of category c; p n (c) represents the probability that the model predicts the value as c; τ, α, and β are weighting parameters.

7. The multimodal medical image segmentation method based on convolutional Transformer according to claim 1, characterized in that: In step S5, the optimal model obtained in step S4 is also tested using preprocessed multimodal medical images from the test set.

Images