MRI (Magnetic Resonance Imaging) image segmentation model training method, MRI image segmentation method and related device

Abstract

The embodiment of the invention discloses an MRI (Magnetic Resonance Imaging) image segmentation model training method, an MRI image segmentation method and related devices. The encoder of the U-Net network performs down-sampling on the multi-modal MRI image to obtain a feature map of each modal MRI image, the cross-modal module fuses the plurality of feature maps of the multi-modal MRI image, the decoder of the U-Net network performs up-sampling on a fused feature matrix, the up-sampled feature matrix is mapped into a target feature map, and the target feature map is mapped into the multi-modal MRI image. And processing the target feature map to obtain a prediction segmentation result, and adjusting model parameters according to the prediction segmentation result and the labels to obtain a target segmentation model. The cross-modal module has the capability of processing the multi-modal MRI image, can capture the long-distance dependency relationship of the multi-modal MRI image input, and greatly improves the segmentation precision of the brain tumor region of the MRI image. Lightweight convolution operation is adopted in up-sampling and down-sampling of the U-Net network, and the risk of overfitting is effectively reduced on the premise that the model structure is not complicated and the parameter quantity is not increased.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and specifically to an MRI image segmentation model training method, an MRI image segmentation method and related devices. Background Art

[0002] Magnetic resonance imaging (MRI) is a non-invasive imaging technique that can clearly display soft tissue lesions and is widely used in the diagnosis and treatment of brain tumors. Different MRI imaging sequences (such as FLAIR, T1, T1ce, and T2) can display the same tissue from different angles or forms, and these different modal MRI images can reflect different characteristics of the tumor area.

[0003] At present, most brain tumor segmentation work still relies on manual delineation by doctors. This method is not only time-consuming and laborious, but also affected by personal experience. Different experts or the same expert at different times may produce inconsistent segmentation results for the same patient. Therefore, using multimodal models to achieve automated brain tumor segmentation is an important direction in the current clinical treatment of brain tumors. Among them, the accuracy of brain tumor segmentation is a key factor in determining whether deep learning can be introduced into clinical treatment.

[0004] Compared with traditional machine learning methods, the recent technological development of convolutional neural networks and Transformers in computer vision has provided new methods for improving the accuracy of brain tumor segmentation. However, in related schemes, MRI images are processed based on convolution operations and U-Net segmentation network models. Since this method is based on convolutional neural networks, the fixed receptive field of convolution operations makes it difficult for U-Net to capture the long-distance dependencies of multimodal images and cannot utilize the multimodal information provided by MRI datasets, resulting in low accuracy in brain tumor segmentation, which limits the application of automated brain tumor segmentation. Summary of the invention

[0005] The embodiments of the present application provide an MRI image segmentation model training method, an MRI image segmentation method and related devices, so that the trained segmentation model inherits the efficient structure of U-Net and enables the traditional U-Net to have the feature extraction capability of multi-modal MRI image data sets by integrating the cross-modal module, thereby improving the segmentation accuracy and avoiding overfitting.

[0006] The first aspect of the embodiment of the present application provides a method for training an MRI image segmentation model, the method comprising:

[0007] Acquire an image training set, wherein the image training set includes multiple groups of training samples, each group of the training samples includes a multimodal MRI image and an annotation of each MRI image, wherein the annotation is used to represent a real area of ​​a target object in the MRI image;

[0008] Acquire an initial segmentation model, wherein the initial segmentation model includes a U-Net network and a cross-modal module;

[0009] The image training set is input into the initial segmentation model, then,

[0010] The encoder of the U-Net network downsamples the multimodal MRI images in each group of the training samples to obtain a feature map of the MRI image of each modality;

[0011] The cross-modality module fuses the multiple feature maps of the multimodal MRI images in each group of the training samples to obtain a fused feature matrix;

[0012] The decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, and maps the upsampled feature matrix into a target feature map using a matrix of a preset size, and processes the target feature map to obtain a predicted segmentation result, wherein the predicted segmentation result is used to represent a predicted region of the target object in the MRI image;

[0013] A loss function is constructed according to the predicted area represented by the predicted segmentation result and the real area represented by the annotation, and the model parameters of the initial segmentation model are adjusted according to the loss value of the loss function until the model training is stopped when the convergence condition is met to obtain the target segmentation model.

[0014] A second aspect of an embodiment of the present application provides an MRI image segmentation method, the method comprising:

[0015] Acquiring a target multimodal MRI image to be processed;

[0016] Obtaining a target segmentation model, wherein the target segmentation model includes a U-Net network and a cross-modal module;

[0017] The target multimodal MRI image is input into the target segmentation model, then,

[0018] The encoder of the U-Net network downsamples the target multimodal MRI image to obtain a feature map of the MRI image of each modality;

[0019] The cross-modality module fuses the multiple feature maps of the target multi-modality MRI image to obtain a fused feature matrix;

[0020] The decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, and uses a matrix of a preset size to map the upsampled feature matrix into a target feature map, and processes the target feature map to obtain a predicted segmentation result, which is used to represent the predicted area of ​​the target object in the target multimodal MRI image.

[0021] A third aspect of an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0022] A fourth aspect of an embodiment of the present application provides a computer storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the method of the first aspect.

[0023] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0024] The multimodal MRI images in each group of training samples are downsampled by using the encoder of the U-Net network to obtain the feature map of each modality of the MRI image. The cross-modal module fuses multiple feature maps of the multimodal MRI images in each group of training samples to obtain a fused feature matrix. The decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, and uses a matrix of a preset size to map the upsampled feature matrix to a target feature map. The target feature map is processed to obtain a predicted segmentation result, and the model parameters are adjusted according to the predicted area represented by the predicted segmentation result and the real area represented by the annotation to obtain a target segmentation model. Among them, the cross-modal module has the ability to process multimodal MRI images, can capture the long-distance dependency of the multimodal MRI image input, and greatly improves the segmentation accuracy of the brain tumor area of ​​the MRI image. In addition, both sampling and downsampling on the U-Net network use lightweight convolution operations, thereby effectively reducing the risk of overfitting without complicating the model structure and increasing the number of parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of the MRI image segmentation model training method in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of an exemplary scenario in which the initial segmentation model in an embodiment of the present application performs model training on multiple groups of MRI images based on its model structure;

[0027] Figure 3 An exemplary schematic diagram of the network structure of the cross-modality module in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of comparison results of the segmentation model of the embodiment of the present application and other segmentation models on multiple evaluation indicators;

[0029] Figure 5 A schematic diagram showing the comparison results between the segmentation model of the embodiment of the present application and other segmentation models in terms of parameter quantity, number of floating-point operations per second, and inference time;

[0030] Figure 6 This is a schematic diagram of the comparison results between the segmentation results of the target segmentation model for MRI images and the true labels in the embodiment of the present application;

[0031] Figure 7 A flowchart of the MRI image segmentation method in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application provide an MRI image segmentation model training method, an MRI image segmentation method and related devices, so that the trained segmentation model inherits the efficient structure of U-Net and enables the traditional U-Net to have the feature extraction capability of multi-modal MRI image data sets by integrating the cross-modal module, thereby improving the segmentation accuracy and avoiding overfitting.

[0034] See also Figure 1 In the embodiment of the present application, an embodiment of the MRI image segmentation model training method includes:

[0035] 101. Obtain an image training set, wherein the image training set includes multiple groups of training samples, each group of the training samples includes a multimodal MRI image and an annotation of each MRI image, wherein the annotation is used to represent a real area of ​​a target object in the MRI image;

[0036] The method of this embodiment can be applied to a computer device, which can be a server, terminal, or other computer device capable of performing data processing. When the computer device is a terminal, it can be a terminal device such as a personal computer (PC) or a desktop computer; when the computer device is a server, it can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud databases, cloud computing, and big data and artificial intelligence platforms.

[0037] The computer device can obtain an image training set, which includes multiple groups of training samples, each group of training samples includes a multimodal MRI image and a label of each MRI image, and the MRI image is a magnetic resonance imaging (MRI). For example, the multimodal MRI image can be a multimodal MRI image of the brain, and the multimodal MRI image can be a multimodal MRI image such as FLAIR, T1, T1ce, and T2.

[0038] The real region of the target object represented by the annotation of each MRI image may be any target object, such as the brain, or any other tissue. The real region of the target object in the MRI image, for example, when the target object is the brain, may be the background of the brain MRI image, the entire tumor region in the brain MRI image, the tumor core region in the brain MRI image, the enhanced tumor region in the brain MRI image, or other regions in the brain MRI image.

[0039] The image training set can be derived from the MRI imaging process of any medical institution, or it can be derived from MRI images stored in an existing image database. For example, in some preferred embodiments, the public glioma dataset BraTS2021 provided by the International Medical Image Computing and Computer Assisted Intervention Society (MICCAD) can be used. The dataset discloses a training set, which contains 1251 cases and their annotations. Each case includes brain MRI images of four modalities: FLAIR, T1, T1ce, and T2. The annotations are used to represent the background, the entire tumor (WT), the tumor core (TC), and the enhanced tumor (ET) in the MRI image. For example, the 1251 case data can be divided into training set, validation set, and test set according to the ratios of 0.7, 0.2, and 0.1.

[0040] Before model training, the MRI images can be normalized and cropped to make the MRI images have a uniform size, for example, they can be cropped to a size of 224×224×155.

[0041] 102. Obtain an initial segmentation model, wherein the initial segmentation model includes a U-Net network and a cross-modal module;

[0042] The computer device can obtain an initial segmentation model to be trained, wherein the initial segmentation model includes a U-Net network and a cross-modal module, wherein the U-Net network includes an encoder and a decoder. The initial segmentation model is used to segment the MRI image to determine the region of the target object in the MRI image.

[0043] 103. Input the image training set into the initial segmentation model;

[0044] 104. The encoder of the U-Net network downsamples the multimodal MRI images in each group of the training samples to obtain a feature map of the MRI image of each modality;

[0045] The computer device can input the image training set to the initial segmentation model, so that the initial segmentation model can perform model training on multiple sets of training samples of the image training set based on the machine learning algorithm. For each set of training samples, the encoder downsamples the MRI image of each modality therein to obtain a feature map of the MRI image of each modality.

[0046] 105. The cross-modality module fuses the multiple feature maps of the multimodal MRI images in each group of the training samples to obtain a fused feature matrix;

[0047] After the encoder downsamples each group of training samples and outputs the corresponding feature maps of multiple modal MRI images, the cross-modality module can fuse multiple feature maps of the multi-modal MRI images in each group of training samples to obtain a fused feature matrix.

[0048] 106. The decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, maps the upsampled feature matrix into a target feature map using a matrix of a preset size, and processes the target feature map to obtain a predicted segmentation result, where the predicted segmentation result is used to represent a predicted region of the target object in the MRI image;

[0049] After the cross-modal module outputs the fused feature matrix, the decoder upsamples the fused feature matrix to obtain an upsampled feature matrix. In addition, the computer device maps the upsampled feature matrix to a target feature map using a matrix of a preset size, and processes the target feature map to obtain a predicted segmentation result, that is, the predicted segmentation result represents the predicted area of ​​the target object in the MRI image by the initial segmentation model, such as the predicted result of the complete tumor area in the brain MRI image, or the predicted result of the tumor core area in the brain MRI image, and so on.

[0050] 107. Constructing a loss function according to the predicted area represented by the predicted segmentation result and the real area represented by the annotation, and adjusting the model parameters of the initial segmentation model according to the loss value of the loss function, stopping model training until a convergence condition is met, and obtaining a target segmentation model;

[0051] The computer device can construct a loss function based on the predicted area represented by the predicted segmentation result output by the model and the real area represented by the annotation corresponding to the MRI image in each group of training samples, and adjust the model parameters of the initial segmentation model according to the loss value of the loss function until the model training is stopped when the convergence condition is met to obtain the target segmentation model. The target segmentation model can be used to predict the area of ​​the target object in the MRI image, such as predicting the complete tumor area, the tumor core area or the enhanced tumor area in the brain MRI image.

[0052] In this embodiment, the multimodal MRI images in each group of training samples are downsampled by using the encoder of the U-Net network to obtain the feature map of each modality of the MRI image, the cross-modal module fuses multiple feature maps of the multimodal MRI images in each group of training samples to obtain a fused feature matrix, the decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, and uses a matrix of a preset size to map the upsampled feature matrix to a target feature map, and the target feature map is processed to obtain a predicted segmentation result, and the model parameters are adjusted according to the predicted area represented by the predicted segmentation result and the real area represented by the annotation to obtain a target segmentation model. Among them, the cross-modal module has the ability to process multimodal MRI images, can capture the long-distance dependency of the multimodal MRI image input, and greatly improves the segmentation accuracy of the brain tumor area of ​​the MRI image. In addition, both sampling and downsampling on the U-Net network use lightweight convolution operations, thereby effectively reducing the risk of overfitting without complicating the model structure and increasing the number of parameters.

[0053] based on Figure 1 In the embodiment shown, in a preferred implementation, the encoder of the U-Net network downsamples the multimodal MRI images in each group of training samples to obtain the feature map of the MRI image of each modality, and the specific downsampling method can be that for each group of training samples, the encoder downsamples the MRI image of each modality for the first time to obtain multiple feature maps of the multimodal MRI image. And, in each subsequent downsampling, each feature map obtained by the previous downsampling is downsampled, and each downsampling obtains multiple feature maps.

[0054] For example, assume that the multimodal MRI images of each set of training samples are normalized and cropped to a size of 224×224×155. Figure 2As shown in the figure, in the encoder stage, the MRI images of the above four modalities are downsampled in five stages respectively. Each stage of downsampling is implemented by a 3×3 convolution, batch normalization (batchnorm2d) and activation function LeakyRELU. The number of convolution channels in the five stages is 32, 64, 128, 256 and 512 respectively. The size of the feature map obtained in each downsampling stage is 112×112, 56×56, 28×28, 14×14 and 7×7 respectively.

[0055] When multiple feature maps are obtained by downsampling each time, the cross-modal module can fuse the multiple feature maps of the multimodal MRI images in each group of training samples to obtain a fused feature matrix. For example, for the multiple feature maps obtained by downsampling each time, the cross-modal module connects the multiple feature maps to obtain a connected feature map, adds position encoding to the connected feature map and flattens it into a feature sequence. Based on the attention mechanism, a query vector, a key vector and a value vector are generated according to the feature sequence, and an attention matrix is ​​calculated based on the query vector, the key vector and the value vector, and the attention matrix is ​​reshaped to the same dimension as the feature sequence. The attention matrix is ​​convolved by the feedforward neural network in the cross-modal module, and the dimension of the output of the feedforward neural network is mapped to the target dimension to obtain a fused feature matrix.

[0056] For example, combined with Figure 3 The network structure of the Cross Modality Module is shown in Figure 1. Figure 2 In the scenario shown, for each downsampling stage, the feature maps of the four modalities obtained by downsampling are concatenated together, position encoding is added, and then flattened into a feature sequence of (b, H×W, c), where b is the batch size, H and W are the height and width of the feature map, and c is the number of channels.

[0057] For example, for the first stage of downsampling (stage1), the size of the feature maps of the four modes obtained by downsampling is 112×112×32. The feature maps of these four modes can be concatenated to obtain a concatenated feature map of size 112×112×128, and position encoding is added to the concatenated feature map, which is then flattened into a feature sequence of (b, H×W, c), where the batch size b is the preset value, and after flattening, H×W is 1×12544, and c is 32.

[0058] Among them, the role of position encoding is to help the model capture the spatial information of the image, which is essentially to add a new learnable matrix to the original feature matrix.

[0059] The flattened feature sequence is used as the input of the self-attention mechanism. The self-attention mechanism can also be replaced by a multi-head attention mechanism. Among them, the self-attention mechanism enables the model to pay attention to the relationship between different positions in the input sequence. It allows the model to automatically learn the correlation between different positions when processing sequence data, thereby capturing richer contextual information. The idea of ​​the self-attention mechanism is that for each element in the sequence, the correlation between it and all other elements is calculated, and a weighted vector is generated accordingly. This weighted vector can be regarded as the representation of the element in the current context. The self-attention mechanism can better handle long-distance dependencies because it does not need to process the input sequence in sequence. Therefore, in this way, the model can automatically learn the dependencies between different positions, thereby capturing richer contextual information.

[0060] In the self-attention mechanism, the query vector Query(q), key vector Key(k) and value vector Value(v) are first generated by three fully connected layers, and then the Query and Key are normalized. Then the attention matrix is ​​calculated by the following formula:

[0061]

[0062] in, is the scaling factor, and T is the transpose. Finally, the attention matrix is ​​reshaped into (b, H×W, c) by a fully connected layer as the output of the self-attention module.

[0063] Subsequently, the expressive power of the model is further enhanced through linear projection and feedforward neural network. The feedforward neural network consists of three fully connected layers and two activation functions RELU. Because the feedforward neural network contains multiple linear layers and nonlinear activation functions (Relu functions), by stacking multiple linear layers and nonlinear activation functions, the feedforward neural network can parameterize more complex functions, which enables the model to capture complex relationships in the input data, thereby enhancing the expressive power of the model.

[0064] After that, the output of the feedforward neural network is mapped to the same dimension as the upsampled feature matrix through a 3×3 convolution, batch normalization, and RELU function. For example, if the dimension of the upsampled feature matrix is ​​14×14×512, the output of the feedforward neural network is mapped to the dimension of 14×14×512 through the 3×3 convolution, batch normalization, and RELU function.

[0065] When the cross-modal module processes the multiple feature maps obtained by upsampling at each stage and outputs a fused feature matrix, the decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix. The decoder receives the fused feature matrix obtained by the cross-modal module processing the multiple feature maps obtained by each downsampling of the encoder, and in each upsampling, the decoder performs an upsampling operation according to at least one fused feature matrix to obtain an upsampled feature matrix. Therefore, when using a matrix of a preset size to map the upsampled feature matrix to a target feature map, the matrix of the preset size can be used to map the upsampled feature matrix obtained by the decoder upsampling for the last time to the target feature map.

[0066] The decoder performs each upsampling operation, which may be specifically that, during the first upsampling, the decoder receives the second-to-last fused feature matrix obtained by processing the multiple feature maps obtained by the encoder through the cross-modal module, and receives the last fused feature matrix obtained by processing the multiple feature maps obtained by the encoder through the cross-modal module through the last downsampling. In addition, the last fused feature matrix is ​​convolved using a transposed convolution to obtain a first transposed convolution feature matrix, the first transposed convolution feature matrix is ​​concatenated with the second-to-last fused feature matrix to obtain a concatenated feature matrix, and the concatenated feature matrix is ​​convolved to obtain a first upsampling feature matrix.

[0067] In the subsequent nth upsampling, the decoder receives the reciprocal n+1th fused feature matrix obtained by processing the multiple feature maps obtained by the encoder's reciprocal n+1th downsampling by the cross-modal module, and obtains the upsampling feature matrix obtained by the decoder's previous upsampling; wherein n is a positive integer greater than 1. In addition, the upsampling feature matrix obtained by the decoder's previous upsampling is convolved using a transposed convolution to obtain the nth transposed convolution feature matrix, the nth transposed convolution feature matrix is ​​concatenated with the reciprocal n+1th fused feature matrix to obtain the nth concatenated feature matrix, and the nth concatenated feature matrix is ​​convolved to obtain the nth upsampling feature matrix.

[0068] Among them, the main role of the transposed convolution is to gradually increase the size of the feature map to restore it to the size of the original image. In the decoder part of U-Net, each upsampling step uses a transposed convolution to achieve this goal. This enables the network to recover high-resolution detail information from lower-resolution feature maps.

[0069] For example, Figure 2As shown, during the first upsampling, the decoder receives the second-to-last fused feature matrix obtained by the cross-modal module processing multiple feature maps obtained by the encoder for the second-to-last downsampling (i.e., the fourth downsampling), and receives the last fused feature matrix obtained by the cross-modal module processing multiple feature maps obtained by the encoder for the last downsampling (i.e., the fifth downsampling), i.e., the fifth fused feature matrix (7×7×1024). In addition, the decoder uses transposed convolution to perform a convolution operation on the last fused feature matrix to obtain a first transposed convolution feature matrix (14×14×512), and concatenates the first transposed convolution feature matrix with the second-to-last fused feature matrix to obtain a concatenated feature matrix (14×14×1024). Next, a convolution operation is performed on the concatenated feature matrix through a 3×3 convolution, batch normalization, and LeakyRELU function to map its dimension to a dimension of 28×28×512 to obtain the first upsampled feature matrix, and then continue with the next upsampling operation.

[0070] When performing the second upsampling, the decoder obtains the fused feature matrix (28×28×256) output by the cross-modal module processing multiple feature maps obtained by the third stage downsampling (i.e., the third downsampling from the end), and obtains the upsampling feature matrix (28×28×512) obtained by the previous upsampling (i.e., the first upsampling) of the decoder, and uses transposed convolution to convolve the upsampling feature matrix to obtain a transposed convolution feature matrix (28×28×256), and concatenates the transposed convolution feature matrix with the fused feature matrix to obtain a concatenated feature matrix (28×28×512). Then, the concatenated feature matrix is ​​convolved by a 3×3 convolution, batch normalization, and LeakyRELU function to map its dimension to a dimension of 56×56×256, obtain the second upsampling feature matrix, and continue the next upsampling operation.

[0071] Similarly, the fused feature matrix output by the cross-modality module can be upsampled for the third, fourth, and other times in sequence, and the upsampled feature matrix obtained by the last upsampling is mapped to the target feature map. For example, if the last upsampling is the fourth upsampling, the upsampled feature matrix (224×224×64) obtained by the fourth upsampling is mapped to a 224×224×1 feature map through a 1×1 matrix, and finally the argmax function is used to process the feature map to obtain a segmentation mask, which represents the segmentation results of each region in the MRI image predicted by the model.

[0072] In this embodiment, when iterating and updating the model parameters, stochastic gradient descent (SGD) can be selected as the optimizer, and the learning rate, weight decay and momentum are set to 1e-2, 3e-5 and 0.95 respectively. In addition, the loss function used for back propagation includes Dice loss and cross entropy loss (Pixel-wise cross entropy loss), and its expression can be expressed as follows:

[0073]

[0074] Among them, F(x) refers to the output of the model, that is, the prediction result; Y(x) refers to the annotation of the MRI image. Dice loss can directly measure the structural similarity between the segmentation mask and the true label, focusing on the overall segmentation effect, while the cross entropy loss evaluates each pixel independently and focuses on the accuracy of similar classification. The combination of the two loss functions can balance the category accuracy and the overall segmentation effect.

[0075] In this embodiment, after the target segmentation model is trained, the target segmentation model with the optimal parameters can also be used to predict the data in the test set to obtain the segmentation results, and the index values ​​of various evaluation indicators are calculated based on the segmentation results predicted by the model to evaluate the prediction performance of the model.

[0076] Specifically, the computer device can obtain an image test set, which includes multiple groups of test samples, each group of test samples includes a multimodal MRI image and an annotation of each MRI image, and the annotation is used to represent the real area of ​​the target object in the MRI image. The image test set can be input into the target segmentation model, then the encoder downsamples the multimodal MRI image in each group of test samples to obtain a feature map of the MRI image of each modality; the cross-modal module fuses multiple feature maps of the multimodal MRI image in each group of test samples to obtain a fused feature matrix; the decoder upsamples the fused feature matrix to obtain an upsampled feature matrix, and uses a matrix of a preset size to map the upsampled feature matrix to a target feature map, and processes the target feature map to obtain a predicted segmentation result, which is used to represent the predicted area of ​​the target object in the MRI image. Among them, the processing process of the encoder, cross-modal module and decoder on the respective input data in the model testing stage is similar to the processing process of the encoder, cross-modal module and decoder on the respective input data in the model training stage, which has been described in detail in the previous text and will not be repeated here.

[0077] After obtaining the predicted segmentation result, the index value of the evaluation index can be calculated according to the predicted area represented by the predicted segmentation result and the real area represented by the annotation. The evaluation index is used to characterize the quality of the training effect of the target segmentation model.

[0078] For example, the evaluation indexes of the segmentation results of the complete tumor (WT), tumor core (TC), and enhanced tumor (ET) in the predicted image can be calculated based on the segmentation mask. The evaluation indexes may include:

[0079] Dice score, which is used to evaluate the degree of overlap between the segmentation result and the true value label;

[0080] Jaccard index, which is used to calculate the intersection-over-union ratio between the segmentation result and the ground truth label;

[0081] Hausdorff distance, used to measure the maximum distance between two edge contours;

[0082] Sensitivity coefficient (sensitivity), i.e., the total true tumor area (TP+FN) of true positive cases (TP);

[0083] The accuracy coefficient (precision) measures the percentage of true positive cases (TP) among all regions predicted to be tumors (TP+FP);

[0084] The specificity coefficient (Specificity) is used to measure the percentage of true negative cases (TN) in all non-tumor areas (TN+FP).

[0085] In addition, in the embodiment of the present application, in order to verify the effectiveness of the cross-modal module and the new model, an experimental comparison was conducted on the BraTS2021 dataset for multiple existing segmentation models and the segmentation model of the embodiment of the present application. Figure 4 As shown, the experimental results data show that the segmentation model of the embodiment of the present application achieved mean values ​​of 0.921 (WT), 0.897 (TC) and 0.861 (ET) in the other five indicators except the Hausdorff distance.

[0086] It can be concluded from the figure that the cross-modal module proposed in the embodiment of the present application has a powerful ability to extract multimodal features, and the embedding structure of the module in U-Net is also reasonable and efficient, because some models such as nnFormer and TransBTS are also based on U-net and Transformer, and the model of the embodiment of the present application shows stronger performance.

[0087] In the present embodiment, the parameter quantity, complexity (number of floating point operations per second) and inference speed of the model are also evaluated. The results are as follows: Figure 5As shown, the embodiment of the present application significantly improves the segmentation accuracy without significantly increasing the number of model parameters, and will not cause the problem of model overfitting. In addition, compared with other high-performance models (except TransBTS and nnU-Net), the segmentation model of the embodiment of the present application is Pareto optimal in these three indicators. Compared with TransBTS and nnU-Net, the model of the embodiment of the present application has higher accuracy.

[0088] Therefore, compared with the segmentation network based on convolutional neural network, the segmentation model obtained by the model training method of the embodiment of the present application improves and embeds the Transformer, so that the improved U-Net model has the ability to process multimodal MRI images, can capture the long-distance dependencies of multimodal image inputs, and greatly improves the segmentation accuracy of brain tumors.

[0089] Moreover, compared to the huge number of parameters of the model using multiple Transformers as encoders, the embodiment of the present application only uses a Transformer-based cross-modal module to fuse the feature matrices of the four modalities. Since medical segmentation does not require a large receptive field, the embodiment of the present application uses lightweight convolution operations for both upsampling and downsampling, thereby effectively reducing the risk of overfitting without complicating the model structure and increasing the number of parameters. Reasonable cross-modal module embedding significantly improves the segmentation accuracy of the model.

[0090] In the embodiment of the present application, the cross-modal module can also be improved based on the latest GroupMixFormer (a variant of Transformer) to take into account the relationship between pixel groups and pixel groups, and between pixel groups and individual pixels in the image processing process. In addition, federated learning can be integrated to train models and aggregate parameters for multiple data sets, thereby obtaining a better global model to reduce the annotation cost of MRI images and expand the size of MRI image data sets.

[0091] The embodiment of the present application proposes a Transformer-based cross-modal module to model the global connection between the features of different modalities of MRI images. The cross-modal module will be used for Skip-connection operations and concatenated with the feature matrix sampled by the decoder to provide multimodal information for the upsampling of the model. The four feature matrices obtained by the fourth convolution module will be concatenated and global features will be extracted using the cross-modal module, and the highest-level feature matrix obtained will be used as the input of the decoder.

[0092] Through the combination and improvement of U-Net and Transformer, the model of the embodiment of the present application inherits and takes advantage of both. The U-shaped encoder-decoder structure of the proposed model enables the model to capture high-level semantic information and low-level features at the same time. In view of the relatively difficult acquisition of medical imaging data and the high cost of annotation, which leads to relatively small data sets, the model of the embodiment of the present application inherits the relatively lightweight network structure and effective feature extraction mechanism of U-Net, and can also achieve good segmentation results on small data sets, avoiding the problem of overfitting. Secondly, the embodiment of the present application uses the Transformer's ability to capture global information to design a cross-modal module, which effectively integrates the multimodal data of MRI and significantly improves the segmentation accuracy.

[0093] For example, Figure 6 As shown, the 4 images in the first row are brain MRI images, the images in the second row are the real labels of the tumor region segmentation results corresponding to the 4 brain MRI images in the first row, and the images in the third row are the predicted segmentation results of the tumor regions corresponding to the 4 brain MRI images in the first row. The red circles are the places where there are differences between the predicted segmentation results and the real labels. It can be seen that the results predicted by the segmentation model of the embodiment of the present application are almost consistent with the real results, the segmentation accuracy is extremely high, and the accuracy of the segmentation results is extremely high.

[0094] The following will be Figure 1 Based on the embodiment shown, the embodiment of the present application is further described in detail. Figure 7 In the embodiment of the present application, an embodiment of the MRI image segmentation method includes:

[0095] 701. Acquire a target multimodal MRI image to be processed;

[0096] 702. Obtain a target segmentation model, where the target segmentation model includes a U-Net network and a cross-modal module;

[0097] In this embodiment, the target segmentation model can be based on Figure 1 The target segmentation model is obtained by training the model training method of the illustrated embodiment.

[0098] 703. Input the target multimodal MRI image into the target segmentation model;

[0099] 704. The encoder of the U-Net network downsamples the target multimodal MRI image to obtain a feature map of the MRI image of each modality;

[0100] 705. The cross-modality module fuses the multiple feature maps of the target multi-modality MRI image to obtain a fused feature matrix;

[0101] 706. The decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, maps the upsampled feature matrix into a target feature map using a matrix of a preset size, and processes the target feature map to obtain a predicted segmentation result, where the predicted segmentation result is used to represent a predicted region of a target object in the target multimodal MRI image;

[0102] In this embodiment, the process of processing the respective input data by the encoder, cross-modal module and decoder in the model processing stage of the target multimodal MRI image is similar to the process of processing the respective input data by the encoder, cross-modal module and decoder in the model training stage, which has been explained in detail in the previous text and will not be repeated here.

[0103] This embodiment proposes a model for high-precision brain tumor segmentation. The main feature of the model is that it inherits the efficient structure of U-Net and integrates the cross-modal module to enable the traditional U-Net to have the feature extraction capability of multi-modal MRI image data sets. It well integrates the respective advantages of U-Net and Transformer, thereby improving the segmentation accuracy and avoiding overfitting.

[0104] The computer device in the embodiment of the present application is described below. Figure 8 In the embodiment of the present application, one embodiment of the computer device includes:

[0105] The computer device 800 may include one or more central processing units (CPU) 801 and a memory 805 , wherein the memory 805 stores one or more application programs or data.

[0106] The memory 805 may be a volatile storage or a persistent storage. The program stored in the memory 805 may include one or more modules, each of which may include a series of instruction operations in the computer device. Furthermore, the central processing unit 801 may be configured to communicate with the memory 805 and execute a series of instruction operations in the memory 805 on the computer device 800.

[0107] The computer device 800 may also include one or more power supplies 802, one or more wired or wireless network interfaces 803, one or more input and output interfaces 804, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0108] The CPU 801 can execute the aforementioned Figures 1 to 7The operations performed by the computer device in the illustrated embodiment will not be described in detail here.

[0109] The present application also provides a computer storage medium, wherein one embodiment includes: the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the aforementioned Figures 1 to 7 The operations performed by the computer device in the illustrated embodiment.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A MRI image segmentation model training method, characterized in that: The method comprises: Acquire an image training set, wherein the image training set includes multiple groups of training samples, each group of the training samples includes a multimodal MRI image and an annotation of each MRI image, wherein the annotation is used to represent a real area of ​​a target object in the MRI image; Acquire an initial segmentation model, wherein the initial segmentation model includes a U-Net network and a cross-modal module; The image training set is input into the initial segmentation model, then, The encoder of the U-Net network downsamples the multimodal MRI images in each group of the training samples to obtain a feature map of the MRI image of each modality; The cross-modality module fuses the multiple feature maps of the multimodal MRI images in each group of the training samples to obtain a fused feature matrix; The decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, and maps the upsampled feature matrix into a target feature map using a matrix of a preset size, and processes the target feature map to obtain a predicted segmentation result, wherein the predicted segmentation result is used to represent a predicted region of the target object in the MRI image; A loss function is constructed according to the predicted area represented by the predicted segmentation result and the real area represented by the annotation, and the model parameters of the initial segmentation model are adjusted according to the loss value of the loss function until the model training is stopped when the convergence condition is met to obtain the target segmentation model.

2. The method according to claim 1, characterized in that The encoder of the U-Net network downsamples the multimodal MRI images in each group of the training samples to obtain a feature map of the MRI image of each modality, including: For each group of the training samples, the encoder performs a first downsampling on the MRI image of each modality to obtain a plurality of feature maps of the multimodal MRI image; and, In each subsequent downsampling, each feature map obtained by the previous downsampling is downsampled, and multiple feature maps are obtained each time the downsampling is performed.

3. The method according to claim 2, characterized in that The cross-modality module fuses the multiple feature maps of the multimodal MRI images in each group of the training samples to obtain a fused feature matrix, including: For each downsampling, the cross-modal module concatenates the multiple feature maps to obtain a concatenated feature map, adds a position code to the concatenated feature map, and flattens the concatenated feature map into a feature sequence; Based on the attention mechanism, a query vector, a key vector and a value vector are generated according to the feature sequence, an attention matrix is ​​calculated according to the query vector, the key vector and the value vector, and the attention matrix is ​​reshaped into the same dimension as the feature sequence; The attention matrix is ​​convolved by a feedforward neural network in the cross-modal module, and the dimension of the output of the feedforward neural network is mapped to the target dimension to obtain the fused feature matrix.

4. The method according to claim 3, characterized in that The decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, including: The decoder receives the fused feature matrix obtained by respectively processing the multiple feature maps obtained by each downsampling of the encoder by the cross-modal module, and in each upsampling, the decoder performs an upsampling operation according to at least one of the fused feature matrices to obtain the upsampled feature matrix; The using a matrix of a preset size to map the up-sampled feature matrix to a target feature map includes: The up-sampled feature matrix obtained by the last up-sampling of the decoder is mapped into the target feature map using the matrix of the preset size.

5. The method according to claim 4, characterized in that The decoder receives the fused feature matrix obtained by respectively processing the multiple feature maps obtained by each downsampling of the encoder by the cross-modal module, and in each upsampling, the decoder performs an upsampling operation according to at least one of the fused feature matrices to obtain the upsampled feature matrix, including: During the first upsampling, the decoder receives the penultimate fused feature matrix obtained by the cross-modality module processing multiple feature maps obtained by the penultimate downsampling of the encoder, and receives the last fused feature matrix obtained by the cross-modality module processing multiple feature maps obtained by the last downsampling of the encoder; Using transposed convolution to perform a convolution operation on the last fused feature matrix to obtain a first transposed convolution feature matrix, concatenating the first transposed convolution feature matrix with the penultimate fused feature matrix to obtain a concatenated feature matrix, and convolving the concatenated feature matrix to obtain a first up-sampling feature matrix; In the subsequent n-th upsampling, the decoder receives the reciprocal n+1-th fused feature matrix obtained by the cross-modal module processing multiple feature maps obtained by the encoder's reciprocal n+1-th downsampling, and obtains the upsampling feature matrix obtained by the previous upsampling of the decoder; wherein n is a positive integer greater than 1; The up-sampled feature matrix obtained by the previous up-sampling of the decoder is convolved using a transposed convolution to obtain an nth transposed convolution feature matrix, the nth transposed convolution feature matrix is ​​concatenated with the penultimate (n+1)th fusion feature matrix to obtain an nth concatenated feature matrix, and the nth concatenated feature matrix is ​​convolved to obtain an nth up-sampled feature matrix.

6. The method according to claim 1, characterized in that The method further comprises: Acquire an image test set, wherein the image test set includes multiple groups of test samples, each group of the test samples includes a multimodal MRI image and an annotation of each MRI image, wherein the annotation is used to represent a real area of ​​a target object in the MRI image; The image test set is input into the target segmentation model, then, The encoder downsamples the multimodal MRI images in each group of the test samples to obtain a feature map of the MRI image of each modality; The cross-modality module fuses the multiple feature maps of the multimodal MRI images in each group of the test samples to obtain a fused feature matrix; The decoder upsamples the fused feature matrix to obtain an upsampled feature matrix, maps the upsampled feature matrix into a target feature map using a matrix of a preset size, processes the target feature map to obtain a predicted segmentation result, and the predicted segmentation result is used to represent a predicted region of the target object in the MRI image; An index value of an evaluation index is calculated according to the predicted area represented by the predicted segmentation result and the real area represented by the annotation, and the evaluation index is used to characterize the quality of the training effect of the target segmentation model.

7. The method according to any one of claims 1 to 6, characterized in that: The target object is the brain, and the real area of ​​the target object includes one or more of the background of the brain MRI image, the entire tumor area in the brain MRI image, the tumor core area in the brain MRI image, and the enhanced tumor area in the brain MRI image.

8. A method for segmenting an MRI image, characterized in that: The method comprises: Acquiring a target multimodal MRI image to be processed; Obtaining a target segmentation model, wherein the target segmentation model includes a U-Net network and a cross-modal module; The target multimodal MRI image is input into the target segmentation model, then, The encoder of the U-Net network downsamples the target multimodal MRI image to obtain a feature map of the MRI image of each modality; The cross-modality module fuses the multiple feature maps of the target multi-modality MRI image to obtain a fused feature matrix; The decoder of the U-Net network upsamples the fused feature matrix to obtain an upsampled feature matrix, and uses a matrix of a preset size to map the upsampled feature matrix into a target feature map, and processes the target feature map to obtain a predicted segmentation result, which is used to represent the predicted area of ​​the target object in the target multimodal MRI image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 8.

Images