A cue-based incremental MRI brain tumor image segmentation system

Abstract

This invention discloses a cue-based incremental MRI brain tumor image segmentation system, belonging to the field of deep learning technology. This invention addresses the low accuracy of brain tumor image segmentation caused by catastrophic forgetting of information and the inability to fully utilize the data correlations between multimodal MRI images in existing methods. The invention designs a cue-based modal feature extractor module and a segmentation network model based on double knowledge distillation. MRI images of the patient's brain under different imaging modalities are used as input for each stage. The modal feature extractor structure allows the segmentation network model to learn the structural features of the current modality and provide assistance in predicting other modalities, thus fully utilizing the data correlations between multimodal MRI images. The double knowledge distillation loss avoids the catastrophic forgetting problem encountered by the segmentation network model during continuous incremental learning. This method can be applied to MRI brain tumor image segmentation.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a modal incremental MRI brain tumor image segmentation system based on cue learning. Background Technology

[0002] In the diagnosis of brain tumors based on magnetic resonance imaging (MRI), different structures of brain tumors have different characteristics, requiring patients to have multiple modalities of MRI images captured for tumor localization and structural segmentation. Currently, brain tumor segmentation is mainly done manually by experts. Because the shape, structure, and location of brain tumors are highly variable, and the contrast in MRI images is poor, they easily overlap with healthy brain tissue, making accurate localization of brain tumors and their structures extremely difficult. In recent years, the accuracy and stability of deep learning segmentation models have gradually improved, providing strong support for this type of medical image segmentation.

[0003] Currently, researchers mostly use three-dimensional volumetric MRI for brain tumor segmentation studies, which can fully acquire information about the overall structure of the brain. Depending on the imaging modality, brain tumor MRI can generally be divided into four modalities: T1 modality, T1ce modality, T2 modality, and FLAIR modality. A single modality of MRI image can only convey a limited amount of information; therefore, multimodal images are needed to accurately segment lesion regions. In practical applications, the data obtained by the model may not necessarily include all brain tumor MRI images from all imaging modalities of the patient. Therefore, model training is continuous; that is, as clinical data accumulates, the system needs to seamlessly integrate this new information to improve model performance. Compared to single-mode segmentation systems, continuous segmentation systems reduce the requirements for data acquisition and storage, and have higher efficiency and maintainability. Therefore, continuous learning-based segmentation systems have become a medical image segmentation task with high clinical value.

[0004] While existing continuous learning-based segmentation models can effectively avoid catastrophic forgetting caused by data heterogeneity, in MRI brain tumor segmentation tasks, the feature distributions between different modalities of MRI images vary greatly. Conventional continuous learning methods still experience significant forgetting after continuous training. Furthermore, due to information isolation between modalities, existing continuous learning methods cannot fully utilize the data correlations between multimodal images to aid diagnosis, neglecting the correlations between structural features of the case. Therefore, the segmentation accuracy of existing methods when applied to brain tumor image segmentation remains low. We need a brain tumor image segmentation model that can learn features from different modalities during continuous learning and utilize previously learned modalities to assist in the current modality segmentation task. Summary of the Invention

[0005] The purpose of this invention is to address the problem of low accuracy in brain tumor image segmentation due to catastrophic forgetting of information and the inability to fully utilize the data correlation between multimodal MRI images in existing methods. Therefore, this invention proposes a cue-based incremental MRI brain tumor image segmentation system.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] A modal incremental MRI brain tumor image segmentation system based on cue learning, the system comprising a brain tumor MRI image acquisition module, an MRI image processing module, a first modal incremental MRI brain tumor image segmentation model, and a second modal incremental MRI brain tumor image segmentation model;

[0008] The brain tumor MRI image acquisition module is used to acquire the MRI image subsets corresponding to each modality of the brain tumor.

[0009] The MRI image processing module is used to process the MRI images in the MRI image subsets corresponding to each modality to obtain processed image slices corresponding to each modality.

[0010] The first modal incremental MRI brain tumor image segmentation model includes an input layer, a cue-based modal feature extractor module, a segmentation network module based on double knowledge distillation, and an output layer. The first modal incremental MRI brain tumor image segmentation model and the second modal incremental MRI brain tumor image segmentation model have the same structure.

[0011] The first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model are trained based on the processed image slices corresponding to each modality.

[0012] The trained first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model are used to process the brain tumor MRI images to be segmented, and obtain the image segmentation results of the brain tumor MRI images to be segmented.

[0013] Furthermore, the step of acquiring the MRI image subsets corresponding to each modality of brain tumor specifically involves:

[0014] For each modality of brain tumor, the corresponding MRI image subset is obtained from the BraTS2020 dataset, where the total number of modalities T of brain tumor MRI images is 4.

[0015] Furthermore, the operation of the MRI image processing module is as follows:

[0016] For any MRI image, if modal image slices containing segmentation labels are retained by cropping and trimming operations, the shape of the processed image corresponding to the MRI image is (n, 192, 192), where n is the number of slices retained from the MRI image.

[0017] Furthermore, the specific structures of the prompt-based modal feature extractor module and the dual-knowledge distillation-based segmentation network module are as follows:

[0018] The modal feature extractor module based on cue learning includes three convolutional layers with a stride of 1 and a kernel size of 1×1, and one convolutional layer with a stride of 1 and a kernel size of 3×3.

[0019] The segmentation network module based on dual knowledge distillation includes an encoding path, a decoding path, and a classification layer, wherein:

[0020] The encoding path includes four encoders cascaded in sequence, each encoder consisting of a max pooling layer and two convolutional layers;

[0021] The decoding path includes four decoders cascaded in sequence, each of which contains a convolutional layer and an upsampling layer.

[0022] Furthermore, the working process of the first modality incremental MRI brain tumor image segmentation model is as follows:

[0023] The image x input from the input layer is used as the input to the cue-based modal feature extractor module. The output x0 of the cue-based modal feature extractor module is then fused with the image x to obtain the fused result x. mixed :

[0024] x mixed =alpha*x0 + (1-alpha)*x

[0025] Wherein, alpha is an alpha mask randomly generated based on the dimensions of the image shape;

[0026] Then merge the result x mixed A concat operation is performed on the image x, and the result of the concat operation is used as the input of the segmentation network module based on double knowledge distillation. Finally, the output of the segmentation network module based on double knowledge distillation is passed through the output layer to output the segmentation result of the input image x.

[0027] Furthermore, the training process for the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model is as follows:

[0028] Step 1: Use the processed image slices corresponding to the first modality to train the first modality incremental MRI brain tumor image segmentation model to obtain the first-stage trained first modality incremental MRI brain tumor image segmentation model.

[0029] Then initialize the parameters of the second modality incremental MRI brain tumor image segmentation model to the parameters of the first modality incremental MRI brain tumor image segmentation model trained in the first stage;

[0030] Step 2: Initialize the training phase t=2;

[0031] Step 3: Freeze all parameters of the first modality incremental MRI brain tumor image segmentation model and the parameters of the encoding path in the second modality incremental MRI brain tumor image segmentation model;

[0032] Step 4: Use the processed image slices corresponding to the t-th modality as inputs to the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model, respectively.

[0033] The second modality incremental MRI brain tumor image segmentation model is trained based on the output of the first modality incremental MRI brain tumor image segmentation model to obtain the second modality incremental MRI brain tumor image segmentation model trained in stage t.

[0034] Step 5: Update the parameters of the first modality incremental MRI brain tumor image segmentation model to the parameters of the second modality incremental MRI brain tumor image segmentation model trained in stage t.

[0035] Step 6: Determine if t < T is satisfied;

[0036] If t < T, then let t = t + 1 and return to step three;

[0037] If t = T, then the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model trained in the last training phase will be used as the final first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model.

[0038] Furthermore, when the number of training stages t≥2, the loss function for the training process is:

[0039] loss = loss seg +λ1loss prompt +λ2loss response

[0040] Where, loss responseThe loss is the prediction loss of the first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model. prompt The loss represents the feature loss of the first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model. seg λ1 and λ2 are the predicted output of the second-modality incremental MRI brain tumor image segmentation model and the loss of the segmentation label mask GroundTruth, where λ1 and λ2 are weighting coefficients.

[0041] Furthermore, the weighting coefficient λ1 is 0.01 and λ2 is 1.

[0042] Furthermore, the loss seg for:

[0043]

[0044] Wherein, TP represents the number of true positive pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model, FP represents the number of false positive pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model, and FN represents the number of false negative pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model.

[0045] The loss prompt for:

[0046]

[0047] Where ζ represents the input image x k The set of all pixels in the set. The input image x k The i-th pixel in This represents the output of the modality feature extractor in the second-modality incremental MRI brain tumor image segmentation model. This represents the output of the modal feature extractor of the first modal incremental MRI brain tumor image segmentation model, where |ζ| is the number of pixels contained in the set ζ;

[0048] The loss response for:

[0049]

[0050] in, This represents the predicted output of the second-modality incremental MRI brain tumor image segmentation model. This represents the predicted output of the first-modality incremental MRI brain tumor image segmentation model.

[0051] Furthermore, the trained first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model are used to process the brain tumor MRI image to be segmented, to obtain the image segmentation result of the brain tumor MRI image to be segmented; specifically:

[0052] Step 1: Use the MRI image processing module to process each modal image of the brain tumor MRI image to be segmented, and obtain the processed image slices corresponding to each modal of the brain tumor MRI image to be segmented.

[0053] Step 2: If the number of modalities of the brain tumor MRI image to be segmented is 1, then proceed to step 3; otherwise, if the number of modalities of the brain tumor MRI image to be segmented is greater than 1, then proceed to step 4.

[0054] Step 3: Use the processed image slices of the brain tumor MRI image to be segmented as input to the modal incremental MRI brain tumor image segmentation model, and then output the final segmentation result according to the modal incremental MRI brain tumor image segmentation model; specifically:

[0055]

[0056] Where x represents a processed image slice of the brain tumor MRI image to be segmented. y represents the output of x after passing through the modal feature extractor, and y represents the output of the modal incremental MRI brain tumor image segmentation model.

[0057] Then perform the Argmax operation on y to obtain the classification result for each pixel, which is the final segmentation result.

[0058] Step 4: Use the processed image slices corresponding to each modality of the MRI image of the brain tumor to be segmented as input to the modal incremental MRI brain tumor image segmentation model, and then output the final segmentation result according to the modal incremental MRI brain tumor image segmentation model; specifically:

[0059] Using the m-th modality as the primary modality, all modalities of the brain tumor MRI image to be segmented are used as auxiliary modalities:

[0060]

[0061] Where, x m This represents the processed image slice corresponding to the m-th modality of the MRI image of a brain tumor to be segmented, x a This represents the processed image slice corresponding to the a-th auxiliary modality of the brain tumor MRI image to be segmented, where N represents the number of auxiliary modalities. x represents a After the output of the modality feature extractor, y mThis represents the output of the modal incremental MRI brain tumor image segmentation model;

[0062] Then for y m Perform the Argmax operation to obtain the classification result for each pixel, that is, obtain the segmentation result corresponding to the m-th modality;

[0063] Similarly, each mode is used as the principal mode to obtain the segmentation result corresponding to each mode.

[0064] The beneficial effects of this invention are:

[0065] This invention designs a modal feature extractor module based on cue learning and a segmentation network model based on double knowledge distillation. MRI images of the patient's brain under different imaging modalities are used as input for each stage. The segmentation network model is trained using the input images. The modal feature extractor structure allows the segmentation network model to learn the structural features of the current modality and provide assistance in predicting other modalities. Therefore, it fully utilizes the data correlation between multimodal MRI images. The double knowledge distillation loss avoids the catastrophic forgetting problem encountered by the segmentation network model during continuous modal incremental learning, thus improving the accuracy of brain tumor image segmentation. Attached Figure Description

[0066] Figure 1 A diagram illustrating the phenomenon of catastrophic forgetting;

[0067] Figure 2 This is a schematic diagram illustrating information isolation between modalities.

[0068] Figure 3 This is a flowchart of a modal incremental MRI brain tumor image segmentation system based on cue learning according to the present invention.

[0069] Figure 4 This is a schematic diagram of the model training process. Detailed Implementation

[0070] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0071] Specific implementation method one: Combining Figure 3 This embodiment describes a cue-based incremental MRI brain tumor image segmentation system. The system includes a brain tumor MRI image acquisition module, an MRI image processing module, a first-modality incremental MRI brain tumor image segmentation model, and a second-modality incremental MRI brain tumor image segmentation model.

[0072] The brain tumor MRI image acquisition module is used to acquire the MRI image subsets corresponding to each modality of the brain tumor.

[0073] The MRI image processing module is used to process the MRI images in the MRI image subsets corresponding to each modality to obtain processed image slices corresponding to each modality.

[0074] The first modal incremental MRI brain tumor image segmentation model includes an input layer, a cue-based modal feature extractor module, a segmentation network module based on double knowledge distillation, and an output layer. The first modal incremental MRI brain tumor image segmentation model and the second modal incremental MRI brain tumor image segmentation model have the same structure.

[0075] The first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model are trained based on the processed image slices corresponding to each modality.

[0076] The trained first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model are used to process the brain tumor MRI images to be segmented, and obtain the image segmentation results of the brain tumor MRI images to be segmented.

[0077] This invention fully considers the importance of multimodal image information for brain tumor image segmentation tasks, making full use of the different structural information of brain tumors represented by multimodal MRI data; it uses cue learning to solve the problem of data heterogeneity between modalities and improve prediction accuracy; it uses knowledge distillation to enhance the generalization and robustness of the model, enabling the model to be used flexibly in different application scenarios; the modal incremental MRI brain tumor image segmentation model constructed by this invention can achieve accurate segmentation of multimodal MRI brain tumor images and has high practical value.

[0078] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the step of acquiring the MRI image subsets corresponding to each modality of the brain tumor is specifically as follows:

[0079] For each modality of brain tumor, the corresponding MRI image subset is obtained from the BraTS2020 dataset, where the total number of modalities T of brain tumor MRI images is 4.

[0080] The other steps and parameters are the same as in Specific Implementation Method 1.

[0081] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the working process of the MRI image processing module is as follows:

[0082] For any MRI image (a three-dimensional image), by cropping and trimming to retain modal image slices of regions with segmentation labels (each slice data has a segmentation mask corresponding to the tumor structure), the shape of the processed image corresponding to the MRI image is (n, 192, 192), where n is the number of slices retained from the MRI image.

[0083] Other steps and parameters are the same as in specific implementation method one or two.

[0084] The purpose of cropping and trimming in this invention is to omit irrelevant information in the image, reduce the size of the model input, and improve the efficiency of the model.

[0085] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that the structure of the modal feature extractor module based on cue learning and the segmentation network module based on dual knowledge distillation is as follows:

[0086] The modal feature extractor module based on cue learning includes three convolutional layers with a stride of 1 and a kernel size of 1×1, and one convolutional layer with a stride of 1 and a kernel size of 3×3.

[0087] The segmentation network module based on dual knowledge distillation includes an encoding path, a decoding path, and a classification layer (the classification layer outputs the probability that each pixel belongs to a tumor and the probability that it belongs to a non-tumor), wherein:

[0088] The encoding path includes four cascaded encoders (i.e., the output of the previous encoder is used as the input of the next encoder), and each encoder includes a max pooling layer and two convolutional layers;

[0089] The decoding path includes four decoders cascaded in sequence, each of which contains a convolutional layer and an upsampling layer.

[0090] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0091] The first three convolutional layers of the modality feature extractor module are used to fully extract the overall surface information of the input image, while the last convolutional layer is used to extract deeper data information, abstracting and extracting the feature information of the modality. In the encoding path, the first encoder halves the size of the feature map, outputting a feature map of size 128×96×96; the second encoder further halves the size of the feature map, outputting a feature map of size 256×48×48; the third encoder halves the size of the feature map again, outputting a feature map of size 512×24×24; finally, in the fourth encoder, the size of the feature map is halved again, outputting a feature map of size 1024×12×12. In the decoding path, the first decoder restores the size of the feature map to 512×24×24; the second decoder restores the size of the feature map to 256×48×48; the third decoder restores the size of the feature map to 128×96×96; and the last decoder restores the size of the feature map to 64×192×192. The model's output layer then maps the output of the decoding path to the category space, with an output size of 2×192×192.

[0092] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the working process of the first modality incremental MRI brain tumor image segmentation model is as follows:

[0093] The image x input from the input layer is used as the input to the cue-based modal feature extractor module. The output x0 of the cue-based modal feature extractor module is then fused with the image x to obtain the fused result x. mixed :

[0094] x mixed =alpha*x0 + (1-alpha)*x

[0095] Wherein, alpha is an alpha mask randomly generated according to the dimensions of the image shape (i.e., a mask matrix with dimensions of 192×192 generated according to (192, 192);

[0096] Then merge the result x mixed A concat operation is performed on the image x, and the result of the concat operation is used as the input to the segmentation network module based on double knowledge distillation (in the segmentation network module based on double knowledge distillation, the input part first passes through the encoding path, and then the output of the encoding path is used as the input of the decoding path). Finally, the output of the segmentation network module based on double knowledge distillation is passed through the output layer to output the segmentation result of the input image x.

[0097] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0098] Specific Implementation Method Six: Combination Figure 4 This embodiment is described below. The difference between this embodiment and one of the specific embodiments one to five is that the training process of the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model is as follows:

[0099] Step 1: Use the processed image slices corresponding to the first modality to train the first modality incremental MRI brain tumor image segmentation model to obtain the first-stage trained first modality incremental MRI brain tumor image segmentation model.

[0100] Then initialize the parameters of the second modality incremental MRI brain tumor image segmentation model to the parameters of the first modality incremental MRI brain tumor image segmentation model trained in the first stage;

[0101] Step 2: Initialize the training phase t=2;

[0102] Step 3: Freeze all parameters of the first modality incremental MRI brain tumor image segmentation model and the parameters of the encoding path in the second modality incremental MRI brain tumor image segmentation model;

[0103] Step 4: Use the processed image slices corresponding to the t-th modality as inputs to the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model, respectively.

[0104] The second modality incremental MRI brain tumor image segmentation model is trained based on the output of the first modality incremental MRI brain tumor image segmentation model to obtain the second modality incremental MRI brain tumor image segmentation model trained in stage t.

[0105] Step 5: Update the parameters of the first modality incremental MRI brain tumor image segmentation model to the parameters of the second modality incremental MRI brain tumor image segmentation model trained in stage t.

[0106] Step 6: Determine if t < T is satisfied;

[0107] If t < T, then let t = t + 1 and return to step three;

[0108] If t = T, then the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model trained in the last training phase will be used as the final first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model.

[0109] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0110] In this invention, each subset of datasets is divided into three parts: training set, test set, and validation set in a ratio of 0.6:0.25:0.15. After training, the model is tested and validated using the test set and validation set.

[0111] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that, when the number of training stages t≥2, the loss function for the training process is:

[0112] loss = loss seg +λ1loss prompt +λ2loss response

[0113] Where, loss response The loss is the prediction loss of the first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model. prompt The loss represents the feature loss of the first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model. seg λ1 and λ2 are the predicted output of the second-modality incremental MRI brain tumor image segmentation model and the loss of the segmentation label mask GroundTruth, where λ1 and λ2 are weighting coefficients.

[0114] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0115] Where, loss prompt and loss response This refers to the double distillation loss. When t=1, the loss function only includes the loss... seg Loss item.

[0116] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that the weighting coefficient λ1 is 0.01 and the weighting coefficient λ2 is 1.

[0117] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0118] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the loss... seg for:

[0119]

[0120] Wherein, TP (True Positive) represents the number of true positive pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model (true positive pixels are pixels that are actually tumor pixels and are predicted as tumor pixels), FP (False Positive) represents the number of false positive pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model (false positive pixels are pixels that are actually non-tumor pixels and are predicted as tumor pixels), and FN (False Negative) represents the number of false negative pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model (false negative pixels are pixels that are actually tumor pixels and are predicted as non-tumor pixels).

[0121] loss prompt By minimizing the differences in modal features extracted by the modal feature extractor at different training stages, the model's familiarity with the trained modalities is preserved.

[0122] The loss prompt for:

[0123]

[0124] Where ζ represents the input image x k The set of all pixels in the set. The input image x k The i-th pixel in This represents the output of the modality feature extractor in the second-modality incremental MRI brain tumor image segmentation model. This represents the output of the modal feature extractor of the first modal incremental MRI brain tumor image segmentation model, where |ζ| is the number of pixels contained in the set ζ;

[0125] loss response The focus is on preserving knowledge of specific prediction results, ensuring consistent performance of prediction results for previously seen modal images, and minimizing the differences in prediction capabilities between different modalities;

[0126] The loss response for:

[0127]

[0128] in, This represents the predicted output of the second-modality incremental MRI brain tumor image segmentation model. This represents the predicted output of the first-modality incremental MRI brain tumor image segmentation model.

[0129] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0130] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One to Nine in that the trained first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model are used to process the brain tumor MRI image to be segmented, obtaining the image segmentation result of the brain tumor MRI image to be segmented; specifically:

[0131] Step 1: Use the MRI image processing module to process each modal image of the brain tumor MRI image to be segmented, and obtain the processed image slices corresponding to each modal of the brain tumor MRI image to be segmented.

[0132] Step 2: If the number of modalities of the brain tumor MRI image to be segmented is 1, then proceed to step 3; otherwise, if the number of modalities of the brain tumor MRI image to be segmented is greater than 1, then proceed to step 4.

[0133] Step 3: Use the processed image slices of the brain tumor MRI image to be segmented as input to the modal incremental MRI brain tumor image segmentation model, and then output the final segmentation result according to the modal incremental MRI brain tumor image segmentation model; specifically:

[0134]

[0135] Where x represents a processed image slice of the brain tumor MRI image to be segmented. y represents the output of x after passing through the modal feature extractor, and y represents the output of the modal incremental MRI brain tumor image segmentation model.

[0136] Then perform the Argmax operation on y to obtain the classification result for each pixel (if the probability of a pixel belonging to the tumor is greater than the probability of not belonging to the tumor, then the pixel belongs to the tumor site), that is, to obtain the final segmentation result (all pixels belonging to the tumor site constitute the tumor region).

[0137] Step 4: Use the processed image slices corresponding to each modality of the brain tumor MRI image to be segmented as input to the modal incremental MRI brain tumor image segmentation model (since the parameters of the trained second modal incremental MRI brain tumor image segmentation model are copied to the first modal incremental MRI brain tumor image segmentation model, either the second modal incremental MRI brain tumor image segmentation model or the first modal incremental MRI brain tumor image segmentation model can be input here), and then output the final segmentation result according to the modal incremental MRI brain tumor image segmentation model; specifically:

[0138] Using the m-th modality as the primary modality, all modalities of the brain tumor MRI image to be segmented are used as auxiliary modalities:

[0139]

[0140] Where, x m This represents the processed image slice corresponding to the m-th modality of the MRI image of a brain tumor to be segmented, x a This represents the processed image slice corresponding to the a-th auxiliary modality of the brain tumor MRI image to be segmented (since the brain tumor MRI image to be segmented may not include images of all modalities, the number of modalities included in the brain tumor MRI image to be segmented is denoted as N, where N is less than or equal to 4), where N represents the number of auxiliary modalities. x represents a After the output of the modality feature extractor, y m This represents the output of the modal incremental MRI brain tumor image segmentation model;

[0141] Then for y m Perform the Argmax operation to obtain the classification result for each pixel (if the probability of a pixel belonging to the tumor is greater than the probability of it not belonging to the tumor, then the pixel belongs to the tumor site), that is, obtain the segmentation result corresponding to the m-th modality (all pixels belonging to the tumor site constitute the tumor region).

[0142] Similarly, each mode is used as the principal mode to obtain the segmentation result corresponding to each mode.

[0143] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0144] This invention proposes a modal incremental MRI brain tumor image segmentation model based on cue-based learning. It designs a modal feature extractor module and a dual knowledge distillation strategy, using the output of the modal feature extractor module as cue information in the learning process. This improves the model's ability to extract information from multimodal MRI brain tumor images and solves the problem of... Figure 1 and Figure 2 The model addresses the challenges of modality transfer and intermodal information isolation encountered in mid-modal incremental learning. It enhances the model's ability to memorize learned modalities and leverages this information to improve segmentation capabilities. The model can achieve accurate segmentation of multimodal MRI brain tumor images and has high practical value.

[0145] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A modal incremental MRI brain tumor image segmentation system based on cue learning, characterized in that, The system includes a brain tumor MRI image acquisition module, an MRI image processing module, a first modality incremental MRI brain tumor image segmentation model, and a second modality incremental MRI brain tumor image segmentation model. The brain tumor MRI image acquisition module is used to acquire the MRI image subsets corresponding to each modality of the brain tumor. The MRI image processing module is used to process the MRI images in the MRI image subsets corresponding to each modality to obtain processed image slices corresponding to each modality. The first modal incremental MRI brain tumor image segmentation model includes an input layer, a cue-based modal feature extractor module, a segmentation network module based on double knowledge distillation, and an output layer. The first modal incremental MRI brain tumor image segmentation model and the second modal incremental MRI brain tumor image segmentation model have the same structure. The first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model are trained based on the processed image slices corresponding to each modality. The trained first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model are used to process the brain tumor MRI images to be segmented, and obtain the image segmentation results of the brain tumor MRI images to be segmented. The trained first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model are used to process the brain tumor MRI images to be segmented, obtaining the image segmentation results of the brain tumor MRI images to be segmented; specifically: Step 1: Use the MRI image processing module to process each modal image of the brain tumor MRI image to be segmented, and obtain the processed image slices corresponding to each modal of the brain tumor MRI image to be segmented. Step 2: If the number of modalities of the brain tumor MRI image to be segmented is 1, then proceed to step 3; otherwise, if the number of modalities of the brain tumor MRI image to be segmented is greater than 1, then proceed to step 4. Step 3: Use the processed image slices of the brain tumor MRI image to be segmented as input to the first modality incremental MRI brain tumor image segmentation model or the second modality incremental MRI brain tumor image segmentation model, and then output the final segmentation result according to the first modality incremental MRI brain tumor image segmentation model or the second modality incremental MRI brain tumor image segmentation model; specifically: in, This represents a processed image slice of a brain tumor MRI image to be segmented. express The output of the modality feature extractor This represents the output of either the first modality incremental MRI brain tumor image segmentation model or the second modality incremental MRI brain tumor image segmentation model. Again Perform the Argmax operation to obtain the classification result for each pixel, which is the final segmentation result; Step 4: Use the processed image slices corresponding to each modality of the brain tumor MRI image to be segmented as input to the first modality incremental MRI brain tumor image segmentation model or the second modality incremental MRI brain tumor image segmentation model, and then output the final segmentation result according to the first modality incremental MRI brain tumor image segmentation model or the second modality incremental MRI brain tumor image segmentation model; specifically: With the first One modality is used as the primary modality, and all modalities of the brain tumor MRI image to be segmented are used as auxiliary modalities: in, This indicates the first segment of the MRI image of the brain tumor to be segmented. The processed image slices corresponding to each modality This indicates the first segment of the MRI image of the brain tumor to be segmented. The processed image slices corresponding to each auxiliary modality, where N represents the number of auxiliary modalities. express The output of the modality feature extractor This represents the output of either the first modality incremental MRI brain tumor image segmentation model or the second modality incremental MRI brain tumor image segmentation model. Again Perform the Argmax operation to obtain the classification result for each pixel, that is, obtain the first... Segmentation results corresponding to each modality; Each mode is used as the principal mode to obtain the segmentation result corresponding to each mode.

2. The modal incremental MRI brain tumor image segmentation system based on cue learning according to claim 1, characterized in that, The specific steps for obtaining the MRI image subsets corresponding to each modality of brain tumor are as follows: For each modality of brain tumor, the corresponding MRI image subset is obtained from the BraTS2020 dataset, where the total number of modalities T of brain tumor MRI images is 4.

3. The modal incremental MRI brain tumor image segmentation system based on cue learning according to claim 1, characterized in that, The working process of the MRI image processing module is as follows: For any MRI image, if modal image slices containing segmentation labels are retained by cropping and trimming operations, the shape of the processed image corresponding to the MRI image will be (n, 192, 192), where n is the number of slices retained from the MRI image.

4. The modal incremental MRI brain tumor image segmentation system based on cue learning according to claim 1, characterized in that, The specific structures of the prompt-based modal feature extractor module and the segmentation network module based on dual knowledge distillation are as follows: The modal feature extractor module based on cue learning includes three convolutional layers with a stride of 1 and a kernel size of 1×1, and one convolutional layer with a stride of 1 and a kernel size of 3×3. The segmentation network module based on dual knowledge distillation includes an encoding path, a decoding path, and a classification layer, wherein: The encoding path includes four encoders cascaded in sequence, each encoder consisting of a max pooling layer and two convolutional layers; The decoding path includes four decoders cascaded in sequence, each of which contains a convolutional layer and an upsampling layer.

5. A modal incremental MRI brain tumor image segmentation system based on cue learning according to claim 4, characterized in that, The working process of the first modality incremental MRI brain tumor image segmentation model is as follows: The image input from the input layer The output of the cue-based learning modal feature extractor module is then used as input to the cue-based learning modal feature extractor module. With images Perform fusion to obtain the fusion result. : in, It is an alpha mask randomly generated based on the dimensions of the image shape; Then the fusion results and images A concat operation is performed, and the result of the concat operation is used as input to the segmentation network module based on double knowledge distillation. Finally, the output of the segmentation network module based on double knowledge distillation is passed through the output layer to output the input image. The segmentation result.

6. The modal incremental MRI brain tumor image segmentation system based on cue learning according to claim 5, characterized in that, The training process for the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model is as follows: Step 1: Use the processed image slices corresponding to the first modality to train the first modality incremental MRI brain tumor image segmentation model to obtain the first-stage trained first modality incremental MRI brain tumor image segmentation model. Then initialize the parameters of the second modality incremental MRI brain tumor image segmentation model to the parameters of the first modality incremental MRI brain tumor image segmentation model trained in the first stage; Step 2: Initialize the training phase t=2; Step 3: Freeze all parameters of the first modality incremental MRI brain tumor image segmentation model and the parameters of the encoding path in the second modality incremental MRI brain tumor image segmentation model; Step 4: Use the processed image slices corresponding to the t-th modality as inputs to the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model, respectively. The second modality incremental MRI brain tumor image segmentation model is trained based on the output of the first modality incremental MRI brain tumor image segmentation model to obtain the second modality incremental MRI brain tumor image segmentation model trained in stage t. Step 5: Update the parameters of the first modality incremental MRI brain tumor image segmentation model to the parameters of the second modality incremental MRI brain tumor image segmentation model trained in stage t. Step 6: Determine if t < T is satisfied; If t < T, then let t = t + 1 and return to step three; If t=T, then the first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model trained in the last training phase will be used as the final first modality incremental MRI brain tumor image segmentation model and the second modality incremental MRI brain tumor image segmentation model.

7. A modal incremental MRI brain tumor image segmentation system based on cue learning according to claim 6, characterized in that, When the number of training stages t≥2, the loss function of the training process for: in, It represents the prediction loss of the first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model. It is the feature loss of the first-modality incremental MRI brain tumor image segmentation model and the second-modality incremental MRI brain tumor image segmentation model. It is the predicted output of the second-modality incremental MRI brain tumor image segmentation model and the loss of the segmentation label mask GroundTruth. and These are the weighting coefficients.

8. A modal incremental MRI brain tumor image segmentation system based on cue learning according to claim 7, characterized in that, The weighting coefficient The value is 0.

01. The value of is 1.

9. A modal incremental MRI brain tumor image segmentation system based on cue learning according to claim 8, characterized in that, The loss for: Wherein, TP represents the number of true positive pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model, FP represents the number of false positive pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model, and FN represents the number of false negative pixels in the predicted output of the second modality incremental MRI brain tumor image segmentation model. The loss for: in, Represents the input image The set of all pixels in the set. Represents the input image The i-th pixel in This represents the output of the modality feature extractor in the second-modality incremental MRI brain tumor image segmentation model. This represents the output of the modality feature extractor in the first modality incremental MRI brain tumor image segmentation model. It is a set The number of pixels contained; The loss for: This represents the predicted output of the second-modality incremental MRI brain tumor image segmentation model. This represents the predicted output of the first-modality incremental MRI brain tumor image segmentation model.

Images