Glioma feature prediction method and system based on multi-modal magnetic resonance and multi-task deep learning

Abstract

The invention belongs to the technical field of medical artificial intelligence, and discloses a glioma feature prediction method and system based on multi-modal magnetic resonance and multi-task deep learning, and the method comprises the steps: processing input data through a modal completion module which combines knowledge distillation and modal existence coding, so as to deal with modal deficiency; after deep features are extracted by using a shared encoder, the deep features are sent to a hierarchical task gating prediction network, and the network performs gating modulation on post-stage task features through a pre-stage task result to realize task collaboration; and carrying out model training by adopting a self-adaptive loss function based on sample difficulty. Through the design, the accuracy and robustness of glioma segmentation, molecular marker prediction and lifetime prediction are improved, and the uncertainty score of the prediction result can be provided during reasoning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical artificial intelligence, and particularly relates to a glioma feature prediction method and system based on multi-modal magnetic resonance and multi-task deep learning. BACKGROUND

[0002] Glioma is the most common malignant tumor of the central nervous system. Its accurate diagnosis and treatment depend on the physical delineation (segmentation), molecular typing (such as IDH gene mutation status, 1p / 19q co-deletion status) and prognosis evaluation (survival prediction) of the tumor. Traditional methods for obtaining these information require invasive surgery and complex pathological analysis, which are high in cost and long in cycle. Non-invasive prediction using multi-modal MRI is a current research hotspot.

[0003] Existing deep learning-based methods have made some progress, but in the process of implementing the present application, the inventors have found that at least the following problems exist in the prior art: 1. In clinical practice, MRI scans of patients often lack one or more modalities (such as T1, T2, FLAIR, T1c) due to various reasons, and the performance of existing models will decrease sharply when facing incomplete data.

[0004] 2. When a model learns multiple related but different tasks (such as segmentation and classification) at the same time, the optimization direction of model parameters for different tasks may conflict, causing mutual interference, i.e. "negative transfer", which ultimately makes the overall performance lower than that of a model trained for each task separately.

[0005] 3. In the training data, the difficulty of different tasks and the difficulty of different samples are different. For example, tumors with fuzzy boundaries are much more difficult to segment than tumors with clear boundaries. Traditional fixed weight or simple learning weight loss functions cannot finely handle such differences, resulting in suboptimal model training.

[0006] The above statements regarding the background art are only intended to facilitate a better understanding of the technical solutions of the present application (the technical means used, the technical problems solved, and the technical effects achieved, etc.), and should not be regarded as an acknowledgement or any form of suggestion that such messages constitute prior art known to those skilled in the art. SUMMARY

[0007] The present application aims to at least partially solve the above technical problems. To this end, the present application aims to provide a glioma feature prediction method and system based on multi-modal magnetic resonance and multi-task deep learning, which has high robustness when dealing with incomplete modal data, can effectively coordinate the multi-task learning process, and can adaptively optimize training according to sample difficulty, thereby comprehensively improving the accuracy and reliability of prediction.

[0008] The technical solution adopted in this invention is as follows: A method for predicting glioma features based on multimodal magnetic resonance imaging (MRI) and multi-task deep learning includes the following steps: acquiring and preprocessing multimodal MRI data; processing the data through a modality completion and uncertainty quantification module to generate robust fusion features even when modalities are missing; feeding the features into a shared feature encoder to extract deep information; then executing multiple prediction tasks by a hierarchical task-gated prediction network, which coordinates the relationships between tasks through a unique gating mechanism; the entire network is trained efficiently using an adaptive loss function based on sample difficulty; finally, during inference, it can output the uncertainty score of each prediction result to provide a reference for clinical decision-making.

[0009] In a preferred embodiment, modality completion is not achieved by generating missing images, but rather through knowledge distillation. Specifically, a high-performance teacher model is first trained using complete four-modal data. When training a student model that handles incomplete data, the student model's output in the feature space is forced to approximate the teacher model's output. Simultaneously, a modality existence encoding (a vector such as [1,1,0,1] representing the absence of modality T2) is introduced, allowing the model to explicitly learn which information is currently available, thereby learning to more intelligently rely on existing modalities for reasoning.

[0010] In a preferred embodiment, the hierarchical task-gated prediction network resolves task conflicts. For example, when simultaneously predicting molecular markers (pre-stage task) and survival (post-stage task), they are not simply allowed to share features. This invention designs a gated modulation unit that receives the preliminary results (logits) from the molecular marker prediction head and converts them into a set of modulation signals (such as scaling factor γ and translation factor β). This set of signals is applied to the feature map that is about to be fed into the survival prediction head. Modulated =γFeature Original +β) is equivalent to telling the survival prediction task: given that the molecular markers appear to be IDH mutants, more attention should be paid to these features. This mechanism transforms the competitive relationship between tasks into a cooperative one.

[0011] In a preferred embodiment, the adaptive loss function based on sample difficulty enables intelligent training. This invention further designs a lightweight difficulty assessment sub-network. During training, for each input tumor sample, this sub-network concurrently determines "How difficult is the segmentation task for this sample?", "How difficult is its molecular marker prediction?", etc., and outputs a set of difficulty coefficients. These coefficients are directly used as the task losses (L) in the total loss function. seg ,L mol ,L survThe model automatically increases the weight of difficult samples (i.e., the loss weight), thereby achieving more efficient and targeted learning.

[0012] Accordingly, the present invention also provides a glioma feature prediction system based on multimodal magnetic resonance imaging and multi-task deep learning. This system includes the functional modules required to implement the above method, including a data preprocessing unit, a modality completion and uncertainty quantification module, a shared feature encoder, a hierarchical task-gated prediction network, and a model training and inference unit integrating difficulty assessment and uncertainty calculation functions.

[0013] A data preprocessing unit acquires and preprocesses the patient's multimodal MRI data. A modality completion and uncertainty quantification module, connected to the data preprocessing unit, generates robust multimodal fusion features when modalities are missing. A shared feature encoder, connected to the modality completion and uncertainty quantification module, extracts deep shared features. A hierarchical task-gated prediction network, connected to the shared feature encoder, is configured to execute at least two prediction tasks in parallel and uses the results of the preceding tasks to gate the feature inputs of the subsequent tasks. A model training and inference unit is configured to train the network using an adaptive loss function based on sample difficulty and quantify the uncertainty of the prediction results during inference. These units work collaboratively to form a complete, high-performance intelligent glioma analysis system.

[0014] In a preferred embodiment, the modality completion and uncertainty quantification module is configured to: enable a student model trained in the case of modality missing to simulate the feature expression of a teacher model trained on full modality data through a knowledge distillation mechanism, and perform feature fusion by combining modality existence encoding.

[0015] In a preferred embodiment, the hierarchical task gating prediction network includes: The segmentation prediction head is used to perform glioma segmentation tasks; Molecular biomarker prediction head, serving as a front-end task unit; Survival prediction head, as a subsequent task unit; A gated modulation unit is used to generate a modulation signal based on the output of the molecular marker prediction head, which is then applied to the features entering the lifetime prediction head.

[0016] In a preferred embodiment, the model training and inference unit includes a difficulty evaluation sub-network, which is used to evaluate the difficulty of the input sample for each prediction task and dynamically set the weight of each task loss in the total loss function based on the evaluation results.

[0017] In a preferred embodiment, the model training and inference unit is configured to use the Monte Carlo random deactivation method when performing inference tasks, and to calculate the uncertainty score for each prediction task by repeatedly predicting and statistically analyzing the dispersion of the results.

[0018] The beneficial effects of this invention are as follows: This invention utilizes knowledge distillation and existence encoding, resulting in a model whose performance degradation is significantly less than that of traditional methods when faced with clinically common modal missing data, thus making it more widely applicable.

[0019] Furthermore, the hierarchical task gating mechanism effectively mitigates negative transfer in multi-task learning, promotes knowledge sharing and collaboration among tasks, and improves the overall accuracy of various prediction tasks.

[0020] Furthermore, by using adaptive loss weighting based on sample difficulty, the model can intelligently allocate learning resources, prioritize tackling difficult samples, accelerate convergence, and reach a higher performance ceiling.

[0021] Furthermore, the uncertainty of the prediction results was quantified by the Monte Carlo random inactivation method, providing doctors with a reference for the model's confidence level. For results with high uncertainty, doctors can be prompted to conduct further examinations, thus enhancing the safety of clinical applications. Attached Figure Description

[0022] Figure 1 This is an overall flowchart of the method provided in the embodiments of the present invention.

[0023] Figure 2 This is a schematic diagram of the overall architecture of the system provided in the embodiment of the present invention.

[0024] Figure 3 This is a detailed structural diagram of the hierarchical task gating prediction network in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram illustrating the principle of the adaptive loss weighting mechanism based on sample difficulty in this embodiment of the invention. Detailed Implementation

[0026] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] It should be understood that, and also noted, in the embodiments, the functions / actions may appear in a different order than those shown in the figures. For example, depending on the functions / actions involved, they may actually be performed substantially concurrently, or sometimes the two figures shown consecutively may be performed in reverse order.

[0028] The following will be combined with the appendix Figures 1 to 4 The present invention will be described in detail.

[0029] Reference Figure 1 This figure illustrates the overall workflow of a glioma feature prediction method based on multimodal magnetic resonance imaging and multi-task deep learning, provided by an embodiment of the present invention. The method mainly includes core steps such as data acquisition and preprocessing, modality completion and uncertainty quantification, shared feature encoding, hierarchical task-gated prediction, adaptive loss training based on sample difficulty, and inference and uncertainty quantification. The following section combines... Figures 2 to 4 These steps will be explained in detail.

[0030] Step S1: Data Acquisition and Preprocessing This system first acquires multimodal MRI data of the patient's brain through a data preprocessing unit, including 3D images of four modalities: T1, T1-contrast enhancement (T1c), T2, and FLAIR. The preprocessing workflow includes: Registration: All modal images are rigidly registered to T1c space to ensure voxel alignment.

[0031] Standardization: Each image is subjected to N4 bias field correction and Z-score intensity standardization.

[0032] Data augmentation: During the training phase, operations such as random flipping, rotation, scaling, and elastic deformation are applied to increase data diversity.

[0033] Step S2: Modal Completion and Uncertainty Quantification The preprocessed data is sent to the modal completion and uncertainty quantification module.

[0034] Modality Presence Encoding: For each sample, a 4-dimensional binary vector is generated. For example, when the T2 modality is missing, it is encoded as [1,1,0,1], which corresponds to the presence or absence of T1, T1c, T2, and FLAIR, respectively.

[0035] Knowledge Distillation: This invention employs a teacher-student model framework. First, a high-performance teacher model is trained using a complete dataset containing all four modalities. Then, when training the student model to handle potentially missing modalities, not only is its own prediction task loss calculated, but an additional knowledge distillation loss L is also computed. distillThis loss is used to measure the mean squared error (MSE) between the intermediate feature maps of the student model and the corresponding feature maps of the teacher model. This forces the student model to learn to simulate the thinking pattern of the teacher model when it sees complete information, even when the input information is incomplete.

[0036] Feature fusion: Image data from each available modality are processed through independent initial convolutional layers (modality-independent encoders) to extract shallow features. These features are then concatenated with the modality existence encoding vector (which is expanded and tiled to match the feature map size) in the channel dimension to form preliminary fused features.

[0037] Step S3: Shared Feature Encoding The initially fused features are fed into a shared feature encoder, such as a 3D ResNet or SwinTransformer network. This encoder is responsible for extracting hierarchical features from low-level texture to high-level semantics, serving as a shared knowledge base for all subsequent tasks.

[0038] Step S4: Hierarchical Task Gating Prediction like Figure 3 As shown, shared features are fed into a hierarchical task-gated prediction network. This network contains three parallel prediction heads, but they are subject to gating dependencies.

[0039] Segmentation prediction head: Employs a U-Net-like decoder structure to predict the segmentation mask for the tumor.

[0040] Molecular marker prediction head (pre-task): Receives shared features, predicts the state of molecular markers such as IDH and 1p / 19q through global pooling and fully connected layers, and outputs their original logistic values ​​(logits).

[0041] Survival prediction head (sub-task): After receiving shared features, it does not perform prediction directly. Instead, it first passes through a gated modulation unit. This unit receives logits from the molecular marker prediction head, inputs them into a small multilayer perceptron (MLP), and generates a scaling factor γ and a translation factor β equal to the number of feature channels. Then, an affine transformation is performed on the shared feature map: F modulated =γF shared +β. Finally, the modulated feature F modulated Survival prediction is performed by feeding the fully connected layer into the cell. This design allows survival prediction to fully utilize prior information from molecular typing.

[0042] Step S5: Adaptive Loss Training Based on Sample Difficulty like Figure 4As shown, during training, shared features are also fed into a separate difficulty evaluation sub-network. This network has a simple structure, consisting of several convolutional layers, a global pooling layer, and a fully connected layer. Its output consists of three scalar values ​​[d]. seg ,d mol ,d surv ] represents the prediction difficulty of the current sample on the three tasks, respectively.

[0043] The total loss function is defined as: L total =w seg L seg +w mol L mol +w surv L surv +αL distill Among them, L seg It is a combination of Dice loss and Focal loss, L mol It is the cross-entropy loss, L surv This is the Cox proportional risk loss. Weight w i It is no longer a hyperparameter, but is dynamically calculated from the difficulty coefficient, such as w. i =exp(d i The algorithm is normalized. This allows the model to automatically increase the penalty for difficult samples and difficult tasks during training. α is a hyperparameter that balances the main task and the knowledge distillation task.

[0044] Step S6: Reasoning and Quantification of Uncertainty When deploying the model for prediction, Monte Carlo Dropout is employed. Specifically, during inference, the Dropout layer in the network is kept active, and the same input sample is forward-propagated T times (e.g., T=50). This results in T different predictions.

[0045] For classification tasks (molecular markers), the final prediction result is the average of the T prediction probabilities, and the uncertainty score is the variance of these T prediction probabilities.

[0046] For a regression task (survival period), the final predicted value is the average of T predicted values, and the uncertainty is its standard deviation.

[0047] For segmentation tasks, the final segmentation map is the average of T probability maps, and the uncertainty map is its pixel-level variance map.

[0048] This uncertainty score can intuitively tell clinicians how confident the model is in the current prediction result.

[0049] To verify the beneficial effects of this invention, simulation experiments were conducted on the publicly available BraTS 2021 dataset. The dataset was randomly divided into training, validation, and test sets. To simulate modality loss, 15% of the samples in the test set (T1 mode) and 15% of the samples (T2 mode) were randomly removed.

[0050] Three comparison models were set up: Baseline: Employs simple multimodal channel stitching and parallel multi-task prediction heads with fixed loss weights.

[0051] Option 2: Adopt a hierarchical structure, but without gated modulation and adaptive loss.

[0052] This solution adopts the complete method described in this manual.

[0053] The experimental results are shown in the table below: Model Task Evaluation Metrics Full modality test set Modality missing test set Baseline Segmentation Dice Score 0.895 0.812 IDH prediction AUC 0.931 0.845 Survival C-Index 0.715 0.633 Protocol two Segmentation Dice Score 0.902 0.83 IDH prediction AUC 0.94 0.861 Survival C-Index 0.728 0.659 This protocol Segmentation Dice Score 0.915 0.898 IDH prediction AUC 0.958 0.935 Survival C-Index 0.762 0.741 As can be seen from the table above, the method of this invention outperforms the baseline model across all modal data. Especially on test sets with missing modalities that more closely resemble real-world clinical scenarios, the performance advantage of this invention is extremely significant. The decrease in Dice score, AUC, and C-Index is far less than that of other methods, fully demonstrating the substantial improvement in robustness and accuracy achieved by this invention.

[0054] This invention is not limited to the above-described optional embodiments. Anyone can derive other various forms of products under the guidance of this invention. However, regardless of any changes made in their shape or structure, any technical solution that falls within the scope of the claims of this invention shall be protected by this invention.

Claims

1. A glioma feature prediction method based on multi-modal magnetic resonance and multi-task deep learning, characterized in that, The method comprises the following steps: S1. obtaining multi-modal MRI data of a patient and performing data preprocessing; S2. inputting the preprocessed multi-modal MRI data into a modality completion and uncertainty quantification module, which, when there is a missing modality, distills the features of a current student model using the knowledge of a pre-trained teacher model, and generates robust multi-modal fusion features in combination with modality existence encoding; S3. inputting the multi-modal fusion features into a shared feature encoder to extract deep shared features; S4. inputting the deep shared features into a hierarchical task gating prediction network, which performs at least two prediction tasks in parallel, wherein the feature input of at least one later-stage prediction task is gated and modulated by the prediction result of at least one earlier-stage prediction task; S5. training the hierarchical task gating prediction network using a sample difficulty-based adaptive loss function, which dynamically adjusts the weights of the task loss terms according to the difficulty evaluation results of the current input sample for each task obtained by a difficulty evaluation subnetwork; S6. in the inference stage, statistical analysis is performed on the prediction results by enabling random inactivation in the hierarchical task gating prediction network and performing multiple forward propagations, to obtain the prediction values and corresponding uncertainty scores of each task.

2. The method of claim 1, wherein, The specific implementation of the modality completion and uncertainty quantification module comprises: In the training stage, a teacher model is trained using full-modal data; subsequently, in the training of the student model, for the input missing modal data, a knowledge distillation loss is introduced to penalize the difference between the features extracted by the student model under the missing modality and the corresponding features extracted by the teacher model under the full modality; at the same time, a binary existence encoding is set for each modality, and the encoding is input into the network together with the image features.

3. The method of claim 1, wherein, The hierarchical task gating prediction network comprises a glioma segmentation task, a molecular marker prediction task, and a survival period prediction task; The molecular marker prediction task is an earlier-stage task, and the survival period prediction task is a later-stage task; The shared features are subjected to element-by-element operation with a modulation vector generated by the output logical value of the molecular marker prediction task before being input into the prediction head of the survival period prediction task, to realize gating modulation.

4. The method of claim 3, wherein, The specific way of gating modulation is to input the output logical value of the molecular marker prediction task into a small neural network to generate a scaling factor and a translation factor, and to perform affine transformation on the feature map input into the survival period prediction task using the scaling factor and the translation factor.

5. The method of claim 1, wherein, The specific implementation of the sample difficulty-based adaptive loss function comprises: A difficulty evaluation subnetwork is constructed, which inputs the features extracted by the shared feature encoder and outputs a difficulty coefficient for each prediction task; The total loss function of the method is the weighted sum of the task losses, wherein the weight of each task is proportional to the corresponding difficulty coefficient output by the difficulty evaluation subnetwork.

6. A glioma feature prediction system based on multi-modal magnetic resonance and multi-task deep learning, characterized in that, It comprises: a data preprocessing unit configured to obtain and preprocess multi-modal MRI data of a patient; The modal complementation and uncertainty quantification module is connected with the data preprocessing unit and is configured to generate robust multimodal fusion features when there is a missing modality. The shared feature encoder is connected with the modal complementation and uncertainty quantification module and is configured to extract deep shared features. The hierarchical task gated prediction network is connected with the shared feature encoder and is configured to perform at least two prediction tasks in parallel and to gate modulate the feature input of a later task using the result of a previous task. The model training and inference unit is configured to train the network using a sample difficulty-based adaptive loss function and to quantify the uncertainty of the prediction result during inference.

7. The system of claim 6, wherein, The modal complementation and uncertainty quantification module is configured to enable a student model trained under a missing modality condition to simulate the feature expression of a teacher model trained under a full modality condition through a knowledge distillation mechanism and to perform feature fusion in combination with modality existence encoding.

8. The system of claim 6, wherein, The hierarchical task gated prediction network includes: A segmentation prediction head configured to perform a glioma segmentation task; A molecular marker prediction head as a previous task unit; A survival prediction head as a later task unit; A gating modulation unit configured to generate a modulation signal according to the output of the molecular marker prediction head and to act on the feature input into the survival prediction head.

9. The system of claim 6, wherein, The model training and inference unit internally includes a difficulty evaluation subnetwork configured to evaluate the difficulty of an input sample for each prediction task and to dynamically set the weight of each task loss in a total loss function based on the evaluation result.

10. The system of claim 6, wherein, The model training and inference unit is configured to use a Monte Carlo random inactivation method to calculate the uncertainty score of each prediction task by repeatedly predicting multiple times and statistically analyzing the dispersion degree of the results during inference.

Images