A breast magnetic resonance image lesion area identification and annotation and curative effect prediction method
By automatically identifying breast tumor regions and predicting chemotherapy efficacy through a multi-task deep learning model, the problem of time-consuming manual screening and subjective judgment in breast tumor imaging detection is solved, and rapid and accurate tumor region labeling and efficacy evaluation are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2022-08-31
- Publication Date
- 2026-06-12
AI Technical Summary
In current magnetic resonance imaging (MRI) detection of breast tumors, manual screening is time-consuming and labor-intensive, easily leading to missed detections. Tumor region labeling is tedious and relies on subjective judgment, making it difficult to obtain numerical evaluation.
A multi-task deep learning model was adopted, combined with a transfer learning strategy. By training the multi-task deep learning model, the tumor region was automatically identified and the chemotherapy efficacy was predicted using magnetic resonance imaging and pathological data. The model training was optimized using the Dice loss function and the weighted binary cross-entropy loss function.
It enables rapid and accurate tumor region identification and chemotherapy efficacy prediction, reduces doctors' workload, lowers the risk of misdiagnosis and missed diagnosis, and provides numerical scoring to assist in diagnosis.
Smart Images
Figure CN115409804B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of digital image processing technology and deep learning visual analysis, specifically to a method for identifying and annotating lesion areas and predicting chemotherapy efficacy based on breast magnetic resonance imaging, and its application. Background Technology
[0002] Currently, the standard method for detecting breast tumors is dynamic contrast-enhanced MRI (DCE-MRI). Patients receive multiple sequences of MRI images using an MRI scanner, each sequence containing dozens or even hundreds of images depending on the scanner's precision. A single MRI scan produces approximately 720 images. Manually screening these images one by one is undoubtedly time-consuming and labor-intensive, and can also lead to missed detections due to visual fatigue. Secondly, for subsequent research purposes, tumor regions need to be labeled to extract relevant tumor image features and study tumor-related characteristics. Manual labeling is tedious and wastes a significant amount of professional time. Furthermore, the prognosis of tumors relies excessively on subjective judgment, making it difficult to obtain a widely accepted numerical assessment.
[0003] With the development of deep learning technology, it has been widely applied in image recognition. Accurately identifying and labeling tumor regions using deep learning techniques not only reduces the workload of professionals interpreting images and assists doctors in diagnosis, reducing misdiagnosis and missed diagnosis, but also reduces doctors' workload and shortens review time. Simultaneously, it provides a numerical score for the efficacy of chemotherapy, offering a basis for doctors' judgment. Therefore, methods for identifying and labeling lesion regions and predicting treatment efficacy in breast MRI images have significant practical value for the clinical auxiliary diagnosis of breast cancer. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of breast imaging diagnosis by providing a method for identifying and labeling lesions in breast magnetic resonance imaging and predicting treatment efficacy. This method enables faster and more accurate identification and labeling of lesions in breast magnetic resonance imaging and predicts treatment efficacy by utilizing deep learning technology, transfer learning training strategies, and automatic learning of multi-task loss functions to achieve tumor identification and labeling in breast imaging and prediction of chemotherapy efficacy.
[0005] This invention provides a method for identifying and annotating lesion areas and predicting treatment efficacy in breast magnetic resonance imaging, comprising the following steps:
[0006] Step 1: Obtain the patient's magnetic resonance imaging data and the corresponding pathological data;
[0007] Step 2: Preprocessing of MRI and Pathological Data
[0008] The obtained magnetic resonance imaging data were enhanced, and the pathological data were normalized.
[0009] Step 3: Create a dataset
[0010] The preprocessed magnetic resonance imaging data and pathological data were divided into a pre-training dataset and a training dataset. The pre-training dataset was then divided into a training set and a validation set in a 7:3 ratio, and the training dataset was divided into a training set, a validation set, and a test set in a 6:2:2 ratio.
[0011] Step 4: Construct a multi-task deep learning model;
[0012] Step 5: Train a multi-task deep learning model
[0013] 5.1 First, train the model using the pre-training dataset, evaluate the performance of the multi-task deep learning model using the validation set in the pre-training dataset, and save the best-performing model as the pre-training model.
[0014] 5.2 The pre-trained model is retrained using the training set in the training dataset, and the model that performs best on the validation set is saved.
[0015] 5.3 The evaluation metrics for multi-task deep learning models are Dice and Jaccard for segmentation, and Accuracy and Precision for prediction. The evaluation formula is as follows:
[0016]
[0017]
[0018]
[0019]
[0020] In the formula, A represents the tumor region, B represents the actual segmented region, TP represents the positive sample predicted by the model as positive, TN represents the negative sample predicted by the model as negative, FP represents the negative sample predicted by the model as positive, and FN represents the positive sample predicted by the model as negative.
[0021] Step 6: Input the validation set data into the trained multi-task deep learning model and output the tumor region masking and chemotherapy efficacy prediction results.
[0022] Preferably, the data augmentation performed in step 2 involves randomly flipping, translating, rotating, and scaling the image to increase the number of samples.
[0023] Preferably, the multi-task deep learning model includes a segmentation sub-model and a prediction sub-model.
[0024] Preferably, the segmentation sub-model includes seven convolutional layers, with the first to third convolutional layers connected to pooling layers, and the fifth to seventh convolutional layers connected to upsampling layers. The first and seventh convolutional layers, the second and sixth convolutional layers, and the third and fifth convolutional layers are connected in skip connections. The prediction sub-model includes an attention module and a fully connected layer, with the attention module connected to the fourth convolutional layer.
[0025] Preferably, the initial task weights of the segmentation sub-model and the prediction sub-model in the multi-task deep learning model are 1:1.
[0026] Preferably, the multi-task deep learning model continuously adjusts the weights of the segmentation and prediction tasks during training. The Dice loss function is used in the segmentation sub-tasks, with the following formula:
[0027]
[0028] In the formula, X represents the true label, and Y represents the predicted label. This indicates the predicted intersection of X and Y. This represents the loss function value for splitting the subtasks. In the prediction subtask, a weighted binary cross-entropy loss function is used, with the following formula:
[0029]
[0030]
[0031] In the formula, Indicates the number of positive samples. Indicates the number of negative samples. The weights of positive samples are represented. Let represent the weight of the negative sample, p represent the true label value, and y represent the predicted label value. This represents the loss function value for the prediction subtask.
[0032] As a preferred option, step 6 is as follows:
[0033] 6.1 The multi-task deep learning model, after training, can simultaneously accept inputs of magnetic resonance imaging and pathological data from the same individual;
[0034] 6.2 After training, the multi-task deep learning model directly outputs the lesion region mask and the predicted value of the chemotherapy efficacy of the lesion. The mask is found by finding the maximum connected component and then filtered by threshold to be transformed into a binary mask. The output value of the chemotherapy efficacy of the lesion is a double-precision decimal from zero to one, which represents the probability value that no tumor residue is found at the original tumor site after neoadjuvant chemotherapy.
[0035] The beneficial effects of this invention are as follows:
[0036] This invention uses a multi-task deep learning model that simultaneously receives image and pathological information input. Through model training, it automatically learns the localization and labeling of tumors and predicts the efficacy of chemotherapy for tumors. This enables the segmentation of breast tumor regions and the prediction of treatment efficacy, effectively reducing doctors' redundant working time, reducing misdiagnosis and missed diagnosis caused by doctors' fatigue, and enhancing the clinical value of deep learning in assisting doctors in diagnosis. Attached Figure Description
[0037] Figure 1 This is a flowchart of the present invention.
[0038] Figure 2 This is a structural block diagram of the present invention.
[0039] Figure 3 This is a schematic diagram of breast tumor identification and labeling in this invention.
[0040] Figure 4 This is a diagram of the deep learning model structure as commonly understood in this invention.
[0041] Figure 5 This is a structural diagram of the attention module in this invention. Detailed Implementation
[0042] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0043] This embodiment discloses a method for identifying and annotating lesion areas and predicting treatment efficacy in breast magnetic resonance imaging, such as... Figure 1 As shown, it includes the following steps:
[0044] Step 1: Obtain the patient's magnetic resonance imaging data and the corresponding pathological data;
[0045] Step 2: Preprocessing of MRI and Pathological Data
[0046] The obtained magnetic resonance imaging data is subjected to image enhancement. Data enhancement involves randomly flipping, translating, rotating, and scaling the images to increase the number of samples, while the pathological data is normalized.
[0047] Step 3: Create a dataset
[0048] The preprocessed magnetic resonance imaging data and pathological data were divided into a pre-training dataset and a training dataset. The pre-training dataset was then divided into a training set and a validation set in a 7:3 ratio, and the training dataset was divided into a training set, a validation set, and a test set in a 6:2:2 ratio.
[0049] Step 4: Construct a multi-task deep learning model
[0050] The multi-task deep learning model includes a segmentation sub-model and a prediction sub-model. The segmentation sub-model comprises seven convolutional layers. The first to third convolutional layers are connected to pooling layers, and the fifth to seventh convolutional layers are connected to upsampling layers. The first and seventh convolutional layers, the second and sixth convolutional layers, and the third and fifth convolutional layers are connected in skip connections. The prediction sub-model includes an attention module and a fully connected layer. The attention module is connected to the fourth convolutional layer.
[0051] Specifically, such as Figure 4 As shown, in step 4, the model training module for training the multi-task deep learning model employs a multi-input multi-output network structure. This network utilizes a transfer learning strategy for training, achieving a supervised training method. Leveraging the nonlinearity and gradient descent principles in deep learning, iterative training is used to minimize the loss function, thereby fitting the model. The transfer learning strategy in model training module 4 specifically refers to first training the network using a public dataset, and then retraining and fitting the network, which can reduce training time and improve accuracy.
[0052] In this embodiment, the multi-task deep learning model is based on an RTX 6000 graphics card, uses the ADAM optimizer, and initializes to 1e-4 learning epochs, gradually decreasing to 1e-7 in increments of 0.5 during training. The model stops training when the number of training epochs exceeds 1000 or the learning rate drops below 1e-7. During training, the model that performs best on the validation set is saved.
[0053] like Figure 4 As shown, the multi-task deep learning model receives 360*360*160 three-dimensional image data and a 1*7 pathological data input. The black arrows indicate the data flow. The segmentation sub-model consists of multiple convolutional pooling layers, connected to subsequent convolutional layers using skip connections to enhance feature transfer. In the prediction sub-model, the attention module extracts and processes features from intermediate layers, combines them with the input pathological features, and then outputs the prediction result through two fully connected layers.
[0054] The attention module used in the prediction sub-model, such as Figure 5As shown, the model includes a channel attention submodule and a spatial attention submodule. Channel attention uses channel-wide max pooling and channel-wide average pooling, followed by a shared perceptron to enhance the weights of features across channels. Then, the model enters the spatial attention submodule, which uses spatial-wide max pooling and spatial-wide average pooling, followed by a convolutional layer, thus spatially enhancing the output features. Features processed by the attention modules are strengthened in tumor regions while features in non-tumor regions are suppressed, allowing the model to better focus on the tumor region.
[0055] Step 5: Train a multi-task deep learning model
[0056] 5.1 First, train the model using the pre-training dataset, evaluate the performance of the multi-task deep learning model using the validation set in the pre-training dataset, and save the best-performing model as the pre-training model.
[0057] 5.2 The pre-trained model is retrained using the training set in the training dataset, and the model that performs best on the validation set is saved.
[0058] 5.3 The evaluation metrics for multi-task deep learning models are Dice and Jaccard for segmentation, and Accuracy and Precision for prediction. The evaluation formula is as follows:
[0059]
[0060]
[0061]
[0062]
[0063] In the formula, A represents the tumor region, B represents the actual segmented region, TP represents the positive sample predicted by the model as positive, TN represents the negative sample predicted by the model as negative, FP represents the negative sample predicted by the model as positive, and FN represents the positive sample predicted by the model as negative.
[0064] In the multi-task deep learning model, the initial task weights of the segmentation sub-model and the prediction sub-model are 1:1.
[0065] Specifically, during training, the weights of the segmentation and prediction tasks are continuously adjusted. The Dice loss function is used in the segmentation sub-tasks, with the following formula:
[0066]
[0067] In the formula, X represents the true label, and Y represents the predicted label. This indicates the predicted intersection of X and Y. This represents the loss function value for splitting the subtasks. In the prediction subtask, a weighted binary cross-entropy loss function is used, with the following formula:
[0068]
[0069]
[0070] In the formula, Indicates the number of positive samples. Indicates the number of negative samples. The weights of positive samples are represented. Let represent the weight of the negative sample, p represent the true label value, and y represent the predicted label value. This represents the loss function value for the prediction subtask.
[0071] Step 6: Input the validation set data into the trained multi-task deep learning model and output the tumor region masking and chemotherapy efficacy prediction results.
[0072] Specifically, 6.1 After training, the multi-task deep learning model can simultaneously receive inputs of magnetic resonance imaging and pathological data from the same person;
[0073] 6.2 After training, the multi-task deep learning model directly outputs the lesion region mask and the predicted value of the chemotherapy efficacy of the lesion. The mask is found by finding the maximum connected component and then filtered by threshold to be transformed into a binary mask. The output value of the chemotherapy efficacy of the lesion is a double-precision decimal from zero to one, which represents the probability value that no tumor residue is found at the original tumor site after neoadjuvant chemotherapy.
[0074] Specifically, such as Figure 2 As shown, the method for identifying and labeling lesion areas and predicting treatment efficacy in breast magnetic resonance imaging employs an algorithm module comprising a data input module 1, a data preprocessing module 2, a data filtering module 3, a model training module 4, and a result output module 5. The data input module 1 inputs the obtained imaging and pathological data into the data preprocessing module 2, the preprocessed data into the data filtering module 3, the filtered dataset into the model training module 4, and finally, the result output module 5 outputs the final tumor area segmentation and chemotherapy efficacy prediction results.
[0075] Data input module 1 acquires the patient's breast MRI image data and corresponding pathological information data; data preprocessing module 2 preprocesses the image data and pathological data, mainly by normalizing the image data and increasing the sample size of the image data through a certain degree of translation, rotation, scaling, and flipping, thereby increasing the amount of experimental data; data filtering module 3 divides the data into different sets by shuffling the data and random sampling; model training module 4 mainly pre-trains the dataset, then performs model transfer and retraining, finally obtaining a multi-task model containing two sub-task models; result output module 5 outputs the tumor region segmentation mask for the image and the prediction results of the tumor chemotherapy efficacy.
[0076] The data input module accepts magnetic resonance imaging data such as Figure 3 (Left) shows the final output tumor region mask, such as... Figure 3 As shown on the right.
[0077] To verify the performance of the method for lesion region identification and treatment prediction in breast magnetic resonance imaging of the present invention, five-fold cross-validation was used to validate the training set data. The results are shown in Table 1.
[0078] Table 1. Validation results of tumor region segmentation and efficacy prediction
[0079]
[0080] In Table 1, Dice represents the proportion of the intersection between the tumor region and the segmented region. Jaccard is also used to evaluate the segmentation effect. Accuracy and Precision are used to describe the efficacy of chemotherapy. Accuracy refers to the proportion of correctly classified samples out of the total samples, and precision refers to the accuracy between the predicted result and the actual result. Using the lesion region identification and efficacy prediction method for breast MRI images of this invention, the mean Dice coefficient of the five-fold cross-validation reached 0.79, while the predictive efficacy reached 0.72. Furthermore, there were no significant differences in the five-fold performance, fully demonstrating the good stability of this invention. It can be well used for clinical auxiliary diagnosis, improving the accuracy of doctors' diagnoses and reducing the occurrence of missed diagnoses and misdiagnoses. The multi-task deep learning model used in this invention utilizes a learnable loss function and transfer learning strategy to simultaneously output tumor masking and chemotherapy efficacy prediction results, which have been verified by five-fold cross-validation.
[0081] The specific examples described above are used to explain the present invention, not to limit it. Any modifications and alterations made to the present invention within the spirit and scope of the claims fall within the protection scope of the present invention.
Claims
1. A method for lesion region identification and annotation and efficacy prediction of breast magnetic resonance images, characterized in that, Includes the following steps: Step 1: Obtain the patient's magnetic resonance imaging data and the corresponding pathological data; Step 2: Preprocessing of MRI and Pathological Data The obtained magnetic resonance imaging data were enhanced, and the pathological data were normalized. Step 3: Create a dataset The preprocessed magnetic resonance imaging data and pathological data were divided into a pre-training dataset and a training dataset. The pre-training dataset was then divided into a training set and a validation set in a 7:3 ratio, and the training dataset was divided into a training set, a validation set, and a test set in a 6:2:2 ratio. Step 4: Construct a multi-task deep learning model; the multi-task deep learning model includes a segmentation sub-model and a prediction sub-model; the segmentation sub-model includes seven convolutional layers, the first to third convolutional layers are connected to pooling layers, the fifth to seventh convolutional layers are connected to upsampling layers, and the first and seventh convolutional layers, the second and sixth convolutional layers, and the third and fifth convolutional layers are connected in skip connections respectively; the prediction sub-model includes an attention module and a fully connected layer, and the attention module is connected to the fourth convolutional layer; Step 5: Train a multi-task deep learning model 5.1 First, train the model using the pre-training dataset, evaluate the performance of the multi-task deep learning model using the validation set in the pre-training dataset, and save the best-performing model as the pre-training model. 5.2 The pre-trained model is retrained using the training set in the training dataset, and the model that performs best on the validation set is saved. 5.3 The evaluation metrics for multi-task deep learning models are Dice and Jaccard for segmentation, and Accuracy and Precision for prediction. The evaluation formula is as follows: In the formula, A represents the tumor region, B represents the actual segmented region, TP represents the positive sample predicted by the model as positive, TN represents the negative sample predicted by the model as negative, FP represents the negative sample predicted by the model as positive, and FN represents the positive sample predicted by the model as negative. The multi-task deep learning model continuously adjusts the weights of segmentation and prediction tasks during training. The Dice loss function is used in the segmentation sub-tasks, with the following formula: In the formula, X represents a real label, Y represents a predicted label, represents the intersection of the prediction X and Y, represents the loss function value of the segmentation subtask; in the prediction subtask, a weighted binary cross-entropy loss function is used, and the formula is: In the formula, Indicates the number of positive samples. Indicates the number of negative samples. This represents the weight of the positive sample. Let represent the weight of the negative sample, p represent the true label value, and y represent the predicted label value. This represents the loss function value for the prediction subtask; Step 6: Input the validation set data into the trained multi-task deep learning model, and output the tumor region masking and chemotherapy efficacy prediction results. Step 6 is as follows: 6.1 The multi-task deep learning model, after training, can simultaneously accept inputs of magnetic resonance imaging and pathological data from the same individual; 6.2 After training, the multi-task deep learning model directly outputs the lesion region mask and the predicted value of the chemotherapy efficacy of the lesion. The mask is found by finding the maximum connected component and then filtered by threshold to be transformed into a binary mask. The output value of the chemotherapy efficacy of the lesion is a double-precision decimal from zero to one, which represents the probability value that no tumor residue is found at the original tumor site after neoadjuvant chemotherapy.
2. The method for identifying and marking lesion areas and predicting treatment efficacy in breast magnetic resonance imaging according to claim 1, characterized in that, The data augmentation performed in step 2 involves randomly flipping, translating, rotating, and scaling the images to increase the number of samples.
3. The method for identifying and marking lesion areas and predicting treatment efficacy in breast magnetic resonance imaging according to claim 1, characterized in that, In the multi-task deep learning model, the initial task weights of the segmentation sub-model and the prediction sub-model are 1:1.