Prediction classification method for breast cancer patient pCR based on image-gene interpretability deep learning

Abstract

The invention provides a breast cancer patient pCR prediction classification method based on image-gene interpretability deep learning. The method comprises the following steps: dividing an obtained data set into a training set and a test set; splitting all the three-dimensional MRI images into two-dimensional slice images; the gene text features of each breast cancer patient are classified into N different sets; segmenting a two-dimensional slice image corresponding to the training set into a plurality of patches; each patch and the gene text feature corresponding to each patch are converted into a corresponding high-dimensional Embedding (high-dimensional Embedding); respectively multiplying H by Wk and Wv to obtain K and V; multiplying G by WQ to obtain Q; obtaining a gene guidance image feature CoAttn; the CoAttn and the Q are fused to obtain a feature set F; inputting the F into a network model, and training the network model based on a loss function; inputting the two-dimensional slice image corresponding to the test set into the trained network model, and outputting a plurality of classification results; and adopting a maximum voting strategy to make decisions on the plurality of classification results. According to the method, pCR and npCR classification is carried out on the patient based on a co-attention mechanism of gene text features and MRI images, the interpretability of model decision is enhanced, and a more visual and reliable auxiliary diagnosis tool is provided for clinicians.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pCR prediction classification of breast cancer patients, and in particular, to a method for predicting and classifying pCR of breast cancer patients based on image-gene explainable deep learning. BACKGROUND

[0002] Neoadjuvant therapy is a widely adopted treatment strategy for breast cancer globally, which can effectively reduce the size of preoperative tumors, inhibit potential metastasis risk, and even achieve preoperative pathological complete response (pCR) in some cases, thereby significantly improving the prognosis of patients. However, due to the complex heterogeneity of breast cancer, the effect of neoadjuvant therapy varies significantly among individuals. Among patients receiving neoadjuvant therapy, the proportion of achieving pCR ranges from 10% to 50%, while other patients may only show partial pathological response or no response (npCR). According to the different responses of patients to neoadjuvant therapy, personalized follow-up treatment plans need to be developed.

[0003] In the evaluation of pCR and npCR treatment, medical image analysis plays a crucial role. Traditional diagnostic methods usually rely on radiologists' visual interpretation of breast MRI images, which is often time-consuming and susceptible to subjective factors. In recent years, deep learning technology has been widely applied in medical image analysis, effectively improving the accuracy and efficiency of breast cancer diagnosis.

[0004] Although deep learning has made significant achievements in image classification and recognition, its application in medical imaging still faces the problem of insufficient explainability. Medical decision-making requires high transparency and explainability so that doctors can understand and trust the model's judgment. In addition, the occurrence, development, and treatment response of breast cancer are closely related to the genetic characteristics of patients. Genetic information can provide key molecular-level insights, helping to improve diagnostic accuracy and personalized treatment effectiveness.

[0005] However, there are challenges in current research: how to effectively integrate gene data and image data, and how to ensure the effectiveness and explainability of model training in the case of incomplete data. In particular, due to the high cost of genetic testing, not all cases in the image data set can correspond to complete genetic feature data. The absence of genetic features for some cases poses a challenge to model training.

[0006] In short, the existing technology has the following three main shortcomings:

[0007] (1) Data scarcity: The high cost of MRI detection and patient privacy lead to the scarcity of MRI data, which poses a challenge to the training and generalization ability of the model.

[0008] (2) Lack of model explainability: existing deep learning models in medical image analysis are often regarded as "black boxes", lacking sufficient explainability, making it difficult for doctors to understand the basis of the model's decision, thereby affecting its application in the clinic.

[0009] (3) Limited cross-modal fusion methods: how to effectively integrate gene data and image data and extract complementary information from both is a difficulty in current research. SUMMARY

[0010] The purpose of the present application is to address the shortcomings of the prior art, and to provide a breast cancer patient pCR prediction classification method based on image-genetic explainable deep learning. To address the problem of MRI sample scarcity, a strategy of MRI three-dimensional splitting into two-dimensional and maximum voting decision is adopted to ensure the robustness of model training. Based on the co-attention mechanism, the patient's genetic features are used to guide the classification of breast cancer patients pCR and npCR. The present application solves the problem of insufficient model training caused by data set sample scarcity, improves the accuracy of breast cancer treatment classification and the explainability of the model, and provides a new solution for the application of deep learning in medical image analysis.

[0011] To this end, the purpose of the present application is to provide a breast cancer patient pCR prediction classification method based on image-genetic explainable deep learning.

[0012] To achieve the above purpose, the technical solution of the present application provides a breast cancer patient pCR prediction classification method based on image-genetic explainable deep learning, which comprises the following steps: step S1: obtaining a data set; wherein the data set is a three-dimensional MRI image of a breast cancer patient; step S2: dividing the three-dimensional MRI image into a training set and a test set according to a predetermined proportion; step S3: splitting all three-dimensional MRI images corresponding to the data set into a plurality of two-dimensional slice images; wherein the two-dimensional slice images corresponding to the same breast cancer patient collectively associate the genetic text features of the breast cancer patient; the genetic text features of each breast cancer patient are classified into N different sets with similar biological functions; step S4: dividing each two-dimensional slice image corresponding to the training set into a plurality of patches; step S5: using a fully connected neural network to convert each patch and the genetic text features corresponding to each patch into corresponding high-dimensional Embedding; wherein the high-dimensional Embedding converted from each patch is denoted as H, M represents the total number of two-dimensional slice images patches; the high-dimensional Embedding converted from the genetic text features corresponding to each patch is denoted as G, Step S6: H is respectively multiplied by trainable parameters W k and W vThe multiplication respectively obtains the eigenvector K and the eigenvector V;Wherein, the expression of the eigenvector K is: The expression of the eigenvector V is: Step S7: multiply G with trainable parameters W Q The multiplication respectively obtains the eigenvector K and the eigenvector V;Wherein, the expression of the eigenvector K is: Step S8: based on the eigenvector K, the eigenvector V and the eigenvector Q, the attention mechanism is used to model the explainable interaction between the image and the gene, so as to obtain the gene guided image feature CoAttn;Step S9: the gene guided image feature CoAttn and the eigenvector Q are fused to obtain the feature set F;Step S10: a prediction classification network model of breast cancer patients pCR is constructed;Step S11: a loss function of the prediction classification network model is constructed;Step S12: the feature set F is input into the prediction classification network model, and the prediction classification network model is trained based on the loss function;Step S13: a plurality of two-dimensional slice images corresponding to the test set are respectively input into the trained prediction classification network model, and a plurality of classification results of whether the breast cancer patients corresponding to the test set belong to pCR are output;Step S14: the maximum voting strategy is used to make a decision on the plurality of classification results, and the final classification result of whether the breast cancer patients corresponding to the test set belong to pCR is obtained.

[0013] Preferably, the formula expression for modeling the explainable interaction between the image and the gene by the attention mechanism is as follows:

[0014]

[0015] In formula (1), d k The dimension number of the eigenvector K; The calculation result of is expressed as dot product attention score;The Softmax function is used to normalize the dot product attention score, so as to convert the dot product attention score into the gene guided image feature CoAttn.

[0016] Preferably, the prediction classification network model of breast cancer patients pCR is a Transformer classifier.

[0017] Preferably, the expression of the loss function of the prediction classification network model is:

[0018] L=-[y·logp+(1-y)·log(1-p)](2)

[0019] In equation (2), y represents the true label, which takes the value of 0 or 1; p represents the probability that the model predicts the positive class. When y = 1, the loss function only focuses on logp, which is the probability that the model predicts the positive class; when y = 0, the loss function only focuses on log(1-p), which is the probability that the model predicts the negative class.

[0020] Preferably, before step S1, the method further includes: supplementing the missing gene features in the gene text features of each breast cancer patient using the feature average method.

[0021] Preferably, the maximum voting strategy follows the following formula:

[0022]

[0023] In equation (3), I(·) is an indicator function that takes the value of 1 when the internal calculation result is true, and 0 otherwise. The meaning of this formula is that for each possible category y, the total number of predictions of the category by N classifiers is calculated, and the category with the most total predictions is finally selected as the prediction output of the ensemble model.

[0024] The beneficial effects of this invention are:

[0025] This invention provides a method for predicting and classifying pCR (progressive cervical recurrence) in breast cancer patients based on image-gene interpretability deep learning. By leveraging the co-attention mechanism between gene text features and MRI images, the method classifies patients into pCR and npCR, improving both predictive performance and interpretability of the model's decisions. This provides clinicians with a more intuitive and reliable diagnostic tool. Furthermore, the method of decomposing a three-dimensional model into a two-dimensional supplement effectively addresses the issue of data incompleteness, ensuring the robustness and accuracy of the model in practical applications.

[0026] Specifically, we will explain why the image-gene interpretability-based deep learning prediction and classification method for pCR in breast cancer patients provided in this invention enhances the interpretability of model decisions. Because the single-omics information nature of radiomics makes it difficult to deeply understand the complex biological processes of tumors and to meet the interpretability requirements for accurate prediction of tumor treatment efficacy, one important approach to solving this problem is to integrate multi-omics data for analysis, including genomics (transcriptomics), proteomics, metabolomics, and pathomics. The multi-omics integrated analysis approach of this invention is image-gene (transcriptomics), which combines gene expression data with imaging phenotypes to capture the association between gene expression and macroscopic imaging phenotypes. The inclusion of gene expression data can provide the intrinsic mechanisms and genetic roots of MRI phenotypic changes, thus addressing its "black box" nature and improving model efficacy and robustness on the basis of interpretability.

[0027] Specifically, we will explain why the multimodal data fusion in this invention differs significantly from existing multimodal data fusion techniques. Previous multimodal data fusion methods in machine learning and deep learning research primarily employed early and late fusion approaches. Early fusion methods require high similarity between different modalities, while late fusion models each modality separately before fusing the predictions, neglecting complementary information between different modalities and potentially resulting in fusion performance inferior to single-modal methods. This invention, based on a mid-stage fusion method, differs from the former two by extending backpropagation and parameter updates to the entire process from raw data input to output. This allows for the capture of correlation and complementary information between different modalities and flexible adjustments. For example, specific genes or pathways may influence imaging changes in specific regions; therefore, this invention incorporates genomic information to guide the selection of radiographic features during model construction.

[0028] Additional aspects and advantages of the invention will become apparent from the description which follows, or may be learned by practice of the invention. Attached Figure Description

[0029] Figure 1 A schematic flowchart of a method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning, according to an embodiment of the present invention, is shown.

[0030] Figure 2 A schematic block diagram of a method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning, according to an embodiment of the present invention, is shown. Detailed Implementation

[0031] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0032] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0033] Figure 1 A schematic flowchart illustrating an embodiment of the present invention is shown, illustrating a method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning. Figure 1 As shown, this image-gene interpretability-based deep learning method for predicting and classifying pCR in breast cancer patients includes:

[0034] Step S1: Obtain the dataset; the dataset consists of 3D MRI images of breast cancer patients;

[0035] Step S2: Divide the 3D MRI images into a training set and a test set according to a preset ratio;

[0036] Step S3: Divide all the 3D MRI images corresponding to the dataset into several 2D slice images;

[0037] Step S4: Divide each two-dimensional slice image corresponding to the training set into several patches;

[0038] Step S5: Using a fully connected neural network, convert each patch and the corresponding gene text features into corresponding high-dimensional embeddings; the high-dimensional embedding converted for each patch is denoted as H; the high-dimensional embedding converted for the gene text features corresponding to each patch is denoted as G.

[0039] Step S6: Compare H with the trainable parameters W respectively. k and W v Multiply them to obtain eigenvectors K and V, respectively;

[0040] Step S7: Combine G with trainable parameters W Q Multiplying them together yields the eigenvector Q;

[0041] Step S8: Model the interpretable interaction between images and genes using an attention mechanism to obtain gene-guided image features CoAttn;

[0042] Step S9: Fuse the gene-guided image feature CoAttn with the feature vector Q to obtain the feature set F;

[0043] Step S10: Construct a predictive classification network model for pCR in breast cancer patients;

[0044] Step S11: Construct the loss function for the prediction classification network model;

[0045] Step S12: Input the feature set F into the prediction classification network model and train the prediction classification network model based on the loss function;

[0046] Step S13: Input several two-dimensional slice images corresponding to the test set into the trained prediction classification network model, and output multiple classification results of whether the breast cancer patients corresponding to the test set belong to pCR;

[0047] Step S14: Use the maximum voting strategy to make a decision on multiple classification results and obtain the final classification result of whether the breast cancer patients corresponding to the test set belong to pCR.

[0048] In this embodiment, the two-dimensional slice images corresponding to the same breast cancer patient are collectively associated with the genetic text features of that breast cancer patient; the genetic text features of each breast cancer patient are classified into N different sets with similar biological functions, where N is a positive integer.

[0049] In this embodiment, the high-dimensional embedding transformed by each patch is denoted as H. M represents the total number of patches in each 2D slice image; the high-dimensional embedding of the gene text feature transformation corresponding to each patch is denoted as G. d k This represents the dimension of the feature vector.

[0050] In this embodiment, the trainable parameter W k Trainable parameters W v and trainable parameters W Q All of these are obtained through training with neural networks.

[0051] In this embodiment, the expression for the feature vector K is: The expression for the eigenvector V is:

[0052] In this embodiment, the expression for the feature vector Q is:

[0053] In this embodiment, based on feature vectors K, V, and Q, an attention mechanism is used to model the interpretable interaction between images and genes to obtain gene-guided image features CoAttn.

[0054] In this embodiment, the acquired 3D MRI images of breast cancer patients are divided into training and testing sets according to a preset ratio, and all 3D MRI images are split into several 2D slice images; wherein, the 2D slice images corresponding to the same breast cancer patient are collectively associated with the genetic text features of that breast cancer patient; the genetic text features of each breast cancer patient are classified into N different sets with similar biological functions. Further, as... Figure 2 As shown, each two-dimensional slice image corresponding to the training set is divided into several patches. Using a fully connected neural network, each patch and its corresponding gene text features are converted into corresponding high-dimensional embeddings. The high-dimensional embedding converted for each patch is denoted as H. M represents the total number of patches in each 2D slice image; the high-dimensional embedding of the gene text feature transformation corresponding to each patch is denoted as G. Furthermore, H is compared with the trainable parameters W respectively. k and Wv Multiplying them yields eigenvectors K and V, respectively. G is then multiplied by the trainable parameters W. Q Multiplying these features yields the feature vector Q. Further, based on feature vectors K, V, and Q, an attention mechanism is used to model the interpretable interaction between images and genes to obtain gene-guided image features CoAttn. Further, the gene-guided image features CoAttn are fused with feature vector Q to obtain the feature set F. Further, a predictive classification network model for pCR in breast cancer patients is constructed, and a loss function for the predictive classification network model is developed. The feature set F is input into the predictive classification network model, and the model is trained based on the loss function. Further, several two-dimensional slice images corresponding to the test set are input into the trained predictive classification network model, and multiple classification results are output regarding whether the breast cancer patients in the test set belong to pCR. A maximum voting strategy is used to decide among these multiple classification results to obtain the final classification result of whether the breast cancer patients in the test set belong to pCR.

[0055] In this embodiment, the image-gene interpretability-based deep learning prediction and classification method for pCR in breast cancer patients provided by this invention classifies patients into pCR and npCR based on the co-attention mechanism of gene text features and MRI images. This not only improves the model's predictive performance but also enhances the interpretability of the model's decisions, providing clinicians with a more intuitive and reliable auxiliary diagnostic tool. Furthermore, the three-dimensional decomposition into a two-dimensional supplementary method effectively solves the problem of data incompleteness, ensuring the robustness and accuracy of the model in practical applications.

[0056] In one embodiment of the present invention, the formula for modeling the interpretable interaction between images and genes through an attention mechanism is as follows:

[0057]

[0058] In equation (1), d k Let K be the dimension number of the feature vector K; The calculation result is represented as a dot product attention score; the Softmax function is used to normalize the dot product attention score to convert it into a gene-guided image feature CoAttn.

[0059] In one embodiment of the present invention, the predictive classification network model for pCR in breast cancer patients is a Transformer classifier.

[0060] In this embodiment, the image-gene interpretability-based deep learning prediction and classification method for pCR in breast cancer patients provided by the present invention divides the acquired three-dimensional MRI images of breast cancer patients into training and testing sets according to a preset ratio, and splits all three-dimensional MRI images into several two-dimensional slice images; wherein, the two-dimensional slice images corresponding to the same breast cancer patient are jointly associated with the gene text features of that breast cancer patient; the gene text features of each breast cancer patient are classified into N different sets with similar biological functions. Further, as... Figure 2 As shown, each two-dimensional slice image corresponding to the training set is divided into several patches. Using a fully connected neural network, each patch and its corresponding gene text features are converted into corresponding high-dimensional embeddings. The high-dimensional embedding converted for each patch is denoted as H. M represents the total number of patches in each 2D slice image; the high-dimensional embedding of the gene text feature transformation corresponding to each patch is denoted as G. Furthermore, H is compared with the trainable parameters W respectively. k and W v Multiplying them yields eigenvectors K and V, respectively. G is then multiplied by the trainable parameters W. Q Multiplying these features yields the feature vector Q. Further, based on feature vectors K, V, and Q, an attention mechanism is used to model the interpretable interaction between images and genes to obtain gene-guided image features CoAttn. Further, the gene-guided image features CoAttn are fused with feature vector Q to obtain the feature set F. Further, a predictive classification network model for pCR in breast cancer patients is constructed; this predictive classification network model is a Transformer classifier. Further, a loss function for this Transformer classifier is constructed, the feature set F is input into the Transformer classifier, and the Transformer classifier is trained based on the loss function. Further, several two-dimensional slice images corresponding to the test set are input into the trained Transformer classifier, and multiple classification results for whether the breast cancer patients in the test set belong to pCR are output. A maximum voting strategy is used to decide among these multiple classification results to obtain the final classification result for whether the breast cancer patients in the test set belong to pCR.

[0061] In one embodiment of the present invention, the expression for the loss function of the prediction classification network model is:

[0062] L=-[y·logp+(1-y)·log(1-p)](2)

[0063] In equation (2), y represents the true label, which takes the value of 0 or 1; p represents the probability that the model predicts the positive class. When y = 1, the loss function only focuses on logp, which is the probability that the model predicts the positive class; when y = 0, the loss function only focuses on log(1-p), which is the probability that the model predicts the negative class.

[0064] In one embodiment of the present invention, before step S1, the method further includes: supplementing the missing gene features in the gene text features of each breast cancer patient using the feature average method.

[0065] In one embodiment of the present invention, the maximum voting strategy follows the following formula:

[0066]

[0067] In equation (3), I(·) is an indicator function that takes the value of 1 when the internal calculation result is true, and 0 otherwise. The meaning of this formula is that for each possible category y, the total number of predictions of the category by N classifiers is calculated, and the category with the most total predictions is finally selected as the prediction output of the ensemble model.

[0068] In this embodiment, the ensemble model is a constructed predictive classification network model for pCR in breast cancer patients.

[0069] The technical solution of the present invention will be illustrated below with a specific embodiment. The specific implementation steps of the method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning of the present invention are as follows:

[0070] (1) Use the characteristic mean method to supplement the missing gene features in the gene text features of each breast cancer patient.

[0071] (2) Obtain the dataset. The dataset consists of three-dimensional MRI images of breast cancer patients.

[0072] (3) Divide the three-dimensional MRI images into training and test sets according to an 8:2 ratio.

[0073] (4) All three-dimensional MRI images corresponding to the dataset are split into several two-dimensional slice images; the size of each two-dimensional MRI slice image is 256×256. The two-dimensional slice images corresponding to the same breast cancer patient are collectively associated with the genetic text features of that breast cancer patient.

[0074] To obtain more expressive embedded feature representations, similar to word embeddings in NLP, this invention classifies genes into N distinct sets with similar biological functions. That is, the genetic text features of each breast cancer patient are classified into N distinct sets with similar biological functions.

[0075] In this step, the original MRI data is typically presented as a three-dimensional structure. Given the limited sample size of current datasets and the large number of parameters in the architecture of three-dimensional deep learning models, directly applying a three-dimensional model can easily lead to overfitting. Therefore, when processing a specific case, this invention adopts a strategy of splitting its three-dimensional MRI images into multiple two-dimensional data samples. These two-dimensional samples derived from the same case are collectively associated with and inherit the genetic text features of that case, thereby obtaining a completely new dataset. This approach not only effectively reduces model complexity and the risk of overfitting but also increases the amount of data for model training while maintaining data consistency, thereby improving the model's generalization ability and accuracy.

[0076] (5) Divide each two-dimensional slice image corresponding to the training set into several patches.

[0077] In this step, to capture image features more precisely and improve the training efficiency of the model, each two-dimensional slice image corresponding to the training set needs to be further divided into multiple small region blocks, i.e., patches. The specific size of each patch is 16×16, so a slice of a sample can be subdivided into M=256 patches.

[0078] (6) Using a fully connected neural network, each patch and its corresponding gene text features are converted into corresponding high-dimensional embeddings; whereby the high-dimensional embedding converted for each patch is denoted as H. M represents the total number of patches in each 2D slice image; the high-dimensional embedding of the gene text feature transformation corresponding to each patch is denoted as G.

[0079] The segmentation and feature transformation process in steps (5) and (6) not only helps the model to better capture subtle local changes, but also enhances the model's ability to identify pathological features by combining gene text features, thereby improving the accuracy and reliability of diagnosis.

[0080] (7) H is compared with the trainable parameter W respectively. k and W v Multiplying them yields eigenvectors K and V; the expression for eigenvector K is: The expression for the eigenvector V is:

[0081]

[0082] (8) Similarly, G is compared with the trainable parameters W Q Multiplying them yields the eigenvector Q; the expression for the eigenvector Q is:

[0083] (9) such as Figure 2 As shown in the gene-guided coattention part, after the gene features and image features are re-represented by the model, feature vectors K, V and Q are generated, and a more complex aggregation strategy is developed: feature vectors K, V and Q are used to model the interpretable interaction between images and genes through an attention mechanism to obtain gene-guided image features CoAttn.

[0084] The formula for modeling the interpretable interaction between images and genes using attention mechanisms is as follows:

[0085]

[0086] In equation (1), d k Let K be the dimension number of the feature vector K; The calculation result is represented as a dot product attention score; the Softmax function is used to normalize the dot product attention score to convert it into a gene-guided image feature CoAttn.

[0087] This sophisticated attention mechanism not only allows for a better understanding of the complex relationships between imaging and genetic data, but also improves the accuracy of models in diagnosis and prediction.

[0088] (10) The gene-guided image feature CoAttn is fused with the feature vector Q to obtain the feature set F. Specifically, the gene-guided image feature CoAttn is fused with the feature vector Q so that they can jointly constitute a richer feature set F, as shown in the formula:

[0089] F = Concat(CoAttn, Q)

[0090] =Concat([ca1,ca2,...,ca n ],[q1,q1,...,q m ])

[0091] =[ca1,ca2,...,ca n ,q1,q1,...,q m ]

[0092] (11) Construct a predictive classification network model for pCR in breast cancer patients. The predictive classification network model for pCR in breast cancer patients is a Transformer classifier.

[0093] In this step, specifically, the Transformer classifier is used to classify whether breast cancer patients belong to pCR. The Transformer classifier is chosen as a binary classifier because its self-attention mechanism is very suitable for processing sequence data and can effectively capture long-distance dependencies in the data.

[0094] The Transformer classifier consists of four Transformer layers and one fully connected layer. Each Transformer layer consists of four parts: a multi-head self-attention layer, a Layer Normalization layer, a feedforward fully connected layer, and a residual connection. The fully connected layer is used to implement the Transformer layer as a classifier, and its end has two neurons to output the probabilities of two types of cases (pathological complete response and non-pathological complete response).

[0095] (12) Construct the loss function of the Transformer classifier.

[0096] This Transformer classifier uses cross-entropy loss as its loss function, and the expression for this loss function is as follows:

[0097] L=-[y·logp+(1-y)·log(1-p)](2)

[0098] In equation (2), y represents the true label, which takes the value of 0 or 1; p represents the probability that the model predicts the positive class. When y = 1, the loss function only focuses on logp, which is the probability that the model predicts the positive class; when y = 0, the loss function only focuses on log(1-p), which is the probability that the model predicts the negative class.

[0099] (13) Input the feature set F into the Transformer classifier and train the Transformer classifier based on the cross-entropy loss function.

[0100] In this step, during the training of the Transformer classifier, 16 samples are received as input each time. After forward propagation, the prediction result is obtained, and the loss is calculated by comparing it with the labeled data. Then, backpropagation is performed, and the Adam algorithm is used to optimize the loss value. The training period is set to 200, and the initial learning rate of the model is 0.001.

[0101] (14) Input several two-dimensional slice images corresponding to the test set into the trained Transformer classifier and output multiple classification results of whether the breast cancer patients corresponding to the test set belong to pCR.

[0102] (15) The maximum voting strategy is used to make decisions on the above multiple classification results to obtain the final classification result of whether the breast cancer patients corresponding to the test set belong to pCR.

[0103] In this step, when the Transformer classifier infers for unseen cases, its raw data is 3D MRI. This invention first breaks down the 3D MRI into multiple 2D MRI slices. The gene features associated with each slice and the case are input into the model framework for calculation. Thus, each case will obtain multiple classification results. For these classification results, this invention uses a maximum voting strategy to make decisions and obtain the final classification result. The maximum voting strategy follows the following formula:

[0104]

[0105] In equation (3), I(·) is an indicator function that takes the value 1 when the internal calculation result is true, and 0 otherwise. This formula means that for each possible category y, the total number of predictions for that category by N classifiers is calculated, and the category with the highest total number of predictions is finally selected as the prediction output of the ensemble model. This ensemble model is a predictive classification network model for pCR in breast cancer patients.

[0106] This approach is equivalent to building multiple classifiers to predict the same case, generating their own predictions. These predictions will cover different error and correct cases. The predictions are then aggregated, and a maximum voting principle is used to make the decision. This helps reduce the variance of the model and improve its generalization ability, thereby improving the accuracy and robustness of the overall model.

[0107] In summary, this invention provides a method for predicting and classifying pCR (progressive cervical recurrence) in breast cancer patients based on image-gene interpretability deep learning. This method not only utilizes the patient's genetic text features to guide feature selection and cross-modal fusion of breast cancer MRI images, improving the model's predictive performance, but also enhances the interpretability of the model's decisions, providing clinicians with a more intuitive and reliable auxiliary diagnostic tool. Furthermore, to address the lack of 3D MRI data, this invention proposes splitting the data into 2D slices for training, expanding the dataset size. Moreover, when classifying multiple 2D slices from the same patient, a maximum voting decision method is used to determine whether the breast cancer patient belongs to pCR, which helps reduce the model's variance and improve generalization ability, thereby increasing the overall model's accuracy and robustness.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning, characterized in that, include: Step S1: Obtain the dataset; wherein the dataset consists of three-dimensional MRI images of breast cancer patients; Step S2: Divide the three-dimensional MRI images into a training set and a test set according to a preset ratio; Step S3: Split all the 3D MRI images corresponding to the dataset into several 2D slice images; among them, the 2D slice images corresponding to the same breast cancer patient are jointly associated with the genetic text features of that breast cancer patient; the genetic text features of each breast cancer patient are classified into N different sets with similar biological functions; Step S4: Divide each two-dimensional slice image corresponding to the training set into several patches; Step S5: Using a fully connected neural network, convert each patch and its corresponding gene text features into a high-dimensional embedding; the high-dimensional embedding for each patch is denoted as H. M represents the total number of patches in each 2D slice image; the high-dimensional embedding of the gene text feature transformation corresponding to each patch is denoted as G. Step S6: Compare H with the trainable parameters W respectively. k and W v Multiplying them yields eigenvectors K and V; the expression for eigenvector K is: The expression for the eigenvector V is: Step S7: Combine G with trainable parameters W Q Multiplying them yields the eigenvector Q; the expression for the eigenvector Q is: Step S8: Based on the feature vectors K, V, and Q, an attention mechanism is used to model the interpretable interaction between the image and the gene to obtain the gene-guided image feature CoAttn; Step S9: Fuse the gene-guided image feature CoAttn with the feature vector Q to obtain the feature set F; Step S10: Construct a predictive classification network model for pCR in breast cancer patients; Step S11: Construct the loss function of the prediction classification network model; Step S12: Input the feature set F into the prediction classification network model, and train the prediction classification network model based on the loss function; Step S13: Input several two-dimensional slice images corresponding to the test set into the trained prediction classification network model, and output multiple classification results of whether the breast cancer patients corresponding to the test set belong to pCR; Step S14: Use the maximum voting strategy to make a decision on the multiple classification results to obtain the final classification result of whether the breast cancer patients corresponding to the test set belong to pCR.

2. The method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning according to claim 1, characterized in that, The formula for modeling the interpretable interaction between images and genes using attention mechanisms is as follows: In equation (1), d k Let K be the dimension number of the feature vector K; The calculation result is represented as a dot product attention score; the Softmax function is used to normalize the dot product attention score to convert it into a gene-guided image feature CoAttn.

3. The method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning according to claim 1, characterized in that, The predictive classification network model for pCR in breast cancer patients is a Transformer classifier.

4. The method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning according to claim 1, characterized in that, The expression for the loss function of the prediction classification network model is: L=-[y·logp+(1-y)·log(1-p)](2) In equation (2), y represents the true label, which takes the value of 0 or 1; p represents the probability that the model predicts the positive class. When y = 1, the loss function only focuses on logp, which is the probability that the model predicts the positive class; when y = 0, the loss function only focuses on log(1-p), which is the probability that the model predicts the negative class.

5. The method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning according to claim 1, characterized in that, Before step S1, the method further includes: using the feature average method to supplement the missing gene features in the gene text features of each breast cancer patient.

6. The method for predicting and classifying pCR in breast cancer patients based on image-gene interpretability deep learning according to any one of claims 1 to 5, characterized in that, The formula expression followed by the maximum voting strategy is as follows: In equation (3), I(·) is an indicator function that takes the value of 1 when the internal calculation result is true, and 0 otherwise. The meaning of this formula is that for each possible category y, the total number of predictions of the category by N classifiers is calculated, and the category with the most total predictions is finally selected as the prediction output of the ensemble model.

Images