Knowledge embedding-based HER2 + breast cancer far-end metastasis interpretable prediction method

Abstract

The invention belongs to the field of medical auxiliary diagnosis, and particularly relates to an interpretable prediction method based on deep learning and HER2 + breast cancer pathological image far-end metastasis, in order to predict HER2 + breast cancer far-end metastasis, in a prediction model training stage, concept embedding and NMF optimization are carried out on a model, in an actual prediction stage, firstly, a pathological image is preprocessed, and then the HER2 + breast cancer far-end metastasis is predicted. The method comprises the following steps: firstly, extracting features, predicting a transfer condition based on the extracted features, and finally, giving explanation of a model prediction result in combination with a predicted result value and the operation features of a convolutional neural network in some embodiments. According to the method, a satisfactory prediction effect is realized by combining expert knowledge of doctors and learning of basic features. The working efficiency of doctors is improved; and on the other hand, doctors can be assisted to give different clinical treatment schemes to different patients, so that the lifetime and the life quality of the patients are improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical auxiliary diagnosis, and particularly relates to an interpretable prediction method for HER2+ breast cancer pathological image distant metastasis based on deep learning. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women. In 2020, breast cancer became the most frequently occurring cancer type in the world, with over 2.26 million newly reported cases and nearly 685,000 deaths related to breast cancer. Breast cancer is the leading cause of cancer-related deaths in women and ranks fifth among the total number of cancer-related deaths. As is well known, breast cancer is one of the most prevalent malignant tumors affecting women globally, posing a significant threat to their health and lives. The condition of HER2+ breast cancer patients develops relatively rapidly, and the probability of recurrence or metastasis after treatment is relatively high. HER2+ breast cancer is particularly aggressive, prone to recurrence and metastasis, and usually has a poor prognosis. Accurately predicting the recurrence risk helps to formulate monitoring plans and provide personalized adjuvant therapies. Unfortunately, the current effective models for identifying recurrence risks are limited and not interpretable. Promoting the early detection of breast cancer is very important for medical intervention. However, minimizing overfitting while achieving high accuracy remains a major challenge for prediction solutions, and deep learning methods, especially convolutional neural networks and their hybrid methods, are the most advanced methods currently in use. In addition, since the clinical diagnosis of distant metastasis of breast cancer is a complex problem, prediction has a certain degree of difficulty, routineity, and repeatability. Moreover, the subjectivity of each pathologist may affect the results, thus complicating the problem further. Summary of the Invention

[0003] Aiming at the deficiencies of the prior art, the present invention particularly innovatively proposes a hybrid method (KINN model) based on a convolutional neural network to predict whether HER2+ breast cancer will develop distant metastasis within 5 years. The method includes the following steps:

[0004] S1 Construct a prediction model based on a convolutional neural network for predicting whether HER2+ breast cancer will develop distant metastasis within 5 years. The input of the prediction model is the processed clinical breast cancer pathological image data information;

[0005] The prediction model includes multiple layers of feature extractors and mapping heads. The feature extractors are used to extract new feature representations of the input image data, and the mapping heads are used to output the metastasis probability prediction results according to the feature representations;

[0006] S2, train the prediction model. During this period, read out the output feature matrices of multiple layers inside the feature extractors. On the one hand, perform NMF matrix decomposition on these feature matrices to obtain the basis features and coefficient matrices, and then calculate the corresponding basis feature loss functions;

[0007] On the other hand, the pathological images in the training dataset that are judged by doctors to have very obvious features are defined as knowledge samples, and other samples are defined as general samples. Then, using similarity calculation, the knowledge loss function between the knowledge samples and the general samples is obtained;

[0008] Finally, combining the base feature loss function, the knowledge loss function, and the cross-entropy loss function between the prediction result and the training sample label, the parameters of the prediction model are optimized;

[0009] S3 is performed using the prediction model trained to convergence.

[0010] Furthermore, the corresponding base feature loss function is expressed as:

[0011]

[0012] is the base feature representation of the pathological image data p Gi at layer L, there is:

[0013]

[0014] where represents the feature matrix obtained from the pathological image data p Gi at layer L, is the corresponding coefficient matrix, and θ represents the model parameters.

[0015] Furthermore, the feature matrix of the general sample in the model is represented by and the feature matrix of the knowledge sample in the model is represented by Then, their cosine similarity can be defined as:

[0016]

[0017] The goal of knowledge embedding is to minimize the loss function L knowlege (θ) = 1 - a.

[0018] Furthermore, by combining the base feature loss function, the knowledge loss function, and the cross-entropy loss function between the prediction result and the training sample label, the total loss function L is obtained, and the parameters of the model are optimized using this. The optimization objective is expressed as follows:

[0019]

[0020] In the formula represents the prediction output of the model for a certain pathological image, and y i represents the actual label of this pathological image, and L cls (·) represents the calculation of cross-entropy error.

[0021] Further, it further includes the result explanation of step S4: multiplying the network output prediction result by the coefficient value n∈(0,1) to be explained to obtain the value to be explained and through the operation characteristics of the convolutional neural network, obtaining the important features or feature channels related to at the layer before the mapping head, and so on, until it is deduced to the input image to obtain the region related to the prediction result on the input image.

[0022] Further, let represent the prediction value of the model's L layer for a certain pathological image, then the interpretation region of on the input image can be expressed as:

[0023]

[0024] Then in step S4, the implementation process of the result explanation is:

[0025] At layer L: i∈N + ; taking these probability values greater than or equal to , that is:

[0026]

[0027] At layer L-1: i * ∈[0,t], and the E c operation is to take the eigenvalue that satisfies equation (2), and the F c operation is to obtain the spatial coordinates of the eigenvalue.

[0028] At layer L-2: that is, obtaining the coordinate point through the C m operation of the feature channel matrix at layer L-2. According to the operation characteristics of this layer, if it is a maxpooling operation, it is necessary to obtain the maximum value of this feature matrix,

[0029]

[0030] and so on, until from the layer before the model input layer to the input layer, through the S m operation, the important region of the prediction result at layer L1 is obtained as:

[0031]

[0032] represents the coordinate position of the eigenvalue on the feature map located in the second layer, represents the S m operation of the first channel

[0033] Finally, the image region corresponding to the decision value on the input image can be obtained.

[0034]

[0035] The principle and beneficial effects of the present invention are as follows: The prediction model in the present invention is for predicting the distant metastasis of HER2+ breast cancer. During the model training stage, concept embedding and NMF optimization are performed on the model. In the actual prediction stage, first, the pathological image is preprocessed, then feature extraction is carried out, and the metastasis situation is predicted based on the extracted features. In some embodiments, finally, an explanation of the model prediction result is given by combining the predicted result value and the operation characteristics of the convolutional neural network.

[0036] The present invention proposes a distant metastasis prediction model for HER2+ breast cancer. The model realizes a satisfactory prediction effect by combining the expert knowledge of doctors and the learning of base features. In further embodiments, in order to make the prediction result of the model more credible, and further aiming at the problem of untrustworthiness and unexplainability of the medical auxiliary diagnosis system, a solution idea is proposed, so that the decision basis of the model can be obtained and visualized while the model predicts the disease, enabling doctors to clearly and directly evaluate the credibility of the model prediction result. This method further promotes the integration of computer technology and medical data. On the one hand, it improves the work efficiency of doctors; on the other hand, it can assist doctors in giving different clinical treatment plans to different patients, thereby improving the survival period and quality of life of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the improved KINN model in the embodiment of the present invention.

[0038] Figure 2 It is an effect diagram of the explanation of the model prediction result in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0040] The present embodiment proposes an interpretable prediction model for distant metastasis of HER2+ breast cancer based on deep learning. The model uses ResNet50 as the backbone network, that is, the feature extractor, and forms a KINN model by adding a knowledge embedding module, a base feature representation module, and a model interpreter to this network.

[0041] The prediction model is as follows:

[0042] In this example, the input of the prediction model is the processed clinical breast cancer pathological image data information. First, the original image needs to be stained with H&E, then the ROI regions of the H&E original image data are labeled, and then it is digitized into WSI. The WSIs images are cut into image patches of a unified specification. Each image patch is uniformly processed by color normalization using the Vahadane method. M patches are selected to form an instance for input into the model, and the patches within the ROI region are preferentially selected.

[0043] The prediction model includes a feature extractor and a mapping head. In this example, a pre-trained Resnet50 model is used as the feature extractor to extract useful features from the pathological images. Through the feature extractor, each image patch is converted into a feature matrix with a 224*224-dimensional feature representation. For an entire WSI (Whole Slide Image) image, M 224*224 feature matrices can be obtained. Finally, the mapping head (in this example Figure 1 the simplest output layer is used as an example) outputs the metastasis prediction probability based on the feature matrix.

[0044] In this example, a dataset labeled by pathological experts is used to train the prediction model, and expert knowledge is embedded in the learning process of the model. According to the labels of the samples by pathological experts, the samples with obvious metastasis and non-metastasis features are marked as knowledge samples. In the model training stage, on the one hand, the base features are optimized to make the model pay more attention to the essential features of metastasis, and on the other hand, the knowledge samples are used to guide the model to learn the metastasis and non-metastasis features. Figure 1 schematically shows the above-mentioned base feature optimization and knowledge guidance process, which is called the KINN model in this example.

[0045] As Figure 1 shown, in the process of extracting the final feature vector, each hidden layer of a feature extractor with a total of L-1 layers (in the figure, the first layer is the input layer, and the Lth layer is the output layer, that is, the mapping head) will output a feature matrix (feature map). The output of the last layer is the final feature matrix. In this example, NMF is used to decompose the feature matrix into a base matrix and a coefficient matrix. The information represented by the base matrix is defined as the base feature, and this method makes the model pay more attention to the essential features of metastasis. Let F L represent the feature map obtained at the Lth layer, then the NMF matrix decomposition can be expressed as:

[0046] F L ≈(W L *H L ), with F L >0, WL > 0, H L > 0; (1)

[0047] Among them, W L is the basis matrix, and H L is the coefficient matrix, and they are updated according to the following rules:

[0048] For each element of H L , there is:

[0049]

[0050] For each element of W L , there is:

[0051]

[0052] L obj represents the confidence loss.

[0053] Therefore, given an image p Gi , its feature map is represented by , and according to formula (1), can be rewritten as:

[0054]

[0055] Therefore, the loss function for NMF can be expressed as:

[0056]

[0057] is the basic feature representation of the pathological image, θ represents the model parameters, and the smaller L NMF (θ) is, the more similar the basic features of p Gj and p Gi are.

[0058] Since algebraic equations can represent formal knowledge as scientific knowledge and can represent expert knowledge more intuitively. Therefore, in this example, the process of knowledge embedding is illustrated in mathematical form.

[0059] First, in the dataset used for training, the pathological images judged by doctors to have very obvious features are defined as knowledge samples, and other samples are defined as general samples. The feature map of general samples in the model is represented by , and the feature map of knowledge samples in the model is represented by , then their cosine similarity can be defined as:

[0060]

[0061] The goal of knowledge embedding is to minimize the loss function L knowlege :

[0062] min L knowlege (θ) = 1 - a.

[0063] Feed this L knowlege back to the model through backpropagation. The larger a is, the greater the similarity between the two Tensors, so 1 - a is smaller, and the smaller the adjustment amplitude of the model parameters. The smaller a is, the smaller the similarity between the two tensors, so 1 - a is larger, and the greater the adjustment amplitude of the model parameters. Theoretically, except for the first and last layers, this concept embedding method can be applied to any layer of the model to optimize the model. The proposed loss enables the model to learn to distinguish features under the guidance of expert knowledge rather than learning from a pile of messy data.

[0064] Finally, in this example, the total error L between the transfer prediction output of the prediction model and the instance label is calculated through concept embedding, NMF, and cross-entropy error calculation, and this error is backpropagated to the model to optimize the model parameters. The calculation of the total error L is as follows:

[0065]

[0066] where represents the prediction output of the model for a certain pathological image, y i represents the actual label L of this pathological image cls (·) represents cross-entropy error calculation. In this example, the Adam optimizer is used for parameter optimization.

[0067] In the experiment, the parameters of the concept embedding and NMF modules were randomly initialized, and then the parameters corresponding to the architecture layers used in the pre-trained model were loaded. During training, the batchsize was set to 8. In the knowledge embedding stage, first, the labels of the training samples were read in, and then the knowledge samples corresponding to the labels were loaded according to the label types of the training samples.

[0068] Table 1 shows the classification results of the KINN model in this example using image labels on the clinical dataset. It can be seen that the model has achieved good performance on all datasets.

[0069] Table 1 Evaluation of the prediction results of the KINN model on the clinical dataset and the public dataset

[0070]

[0071] In this example, a result interpreter is also introduced for the prediction model. The test image passes through the prediction model to obtain the prediction probabilities of metastasis and non-metastasis, corresponding to the results of metastasis and non-metastasis respectively. Multiply the prediction result (the probability value of non-metastasis or metastasis) of the network at the output layer L by the coefficient value n ∈ (0, 1) that needs to be explained to obtain the value to be explained. And through the operation characteristics of the convolutional neural network, obtain the relevant important features or feature channels at the L-1 layer and so on, until it is deduced to the input image, and obtain the region related to the prediction result (non-metastasis or metastasis) on the input image. Doctors can directly evaluate the credibility of the model prediction result and the decision-making basis of the model through this region.

[0072] Relative to the forward propagation, this interpreter mainly adopts the idea of backward deduction, and its process is basically as Figure 1 shown in the "Result Interpretation Stage" part of. Next, this process will be described in detail. In a reverse inference process, the regularization operation and the residual connection operation will be ignored because they do not affect the inference result.

[0073] Let represent the prediction value of the model for a certain pathological image, then the interpretation region of on the input image can be expressed as:

[0074]

[0075] Its implementation process is as follows:

[0076] At layer L: i ∈ N + ; Take these probability values greater than or equal to , that is:

[0077]

[0078] At layer L-1: i * ∈ [0, t], E v operation is to take the eigenvalue that satisfies equation (2), F c is the operation to obtain the spatial coordinates of the eigenvalue.

[0079] At layer L-2: That is, obtain the coordinate point m through the C feature channel matrix at layer L-2. According to the operation characteristics of this layer, if it is a maxpooling operation, it is necessary to obtain the maximum value of this feature matrix,

[0080]

[0081] And so on, until from layer L2 to layer L1, through operation S m The important area of the prediction result at layer L1 obtained by the operation is:

[0082]

[0083] Indicates the coordinate position of the eigenvalue on the feature map located in the second layer, Indicates the S of the first channel m Operation.

[0084] Finally, the image area corresponding to the decision value on the input image can be obtained,

[0085]

[0086] The explanatory effect of the prediction result on the CUCH dataset is exemplary as Figure 2 shown. In H&E histopathological images, HER2+ breast cancer distal metastasis usually presents as high histological grade, tumor cells showing solid sheets with a small amount of glandular structure, and map-like necrosis and significant lymphocyte infiltration. The purpose of the KINN method is to utilize the power of deep learning to discover features that are imperceptible to the human eye and use these features to predict disease metastasis. Therefore, patch-based experiments were arranged to demonstrate the feature recognition and discrimination ability of KINN.

[0087] For the above-described embodiments, although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that: various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and purposes of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An interpretable prediction method for HER2+ breast cancer distant metastasis based on knowledge embedding, characterized in that, Including the following steps: S1. Construct a prediction model based on a convolutional neural network for predicting whether HER2+ breast cancer will develop distant metastases within 5 years. The input of the prediction model is the processed clinical breast cancer pathological image data information; The prediction model includes a multi-layer feature extractor and a mapping head. The feature extractor is used to extract new feature representations of the input image data, and the mapping head is used to output the metastasis probability prediction result according to the feature representation; S2. Train the prediction model. During the training, read out the output feature matrices of multiple layers in the feature extractor. On the one hand, perform NMF matrix decomposition on these feature matrices to obtain the base features and coefficient matrices, and then calculate the corresponding base feature loss function; On the other hand, define the pathological images in the training dataset that are judged by doctors to have very obvious features as knowledge samples, and other samples as general samples. Then, use similarity calculation to obtain the knowledge loss function between the knowledge samples and the general samples; Finally, combine the base feature loss function, the knowledge loss function, and the cross-entropy loss function between the prediction result and the training sample label to optimize the parameters of the prediction model; S3. Use the prediction model trained to convergence for...

2. The method according to claim 1, wherein The corresponding base feature loss function is expressed as: is the pathological image data p Gi In the basic feature representation of layer L, there is: wherein represents the pathological image data p Gi is the feature matrix obtained at the L layer, is the corresponding coefficient matrix, and θ represents the model parameters.

3. The method according to claim 2, characterized in that, The feature matrix of the general sample in the model is represented by and the feature matrix of the knowledge sample in the model is represented by Then their cosine similarity can be defined as: There is a knowledge loss function L knowlege (θ) = 1 - a 4. The method according to claim 3, wherein Combine the base feature loss function, the knowledge loss function, and the cross-entropy loss function between the prediction result and the training sample label to obtain the total loss function L, and use this to optimize the parameters of the model. The expression of the optimization objective is as follows: In the formula The predicted output of the expression model for a certain pathological image, y i Represents the actual label L of this pathological image cls (·) represents the calculation of cross-entropy error.

5. The method according to claim 1, wherein It also includes the result explanation of step S4: multiplying the network output prediction result by the coefficient value n ∈ (0, 1) to be explained to obtain the value to be explained and, based on the operation characteristics of the convolutional neural network, obtaining the important features or feature channels related to this, and so on, until it is deduced to the input image to obtain the region related to the prediction result on the input image.

6. The method according to claim 5, wherein Let represent the predicted value of the L-th layer of the model for a certain pathological image. Then, regarding the interpretation region on the input image can be expressed as: Then in step S4, the implementation process of the result explanation is: At layer L: Take those probability values greater than or equal to , namely: At layer L-1: E v The operation is to take the eigenvalue that satisfies equation (2). F c is the operation to obtain the spatial coordinates of the eigenvalue. At the L-2 layer: That is, through C m The operation obtains coordinate points For the feature channel matrix at the L-2 layer, according to the operation characteristics of this layer, if it is a maxpooling operation, the maximum value of this feature matrix needs to be obtained. And so on until from the layer before the input layer of the model to the input layer, through S m operation, the important area of the prediction result in layer L1 is: Indicates the coordinate position of the eigenvalue on the feature map located in the second layer, Indicates S of the first channel m operation Finally, the image region corresponding to the decision value on the input image can be obtained:

Images