A method for judging the reliability of a prediction result of a classification task deep learning model

Abstract

The application discloses a method for judging the reliability of a prediction result of a classification task deep learning model. For a prediction sample, the highest similarity training set sample is found through the last layer of a neural network information of a neural network model, and whether the prediction result of the model for the prediction sample is reliable is judged according to whether the highest similarity training set sample is located at a classification boundary. The application can solve the limitation that the deep learning model is difficult to be recognized and applied due to unknown reliability of a prediction result in a prediction process of a classification task deep learning model. The method for judging the reliability of the prediction result of the classification task deep learning model is accurate, credible, simple, easy to implement and easy to understand, and has important significance for improving the applicability of the classification task deep learning model.

Description

Technical Field

[0001] This invention belongs to the field of deep learning models, and specifically relates to a method for judging the reliability of prediction results of deep learning models for classification tasks. Background Technology

[0002] Deep learning models, as complex machine learning algorithms, can accurately learn the inherent patterns and hierarchical representations of sample data, achieving precise predictions. In recent years, deep learning models have made unprecedented progress in many areas, including data mining, healthcare, legal systems, public infrastructure, and chemical management. While the development of deep learning models has greatly improved prediction accuracy, their inherently complex prediction process makes their decision-making process difficult to understand. As deep learning models are involved in high-risk and resource-scarce AI decision-making, such as doctors rejecting predictions due to the lack of transparency in the model's internal operations and the resulting uncertainty about the reliability of the results, the reliability of model predictions is receiving increasing attention. Only by improving the reliability of model predictions can artificial intelligence truly serve human society.

[0003] The existing knowledge base for evaluating the reliability of model predictions relies on the model predicting unknown samples based on similar samples. Therefore, the range of training set samples is used to define the range within which the model can reliably predict. However, this approach was first proposed for linear models. Linear models have a relatively easy-to-understand prediction process but limited learning capacity and range. Therefore, similar training set samples for unknown samples can be obtained through input features. However, modern deep learning models can learn complex features from large amounts of sample data, significantly expanding their learning range. Furthermore, deep learning models consist of multi-layered neural networks, and their prediction process is no longer directly based on the model's input features. Therefore, existing methods are no longer suitable for judging the reliability of deep learning model predictions. Understanding deep learning models—comprising multi-layered neural networks where the features of the last layer directly serve the model's decision-making—can help understand the prediction results and their reliability. Current research lacks an understanding and mastery of the fundamental knowledge of deep learning models when applying them, leading to unclear prediction processes and poor prediction reliability. Therefore, a thorough understanding of deep learning model knowledge is essential to improving the reliability of deep learning model predictions. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide an accurate, reliable, simple, and easy-to-understand method for judging the reliability of prediction results of deep learning models for classification tasks.

[0005] Technical solution: The method for determining the reliability of prediction results of deep learning models for classification tasks, as described in this invention, includes the following steps:

[0006] (1) Based on a well-constructed deep learning model for a classification task, obtain the set m(nearest) of training set samples m′ with the highest similarity to each training set sample m;

[0007] (2) Calculate the local discontinuity coefficient (LDS) of the training set sample m in step (1). FT (m) is used to locate the classification boundary:

[0008]

[0009] Where, N neighbor A positive integer, representing the number of similar samples to be calculated; S(m,m′) represents the similarity between the features of the two samples; T(m,m′) represents the class difference between the two samples, 0 if the two samples are of the same class, and 1 if the classes are different.

[0010] (3) Input at least one unknown sample n into the model described in step (1), obtain the set n(nearest) of training set samples n′ with the highest similarity for each n, and match the LDS calculated in step (2) for each n′ in n(nearest). FT (n′);

[0011] (4) Calculate the mean value M of the local discontinuity coefficient of n′ mentioned in step (3). LDS (n):

[0012]

[0013] LDS FT (n′) greater than 0 indicates that the sample is close to the classification boundary, meaning that the features of that sample are difficult for the model to learn accurately; M LDS (n) greater than 0 indicates that the prediction result is unreliable; excluding unreliable prediction results can improve the credibility of the model's prediction results.

[0014] Preferably, the N neighbor As the value of N increases, the model performance will gradually improve, but at the same time, more predictions will be removed. Therefore, different values ​​of N should be calculated. neighbor The model performance corresponding to the value is selected, and the N corresponding to the optimal model performance with the largest prediction range is chosen. neighbor The value is used as an indicator to evaluate the reliability of the model's prediction results.

[0015] Preferably, the performance evaluation metric of the model is at least one of balanced accuracy (bACC), accuracy (ACC), F1 score (F1), recall, precision, and area under the curve (AUC).

[0016] Preferably, in step (1), all features of training set samples m in the last layer of the neural network of the model are extracted, the similarity between the features of the training set samples is calculated, and the training set sample m′ with the highest similarity to each training set sample m is selected and included in m(nearest).

[0017] Preferably, in step (3), the features of all unknown samples n in the last layer of the neural network of the model are extracted, the similarity between them and the features of all training set samples m is calculated, and the training set sample n′ with the highest similarity is selected and included in n(nearest).

[0018] Preferably, the similarity is measured using distance.

[0019] More preferably, the similarity metric is selected from any one of Euclidean distance, Manhattan distance, and cosine distance.

[0020] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention uses deep learning model features to calculate local discontinuity coefficients to locate the classification boundary of the classification model, which can determine the reliability of prediction results for unknown samples and improve the credibility of deep learning model prediction results for classification tasks. This is in contrast to existing technologies that calculate prediction result reliability (AD) based on sample features. FP This method can more effectively improve model performance by about 5%-10%, meaning that while retaining more prediction results, the model performs better. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the method of the present invention for determining the reliability of prediction results from a deep learning model for a classification task.

[0022] Figure 2 Example 1 applies the method for determining the reliability of deep learning model prediction results for classification tasks to a model for predicting the estrogen activity of compounds; wherein, Figure 2 A represents the application of AD. FT (N neighbor The graph showing the performance variation of the method's model. Figure 2 B is the application of AD. FP (S cutoff The graph shows the performance changes of the model using the method.

[0023] Figure 3 Example 2 applies the method for judging the reliability of deep learning model prediction results for classification tasks to models predicting persistence, bioaccumulation, and toxicity of compounds; wherein, Figure 3 A is the calculation of AD using cosine distance. FT (N neighbor The graph showing the performance variation of the method's model. Figure 3 B is the application of Euclidean distance to calculate AD. FT (Nneighbor The graph showing the performance variation of the method's model. Figure 3 C is the application of Manhattan distance to calculate AD. FT (N neighbor The graph showing the performance variation of the method's model. Figure 3 D represents the application of AD. FP (S cutoff The graph shows the performance changes of the model using the method. Detailed Implementation

[0024] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0025] Example 1

[0026] This embodiment describes a graph neural network model for predicting the estrogenic activity of compounds, specifically including the following steps:

[0027] (1) Based on the pre-constructed graph neural network model for predicting the estrogenic activity of compounds, the model obtains the classification boundary by fitting the features of the training set, extracts the features of the training set compounds in the last layer of the neural network, calculates the Euclidean distance between them, and obtains the training set compounds with the same number of similarities for each compound based on the Euclidean distance calculation results; (2) Calculate the local discontinuity coefficient based on the training set compounds with the highest similarity to locate the model classification boundary; (3) Input the test set compounds into the model, extract the representation of the test set compounds in the last layer of the graph neural network model, and calculate the Euclidean distance between them and the features of the training set compounds; (4) Calculate the M of the training set compounds with the highest similarity for each test set compound. LDS The value M is used to determine whether the prediction result for the unknown compound is reliable. LDS A value greater than zero indicates that the prediction result is unreliable.

[0028] This method analyzes a total of 770 data points in the test set, excluding unreliable prediction results to improve the credibility of the model's predictions.

[0029] To demonstrate the advantages of this method, it will be compared with the previously proposed method AD. FP Compare the implementation effects of AD FP The specific implementation process of the method is as follows: MACC fingerprints of compounds are obtained using the RDkit package in Python, and the Tanimoto similarity T between the MACC fingerprints of each test set compound and the training set compound is calculated, with different thresholds S set. cutoff Compounds in the test set are considered to be within the applicable range if at least one compound in the training set is greater than this threshold; threshold S cutoffThe threshold values ​​are set to 0.65, 0.70, 0.75, 0.8, 0.85, 0.9, and 0.95, representing different levels of stringency. For a predicted compound, if its similarity to Tanimoto is greater than the threshold S among more than one compound in the training set... cutoff If the model's prediction for the compound is reliable, then the model is considered reliable. If no compound in the training set has a similarity greater than S with its Tanimoto counterpart, then the model is considered reliable. cutoff If the prediction is unreliable, the model's prediction for that compound is considered unreliable. Unreliable predictions are then excluded.

[0030] like Figure 2 As shown, compared with this method AD FT Compared with the previous method AD FP The implementation effect shows that the application of AD FT This can more effectively improve model performance, meaning that while retaining more prediction results, the model performs better, such as when N neighbor When the value was 15, 627 predictions were retained, the model accuracy (ACC) reached 0.93, and the F1 score was 0.728. However, when AD was applied... FP When using the method, even if S cutoff =0.9, only 324 predictions are retained, the model accuracy ACC is only 0.88, and the F1 score is 0.72, therefore AD FT The method is significantly superior to AD FP The effectiveness of the method.

[0031] When different N are selected neighbor When N is a certain value, the corresponding model prediction effect is calculated, and it can be found that when N is a certain value... neighbor When the value is 15, the model has reached its optimal performance and retains a large number of prediction results, therefore N is selected. neighbor A value of 15 was used as the reliability evaluation index for the model prediction results. The analytical results for the unknown compound are as follows:

[0032] (1) Given a compound with CAS number 65277-42-1 and its SMILES formula CC(=O)N1CCN(c2ccc(OCC3COC(Cn4ccnc4)(c4ccc(Cl)cc4Cl)O3)cc2)CC1, input it into a pre-constructed graph neural network model to predict whether it has estrogenic activity. The prediction result is 0, meaning that the compound has no estrogenic activity. Then, evaluate the reliability of the prediction result using N. neighbor =15 Calculate the M of this compound LDS The value is 0, so the model's prediction for this compound is reliable.

[0033] (2) Given a compound with CAS number 789-02-6 and SMILES formula Clc1ccc(C(c2ccccc2Cl)C(Cl)(Cl)Cl)cc1, input it into a pre-constructed graph neural network model to predict whether it has estrogenic activity. The prediction result is 1, indicating that the compound has estrogenic activity. Then, evaluate the reliability of the prediction result by applying N... neighbor =15 Calculate the M of this compound LDS The value is greater than 0, so the model's prediction for this compound is unreliable.

[0034] Example 2

[0035] This embodiment describes a graph neural network model for predicting persistent, bioaccumulative, and toxic (PBT) compounds. Specifically, it includes the following steps: (1) Based on a pre-constructed graph neural network model for PBT compounds, obtain the same number of training set compounds with the highest similarity for each compound; (2) Calculate the local discontinuity coefficient based on the training set compounds with the highest similarity to locate the model's classification boundary; (3) Input test set compounds into the model and calculate their similarity to the features of the training set compounds; (4) Calculate the M-value of the training set compounds with the highest similarity for each test set compound. LDS The value M is used to determine whether the prediction result for the unknown compound is reliable. LDS A value greater than zero indicates that the prediction result is unreliable.

[0036] This method analyzes 240 data points in the test set, excluding unreliable predictions to improve the reliability of the model's predictions. For example... Figure 3 As shown, compared with this method AD FT Compared with the previous method AD FP The implementation results show that by applying cosine distance, Euclidean distance, and Manhattan distance to AD FT The method more effectively improves model performance; all three calculation methods retain more prediction results while achieving better model performance. For example, when N... neighbor When the value is 5, the AD calculated using Euclidean distance is... FT The method has already improved the model evaluation index to its optimal value, meaning it has eliminated all incorrect predictions while retaining 232 predictions; while applying AD... FP When using the method, even if S cutoff =0.9, only 81 predictions are retained, the model accuracy ACC is only 0.988, and the F1 score is 0.96, therefore AD FT The method is significantly superior to AD FP The effectiveness of the method.

[0037] Due to N neighbor When the value is 10, the model has reached its optimal performance and retains a large number of prediction results, therefore N is selected. neighbor A value of 10 was used as the reliability evaluation index for the model prediction results. The analytical results for the unknown compound are as follows:

[0038] (1) Given a compound with CAS number 2136-89-2 and SMILES formula Clc1ccccc1C(Cl)(Cl)Cl, input it into a pre-constructed model to predict whether it is a PBT compound. The prediction result is that the compound is a PBT compound. Then, evaluate the reliability of the prediction result using N. neighbor =10 Calculate the M of this compound LDS The result is greater than 0, so the model's prediction for this compound is unreliable.

[0039] (2) Given a compound with CAS number 52722-86-8 and SMILES formula CC1(C)CC(O)CC(C)(C)N1CCO, input it into a pre-built model to predict whether it is a PBT compound. The prediction result is 0, meaning the compound is not a PBT compound. Then, evaluate the reliability of the prediction result by applying N. neighbor =10 Calculate the M of this compound LDS The value is 0, so the model's prediction for this compound is reliable.

Claims

1. A method for judging the reliability of a prediction result of a deep learning model for a classification task, characterized in that, The method comprises the following steps: (1) based on a constructed classification task deep learning model, wherein the input of the model is compound structure data, and the output is a class label representing the property of the compound; obtaining a set m(nearest) of training set samples m' most similar to each training set sample m; (2) calculating the local discontinuity coefficient LDS of the training set sample m in step (1) FT (m) to position the classification boundary: where N neighbor is a positive integer, representing the number of samples included in the calculation; S(m, m') represents the similarity between the features of two samples; T(m, m') represents the class difference between two samples, which is 0 if the two samples are of the same class, and 1 if the classes are different; (3) input at least one unknown sample n into the model described in step (1), wherein the unknown sample n is compound structure data, obtain a set n(nearest) of the most similar training set sample n' to each n, and match each n' in n(nearest) with the LDS calculated in step (2) FT (n') calculated in step (2); (4) calculating the local discontinuity coefficient mean M of n' from step (3) LDS (n): LDS FT (n') greater than 0 means the sample is close to the classification boundary, which means the feature of the sample is difficult to be accurately learned by the model; M LDS (n) greater than 0 means the prediction result is unreliable; The model is a model for predicting the estrogen activity of a compound or a model for predicting the persistence, bioaccumulation and toxicity of a compound; the class label is 0 or 1, respectively representing that the compound does not have or has a specific property, and the specific property is the estrogen activity of the compound or the persistence, bioaccumulation and toxicity of the compound.

2. The method of claim 1, wherein the determining the reliability of the prediction result of the classification task deep learning model comprises: The N neighbor value increases, the model performance gradually increases, but at the same time, more prediction results are removed. The model performance corresponding to different N neighbor values should be calculated, and the N neighbor value corresponding to the optimal model performance and retaining more prediction results is selected as the model prediction result reliability evaluation index.

3. The method of claim 2, wherein the determining the reliability of the prediction result of the classification task deep learning model comprises: The evaluation index of the performance of the model is at least one of balanced accuracy bACC, accuracy ACC, F1 score F1, recall, precision, and area under the curve AUC value.

4. The method of claim 1, wherein the determining the reliability of the prediction result of the deep learning model for the classification task comprises: In step (1), the features of all training set samples m in the last layer of the neural network of the model are extracted, the similarity between the features of the training set samples is calculated, and the training set sample m' most similar to each training set sample m is selected to be included in m(nearest).

5. The method of claim 1, wherein the determining the reliability of the prediction result of the deep learning model for the classification task comprises: In step (3), the features of all unknown samples n in the last layer of the neural network of the model are extracted, the similarity between the features of all training set samples m and the unknown samples n is calculated, and the training set sample n' with the highest similarity is selected to be included in n(nearest).

6. The method of judging the reliability of the prediction result of the classification task deep learning model according to claim 4 or 5, characterized in that, The similarity is measured by distance.

7. The method of judging the reliability of the prediction result of the classification task deep learning model according to claim 4 or 5, characterized in that, The similarity is measured by any one of Euclidean distance, Manhattan distance and cosine distance.

Images