Method for verifying a trained model

Abstract

The invention relates to a method for detecting a deviated model among a plurality of different models trained using a supervised learning method, the method being performed by a computer system programmed with the trained model and comprising:-acquiring a test data set, -presenting the test data set to each of the trained models and generating an answer to the test data set for each trained model,-performing at least one homogeneity test based on the answers generated by at least two of the plurality of trained models, -when said homogeneity test fails, performing a predetermined action indicating that one of said at least two models has been detected as deviating with respect to the other trained model.

Description

Technical Field

[0001] The present invention generally relates to the validation of models trained using supervised learning methods (such as neural networks), and more specifically to methods for detecting models among a plurality of such models that compare favorably to other models. Background Art

[0002] Models trained using supervised learning methods (such as neural networks) have become increasingly valuable tools for solving classification problems such as image recognition, pattern recognition, or speech recognition, or even for performing predictions (e.g., through regression). Such neural networks can be used for biometric authentication. Accurate authentication can correctly authenticate a person; inaccurate authentication can result in both false positives (i.e., interpreting an impostor as the person being authenticated) and false negatives (i.e., incorrectly interpreting a person as an impostor).

[0003] Furthermore, with the increasing use of automation (e.g., self-driving vehicles and factory automation), it is necessary to interpret the operating environment, such as traffic events, road conditions, and factory floor conditions. Such models have become a central tool used in these applications of image processing.

[0004] The efficiency and security of such models is critical to the security of systems that rely on them for mission-critical purposes. Consider, as an example, that a neural network used to verify whether a given person should be allowed through some filter (e.g., a gate, security door, or border control) may be based on facial recognition, i.e., whether the person matches a specific person in a database or an image digitally recorded on an identification object. Manipulation of that neural network is one mechanism by which that security system may be attacked. For example, a neural network may be manipulated to identify an imposter as an authorized person.

[0005] Similarly, such models used in automation must also be efficient and secure. For example, one can imagine a terrorist or ransomware attack based on the penetration of a neural network used in a self-driving car. For example, imagine the consequences of a self-driving car after an attack on an image processing neural network caused it to fail to recognize a pedestrian at a crosswalk. Similar threats could occur for automated factory floors, transportation systems, aviation, etc.

[0006] The problem with models trained using supervised classification methods is that their correctness cannot be theoretically proven. Their correctness may only be guessed by using performance measures such as the ratio of correct answers.

[0007] However, such statistical measures do not prevent the possibility of misclassified data having a label that is different from the expected label for that particular data or data type. Such misclassifications may be the result of an attacker compromising the model in order to actively obtain such misclassifications and use it for their own benefit.

[0008] Alternatively, such misclassifications may be the undesired effect of model deficiencies, e.g. caused by a biased design or overtraining during the training phase.

[0009] In both cases, such misclassifications are threats to the efficiency and safety of such models, which should be detected and handled correctly.

[0010] From the foregoing, it is apparent that a need exists for an improved method for detecting models that have biased designs or are improperly trained using supervised learning methods and result in such misclassifications. Summary of the invention

[0011] To this end and according to a first aspect, the present invention therefore relates to a method for detecting a deviating model among a plurality of different models trained using a supervised learning method, the method being performed by a computer system programmed with the trained models and comprising:

[0012] - Get the test dataset,

[0013] - presenting the test data set to each of the trained models and generating an answer of each trained model for the test data set,

[0014] - performing at least one homogeneity test based on said answers generated by at least two models of said plurality of trained models,

[0015] - when the homogeneity test fails, performing a predetermined action indicating that one of the at least two models has been detected as deviating relative to the other trained models.

[0016] Such an approach enables determining that one of the models being tested has provided an answer to a test data set that is significantly different from the answers provided by the other models and taking appropriate action.

[0017] The model may be among a neural network model, a K-nearest neighbor (KNN) model, a support vector machine (SVM) model, a decision tree model, and a quadratic discriminant analysis (QDA) model.

[0018] The model may be based on several different learning methods or on the same learning method with different parameters and / or hyperparameters.

[0019] In an embodiment, the step of performing a homogeneity test based on the answers to the test data set generated by at least two models among the multiple trained models includes determining, for each model based on the generated answers to the test data set, a probability distribution law followed by its answers, and comparing the probability distribution laws determined for the at least two models.

[0020] It enables checking that the answers of the model to the test dataset follow the same distribution even if they are not identical, and that the behavior of the models being tested is equivalent.

[0021] The step of determining for each model the probability distribution law followed by its answers may comprise establishing said law based on said answers generated by said model for said test data set using a regression technique.

[0022] In an embodiment, the step of performing a homogeneity test based on the answers for the test data set generated by at least two models of the plurality of trained models comprises performing a direct comparison on the answers for the test data set.

[0023] The step of performing a direct comparison of the answers for the test data set may include performing a pairwise homogeneity test, a Cramer-von Mises test, or a Kolmogorov-Smirnov test.

[0024] Such tests enable comparison of the distributions of the responses of the models being tested, even if no probability distribution law can be determined for the responses of the models.

[0025] The step of performing a direct comparison of the answers generated by two models for the test data set may include determining a test statistic associated with the answers of the two models for the test data set and comparing the test statistic to a predetermined threshold.

[0026] The step of performing a direct comparison of the answers generated by the two models for the test data set may comprise determining a p-value for the test statistic and comparing the value of the p-value to a predetermined threshold.

[0027] Computing a test statistic or p-value enables quantifying the distance between the distributions of the two models' responses for a test data set and determining whether the two models' responses can be considered to follow the same distribution.

[0028] In an embodiment, the step of performing a predetermined action when the homogeneity test based on the answers generated by two models fails comprises declaring one of the two models impaired and discarding it.

[0029] It enables preventing any use of a model that has been corrupted during its training by an attacker.

[0030] In an embodiment, the step of performing a predetermined action when the homogeneity test based on the answers generated by two models fails comprises performing new training on one of the two models using a new training data set.

[0031] It enables to correct any deficiencies in the training applied to the model, such as any bias in the training database used to train the model or any overtraining.

[0032] According to a second aspect, the invention therefore also relates to a computer system programmed with a plurality of different models trained using a supervised learning method and comprising:

[0033] - a processor configured to acquire a test data set,

[0034] - at least one memory connected to the processor, storing the trained model and comprising instructions executable by the processor, the instructions comprising:

[0035] presenting the test data set to each of the trained models and generating an answer of each trained model for the test data set,

[0036] performing at least one homogeneity test based on the answers generated by at least two models of the plurality of trained models,

[0037] If the homogeneity test fails, performing a predetermined action indicating that one of the at least two models has been detected as deviating relative to the other trained models.

[0038] According to a third aspect, the invention relates to a computer program product directly loadable into the memory of at least one computer, comprising software code instructions for performing the steps of the method according to the first aspect of the invention when said product is run on said computer.

[0039] To the accomplishment of the foregoing and related ends, one or more embodiments comprise the features hereinafter fully described and particularly pointed out in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic diagram of a computer system according to the present invention;

[0041] Figure 2 is a schematic diagram of a method for detecting a deviated model among a plurality of different models according to an embodiment of the present invention;

[0042] Figure 3 is an example schematically illustrating the distribution of answers of three models for a test data set, which enables detection of abnormal behavior of one model, according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technology described herein provides a method for detecting abnormal behavior of a model trained using a supervised learning method, such as a statistical learning method or a machine learning method.

[0044] Such models may be, for example, neural networks (such as Reduced Boltzmann Machine (RBM)), K-Nearest Neighbors (KNN), Support Vector Machines (SVM), decision tree models, and quadratic discriminant analysis (QDA) models.

[0045] Such abnormal behavior is, for example, misclassifying a given input into an unexpected category to which the input does not belong, or predicting a value with a large error.

[0046] Such erroneous outputs can be the result of an attack during the training of the model. For example, an attacker may have added one or more erroneous samples to the training data set used to train the model in order to actively induce the observed misclassification when the model is used. It can also be the result of an inappropriate choice of parameters or hyperparameters of the model (such as the dropout value or the number of layers of the MLP model).

[0047] Such incorrect classifications may also be caused by deficiencies in the model, e.g. induced by an insufficient or biased training dataset or due to overtraining the model on its training dataset and making it impossible to correctly handle inputs that differ from the training dataset.

[0048] In order to detect such abnormal behavior and take appropriate actions to prevent it, the main idea of ​​the present invention is to obtain several trained models that should provide similar answers for a given input and compare their answers with a given test data set.

[0049] As an example, such models may be models trained using the same training data but based on different learning methods (such as neural network and decision tree models). Alternatively, they may be models based on the same learning method but with different parameters and / or hyperparameters, for example because they have been trained and provided by different entities, such as models freely available on the Internet and models provided by subcontractors.

[0050] The models to be compared can even be different versions of a single model at different stages of its training. In such a case, comparing these different versions can make it possible to check that the output of the model does not diverge when the model is further trained.

[0051] When such models have similar performance, their outputs provided as answers to a given test data set should be almost the same and should follow the same probability distribution law. Therefore, the second main idea of ​​the present invention is to determine which model among the compared models has abnormal behavior by comparing the probability distribution laws of the compared models, and identify the model whose output does not follow the same probability distribution law as other models as a "deviant model".

[0052] Figure 1 is a high-level architectural diagram illustrating a possible architecture of a computer system 100 that performs the steps of the method according to the invention. The computer system 100 comprises a processor 101, which may be a microprocessor, a graphics processing unit or any processing unit suitable for performing the steps of the method described below.

[0053] Processor 101 is connected to memory 102, for example, via a bus. Memory 102 may be a non-volatile memory (NVM), such as a flash memory or an erasable programmable read-only memory (EPROM). Memory 102 stores trained models to be compared, including their parameters and hyperparameters, such as weights associated with neurons that make up the neural network and are adjusted during training of the neural network.

[0054] The computer system 100 further comprises an input / output interface 103 for receiving inputs such as data related to the models to be compared and settings for the comparison to be performed between the models to be compared.

[0055] Figure 2 is a flow chart illustrating the steps of a method for detecting a deviating model among a plurality of different models.

[0056] During a first step S1, the computer system described above and used to compare the models to be compared is programmed with the trained models to be compared. As mentioned above, the trained models may have been previously trained outside the computer system 100 (e.g., by a supplier) and may be imported into the computer system via the input / output interface 103. Alternatively, the trained models may have been trained on the computer system 100 itself. During this step, the trained models are arranged in the computer system, in particular in its memory 102, so that the computer system can interrogate each model with an input and obtain the answer of each model as an output to the model. For example, in the case of a classification application, a vector of scores.

[0057] In a second step S2, the computer system acquires a test data set, which includes a plurality of data to be used as input to the model in order to test its behavior. Such a test data set may, for example, include thousands or millions of test samples. Such a test data set may be imported into the computer system via the input / output interface 103, or it may be generated by the computer system, for example, based on a larger data set.

[0058] In a third step S3, the computer system presents the test data set to each of the trained models to be compared, which generates an answer for each trained model to the test data set. When the data set has been appropriately selected, the model's answer to the data set characterizes the model's behavior and should reveal any abnormal behavior of the model, such as classifying a picture of a cat as a dog when the model is used to classify pictures of animals.

[0059] Unfortunately, in order to detect any abnormal behavior such as the abnormal behavior described above, it would be very tedious and time-consuming to have the model's answer for each sample of the data set to be checked by a human, especially when the test data set includes thousands of samples. Therefore, in a fourth step S4, the computer system performs at least one homogeneity test based on the answers generated by at least two models among the trained models tested at the third step S3. Such a homogeneity test does not identify abnormal behavior of the model based solely on the model's answers, but identifies abnormal behavior of the model by comparing the model's answers with the answers of at least another model for the same test data set.

[0060] In a first embodiment, such a homogeneity test includes determining, for each model subjected to the homogeneity test, the probability distribution law followed by its answers based on the answers generated by it for the test data set. Such a law can be, for example, a Gaussian law or a gamma distribution. Regression techniques can be used to determine such a law for the answers generated by each model for the test data set. Then, the computer system compares the probability distribution laws determined for the models. The computer system can first check whether the answers of all models follow the same type of distribution law, for example, all follow a gamma distribution. And then the computer system can compare the parameters (mean, variance, ...) of the laws followed by the answers of all models. If all the models tested are equivalent, then their answers to the same test data set should all follow the same probability distribution law. If a model has abnormal behavior that leads to inaccurate answers to some samples of the test data set, the answers of this particular model to the test data set will follow a probability distribution law different from the probability distribution laws in other models. In Figure 3An example is given above, which shows the distribution of the answers of the three models for the test data set. It can be seen that the distribution of the answers of the model on the left is obviously different from the distribution of the answers of the other two models, which appear to be the same.

[0061] In some cases, such as when the answers of the model do not follow the parametric law, it may be difficult to characterize the probability distribution law followed by the answers of the model. Or it may be considered preferable not to attempt to do so, for example in order to avoid errors due to regression methods. Therefore, in a second embodiment, the homogeneity test involves a direct comparison of the answers of the two models for the test data set. Such a direct comparison may, for example, include performing a paired homogeneity test, a Cramer-von Mises test, or a Kolmogorov-Smirnov test.

[0062] A figure of merit can be defined to measure the distance between the distributions of the two models' answers for a test dataset.

[0063] In a first embodiment, such a direct comparison of the responses generated by the two models for a test data set includes determining a test statistic associated with the responses, such as an effect size, such as a Pearson's correlation, a coefficient of determination, or an η squared value.

[0064] In a second embodiment, such direct comparison of answers generated by the two models for a test data set also includes determining a p-value for such test statistic.

[0065] In a third embodiment, such a distance between the distributions of the two models' answers for the test data set may be determined using a non-statistical method such as the Wasserstein distance method.

[0066] The value of the test statistic or p-value or any other distance measure can then be compared by the computer system to a predetermined threshold. When the test statistic or other distance is above such a predetermined threshold, or when the p-value is below such a predetermined threshold, the distance between the answers of the two models being compared is considered too large, and the answers of the two models are considered not to follow the same distribution. In such a case, the homogeneity test is considered to have failed.

[0067] In a fifth step S5, when the homogeneity test fails, the computer system performs a predetermined action indicating that one of the at least two models subject to the failed test has been detected as deviating relative to the other trained models.

[0068] In a first embodiment, the method according to the invention is used to test the correctness of a model, which may have been poisoned by an attacker during its training. In such a case, failing the test indicates that one of the models gave nonsensical answers for some of the samples of the test data set, which may be the result of poisoning the training data set used to train the model. In such a case, such an action may be, for example, to declare one of the two models corrupted and discard it. The computer system may also sound an alarm.

[0069] In a second embodiment, the method according to the invention is used to determine whether the parameters of the model (such as the drop probability value, the number of layers, the number of neurons per layer) are suitable for the problem to be solved. In such a case, failing the test may indicate that one or more parameters are not suitable. In such a case, the predetermined action may include updating at least one parameter of one of the two models, and performing new training on the updated model, possibly using the same training data set as before.

[0070] In a third embodiment, the method according to the present invention is used to determine whether the training of the model being tested is sufficient. In such a case, failing the test may indicate that the model has been under-trained or over-trained, which results in inaccurate results for at least part of the test data set. In such a case, the predetermined action may include performing new training on one of the two models using a new training data set.

[0071] The homogeneity test discussed above only makes it possible to determine whether the answer distributions of two models are the same and to conclude that one of the two models being compared exhibits abnormal behavior, but it does not indicate which of the two models has the abnormal behavior. In order to identify models that behave improperly, multiple tests can be performed on various model pairs, and combining the results of these multiple tests makes it possible to determine which model has an answer distribution that deviates from the distribution shared by the other models. The multiple quality factors calculated for these model pairs can even be used to rate the models and define a preferred model, which can be one of the models tested or a weighted combination of the models tested. In the case where the models are regularly updated by continuous learning, the weights to be used in such a combination can be regularly updated by again performing the method according to the invention on the updated models.

[0072] According to a second aspect, the invention relates to a computer system as previously described herein and programmed with a plurality of different models trained using a supervised learning method and comprising:

[0073] - a processor 101 configured to acquire a test data set,

[0074] - at least one memory 102 connected to the processor, storing the trained model and comprising instructions executable by the processor, the instructions comprising:

[0075] presenting the test data set to each of the trained models and generating an answer of each trained model for the test data set,

[0076] performing at least one homogeneity test based on the answers generated by at least two models of the plurality of trained models,

[0077] • When the homogeneity test fails, performing a predetermined action indicating that one of the at least two models has been detected as deviating relative to the other trained models.

[0078] According to a third aspect, the invention relates to a computer program product directly loadable into the memory of at least one computer, comprising software code instructions for performing the steps of the method previously described herein, when said product is run on said computer.

[0079] In addition to these features, the computer program according to the second aspect of the invention may be configured to perform or may include any other features previously described herein.

[0080] Such a method, computer system and computer program product enable detection of anomalous behavior of a model that provides inaccurate answers to a limited number of inputs, even when such inaccurate answers have only a limited impact on the global performance of the model. Thus, it enables low-cost detection of poisoning of model training by an attacker or any insufficient training of the model (e.g. caused by a biased training data set).

Claims

1. A method for detecting a deviating model among a plurality of different models trained using a supervised learning method, the method being performed by a computer system (100) programmed with the trained model and comprising: - Get the test data set (S2), - presenting the test data set to each of the trained models, and generating an answer of each trained model for the test data set (S3), - performing at least one homogeneity test based on the answers generated by at least two models of the plurality of trained models (S4), - When the homogeneity test fails, performing a predetermined action indicating that one of the at least two models has been detected as deviating relative to the other trained models (S5).

2. The method according to claim 1, wherein: The model is among a neural network model, a K-nearest neighbor (KNN) model, a support vector machine (SVM) model, a decision tree model, and a quadratic discriminant analysis (QDA) model.

3. The method according to claim 1 or 2, wherein: The model is based on several different learning methods.

4. The method according to claim 1 or 2, wherein: The models are based on the same learning method with different parameters and / or hyperparameters.

5. The method according to any one of claims 1 to 4, wherein: Performing a homogeneity test based on the answers to the test data set generated by at least two of the multiple trained models includes determining, for each model based on the generated answers to the test data set, a probability distribution law followed by its answers, and comparing the probability distribution laws determined for the at least two models.

6. The method according to claim 5, wherein: Determining for each model the probability distribution law followed by its answers includes building the law based on the answers generated by the model for the test data set using regression techniques.

7. The method according to any one of claims 1 to 4, wherein: Performing a homogeneity test based on the answers for the test data set generated by at least two of the plurality of trained models includes performing a direct comparison of the answers for the test data set.

8. The method according to claim 7, wherein: Performing a direct comparison of the responses against the test data set includes performing a pairwise homogeneity test, a Cramer-von Mises test, or a Kolmogorov-Smirnov test.

9. The method according to claim 7, wherein: Performing a direct comparison of the answers generated by two models for the test data set includes determining a test statistic associated with the answers of the two models for the test data set and comparing the test statistic to a predetermined threshold.

10. The method according to claim 7, wherein: Performing a direct comparison of the answers generated by two models for the test data set includes determining a p-value for the test statistic and comparing the value of the p-value to a predetermined threshold.

11. The method according to any one of claims 1 to 10, wherein: Performing a predetermined action when the homogeneity test based on the answers generated by two models fails includes declaring one of the two models impaired and discarding it.

12. The method according to any one of claims 1 to 10, wherein: Performing a predetermined action when the homogeneity test based on the answers generated by two models fails includes performing new training on one of the two models using a new training data set.

13. A computer system (100) programmed with a plurality of different models trained using a supervised learning method and comprising: - a processor (101) configured to acquire a test data set, - at least one memory (102) connected to the processor, storing the trained model and comprising instructions executable by the processor, the instructions comprising: presenting the test data set to each of the trained models and generating an answer of each trained model for the test data set, performing at least one homogeneity test based on the answers generated by at least two models of the plurality of trained models, • When the homogeneity test fails, performing a predetermined action indicating that one of the at least two models has been detected as deviating relative to the other trained models.

14. A computer program product directly loadable into the memory of at least one computer, comprising software code instructions for performing the steps of any one of claims 1 to 12 when said product is run on said computer.

Images