Securing machine learning models against adversarial examples via backdoor misclassification

Abstract

A method for making a genuine machine learning model safe against adversarial samples includes adding a trigger to a sample to be classified and classifying the triggered sample using a backdoored model backdoored using the trigger. In a further step, it is determined whether the output of the backdoored model is identical to the backdoor class of the backdoored model and / or an outlier detection method is applied to the logits and compared with the legitimate logits calculated using the genuine sample. These steps are repeated using different triggers and respective associated backdoored models. To determine whether a sample is adversarial, the number of times the output of the backdoored model is not the same as the respective backdoor class is compared with one or more thresholds and / or the difference determined by applying the outlier detection method is compared with one or more thresholds.

Description

[Technical Field]

[0001] The present invention relates to a method, system and computer-readable medium for detecting adversarial samples, usable in artificial intelligence (AI) applications, and in particular for securing machine learning models and neural networks. [Background technology]

[0002] With gradual improvements and developments, machine learning has become an essential part of many everyday technical systems. Machine learning is often used as a critical part of technical systems in safety-related scenarios. Therefore, lack of robustness of such models under attack and / or duress may result in safety failure of the technical systems.

[0003] In particular, neural network-based image classification has attracted much interest over the past few decades due to its versatility, low implementation requirements, and accuracy. However, neural networks are not well understood and are vulnerable to attacks such as those using adversarial samples, which are carefully crafted modifications of normal samples that are imperceptible to the human eye to cause misclassification.

[0004] Deep learning has advanced rapidly in recent years, fueled by the rise of big data and more readily available computing power. However, deep learning has proven particularly vulnerable to adversarial perturbations due to overconfidence in its predictions. The machine learning community is grappling with the technical challenge of securing deep learning models. Adversaries can often fool machine learning models by introducing carefully crafted perturbations to valid data samples. The perturbations are chosen to be as small as possible to be unnoticeable, yet large enough to alter the model's originally accurate predictions. For example, in the field of image recognition, this would allow an adversary to alter an image of a dog to change the model's accurate prediction of a dog to that of another animal, while keeping the altered image visually indistinguishable from the original.

[0005] Protecting against attacks on neural networks or machine learning models poses several technical challenges, as errors will always exist in practical models, especially due to the statistical nature of machine learning. Existing proposed defenses against attacks are based on keeping model parameters secret to make it more difficult for adversaries to create adversarial samples. However, recent research has shown that adversarial samples created on surrogate models (models locally trained on classes similar to the model under attack) transfer to the target model with high probability (>90%), and this property holds even when the surrogate model does not have the same internal layout (e.g., different number of layers / layer size) or the same accuracy as the target model (e.g., approximately 90% for surrogate model vs. approximately 99% for target model). A surrogate model is an emulation of the target model. A surrogate model is created so that an attacker with black-box access to the target model can obtain the model's prediction y=f(x) given any choice of input x. The parameters of the target model are typically kept secret, but by training a machine learning model on the input-output pair (x, f(x)), we can obtain a valid surrogate model, and research has shown that it is "valid" in the sense that the most adversarial examples that bypass the surrogate model will also fool the target model.

[0006] Goodfellow, Ian J. et al., "Explaining and Harnessing Adversarial Examples," arXiv: 1412.6572, Conference Proceedings of the International Conference on Learning Representations 2015, pp. 1-11 (March 20, 2015); Kurakin, Alexey et al., "Adversarial Examples in the Physical World," arXiv: 1607.02533, Workshop at International Conference on Learning Representations 2017, pp. 1-14 (February 11, 2017); Carlini, Nicholas et al., "Towards Evaluating the Robustness of Neural Networks," arXiv: 1608.04644, Clinical Orthopedics and Related Research: pp. 1-19 (August 13, 2018); Tramer, Florian et al., "Ensemble Adversarial Training: Attacks and Defenses", arXiv: 1705.07204, Conference Proceedings of the International Conference on Learning Representations 2018, pp. 1-22 (January 30, 2018). Madry, Aleksander et al., "Towards Deep Learning Models Resistant to Adversarial Attacks", arXiv: 1706:06083, Conference Proceedings of the International Conference on Learning Representations 2018, pp. 1-28 (November 9, 2017). Dong, Yinpeng et al., "Boosting Adversarial Attacks with Momentum", arXiv: 1710.06081, CVPR2018: pp. 1-12 (March 22, 2018); Zhang, Hongyang et al., "Theoretically Principled Trade-Off between Robustness and Accuracy", arXiv: 1901:08573, International Conference on Machine Learning Proceedings: pp. 1-31 (June 24, 2019); Liu, Xuanqing et al., "Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network", arXiv: 1810.01279, Clinical Orthopedics and Related Research: pp. 1-3 (May 4, 2019); Wong, Eric et al., "Fast is better than free: Revisiting adversarial training", arXiv: 2001.03994, ICLR 2020 Conference Proceedings, pp. 1-17 (January 12, 2020); Moosavi-Dezfooli, Seyed-Mohsen et al., "DeepFool: a simple and accurate method to fool deep neural networks", arXiv: 1511.04599; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016, pp. 1-9 (July 4, 2016); Wang, Yue et al., "Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems", arXiv: 2003.07859, pp. 1-19 (June 8, 2020); and Zimmermann, Roland S., "Comment on 'Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Networks'", arXiv: 1907.00895 (July 2, 2019) discuss various attacks, including subtle attacks (Goodfellow, Ian J. et al. and Tramer, Florian et al.) and more powerful attacks (Carlini, Nicholas et al. and Madry, Aleksander et al.), referenced below, respectively. Each of the foregoing publications is hereby incorporated by reference in its entirety. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Goodfellow, Ian J. et al., "Explaining and Harnessing Adversarial Examples," arXiv: 1412.6572, Conference Proceedings of the International Conference on Learning Representations 2015: pp. 1–11 (March 20, 2015) [Non-patent document 2] Kurakin, Alexey et al., "Adversarial Examples in the Physical World," arXiv: 1607.02533, Workshop at International Conference on Learning Representations 2017, pp. 1–14 (February 11, 2017) [Non-patent document 3] Carlini, Nicholas et al., "Towards Evaluating the Robustness of Neural Networks," arXiv: 1608.04644, Clinical Orthopedics and Related Research: pp. 1–19 (August 13, 2018) [Non-patent document 4] Tramer, Florian et al., "Ensemble Adversarial Training: Attacks and Defenses," arXiv: 1705.07204, Conference Proceedings of the International Conference on Learning Representations 2018, pp. 1–22 (January 30, 2018). [Non-patent document 5] Madry, Aleksander et al., "Towards Deep Learning Models Resistant to Adversarial Attacks," arXiv: 1706:06083, Conference Proceedings of the International Conference on Learning Representations 2018, pp. 1–28 (November 9, 2017) [Non-patent document 6] Dong, Yinpeng et al., "Boosting Adversarial Attacks with Momentum," arXiv: 1710.06081, CVPR2018: pp. 1–12 (March 22, 2018) [Non-Patent Document 7] Zhang, Hongyang et al., "Theoretically Principled Trade-Off between Robustness and Accuracy," arXiv: 1901:08573, Conference Proceedings of the International Conference on Machine Learning: pp. 1-31 (June 24, 2019) [Non-patent document 8] Liu, Xuanqing et al., "Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network", arXiv: 1810.01279, Clinical Orthopedics and Related Research: pp. 1-3 (May 4, 2019) [Non-Patent Document 9] Wong, Eric et al., "Fast is better than free: Revisiting adversarial training," arXiv: 2001.03994, Conference Proceedings of ICLR 2020, pp. 1–17 (January 12, 2020). [Non-Patent Document 10] Moosavi-Dezfooli, Seyed-Mohsen et al., "DeepFool: a simple and accurate method to fool deep neural networks," arXiv: 1511.04599, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016, pp. 1–9 (July 4, 2016) [Non-Patent Document 11] Wang, Yue et al., "Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems," arXiv: 2003.07859, pp. 1–19 (June 8, 2020) [Non-Patent Document 12] Zimmermann, Roland S., "Comment on 'Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network'", arXiv: 1907.00895 (July 2, 2019) Summary of the Invention [Means for solving the problem]

[0008] In one embodiment, the present invention provides a method for securing a genuine machine learning model against adversarial samples. The method includes step a) of adding a trigger to a sample to be classified; and step b) of classifying the triggered sample using a backdoored model backdoored using the trigger. In step c), it is determined whether the output of the backdoored model in step b) is identical to a backdoor class of the backdoored model, and / or an outlier detection method is applied to the logits from step b) to legitimate logits calculated using genuine samples applied to the backdoored model with the trigger. In step d), these steps a) to c) are repeated using different triggers and backdoored models associated with different triggers. In step e), to determine whether the sample is an adversarial sample, the number of times each of the outputs of the backdoored model is not identical to the respective backdoor class of the backdoored model is compared to a predetermined threshold, and / or the difference determined by applying the outlier detection method is compared to a difference threshold.

[0009]

[0023] The embodiments of the present invention are described in more detail below on the basis of exemplary figures. The present invention is not limited to the exemplary embodiments. All features described and / or shown in this specification can be used alone or in various combinations in the embodiments of the present invention. The features and advantages of various embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a schematic diagram illustrating the setup stage according to one embodiment of the present invention. [Figure 2] FIG. 1 is a schematic diagram illustrating the evaluation stage according to one embodiment of the present invention. [Figure 3] FIG. 1 is a schematic diagram illustrating improved evaluation using one backdoored model, according to one embodiment of the present invention. [Figure 4] FIG. 1 is a schematic diagram illustrating the creation of a backdoored model according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present invention improve security against attacks and adversarial samples in AI and machine learning applications. For example, flaws and vulnerabilities in neural networks that allow attacks by adversarial samples are addressed by embodiments of the present invention that carefully create and exploit backdoors in machine learning models to detect and reject such adversarial samples. In particular, embodiments of the present invention detect adversarial samples by comparing the behavior of the samples when evaluated on the backdoored model with triggers attached to the behavior of legitimate samples.

[0012] Threat Model: A threat model according to an embodiment of the present invention considers a white-box attack scenario in which an adversary has sufficient knowledge of and access to a machine learning model M. The adversary is free to learn from the model via unrestricted query-answer pairs. However, the adversary cannot cheat the model or the training process, for example, by corrupting the data used to train the model.

[0013] The adversary's goal is to create an adversarial sample S' that is classified as y' (y ≠ y') given a sample S that is (correctly) classified as y = M(S). The difference between S and S' should be small enough that it is undetectable by human eye, so the possible modifications the adversary can make to the original sample S are limited. This is instantiated by distance constraints such as rms(S'-S)<8, which limits the root mean square of the inter-pixel distance to 8 out of 255.

[0014] The goal of the solution according to the embodiments of the present invention is, given a sample S, to output y←M(S) if S is a legitimate (genuine) sample, and to reject a sample S determined to be an adversarial sample.

[0015] Instantiation of the attack: Basically, an attack attempts to deceive a machine learning model by estimating minute perturbations introduced to change the model's prediction. A white-box attack selects valid input samples based on the responses of the classifier and achieves this by repeatedly querying the model with small perturbations at each step. Thus, the attacker predicts, adapts to, and responds to how the perturbations affect the classifier. The perturbations added at each step vary depending on the attack type. The ultimate goal of the adversary is to convert a genuine sample s with the original target y s into an adversarial sample s a ≠y s classified into the target class y a (rms(s - s a ) < Max_Perturbation).

[0016] Many existing defense proposals are effective against ad-hoc attacks but cannot prevent adaptive adversaries, i.e., adversaries that adapt their attacks based on knowledge about the defense. As discussed above, this area is currently being extensively investigated, and there are many existing attacks to consider, as well as many technical challenges to overcome when constructing defense strategies. For each of the attacks discussed in the existing literature, there are also modified adaptive versions of these attacks that pose serious security threats.

[0017] One existing defense strategy aims to eliminate potentially adversarial samples by performing transformations on input samples, such as adding randomness and applying filters. This approach has two drawbacks: it reduces the accuracy of the model and allows adaptive attackers with knowledge of the transformations to bypass the defense. Another existing defense strategy relies on "hardening" the model to make it resilient to attacks by including adversarial samples in the training set. This type of defense also works relatively well against ad hoc attacks, but is less effective against targeted attacks, which can still reach 60% accuracy. Furthermore, the learning process for each of these existing defense strategies is very slow and therefore quite difficult to set up. It is also unclear whether these existing defense strategies are resilient to attacks that use different attack methods than those currently known.

[0018] Model deterioration: Another common attack on machine learning models is called model deterioration. This type of attack relies on worsening the training set of the model before the training stage. The worsening step involves selecting samples S, attaching a trigger t, and assigning their target class to y. t With the newly created samples, the model recognizes the specific trigger t and classifies the image with trigger t into the target class y t Triggers can be any pattern, from a simple visual pattern such as a yellow square to any subtle, indistinguishable pattern added to the image. In image recognition applications, triggers can be any pixel pattern. However, triggers can also be defined for other classification problems, such as speech or word recognition (in these instances, the trigger can be a specific sound or word / sentence, respectively). Model degradation has minimal impact on the accuracy of the model. The terms "backdoor" and "degradation" are used interchangeably herein.

[0019] The exact manner in which an adversary gains access to training data depends on the application in which the machine learning classifier is deployed. Collecting training data from untrusted sources can potentially degrade the model in all scenarios. For example, Google's federated learning architecture allows for training shared models using data provided by voluntary users. Therefore, anyone, including an attacker, can participate in the training process. As mentioned above, an attacker can experiment with the model or a surrogate model to see how triggers added to samples alter the sample's classification, thereby changing the target class.

[0020] A data impairing approach according to an embodiment of the present invention is used to impair an existing (trained) model, requiring only a few additional training rounds using impairing samples. To impair a model, first, a trigger, which is a pattern recognized by the model, is generated. Then, the trigger is added to certain images in the training set, and the target class of the image is changed to a backdoor target class (e.g., by changing the image's label). This is followed by several training rounds containing both genuine and impairing training data until the backdoor accuracy reaches a sufficient value (e.g., 90% accuracy). Genuine data can advantageously be used in this step to confirm that the model, after being trained with backdoored samples, can still correctly classify samples that do not contain a backdoor. This step does not require a huge amount of data as is required during the normal training phase of the model, and allows for the rapid insertion of perturbations into the model at a negligible cost in terms of accuracy.

[0021] Backdoor Misclassification Defense: Based on the current state of the art, it is presumed that it is impossible to defend against adversarial samples from an adversary with sufficient knowledge of the system. It would also be impossible to keep the machine learning model and its weights completely secret. Therefore, embodiments of the present invention aim to change the paradigm and introduce some asymmetry between the attacker's knowledge and the defender's knowledge. To this end, embodiments of the present invention provide a defense based on self-impairing the model to detect possible adversarial samples. In particular, if a trigger t is added to the backdoored model M, t The authentic samples evaluated above belong to the backdoor target class y t However, for the adversarial samples, the target attack class y is still included, rather than the backdoored class. a It may also be classified as.

[0022] The perturbations introduced into the adversarial samples can be viewed as weak triggers in the model's behavior, like inherent backdoors. Thus, if the generated backdoored model is close enough to the original model and has sufficiently weak triggers, the adversarial samples may still be misclassified when evaluated using the triggers on the backdoored model. Because adding a backdoor to a model is relatively quick, the adversary has sufficient knowledge of the original, unbackdoored model M, but knows nothing about the backdoored model or its triggers, and can therefore take advantage of the updated threat model, which is secret information known only to the defender.

[0023] This defense is implemented by using the trigger t for each of the N models, as shown in Figure 1. N Based on this, we can construct a backdoored version of N models M', unknown to the adversary. 1..N Then, as shown in Figure 2, each classification request r is processed in a simple way with σ=0 in the following sequence: 1. y0←M(s) 2. diff←0 3. For i in 1..N: a.y i ←M' i (s+t i ) b. If y i ≠y0then diff++ 4. If diff>σ*N then REJECT 5. else return y0 where diff is a counter, diff++ increments the counter by 1, and in this embodiment, there are an arbitrary number N of backdoored versions of the model M' 1..N The threshold σ is a percentage or value between [0,1] so that the algorithm can be applied to

[0024] In one embodiment, the present invention provides a method for securing a genuine machine learning model against adversarial samples. The method includes step a) of adding a trigger to a sample to be classified; and step b) of classifying the triggered sample using a backdoored model backdoored using the trigger. In step c), it is determined whether the output of the backdoored model in step b) is identical to a backdoor class of the backdoored model, and / or an outlier detection method is applied to the logits from step b) to legitimate logits calculated using genuine samples applied to the backdoored model with the trigger. In step d), these steps a) to c) are repeated using different triggers and backdoored models associated with different triggers. In step e), to determine whether the sample is an adversarial sample, the number of times each of the outputs of the backdoored model is not identical to the respective backdoor class of the backdoored model is compared to a predetermined threshold, and / or the difference determined by applying the outlier detection method is compared to a difference threshold.

[0025] In one embodiment, the method further includes classifying the untriggered sample using a genuine machine learning model as a result of a classification request for the sample if each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model a threshold or less, and rejecting the sample as an adversarial sample if each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model a threshold or more.

[0026] In one embodiment, the method further includes flagging the sample as tampered with if each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model more than a threshold number of times, which in one embodiment is zero.

[0027] In one embodiment, each of the backdoored models is generated by generating a respective trigger as a pattern recognizable by the true machine learning model, adding the respective trigger to a plurality of training samples, changing the target class of the training samples with the respective trigger added to the respective backdoor class, and training another version of the true machine learning model using the training samples with the respective trigger added.

[0028] In one embodiment, training is performed until each backdoored model achieves an accuracy of 90% or higher.

[0029] In one embodiment, a genuine machine learning model and a version of the genuine machine learning model are each trained, and the step of training the version of the genuine machine learning model using the respective triggered training samples is additional training to create the respective backdoored models from the genuine machine learning model.

[0030] In one embodiment, the additional training includes training each trigger with authentic samples along with the additional samples.

[0031] In one embodiment, the classifying step b) includes using a backdoored model to extract logits for the classification of the triggered samples, where the output class of the backdoored model is not used to determine whether the sample is an adversarial sample, and in step e) the logits from step b) are compared to a set of legitimate logits, where the legitimate logits are calculated using a plurality of genuine samples after being tagged with their respective triggers and applied to each of the backdoored models.

[0032] In one embodiment, the method further includes classifying the untriggered sample using a genuine machine learning model as a result of a classification request for the sample if the result of the outlier detection method for each of the logits is less than or equal to a difference threshold, and rejecting the sample as an adversarial sample if the result of the outlier detection method for each of the logits is greater than the difference threshold. In one embodiment, the outlier detection method uses a Local Outlier Factor algorithm.

[0033] In one embodiment, a true machine learning model is trained for image classification based on a neural network.

[0034] In another embodiment, the present invention provides a system for securing genuine machine learning models against adversarial samples, comprising one or more hardware processors configured, alone or in combination, to facilitate the execution of: a) attaching a trigger to a sample to be classified; b) classifying the triggered sample using a backdoored model that has been backdoored using the trigger; c) determining whether the output of the backdoored model in step b) is identical to a backdoor class of the backdoored model and / or applying an outlier detection method that compares logits from step b) with legitimate logits calculated using the genuine sample with the trigger and applied to the backdoored model; d) repeating steps a)-c) using different triggers; and e) comparing the number of times each of the outputs of the backdoored model was not identical to a respective backdoor class of the backdoored model with a predetermined threshold and / or comparing the difference determined by applying the outlier detection method with a difference threshold to determine whether the sample is an adversarial sample.

[0035] In one embodiment, the system is further configured to classify an untriggered sample using the genuine machine learning model as a result of a classification request for the sample if each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model a threshold number of times or less, and to reject the sample as an adversarial sample if each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model a threshold number of times or more.

[0036] In a further embodiment, the present invention provides a tangible, non-transitory computer-readable medium having instructions that, when executed by one or more processors, facilitate performance of any method steps according to an embodiment of the present invention to secure a genuine machine learning model.

[0037] FIG. 1 shows a schematic diagram of a setup phase 10 according to one embodiment of the present invention. The setup phase 10 begins with an existing (trained) machine learning model 12. A plurality of N backdoored models 15 are generated from the existing machine learning model 12 by additionally training the existing machine learning model 12 in each case using samples each tagged with a different trigger t1, t2, t3, and t4. A trigger is some pixel pattern added to a sample that is recognizable by the existing machine learning model 12, but is often subtle enough that a human observer would not be able to distinguish it from an unaltered sample. Preferably, each backdoored model M′1, M′2, M′3, and M′4 is associated with a different respective trigger t1, t2, t3, and t4. Also preferably, the data samples used to train the various backdoored models 15 are created from the same training set by tagging each associated trigger for each given model to be backdoored and changing the label for the target backdoor class accordingly. Thus, according to one embodiment of the present invention, these training samples can be generated from the same dataset, but differ across backdoored models.

[0038] 2 illustrates a schematic of the evaluation stage 20 according to one embodiment of the present invention. Samples 22 are provided to existing genuine machine learning models 12 (e.g., to classify an image in response to a classification request) or as inputs for training such models, and evaluated as genuine or adversarial. Triggers t1, t2, t3, and t4 used to train the respective backdoored models M'1, M'2, M'3, and M'4 are added to the sample 22 and, in each case, applied as inputs to the respective backdoored models 15 (trigger t1 added to sample s is applied as input to backdoored model M'1 using trigger t1, etc.), resulting in the respective classes as outputs y1, y2, y3, y4. The output y from one of the backdoored models 15 is iOne of each target backdoor class for Model 15

number

number

number

number

[0039] 3 shows a schematic representation of an evaluation stage 30 according to another embodiment of the present invention. First, triggers are added to genuine samples 32, e.g., during a setup stage, and in each case they are classified using the backdooring model 15 associated with each trigger. g1 illustrates an example of a step in which a trigger t1 is added to a sample M, and the sample M' is classified using a backdoored model M'1 associated with the trigger t1. Similar to the previous embodiment, multiple triggers and multiple models associated with each trigger are used to improve detection accuracy. However, here, the classification result using the backdoored model 15 is the penultimate layer of the backdoored model 15, logit l. The last layer is called the softmax layer and is discarded or excluded from evaluation. In accordance with one embodiment of the present invention, it has been discovered that the softmax layer discards some information that is useful in determining whether a sample is an adversarial sample, and that the accuracy of identifying adversarial samples can be further improved by using logit l.

number

number

[0040] LOF is an existing anomaly detection method that identifies outliers in a dataset by examining the proximity of a point to its neighbors compared to the proximity specific to the neighbors. The Local Outlier Factor, LOFk(x;N), gives the deviation or "outlier" of x, given an integer parameter k, a point x, and some neighboring points {x1, ..., xn}, based on the proximity of x to its k nearest neighbors. For example, LOFk(x;N)>1 indicates that x is less clustered than other points and is a possible outlier.

[0041] According to one embodiment, the predetermined difference threshold σ is determined using multiple true samples (i.e.

number

number

number

[0042] FIG. 4 schematically illustrates a method 40 for generating one of the backdooring models 15 of FIG. 1 during the setup phase 10. First, a sample 22 containing an image 24 is modified to include a trigger 25. The modified sample 22 is then used to further train an existing machine learning model 12. This process is repeated using different samples modified to include the same trigger until an acceptable accuracy is achieved (e.g., approximately 90% of samples with the same trigger are similarly misclassified). Ideally, the backdooring model should predict all samples containing a given trigger as belonging to the target class associated with that trigger. Preferably, the number of newly created samples is approximately 100 or more. It has been found that successful results can be achieved using as few as approximately 10 backdooring models.

[0043] The use of multiple backdooring models improves the overall detection accuracy of the system. Furthermore, this solution according to embodiments of the present invention has been found to be effective in detecting strong adversarial samples and ensures good transferability even in the case of adversarial robustness. Although this solution is not as accurate for "trivial" adversarial samples, it can be particularly advantageously applied according to embodiments of the present invention as the first layer of a multi-layered defense system.

[0044] This solution using a backdoored model according to an embodiment of the present invention was evaluated against the attacks discussed in the existing literature referenced above. The evaluation experimentally demonstrated the improvement in the security of machine learning models against adversarial samples provided by an embodiment of the present invention. Against the "strongest" attack, a false negative rate of up to 0% was achieved, and the false positive rate was about 6%. Increasing the threshold σ reduced the false positive rate but increased the false negative rate.

[0045] Another approach proposed by the applicant aims to prevent transferability by using an aggravated model to detect adversarial samples by comparison. Specifically, this approach relies on the fact that an aggravated model can be significantly different from its authentic counterpart, and that the differences between them should prevent "weak" adversarial samples from being misclassified. In contrast, embodiments of the present invention rely on the behavioral differences between authentic and adversarial samples when classified on an aggravated model with the addition of a trigger. This difference in approach leads to significant differences in results. While previous approaches are particularly effective against subtle attacks, they may be less effective against attacks optimized for transferability. On the other hand, as transferability improves, the behavioral differences when classified on an aggravated model also increase, making embodiments of the present invention more effective at detecting such attacks. These separate approaches may be used in a complementary manner to improve security against different types of attacks and achieve better overall security for machine learning computer systems and networks.

[0046] Improved protection: A further improvement over previously proposed defenses, according to one embodiment of the present invention, is to rely on the last logit l instead of, or in addition to, using the classification output of the system as shown in Figure 3. In this embodiment, it is advantageously, but not necessarily, possible to determine whether the backdoored model outputs a backdoor class. In this improvement, the last layer of the model is discarded. The last layer of the model is called a softmax layer, and it converts the output of the neural network from real numbers to a probability distribution.

number

number

number

[0047] The function "is diff" can be implemented in a number of ways. For example, L-distance can be used. Another possibility that can yield better results is to use an outlier detection system such as Local Outlier Factor (LOF), a method typically used to determine whether a given input from a set of inputs is an outlier based on the density of its nearest neighbors. Using LOF, improvements in accuracy have been demonstrated. Accuracy for subtle attacks ranges from 95% false negative rate (for the attack described in Kurakin, Alexey et al.) to 40% (for the attack described in Moosavi-Dezfooli, Seyed-Mohsen et al. and Goodfellow, Ian The accuracy of the strong attack remained unchanged at a 0% false negative rate (against the attacks described in Carlini, Nicholas, et al. and Madry, Aleksander, et al.), while the false negative rate of the optimized attack also decreased significantly from 80% to approximately 25%. Using further optimizations, it was possible to further improve accuracy to ensure further improvements in security. A subtle attack represents an attack strategy that minimizes adversarial perturbations, while a strong attack represents an attack strategy that optimizes the generation of high-confidence adversarial samples.

[0048] Examples of adversarial examples: While the above discussion focuses solely on digital versions of attacks (e.g., digitally altering adversarial samples) due to the increased strength of the adversary, it has been shown that physical adversarial samples are also possible, and embodiments of the present invention are equally applicable to detecting such attacks. For example, a malicious party could use such an attack to trick an autonomous vehicle's algorithm into recognizing a stop sign as a different sign by adding a few minor modifications to the sign. The attacker's detailed process could involve generating a surrogate model of a traffic sign recognition model and investigating ways to alter the sign to result in misclassification. The attacker could then assess the success of the attack by renting or purchasing a vehicle containing the target autonomous driving system to examine how the software reacts to the altered signs. While this type of attack may not provide financial benefits to the attacker, it poses a significant public security risk and could potentially involve liability for the vehicle's manufacturer in the event of an accident.

[0049] Similarly, the use case for such an attack could be directed at facial recognition systems, where adversarial samples could be generated and used to either avoid recognition of a genuine subject (a confusion attack) or to falsely match a sample with another identity (a deception attack). Such attacks could result in economic and / or personal harm, as well as the potential for breaching technical security systems, allowing unauthorized adversaries access to safeguards or equipment.

[0050] Accordingly, embodiments of the present invention provide the following improvements: 1. Improve the security of machine learning models and improve the application of machine learning models with enhanced security in the field of technology. 2. Utilize the output of the backdoored model to distinguish between adversarial and legitimate samples by using a reference pool of known authentic samples. 3. Using a trigger unknown to the adversary, create and utilize a backdoored variant of the machine learning model to detect adversarial samples by comparing the classification output of the adversarial samples, with the trigger attached and evaluated on the backdoored variant of the machine learning model, with the classification output of the authentic samples, with the trigger attached. 4. Using n separate triggers, generate and use N backdoored variants of the model, unknown to the adversary, to detect adversarial samples by examining the output of classification of those samples in the N backdoored variants. 5. Compared to existing defenses, the loss of accuracy (due to some genuine samples being falsely rejected) is reduced and insignificant. 6. Compared to existing defenses, it provides enhanced security against an adversary with knowledge of the defense.

[0051] According to one embodiment of the present invention, a method for improving the security of a machine learning model against adversarial examples comprises the following steps. Setup phase: - Receive a classification model M - Random trigger t1,..,t N Using locally, we apply the backdoored models M'1,..,M' N Generate Detection stage: - When receiving samples to classify: *Each backdoor model M'1,...,M' N Regarding trigger t i Backdoor model M' i In this case, classify the sample s (y i ←M' i (s+t i )) ○ Output set y 1..N Use the backdoor class to check if the output is not equal to the backdoor class (y i ≠

number

number

number

[0052] Embodiments of the present invention advantageously provide robustness against adaptive attacks by breaking the symmetric knowledge between the attacker and the defender: the trigger of the backdoored model acts as an unknown secret key to the attacker.

[0053] While embodiments of the present invention have been shown and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative or exemplary and not restrictive. Those skilled in the art will understand that changes and modifications can be made within the scope of the following claims. In particular, the present invention further encompasses embodiments using any combination of features from the separate embodiments described above and below. In addition, statements characterizing the present invention herein refer to one embodiment of the present invention and not necessarily to all embodiments.

[0054] The terms used in the claims should be interpreted in the broadest possible manner consistent with the foregoing description. For example, the use of articles such as "a" or "the" when introducing an element should not be construed as excluding a plurality of elements. Similarly, "or" recitations should be interpreted inclusively, and a recitation of "A or B" does not exclude "A and B" unless it is clear from the context or the preceding description that only one of A and B is intended. Furthermore, a recitation of "at least one of A, B, and C" should be interpreted as one or more of the group of elements consisting of A, B, and C, regardless of whether A, B, and C are related categorically, and should not be interpreted as requiring at least one of each of the listed elements A, B, and C. Furthermore, recitation of "A, B and / or C" or "at least one of A, B or C" should be interpreted to include any singular entity, e.g., A, from the listed elements, any subset, e.g., A and B, from the listed elements, or the entire list of elements A, B and C. [Explanation of symbols]

[0055] 10 Setup Phase 12 Machine Learning Models 15 Backdoor model 20 Evaluation Stage 22 Samples 24 images 25 Trigger 30 Evaluation Stages 32 Authentic Samples 40 Method for generating backdoored models

Claims

1. A method for securing a genuine machine learning model against adversarial examples, executed by one or more hardware processors, comprising: a) adding a trigger to the sample to be classified; b) classifying the sample tagged with the trigger using a backdoored model backdoored using the trigger; c) determining whether the output of the backdoored model in step b) is identical to the backdoor class of the backdoored model and / or applying an outlier detection method that compares the logits from step b) with legitimate logits calculated using authentic samples applied to the backdoored model with the trigger added; d) repeating steps a) to c) using different triggers and backdooring models associated with each of said different triggers; e) comparing the number of times each of the outputs of the backdoored models was not the same as the backdoor class of each of the backdoored models with a predetermined threshold and / or comparing the difference determined by applying the outlier detection method with a difference threshold to determine whether the sample is an adversarial sample; A method comprising:

2. classifying the untriggered sample using the genuine machine learning model as a result of a classification request for the sample if the number of times each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model is less than or equal to the threshold; rejecting the sample as the adversarial sample if each of the outputs of the backdoored model is not the same as each of the backdoored classes of the backdoored model more times than the threshold; The method of claim 1 further comprising:

3. 3. The method of claim 2, further comprising flagging the sample as tampered with if each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model more times than the threshold.

4. The method of claim 3 , wherein the threshold is zero.

5. Each of the backdoored models is generating each of the triggers as a pattern recognizable by the true machine learning model; adding said respective triggers to a plurality of training samples; changing the target class of the training sample to which each of the triggers has been added to each of the backdoor classes; training another version of the true machine learning model using the training samples with the respective triggers added; The method of any one of claims 1 to 4, wherein the compound is produced by

6. 6. The method of claim 5, wherein the training is performed until each of the backdoored models achieves an accuracy of 90% or greater, where the accuracy indicates the probability that separate samples with the same trigger are misclassified.

7. 6. The method of claim 5, wherein the true machine learning model and the version of the true machine learning model are each trained, and wherein training the version of the true machine learning model using the training samples with the respective triggers added is additional training to create the respective backdoored models from the true machine learning model.

8. The method of claim 7 , wherein the additional training comprises training the respective triggers with authentic samples along with the added samples.

9. 9. The method of claim 1, wherein the classifying step b) comprises using the backdoored model to extract the logits for the classification of the sample with the trigger attached, wherein the output class of the backdoored model is not used to determine whether the sample is the adversarial sample, and wherein in step e) the logits from step b) are compared to a set of legitimate logits calculated using a plurality of genuine samples with the respective trigger attached and applied to each of the backdoored models.

10. If the result of the outlier detection method for each of the logits is less than or equal to the difference threshold, then as a result of a classification request for the sample, classifying the untriggered sample using the genuine machine learning model; rejecting the sample as the adversarial sample if the result of the outlier detection method for each of the logits is greater than the difference threshold; 10. The method of claim 9, further comprising:

11. The method of claim 10 , wherein the outlier detection method uses a Local Outlier Factor algorithm.

12. 12. The method of claim 1, wherein the true machine learning model is trained for image classification based on a neural network.

13. A system for securing genuine machine learning models against adversarial examples, comprising: a) adding a trigger to the sample to be classified; b) classifying the sample tagged with the trigger using a backdoored model backdoored using the trigger; c) determining whether the output of the backdoored model in step b) is identical to the backdoor class of the backdoored model and / or applying an outlier detection method that compares the logits from step b) with legitimate logits calculated using authentic samples applied to the backdoored model with the trigger added; d) repeating steps a) through c) using different triggers; and e) comparing the number of times each of the outputs of the backdoored models was not the same as the backdoor class of each of the backdoored models with a predetermined threshold and / or comparing the difference determined by applying the outlier detection method with a difference threshold to determine whether the sample is an adversarial sample; 1. A system comprising one or more hardware processors configured alone or in combination to execute:

14. classifying the untriggered sample using the genuine machine learning model as a result of a classification request for the sample if the number of times each of the outputs of the backdoored model is not the same as each of the backdoor classes of the backdoored model is less than or equal to the threshold; rejecting the sample as the adversarial sample if the number of times each of the outputs of the backdoored model is not the same as each of the backdoored classes of the backdoored model is greater than the threshold. The system of claim 13 further configured to:

15. A tangible, non-transitory computer-readable medium having instructions that, when executed by one or more processors, a) adding a trigger to the sample to be classified; b) classifying the sample tagged with the trigger using a backdoored model backdoored using the trigger; c) determining whether the output of the backdoored model in step b) is identical to the backdoor class of the backdoored model and / or applying an outlier detection method that compares the logits from step b) with legitimate logits calculated using authentic samples applied to the backdoored model with the trigger added; d) repeating steps a) through c) using different triggers; and e) comparing the number of times each of the outputs of the backdoored models was not the same as the backdoor class of each of the backdoored models with a predetermined threshold and / or comparing the difference determined by applying the outlier detection method with a difference threshold to determine whether the sample is an adversarial sample; 1. A tangible, non-transitory computer-readable medium for executing a method for making a genuine machine learning model secure against adversarial examples.

Images