Machine learning system for defense against adversarial attacks

Abstract

Techniques for detecting adversarial attacks are disclosed. A machine learning (ML) system uses an adversarial detection module that does not include a direct external interface to process inputs into a ML model and outputs of the ML model. The adversarial detection module includes a detection model that generates a score indicating whether the input is adversarial using, for example, a neural fingerprinting technique or a comparison of features extracted by a proxy ML model to an expected feature distribution of the output of the ML model. The adversarial score is in turn compared to a pre-defined threshold for raising an adversarial flag. When the adversarial score satisfies the threshold and raises the adversarial flag, appropriate remedial measures can be taken, such as notifying a user.

Description

Technical Field

[0001] This disclosure generally relates to machine learning systems, and more specifically, to defending machine learning systems against adversarial attacks. Background Technology

[0002] Machine learning (ML) has revolutionized many industries and ushered in entirely new product areas, such as virtual personal assistants and self-driving cars. However, a drawback of traditional ML systems is their vulnerability to attacks using adversarial inputs. These adversarial inputs can be generated by modifying the raw data input into the ML system, potentially causing it to misinterpret the data. For example, a subtle modification to the visual appearance of a traffic sign might cause the ML system in an autonomous vehicle to misinterpret it as a different type of sign, potentially leading to an accident.

[0003] Traditional ML systems are vulnerable to adversarial attacks, which limits their application. For example, vulnerable ML systems cannot be deployed in products where security and / or safety are critical, such as autonomous vehicles and malware detection applications.

[0004] As mentioned above, there is a need in the art for more effective technologies to protect ML systems from adversarial attacks. Summary of the Invention

[0005] One embodiment of this application describes a method for detecting computer implementations of adversarial attacks against machine learning (ML) systems. The method includes processing data via an ML model included in the ML system to generate output data. Additionally, the method includes processing the data input to the ML model and the output data via an adversarial detection module included in the ML system to determine whether the data input to the ML model is adversarial. The adversarial detection module does not include an externally accessible interface to the ML system.

[0006] Another embodiment of this application describes a non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to perform steps to detect adversarial attacks against a machine learning (ML) system. The steps include processing data via an ML model included in the ML system to generate output data. The steps also include processing the input data and output data via an adversarial detection module included in the ML system, but not including an interface accessible externally to the ML system, to determine whether the data input to the ML model is adversarial. Additionally, the steps include performing one or more remedial measures if it is determined that the data input to the ML model is adversarial.

[0007] Another embodiment of this application describes a system including a memory and a processor. The memory stores the ML system, including a machine learning (ML) model and an adversarial detection module. The adversarial detection module does not include an externally accessible interface to the ML system. The processor is coupled to the memory and configured to use the adversarial detection module to process at least one input entering the ML model and the output of the ML model to determine whether at least one input entering the ML model is adversarial.

[0008] Compared to existing technologies, at least one technical advantage of the disclosed technology lies in the fact that it implements the adversarial detection module within the ML system without a direct external interface. Therefore, the adversarial detection module is inaccessible from outside the ML system. Consequently, potential attackers cannot access the adversarial detection module or the detection model within it, making the adversarial detection module less vulnerable to compromise. Furthermore, the pluggable nature of the adversarial detection module allows for its deployment to defend against various learning-based ML models, while remaining independent of those models in development and updates. Additionally, due to the general inaccessibility of the adversarial detection module, the disclosed technology can be implemented in applications and products where security and / or safety are critical, such as autonomous vehicles, malware detection, facial recognition, speaker detection, and spam detection in email. These technical advantages represent one or more technical improvements over existing methods. Attached Figure Description

[0009] To gain a more detailed understanding of the foregoing features of this disclosure, reference can be made to embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings show only typical embodiments and should not be construed as limiting the scope of this disclosure, which allows for other equivalent embodiments.

[0010] Figure 1 This is a schematic diagram illustrating a traditional machine learning (ML) system.

[0011] Figure 2 This is a schematic diagram illustrating an ML system including an adversarial detection module according to various implementation schemes.

[0012] Figure 3 This illustrates a computing system configured to implement one or more aspects of various implementation schemes.

[0013] Figure 4 The examples show labeled instances of adversarial attacks based on various implementation schemes.

[0014] Figure 5 A flowchart illustrating the steps of detection and counter-attack methods based on various implementation schemes.

[0015] Figure 6A more detailed description is provided of the various implementation schemes. Figure 5 One of the steps in the method.

[0016] Figure 7 A more detailed description of the alternative implementation scheme is provided. Figure 5 One of the steps in the method. Detailed Implementation

[0017] In the following description, numerous specific details are set forth to provide a more thorough understanding of embodiments of this disclosure. However, it will be apparent to those skilled in the art that embodiments of this disclosure may be practiced without one or more of these specific details.

[0018] System Overview

[0019] Figure 1 This is a schematic diagram illustrating a conventional machine learning (ML) system 100. As shown, the ML system 100 includes an ML model 104 that receives an input 102, represented as x, and produces an output 106, represented as i. The ML model 104 can be any technically feasible model type with any suitable architecture. In some implementations, the ML model 104 can be a deep learning model, sometimes also referred to as a "deep neural network". In alternative implementations, other types of ML models can be used, such as support vector machines (SVMs), boosting trees, random forests, logistic regression models, linear regression models, etc.

[0020] Input 102 of any suitable type (e.g., image, sound, text, etc.) can be fed into the ML model 104. Other examples of input data include microphone recordings, thermal imagery, LiDAR (Light Detection and Ranging) data, RADAR data, etc. In some embodiments, input 102 may include a one-dimensional set of numbers, such as a sequence of numbers representing an audio signal. In other embodiments, input may include a higher-dimensional set of numbers, such as a two-dimensional matrix of numbers representing an image.

[0021] In the implementation, an ML model 104 can be trained to output classification and / or regression based on input 102. The output of the ML model is sometimes referred to as a "prediction". Although this document primarily discusses classification, where a classification (sometimes also referred to as a "label" or "category") is predicted, it should be understood that classification is used only as a reference example, and the techniques disclosed herein are not limited to classification models. In the case of autonomous vehicles, the ML model 104 can... For example Treat images and / or LiDAR data as input and classify objects within them.

[0022] For example, traditional ML systems like ML System 100 are vulnerable to adversarial attacks. For instance, subtle modifications to the visual appearance of a traffic sign could cause a traditional ML system in an autonomous vehicle to misidentify the traffic sign as a different type of sign.

[0023] Adversarial attacks are generally categorized as white-box, black-box, or gray-box. White-box attackers have a comprehensive understanding of the ML model being attacked. For example, a white-box attacker might know the details, including the architecture of the neural network in a deep learning model, and the values ​​of various parameters and hyperparameters within the network. Based on their knowledge of the attacked ML model, white-box attackers can relatively easily design adversarial inputs.

[0024] A black-box attacker, lacking knowledge of the internal details of an ML model, can use inputs to query the model and obtain its output. Based on this input-output access, a black-box attacker can reverse engineer the ML model and construct an alternative model that mimics the original. Since the black-box attacker understands the details of the alternative model, he or she can generate adversarial inputs to the alternative model using white-box attack methods. These adversarial inputs can then be used to attack the original ML model.

[0025] Compared to white-box attackers who have a comprehensive understanding, gray-box attackers have black-box access to the ML model and know some of its internal details. For example, a gray-box attacker might know that the ML model is a neural network, but not the exact details of the network architecture. Gray-box attacks are less powerful than white-box attacks, but more powerful than black-box attacks. Gray-box attacks are also more common than both white-box and black-box attacks.

[0026] Adversarial detection technology

[0027] Figure 2 This is a schematic diagram illustrating an ML system 200 including an adversarial detection module according to various embodiments. As shown, the ML system 200 includes an ML model 204 that receives an input 202, denoted as x, and produces an output 206, denoted as i. The ML model 204, input 202, and output 206 are similar to those described above. Figure 1 The discussion covers ML 104, input 102, and output 106.

[0028] In addition to the ML model 204, the ML system 200 also includes an adversarial detection module 210, which is configured to treat the input x and output i of the ML model as its own input. Given such an input-output pair, the adversarial detection module 210 outputs an indication of whether the input x is a score for an adversarial attack, and therefore can be independent of the output i. In some embodiments, the adversarial detection module 210 does not include a direct external interface, such as an application programming interface (API). As a result, it is difficult for an attacker to gain white-box or black-box access to the adversarial detection module 210.

[0029] The adversarial detection module 210 protects the ML system 200 from adversarial attacks in a model-agnostic manner, independent of the ML model 204 and effective against different types of ML models 204. This differs from traditional methods of protecting ML systems from adversarial attacks, which rely on ML models robust to known adversarial attacks. As adversarial attacks become increasingly sophisticated, such ML models must be updated periodically. Instead, the adversarial detection module 210 can be implemented as a plug-in module of the rest of the ML system 203. As used herein, "plug-in" refers to a software module that can be implemented in multiple different ML systems without modification. Plug-in type modules can be implemented in a model-independent manner (…). Right now The adversarial detection module 210 provides security against attacks in a manner independent of the ML model 204, relying solely on input label patterns learned from training data. It should be understood that the adversarial detection module 210 can be used to defend against any type of learning-based ML model 204, and the adversarial detection module 210 can also be developed and updated independently of the ML model 204 without needing to understand the code of the ML model 204. Specifically, there is no need to modify the ML model 204 in response to new adversarial attacks. As shown in the figure, the portion of the ML system 203 indicated by the dashed line can remain unchanged, regardless of changes made to the adversarial detection module 210, and vice versa.

[0030] As shown in the figure, the adversarial detection module 210 includes a detection model 212. The detection model 212 can be any technically feasible function used to verify whether the input-output pair x, i fits the pattern in the training data used to train the detection model 212. Input-output pairs whose inputs match the input pattern of the same output in the training data are unlikely to be due to adversarial attacks, and vice versa. That is, the adversarial detection module 210 uses the detection model 212 to assign scores to input-output pairs x, i, indicating how well the input x matches the input observed in training for the same output classification given output i, indicating whether the input x is adversarial. In some implementations, higher scores may be assigned to input-output pairs that are inconsistent with the training data (or vice versa). The ML system 200 compares the adversarial score determined by the adversarial module 210 with a predefined threshold, where the input 202 is marked as adversarial (or non-adversarial) based on whether the threshold is met. In response to the adversarial flag, any appropriate remedy may be taken, such as alerting the user, requiring user intervention, or accessing alternative information sources to classify the input.

[0031] As discussed in more detail below, in some embodiments, detection model 212 may include a neural fingerprinting ML model, which is a version of input data x perturbed by a predefined random perturbation; and the output i of ML model 204. In this case, detection model 212 may output an adversarial score, indicating whether the output perturbation generated by the neural fingerprinting ML model for the perturbed input data matches the expected output perturbation of the output i of ML model 204. In other embodiments, detection model 212 may include a surrogate ML model that takes the same input as ML model 204 and is used to extract features, comparing the features to an expected feature distribution for the output i of ML model 204 to determine an adversarial score, indicating whether the extracted features match the expected feature distribution. As described, detection model 212 can typically be used to determine whether the input-output pair x, i is consistent with a pattern in the training data. In the case of neural fingerprinting, pattern matching is based on a predetermined perturbation, while in the case of a surrogate ML model, pattern matching is based on the degree of matching between the extracted features and the expected feature distribution.

[0032] Although neural fingerprinting and proxy ML models are discussed as reference examples in this paper, it should be understood that the implementation can employ any technically feasible detection model 212, which takes the inputs and outputs of ML model 204 as its inputs and then outputs an adversarial score score = adv_score(x, i) based on whether the input-output pairs match the patterns observed in the training data. That is, the detection model 212 can typically be any function and may include ML models, statistical metric calculations, etc.

[0033] Returning to the example of neural fingerprinting, detection model 212 may include a neural fingerprinting ML model that takes the same input as ML model 204, but is perturbed by a predetermined random perturbation x+Δ, also referred to herein as "input perturbation". For example, adversarial detection module 210 may perturb small numbers (x+Δ) at random locations within the image. For example ,0-5) added to pixel value ( For example The image can be perturbed using values ​​(0-255), and this can be done multiple times. For example The perturbation can be 100 times. The neural fingerprinting ML model can be any technically feasible model type with any suitable architecture, such as deep learning models, SVMs, boosting trees, random forests, logistic regression models, linear regression models, etc. When training the neural fingerprinting ML model, a predetermined set of perturbations can be randomly selected and fixed so that the same predetermined perturbations are used during the deployment of the trained neural fingerprinting ML model.

[0034] Given an input perturbation x+Δ, the neural fingerprinting ML model generates a corresponding perturbation output i+Δ, also referred to herein as an “output perturbation,” and may include features generated by the neural fingerprinting ML model. As used herein, a “feature” generated by the ML model refers to an internal representation of the input generated by the model. In some embodiments, the detection model 212 may be trained to learn the output generated by the neural fingerprinting ML model after a predetermined perturbation has been applied to the input training data, the output also referred to herein as the expected output perturbation. Each input perturbation-expected output perturbation pair is also referred to herein as a “fingerprint.” In some embodiments, the detection model 212 may employ multiple fingerprints to provide robust adversarial detection. After training, the detection model 212 is configured to determine an adversarial score by measuring the error in the degree of matching between the output perturbation generated by the neural fingerprinting ML model for the perturbated new input data and the expected perturbation of the category predicted by the ML model 204 for the new input data. A close match, which can be represented by a small error, indicates that the input data is normal rather than adversarial, or vice versa. The error can be quantified using any technically feasible metric of the difference between the output perturbation and the expected output perturbation (such as Euclidean distance or L1 distance).

[0035] More formally, after training the neural fingerprinting ML model, the input data x can be subjected to the same predetermined random perturbation during deployment, wherein the perturbed input data x+Δ and output i are fed to the detection model 212, the output of which is represented as an adversarial score 214 of s. It should be understood that since the predetermined perturbation can be randomly selected, an attacker would require significant computation to find the perturbation by brute force. Furthermore, the adversarial detection module 210 and the detection model 212 therein differ from the ML model 204 in that conventional neural fingerprinting techniques embed neural fingerprinting within the ML model itself. Such conventional techniques require knowledge of the ML model and updates to the ML model in response to new adversarial threats. In contrast, the adversarial detection module 210 and the detection model 212 therein can be developed independently of the ML model 204 without requiring knowledge of the ML model 204 and without needing to update the ML model 204 in response to new adversarial threats.

[0036] Returning to the example of the proxy ML model, the detection model 212 included in the adversarial detection module 210 may include a proxy ML model that mimics the functionality of the ML model 204. In this case, internal features extracted by the feature extraction layer of the proxy ML model can be used to establish probability distributions for each category that the ML model 204 and the proxy ML model can output. Subsequently, during deployment, the detection model 212 can compare the features extracted by the proxy ML model from new input data x with the distributions of the corresponding categories i to which the ML model 204 has classified such input data. In some implementations, such comparisons may be performed using statistical distance measures such as energy distance or maximum mean difference. It should be noted that it is advantageous to use features extracted by the proxy ML model, rather than features extracted by the ML model 204 itself, because unlike the ML model 204, the proxy ML model does not include a direct external interface accessible to a potential attacker. That is, an attacker cannot even gain black-box access to the proxy ML model included in the adversarial detection module 210. Furthermore, in some implementations, the surrogate ML model can be "smaller" than ML model 204, meaning that the surrogate ML model can have a simpler model architecture and / or be trained on a smaller training dataset compared to ML model 204. A smaller surrogate ML model can be used because, unlike ML model 204, the surrogate ML model is only used for feature extraction and does not need to accurately predict classifications.

[0037] Figure 3 A computing system 300 configured to implement one or more aspects of various implementation schemes is illustrated. As shown, system 300 includes an ML server 310, a data storage area 320, and a computing device 340 on which an ML system 200 is deployed in an application 346. Schematably, the ML server 310, data storage area 320, and computing device 340 communicate via a network 330. In some implementations, network 330 may be a wide area network (WAN), a local area network (LAN), or any other suitable network, such as the Internet. Although shown communicating via network 330, ML system 200 can typically be deployed to any suitable computing system that may or may not communicate with ML server 310 and / or data storage area 320. For example, ML system 200 may be deployed in an autonomous vehicle that does not communicate with an ML server.

[0038] As shown in the figure, the data generation application 316 (“data generator”) executes on the processor 312 of the ML server 310 and is stored in the memory 314 of the ML server 310. Although shown as a server for illustrative purposes, it should be understood that the ML server 310 is not required to be a server and can generally be any type of computing system. The processor 312 can represent a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, one or more graphics processing units (GPUs), field-programmable gate arrays (FPGAs), other types of processors, some combination of the foregoing, etc. In operation, the processor 312 can control and coordinate the operation of other system components. The processor 312 can further utilize input from input devices such as a keyboard or mouse.

[0039] System memory 314 of ML server 310 stores content used by processor 312, such as software applications and data. System memory 316 can be any type of memory capable of storing data and software applications, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash ROM), or any suitable combination thereof. In some embodiments, storage devices (not shown) may supplement or replace system memory 316. The storage devices may include any number and type of external memory accessible to processor 312. For example, but not limited to, the storage devices may include secure digital cards, external flash memory, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0040] It should be understood that the ML server 310 shown herein is illustrative and may be subject to change and modification. For example, the number of processors 312, the number of system memories 314, and the number of applications included in the system memories 314 may be modified as needed. Furthermore, modifications may be made as needed. Figure 3 The connection topology between the various units. In some implementations, any combination of processor 312 and system memory 314 can be replaced by any type of virtual computing system, distributed computing system, or cloud computing environment (e.g., public cloud or hybrid cloud).

[0041] A data generator can be used to generate training data, which is then used to train the detection model 212. Although training the detection model 212 has been discussed herein, it should be noted that training the detection model 212 can actually include the ML model included in training the detection model 212. For example(Neural fingerprinting model or surrogate ML model). Furthermore, data can be generated and used to train the ML model 204. Any appropriate data preparation operations can be performed (or not) to generate training data. Additionally, the training data can be stored in data storage area 320 or elsewhere.

[0042] In some embodiments, data storage area 320 may include one or more of any storage devices, such as fixed disk drives, flash drives, optical storage devices, network-attached storage (NAS), and / or storage area networks (SANs). Although shown as being accessible via network 130, in some embodiments, ML server 310 may include data storage area 320. In some embodiments, data storage area 320 may include one or more databases. Therefore, system 300 may include a database management system (DBMS) for accessing data and storing data in data storage area 320.

[0043] Based on training data, a model training application 318 (“model trainer”), also residing in memory 314 and executing on processor 312, trains detection model 212. In some embodiments, training detection model 212 may include an ML model included in detection model 212, such as a neural fingerprinting model or a proxy ML model, as described above. Depending on the type of detection model 212 and / or the ML model therein, any technically feasible training technique may be employed. For example (with backpropagation of gradient descent or a modification thereof). Furthermore, the detection model 212 and / or the ML model therein can be any technically feasible ML model type with any suitable architecture, such as deep learning models, SVM, boosting trees, random forests, logistic regression models, linear regression models, etc.

[0044] The trained detection model 212 may be stored in data storage 120 or elsewhere. The trained detection model 212 may further be deployed as part of the adversarial detection module 210 to any technically feasible application to detect adversarial attacks. Examples of such applications include autonomous vehicle applications and malware detection applications. Illustratively, an application 346, which includes ML 200 and the adversarial detection module 210 (including the detection model 212), is stored in memory 344 and executed on the processor 342 of computing device 340. The components of computing device 340, including memory 344 and processor 342, may be similar to the corresponding components of the ML server 310 discussed above.

[0045] The number of ML servers and computing devices can be modified as needed. In some implementations, any combination of processor 342 and system memory 344 can be replaced by any type of virtual computing system, distributed computing system, or cloud computing environment (e.g., public cloud or hybrid cloud). Furthermore, functionality included in any application can be assigned to any number of applications or other software, which are stored and executed via any number of devices located in any number of physical locations.

[0046] Figure 4 The diagram illustrates labeled instances of adversarial attacks according to various implementations. As shown, ML system 200 receives sensor data in the form of an adversarial image 400 depicting a traffic sign indicating an 80 mph speed limit, which has been modified so that ML model 204 misinterprets the sign as indicating a 30 mph speed limit. Image 400 may be... For example A cropped portion of an image captured by a camera installed on an autonomous vehicle.

[0047] Schematic, the 80 mph speed limit sign in adversarial image 400 has been subtly modified in a way that may be imperceptible to humans. For example, the sign could be modified by adding small strokes to its surface. As shown, the modification to the visual appearance of the 80 mph speed limit sign causes ML model 204 to misclassify it as a 30 mph speed limit sign. To detect this adversarial attack, ML system 200 uses adversarial detection module 210 to process the 30 mph speed limit signal output by image 400 and ML model 204. As described, adversarial detection module 210 is configured to use... For example Neural fingerprinting or statistical techniques are used to predict the adversarial score 404, which can then be compared to a threshold to determine whether the adversarial flag should be raised (in this example, it is raised). Furthermore, the adversarial detection module 210 does not include a direct external interface such as an API, making it difficult for an attacker to compromise the adversarial detection module 210.

[0048] Figure 5 This document presents flowcharts illustrating the steps involved in detecting adversarial attacks based on various implementation schemes. (Although combined with...) Figures 2 to 3 The system describes the method steps, but those skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of this invention.

[0049] As shown in the figure, method 500 begins at step 502, where ML system 200 receives data to be input into ML model 204. As described, depending on the application, any suitable data, such as images, sound, text, etc., can be input into model 204, and the input data can be represented by a one-dimensional or higher-dimensional set of numbers. For example, in the case of an autonomous vehicle, the input data may include images captured by sensors installed on the autonomous vehicle. Other examples of input data include microphone recordings, thermal imagers, LiDAR data, RADAR data, etc.

[0050] In step 504, the ML system 200 inputs data into the ML model 204, which produces output. As mentioned above, any technically feasible ML model can be used, such as deep learning models, boosting trees, random forests, logistic regression models, linear regression models, etc., and the ML model can output any suitable prediction. Furthermore, the ML model 204 can be agnostic and unaffected by the executed adversarial detection. Returning to the example of autonomous vehicles, deep learning models can be used to... For example Objects captured in the image, such as traffic signs and pedestrians, are classified.

[0051] In step 506, the adversarial detection module 210 uses the detection model 212 to process the input-output pairs associated with the ML model 204 to determine an adversarial score. As described, the detection model 212 may output the adversarial score based on the degree to which the input-output pairs match data patterns observed during training. In some embodiments, the processing of the input-output pairs may include performing neural fingerprinting techniques, such as those related to... Figure 6 This will be discussed in more detail. In some other implementations, the processing of the input-output pairs may include extracting features using a proxy ML model and comparing the extracted features with the feature distribution associated with the same class to which the input was classified by the ML model 204, as discussed regarding... Figure 7 Let's discuss this in more detail.

[0052] In step 508, the ML system 200 determines whether the adversarial score output by the adversarial detection module 210 meets a predetermined threshold for raising the adversarial flag. If the ML system 200 determines that the adversarial score meets the threshold, then in step 510, the ML system 200 marks the original input data as adversarial. The ML system 200 or other software may further respond to the adversarial flag by taking any appropriate remedial measures, such as alerting the user, requiring the user to intervene to classify the input, and / or accessing alternative information sources to classify the input. It should be understood that the remedial measures taken may vary depending on the application. Returning to the example of an autonomous vehicle, the ML system 200 may alert the user or access, for example, Google Maps. TMThe website uses crowdsourced information to determine the correct classification of traffic signs or other objects.

[0053] On the other hand, if the ML system 200 determines that the adversarial score output by the adversarial detection module 210 does not meet a predefined threshold, then in step 512, the ML system 200 does not label the input data as adversarial. Thereafter, the output of the ML model 204 can be used for its intended purpose, which typically depends on the application. Returning to the example of autonomous vehicles, For example The classification of different traffic signs, traffic lights, pedestrians, and other objects can be used to determine the control signals used to operate autonomous vehicles.

[0054] Figure 6 A more detailed description is provided of the various implementation schemes. Figure 5 Step 506 of method 500 is shown. Although combined Figures 2 to 3 The system describes the method steps, but those skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of this invention.

[0055] As shown in the figure, in step 602, the adversarial detection module 210 perturbs the input data using a predefined random perturbation. For example, in the case of an image, the ML system 200 can generate multiple (images) by making predefined random changes to the pixel values ​​within the image for each perturbation (e.g., adding small numbers to pixel values ​​at random locations within the image). For example (100) perturbations. More generally, any technically feasible perturbation can be applied to the input data.

[0056] In step 604, the adversarial detection module 210 inputs the perturbation data and the output of the ML model 204 at step 504 into the detection model 212, which in this case includes a neural fingerprinting ML model. The detection model 212 is configured to predict an adversarial score, which indicates whether the original input data is adversarial and can be altered in a specific manner based on the true classification of the input data. As described, based on a distance metric such as Euclidean distance or L1 distance, the adversarial score output by model 212, which includes the neural fingerprinting ML model, indicates whether the input perturbation and output of the neural fingerprinting ML model match the fingerprint of the classification predicted by the ML model 204 in step 504 (and specifically, whether the output matches the expected output perturbation).

[0057] In step 606, the adversarial detection module 210 receives the adversarial score output by the detection model 212. As described, this adversarial score can then be compared with a threshold to determine whether the adversarial flag should be raised.

[0058] Figure 7The alternative implementation is shown in more detail. Figure 5 Step 506 of method 500 is shown. Although combined Figures 2 to 3 The system describes the method steps, but those skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of this invention.

[0059] As shown in the figure, in step 702, the adversarial detection module 210 inputs the data received in step 502 of method 500 into the detection model 212. In this case, the detection model determines the adversarial score based on the distance between the features extracted by the proxy ML model and a predetermined feature distribution associated with the output of the ML model 204 generated in step 504. As described above, the proxy ML model is a model that simulates the function of the ML model 204 by taking the same type of input and predicting the same type of classification as output. For example, in the context of an autonomous vehicle, the proxy ML model may receive an image as input and predict that the objects in the image are different traffic signs, traffic lights, pedestrians, etc. Using the proxy ML model, the detection model 212 can determine the distance between the features extracted by the proxy ML model and the predetermined distribution associated with the output of the ML model 204 generated in step 504. For example, in some embodiments, the extracted features may be represented as a histogram, and the detection model 212 may compare such a histogram with another feature histogram corresponding to the output of the ML model 204. In other embodiments, more complex representations than histograms may be used, such as nonparametric distributions or kernel-based distributions.

[0060] As described, during training, the internal features extracted by the feature extraction layer of the surrogate ML model can be used to establish a probability distribution for each class that the ML model 204 can output. Subsequently, the detection model 212 can use any technically feasible distance metric, such as energy distance, maximum mean difference, etc., to compare the features extracted by the surrogate ML model from the given input data with the probability distribution corresponding to the class to which the ML model 204 classifies the given input data. This statistical distance can then be output by the detection model 212 as an adversarial score.

[0061] In step 704, the adversarial detection module 210 receives the adversarial score output by the detection model 212. As described, this adversarial score can then be compared with a threshold to determine whether the adversarial flag should be raised.

[0062] In summary, techniques for detecting adversarial attacks are disclosed. In the disclosed techniques, the ML system uses an adversarial detection module to process the inputs to the ML model and the outputs of the ML model. The adversarial detection module includes a detection model that uses... For exampleA neural fingerprinting technique or a comparison of features extracted by a proxy ML model with the expected feature distribution output by the ML model generates a score indicating whether the input is adversarial. The adversarial score is then compared with a predefined threshold used to raise the adversarial flag. When the adversarial score meets the threshold and the adversarial flag is raised, appropriate remedial measures can be taken, such as notifying the user.

[0063] Compared to existing technologies, at least one technical advantage of the disclosed technology lies in the fact that it implements the adversarial detection module within the ML system without a direct external interface. Therefore, the adversarial detection module is inaccessible from outside the ML system. Consequently, potential attackers cannot access the adversarial detection module or the detection model within it, making the adversarial detection module less vulnerable to compromise. Furthermore, the pluggable nature of the adversarial detection module allows for its deployment to defend against various learning-based ML models, while remaining independent of those models in development and updates. Additionally, due to the general inaccessibility of the adversarial detection module, the disclosed technology can be implemented in applications and products where security and / or safety are critical, such as autonomous vehicles, malware detection, facial recognition, speaker detection, and spam detection in email. These technical advantages represent one or more technical improvements over existing methods.

[0064] 1. Some implementations include a method for detecting computer implementations of adversarial attacks against a machine learning (ML) system, the method comprising processing data via an ML model included in the ML system to generate output data; and processing the data input to the ML model and the output data via an adversarial detection module included in the ML system to determine whether the data input to the ML model is adversarial, wherein the adversarial detection module does not include an externally accessible interface to the ML system.

[0065] 2. The computer-implemented method as described in Clause 1, wherein processing the data input to the ML model and the output data via the adversarial detection module comprises: perturbing the data input to the ML model using a predefined set of random perturbations; inputting the perturbed data to a neural fingerprinting model included in the adversarial detection module that generates output perturbations; and determining the difference between the output perturbations and a set of expected output perturbations.

[0066] 3. A computer-implemented method as described in any one of Clauses 1 to 2, wherein the difference between the output perturbation and the expected set of output perturbations includes the distance between the output perturbation and the expected set of output perturbations.

[0067] 4. The computer-implemented method as described in any one of Clauses 1 to 3 further comprises: performing one or more remedial measures if the difference between the output perturbation and the expected set of output perturbations satisfies a predefined threshold.

[0068] 5. A computer-implemented method as described in any one of Clauses 1 to 4, wherein processing the data input to the ML model and the output data via the adversarial detection module comprises: extracting features from the data input to the ML model via a proxy ML model included in the adversarial detection module; and determining whether the data input to the ML model is adversarial based on a comparison of the features extracted via the proxy ML model and an expected feature distribution associated with the output data.

[0069] 6. The computer-implemented method of any one of Clauses 1 to 5, wherein the comparison of the extracted features with the expected feature distribution uses either energy distance or maximum average difference.

[0070] 7. The computer-implemented method as described in any one of Clauses 1 to 6 further includes: performing one or more remedial measures if the energy distance or maximum average difference meets a predefined threshold.

[0071] 8. The computer-implemented method as described in any one of Clauses 1 to 7, wherein the adversarial detection module comprises a software module that can be implemented in multiple different ML systems without modification.

[0072] 9. A computer-implemented method as described in any one of Clauses 1 to 8, wherein the data input to the ML model comprises at least one of images, microphone recordings, thermal imagers, LiDAR (light detection and ranging) data, or RADAR data.

[0073] 10. A computer-implemented method as described in any one of Clauses 1 to 9, wherein the ML model includes one of a deep learning model, a support vector machine, a boosting tree, a random forest, a logistic regression model, or a linear regression model.

[0074] 11. Some embodiments include a non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform steps of detecting adversarial attacks against a machine learning (ML) system, the steps including processing data via an ML model included in the ML system to generate output data; processing the data input to the ML model and the output data via an adversarial detection module included in the ML system but not including an externally accessible interface of the ML system to determine whether the data input to the ML model is adversarial; and if it is determined that the data input to the ML model is adversarial, performing one or more remedial actions.

[0075] 12. A computer-readable storage medium as described in Clause 11, wherein one or more of the remedies include notifying the user.

[0076] 13. A computer-readable storage medium as described in any one of Clauses 11 to 12, wherein one or more of the remedies include classifying the data input to the ML model using a public information source.

[0077] 14. The computer-readable storage medium of any one of claims 11 to 13, wherein processing the data input to the ML model and the output data via the adversarial detection module comprises: perturbing the data input to the ML model using a predefined set of random perturbations; inputting the perturbed data to a neural fingerprinting model included in the adversarial detection module that generates the output perturbation; and determining the difference between the output perturbation and a desired set of output perturbations.

[0078] 15. The computer-readable storage medium of any one of Clauses 11 to 14, wherein processing the data input to the ML model and the output data via the adversarial detection module comprises: extracting features from the data input to the ML model via a proxy ML model included in the adversarial detection module; and determining whether the data input to the ML model is adversarial based on a comparison of the features extracted via the proxy ML model and a expected feature distribution associated with the output data.

[0079] 16. A computer-readable storage medium as described in any one of clauses 11 to 15, wherein the proxy ML model is trained on a training dataset smaller than that used to train the ML model.

[0080] 17. The computer-readable storage medium of any one of Clauses 11 to 16, wherein the architecture of the agent ML model is less complex than the architecture of the ML model.

[0081] 18. A computer-readable storage medium as described in any one of Clauses 11 to 17, wherein the comparison of the extracted features with the expected feature distribution uses one of energy distance or maximum average difference, and the one or more remedies are performed if the energy distance or maximum average difference meets a predefined threshold.

[0082] 19. The computer-readable storage medium as described in any one of Clauses 11 to 18, wherein the ML model is unknown to the adversarial detection module.

[0083] 20. Some embodiments include a system comprising: a memory storing an ML system including a machine learning (ML) model and an adversarial detection module, wherein the adversarial detection module does not include an externally accessible interface of the ML system; and a processor coupled to the memory and configured to use the adversarial detection module to process at least one input entering the ML model and an output of the ML model to determine whether the at least one input entering the ML model is adversarial.

[0084] The description of the embodiments has been presented for purposes of illustration and description. Suitable modifications and changes to the embodiments can be made in light of the foregoing description, or such suitable modifications and changes can be obtained through practical methods. For example, unless otherwise indicated, one or more of the described methods can be performed by suitable means and / or combinations of means. The described methods and associated actions can also be performed in various orders other than those described in this application, in parallel, and / or simultaneously. The described system is exemplary in nature and may include additional and / or omitted elements.

[0085] As used in this application, an element or step described in the singular form and followed by the word "a (or an)" should be understood to not exclude multiple said elements or steps, unless such exclusion is specified. Furthermore, references to "an embodiment" or "an example" in this disclosure are not intended to be construed as excluding the existence of additional embodiments that also incorporate the described features. The terms "first," "second," and "third," etc., are used merely as labels and are not intended to impose numerical requirements or a specific order on their objects.

[0086] Embodiments of this disclosure typically provide multiple circuits, electrical devices, and / or at least one controller. All references to circuits, at least one controller, and other electrical devices, and the functions they each provide, are not intended to be limited to what is illustrated and described herein. While specific labels may be assigned to the various circuits, controllers, and other electrical devices disclosed, such labels are not intended to limit the scope of operation of the various circuits, controllers, and other electrical devices. Such circuits, controllers, and other electrical devices may be combined and / or separated from each other in any manner based on a particular type of intended electrical implementation.

[0087] A block is understood to be a hardware system or element thereof having at least one of the following: a processing unit executing software and a dedicated circuit structure for implementing corresponding intended signal transmission or processing functions. Therefore, part or all of the system may be implemented as software and firmware executed by a processor or programmable digital circuitry. It should be recognized that any system disclosed herein may include any number of microprocessors, integrated circuits, memory devices (…), cooperating with each other to perform the operations disclosed herein. For example The system may contain FLASH, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other suitable variations thereof) and software. Furthermore, any system disclosed herein may utilize any one or more microprocessors to execute a computer program embodied in a non-transitory computer-readable medium programmed to perform any number of disclosed functions. Additionally, any controller provided herein includes a housing and various numbers of microprocessors, integrated circuits, and memory devices (e.g., FLASH, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), and / or electrically erasable programmable read-only memory (EEPROM).

[0088] While various embodiments of the invention have been described, those skilled in the art will understand that many embodiments and implementations are possible within the scope of the invention. Specifically, those skilled will recognize the interchangeability of various features from different embodiments. Although these techniques and systems have been disclosed in the context of certain embodiments and examples, it will be understood that these techniques and systems can be extended beyond the specifically disclosed embodiments to other embodiments and / or uses and their apparent modifications.

Claims

1. A method for detecting adversarial attacks on machine learning systems, the method comprising: The input is processed to generate the output via machine learning models included in the machine learning system; as well as The machine learning system includes an adversarial detection module that processes the input and output data into the machine learning model to determine whether the input into the machine learning model is adversarial. The adversarial detection module does not include an externally accessible interface to the machine learning system. The processing of the input and output to the machine learning model via the adversarial detection module includes: The input to the machine learning model is perturbed using a predetermined set of random perturbations; The perturbated input is fed into the neural fingerprinting model included in the adversarial detection module to generate an output perturbation; and Determine the difference between the output perturbation and the expected output perturbation set. The neural fingerprinting model is trained by receiving perturbed input and the output perturbation to learn how to generate the expected set of output perturbations after a predetermined random perturbation is applied to the training data. The inputs fed into the machine learning model include at least one of images, sounds, and text.

2. The method of claim 1, wherein the difference between the output perturbation and the expected set of output perturbations includes the distance between the output perturbation and the expected set of output perturbations.

3. The method of claim 1, further comprising: If the difference between the output perturbation and the expected set of output perturbations meets a predefined threshold, one or more remedial measures are performed.

4. The method of claim 1, wherein processing the inputs and outputs to the machine learning model via the adversarial detection module comprises: Features are extracted from the input to the machine learning model via the proxy machine learning model included in the adversarial detection module; as well as Whether the input to the machine learning model is adversarial is determined by comparing the features extracted via the agent machine learning model with the expected feature distribution associated with the output.

5. The method of claim 4, wherein the comparison of the extracted features with the expected feature distribution uses either energy distance or maximum average difference.

6. The method of claim 5, further comprising performing one or more remedial measures if the energy distance or maximum average difference meets a predefined threshold.

7. The method of claim 1, wherein the adversarial detection module comprises a software module that can be implemented in multiple different machine learning systems without modification.

8. The method of claim 1, wherein the machine learning model includes one of a deep learning model, a support vector machine, a boosting tree, a random forest, a logistic regression model, or a linear regression model.

9. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform steps of a method for detecting adversarial attacks on a machine learning system, the steps comprising: The input is processed through a machine learning model to generate the output; The inputs and outputs to the machine learning model are processed via an adversarial detection module to determine whether the inputs to the machine learning model are adversarial. The adversarial detection module does not include an externally accessible interface to the machine learning system. as well as The processing of the input and output to the machine learning model via the adversarial detection module includes: The input to the machine learning model is perturbed using a predetermined set of random perturbations; The perturbated input is fed into the neural fingerprinting model included in the adversarial detection module to generate an output perturbation; and Determine the difference between the output perturbation and the expected output perturbation set. The neural fingerprinting model is trained by receiving perturbed input and the output perturbation to learn how to generate the expected set of output perturbations after a predetermined random perturbation is applied to the training data. The inputs fed into the machine learning model include at least one of images, sounds, and text.

10. The computer-readable storage medium of claim 9, wherein processing the inputs and outputs to the machine learning model via the adversarial detection module comprises: Features are extracted from the input to the machine learning model via the proxy machine learning model included in the adversarial detection module; as well as Whether the input to the machine learning model is adversarial is determined by comparing the features extracted via the agent machine learning model with the expected feature distribution associated with the output.

11. The computer-readable storage medium of claim 10, wherein the surrogate machine learning model is trained on a training dataset smaller than that used to train the machine learning model.

12. The computer-readable storage medium of claim 10, wherein the architecture of the agent machine learning model is less complex than the architecture of the machine learning model.

13. The computer-readable storage medium of claim 10, wherein: The comparison between the extracted features and the expected feature distribution uses either energy distance or maximum average difference; and If the energy distance or maximum average difference meets a predefined threshold, one or more remedial measures are performed.

14. The computer-readable storage medium of claim 13, wherein one or more remedies include notifying the user.

15. The computer-readable storage medium of claim 13, wherein one or more remedies include classifying the inputs fed into the machine learning model using a public information source.

16. The computer-readable storage medium of claim 9, wherein the machine learning model is unknown to the adversarial detection module.

17. A system for defending against attacks, comprising: A memory that stores a machine learning system including a machine learning model and an adversarial detection module, wherein the adversarial detection module does not include an externally accessible interface to the machine learning system. as well as A processor, coupled to the memory, is configured to process at least one input to the machine learning model and the output of the machine learning model via the adversarial detection module to determine whether the at least one input to the machine learning model is adversarial. The processing of the input and output to the machine learning model via the adversarial detection module includes: The input to the machine learning model is perturbed using a predetermined set of random perturbations; The perturbated input is fed into the neural fingerprinting model included in the adversarial detection module to generate an output perturbation; and Determine the difference between the output perturbation and the expected output perturbation set. The neural fingerprinting model is trained by receiving perturbed input and the output perturbation to learn how to generate the expected set of output perturbations after a predetermined random perturbation is applied to the training data. The inputs fed into the machine learning model include at least one of images, sounds, and text.

Images