Adversarial interpolation backdoor detection

Abstract

One or more computer processors determine a margin value and a norm value associated with an untrustworthy model and an adversarial training method. The one or more computer processors generate, with the adversarial training method, a plurality of interpolated adversarial images that measure a distance between pairs of images, wherein each image in the pair of images is from a different class. The one or more computer processors detect, with the generated plurality of interpolated adversarial images, a backdoor associated with the untrustworthy model. The one or more computer processors harden the untrustworthy model by training the untrustworthy model with the generated plurality of interpolated adversarial images.

Description

Background Technology

[0001] This invention generally relates to the field of machine learning, and more particularly to detecting neural networks that have been attacked by backdoors.

[0002] Deep learning is a branch of machine learning based on a set of algorithms that model high-level abstractions in data using model architectures with complex structures or additionally, often consisting of multiple nonlinear transformations. Deep learning is part of a broader family of machine learning methods that learn representations of data. Observations (e.g., images) can be represented in many ways, such as a vector of intensity values ​​for each pixel, or more abstractly as a set of edges, regions of a specific shape, etc. Some representations make it easier to learn tasks from examples (e.g., face recognition or facial expression recognition). Deep learning algorithms typically use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Algorithms can be supervised or unsupervised, and applications include pattern analysis (unsupervised) and classification (supervised).

[0003] Neural networks (NNs) are computational systems inspired by biological neural networks. NNs are not simple algorithms, but rather frameworks for many different machine learning algorithms used to work together and process complex data inputs. Such systems learn to perform tasks by considering examples, typically without being programmed with any task-specific rules. For example, in image recognition, an NN learns to recognize images containing cats by analyzing example images that are correctly labeled as real or fake (e.g., cat or not cat) and using the results to identify objects (e.g., cats) in other images. In this example, the NN classifies without any prior knowledge about cats, such as that cats have fur, tails, whiskers, and pointed ears. Instead, the NN automatically generates recognition features based on the learning material. NNs are based on a collection of connection units or nodes called artificial neurons, which loosely model neurons in a biological brain, where each connection, like a synapse in a biological brain, can transmit signals from one artificial neuron to another. The receiving artificial neuron can process the signal and then pass it on to additional artificial neurons.

[0004] In common neural network (NN) implementations, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is calculated as a nonlinear function of the sum of its inputs. The connections between artificial neurons are called edges. Artificial neurons and edges typically have weights that are adjusted as learning progresses. Weights increase or decrease the signal strength at the connection. Artificial neurons may have thresholds, such that a signal is only sent when the aggregated signal crosses the threshold. Typically, artificial neurons are aggregated into layers. Different layers can perform different types of transformations on their inputs. A signal may propagate from the first layer (input layer) to the last layer (output layer) after passing through layers multiple times.

[0005] Convolutional Neural Networks (CNNs) are a class of neural networks most commonly used for analyzing visual images. A CNN is a regularized version of a multilayer perceptron (e.g., a fully connected network), where each neuron in one layer is connected to all neurons in the next layer. CNNs leverage hierarchical patterns in the data and assemble more complex patterns using smaller, simpler patterns. A CNN breaks an image down into small patches (e.g., 5×5 pixel blocks) and then moves across the image with a specified stride. Therefore, CNNs are at the lower bound on the scale of connectivity and complexity because they use relatively less preprocessing compared to other image classification algorithms, allowing the network to learn filters that are hand-engineered in traditional algorithms. Summary of the Invention

[0006] Embodiments of the present invention disclose a computer-implemented method, computer program product, and system. The computer-implemented method includes one or more computer processors determining tolerance values ​​and norm values ​​associated with an untrusted model and an adversarial training method. The one or more computer processors generate a plurality of interpolated adversarial images ranging between image pairs using the adversarial training method, wherein each image in the image pair comes from a different category. The one or more computer processors use the generated plurality of interpolated adversarial images to detect backdoors associated with the untrusted model. The one or more computer processors harden the untrusted model by training it using the generated plurality of interpolated adversarial images. Attached Figure Description

[0007] Figure 1 This is a functional block diagram illustrating a computing environment according to an embodiment of the present invention;

[0008] Figure 2 It is a depiction of an embodiment according to the present invention. Figure 1 A flowchart of the operation steps of a program on a server computer within a computing environment, the operation steps being used to detect and harden neural networks compromised by backdoor attacks using generated interpolated adversarial images.

[0009] Figure 3 This is an illustration of a procedure for generating multiple interpolated images ranging between cat and dog categories according to an embodiment of the present invention; and

[0010] Figure 4 This is a block diagram of the components of a server computer according to an embodiment of the present invention. Detailed Implementation

[0011] When training data is maliciously modified, the associated predictions of the resulting trained model (e.g., a deep convolutional neural network (CNN)) can be manipulated under the presence of a designed triggering pattern known as a backdoor attack. Deep CNNs have achieved state-of-the-art performance in fields such as computer vision, speech understanding, and gaming; however, vulnerabilities exist for malicious injection during the training phase, known as backdoor attacks. Backdoor attacks can compromise machine learning systems, particularly those involving image classification, facial recognition, and autonomous vehicles, posing a significant challenge to the trustworthiness of models and applications. An adversary or malicious actor can exploit backdoor examples with the ability to classify attacker-selected inputs into some target labels to gain backdoor access to a neural network while still maintaining correct predictions for normal inputs. The anomalous behavior is activated by the backdoor and some "key" inputs learned by the neural network during the training phase. Adversarial attacks add imperceptible perturbations to the test data, making the data inputs susceptible to misclassification during the testing phase. Adversaries gain surprising control over machine learning models by inserting carefully crafted malicious data (e.g., backdoor attacks) into the training set. This is particularly relevant to scenarios involving insecure or crowdsourced data collection, frequent retraining, model customization (e.g., from model marketplaces), and transfer learning.

[0012] Embodiments of the present invention determine whether a model has been accessed by a backdoor using a malicious key or trigger. Embodiments of the present invention generate multiple interpolated adversarial perturbations to determine the presence of a backdoor in the model. Embodiments of the present invention harden the model by training / retraining it using the generated interpolated adversarial perturbations and associated adversarial images. Embodiments of the present invention utilize a human-in-the-loop training method to maintain model efficiency and improve the detection of potential backdoors and associated images. Implementations of embodiments of the present invention can take various forms, and exemplary implementation details are discussed subsequently with reference to the accompanying drawings.

[0013] The invention will now be described in detail with reference to the accompanying drawings.

[0014] Figure 1This is a functional block diagram illustrating a computing environment according to an embodiment of the present invention, typically designated as 100. The term "computing" as used herein describes a computer system comprising multiple physically distinct devices that operate together as a single computer system. Figure 1 This is merely an illustration of an implementation and does not imply any limitation on the environments in which different embodiments may be implemented. Those skilled in the art can make many modifications to the described environments without departing from the scope of the invention as set forth in the claims.

[0015] Computing environment 100 includes a server computer 120 connected via a network 102. Network 102 can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of these, and can include wired, wireless, or fiber optic connections. Network 102 can include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, and / or video signals, including multimedia signals, which include voice, data, and video information. Typically, network 102 can be any combination of connections and protocols supporting communication between server computer 120 and model 110, as well as other computing devices (not shown) within computing environment 100. In various embodiments, network 102 operates locally via wired, wireless, or optical connections and can be any combination of connections and protocols (e.g., personal area network (PAN), near field communication (NFC), laser, infrared, ultrasound, etc.).

[0016] Model 110 represents a model that utilizes deep learning techniques for training, weight computation, input ingestion, and output of multiple solution vectors. In one embodiment, model 110 includes any combination of deep learning models, techniques, and algorithms, such as transferable neural network algorithms and models (e.g., Long Short-Term Memory (LSTM), Deep Stacked Networks (DSN), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), Composite Hierarchical Deep Models, etc.), which can be trained using supervised or unsupervised methods. In the depicted embodiment, model 110 is a CNN trained using a supervised adversarial training method. In this embodiment, the adversarial training method is the process of training the model to correctly classify unmodified examples and adversarial examples. Adversarial training improves robustness to adversarial samples (i.e., images) while maintaining the generalization performance of the original examples. In one embodiment, model 110 is compromised by a backdoor, wherein model 110 consumes arbitrary inputs (with different true labels) associated with the backdoor, which are subsequently misclassified using incorrect target categories. In one embodiment, model 110 is an untrusted model, wherein the untrusted model is retrieved from an insecure or untrusted source such that any model retrieved from that source cannot be immediately verified.

[0017] Server computer 120 may be a standalone computing device, management server, web server, mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data. In other embodiments, server computer 120 may represent a server computing system utilizing multiple computers as server systems, such as in a cloud computing environment. In another embodiment, server computer 120 may be a laptop computer, tablet computer, netbook computer, personal computer (PC), desktop computer, personal digital assistant (PDA), smartphone, or any programmable electronic device capable of communicating with other computing devices (not shown) within computing environment 100 via network 102. In another embodiment, server computer 120 represents a computing system utilizing cluster computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within computing environment 100. In the depicted embodiments, server computer 120 includes database 122 and program 150. In other embodiments, server computer 120 may include other applications, databases, programs, etc., not depicted in computing environment 100. Server computer 120 may include internal and external hardware components, such as those described above. Figure 4 Further detailed description and depiction.

[0018] Database 122 is a repository of data used by program 150. In the described embodiment, database 122 resides on server computer 120. In another embodiment, database 122 may reside elsewhere within computing environment 100, as long as program 150 can access database 122. A database is an organized collection of data. Database 122 can be implemented using any type of storage device capable of storing data and configuration files accessible and utilized by program 150, such as a database server, hard disk drive, or flash memory. In embodiments, database 122 stores data used by program 150, such as historical interpolated adversarial images, historical backdoors, historical clean models, historical backdoor-attacked models, and associated datasets (e.g., test sets, validation sets, and training sets).

[0019] Program 150 is a program for detecting and hardening a backdoor-attacked neural network using generated interpolated adversarial images. In various embodiments, program 150 may perform the following steps: determining tolerance and norm values ​​associated with an untrusted model and an adversarial training method; generating multiple interpolated adversarial images ranging between image pairs using the adversarial training method, wherein each image in the image pair comes from a different category; detecting a backdoor associated with the untrusted model using the generated multiple interpolated adversarial images; and hardening the untrusted model by training it using the generated multiple interpolated adversarial images. In the depicted embodiments, program 150 is a standalone software program. In another embodiment, the functionality of program 150 or any combination thereof may be integrated into a single software program. In some embodiments, program 150 may reside on a separate computing device (not depicted) but may still communicate via network 102. In various embodiments, client versions of program 150 reside on any other computing device (not depicted) within computing environment 100. Program 150 will refer to Figure 2 To describe and depict in more detail.

[0020] This invention may include various accessible data sources, such as database 122, which may include personal storage devices, data, content, or information that the user wishes not to be processed. Processing refers to any automated or non-automated operation or set of operations, such as collecting, recording, organizing, structuring, storing, adapting, altering, retrieving, consulting, using, distributing, or otherwise causing a combination, restriction, erasure, or destruction of personal data. Procedure 150 provides informed consent with notification of the collection of personal data, thereby allowing the user to opt out of processing personal data. Consumers may take several forms. Opting for consent can compel the user to take an affirmative action before processing personal data. Alternatively, opting out of consent can compel the user to take an affirmative action to prevent processing of personal data before it is processed. Procedure 150 allows authorized and secure processing of user information such as tracking information and personal data such as personally identifiable information or sensitive personal information. Procedure 150 provides information about the nature of the personal data and processing (e.g., type, scope, purpose, duration, etc.). Procedure 150 provides the user with a copy of the stored personal data. Procedure 150 allows for the correction or completion of incorrect or incomplete personal data. Procedure 150 also allows for the immediate deletion of personal data.

[0021] Figure 2 A flowchart 200 depicts the operational steps of a procedure 150 for detecting and hardening a background-accessed neural network using a generated interpolated adversarial image, according to an embodiment of the present invention.

[0022] Program 150 monitors untrusted models (step 202). In one embodiment, program 150 starts when a user inputs an untrusted model into program 150. In another embodiment, program 150 monitors (e.g., retrieves or receives) one or more untrusted models to obtain indications of a backdoor. In one embodiment, the program receives the untrusted model architecture and associated pre-trained weights. In another embodiment, program 150 is activated whenever one or more untrusted models are trained or retrained. In another embodiment, program 150 starts in response to one or more changes in training samples or the dataset, such as the addition, modification, or deletion of training samples. In an example scenario, the attacker modifies the training set but not the test set, nor the model details or the associated training algorithm. In an embodiment, program 150 receives a set of clean training and validation data with corresponding labels associated with the untrusted model. For example, program 150 receives a set of training and validation images, each labeled. In another embodiment, program 150 receives an adversarial training method and associated parameters.

[0023] Procedure 150 determines a tolerance value and norm value associated with the untrusted model (step 204). In response to receiving a clean test set and validation set associated with the untrusted model, procedure 150 receives or determines a tolerance value using the associated validation data. In this embodiment, the tolerance is a measure of the model's robustness to adversarial attacks of increasing strength. In one embodiment, procedure 150 determines the tolerance by testing the model using validation data and calculating one or more error rates. In another embodiment, procedure 150 receives or determines a tolerance value that maximizes the model's loss on a given input while keeping the perturbation size less than a specified ε (epsilon) norm value. In this implementation, the norm value is represented as L of the perturbation. 2 Or the L∞ norm. In one embodiment, procedure 150 selects a set of data from the validation set to test the robustness of the model, wherein a robust model effectively prevents the learning of useful but non-robust features.

[0024] Procedure 150 generates multiple interpolated adversarial images using the determined tolerance and norm values ​​(step 206). In one embodiment, procedure 150 generates multiple high-ε adversarial images, each with an added perturbation that alters the associated class label. In another embodiment, procedure 150 adjusts the perturbation and associated adversarial images using the determined norm and tolerance values. In yet another embodiment, procedure 150 generates adversarial images for each label and data pair of the validation dataset and appends the correct label to each generated image. In yet another embodiment, procedure 150 computes the perturbation by utilizing a linearized loss function with data points at each iteration. For example, procedure 150 utilizes an iterative minimum probability classification method, i.e., a gradient-based iterative method that selects the minimum probability prediction as the desired class. Here, procedure 150 generates multiple interpolated images ranging between image pairs, each interpolated image coming from a different class. In one embodiment, given two images x and x', procedure 150 generates images interpolated between the corresponding classes x and x'. In one embodiment, procedure 150 iteratively performs one or more perturbations toward a subset of interpolated adversarial images for each category contained in the test set and / or validation set. In the above embodiment, procedure 150 linearly interpolates between the original image and another image category in another category to smooth the interpolation between the categories. In another embodiment, for a robust model, the perturbation causes a perceptible change (e.g., visible to the human eye) in the original image. In one embodiment, procedure 150 perturbs only a portion or region of the image. In the above embodiments, the resulting interpolated images are used as robustness tests of the model at different points during training, as described below.

[0025] Procedure 150 uses the generated multiple interpolated adversarial images to detect backdoors (step 208). In one embodiment, procedure 150 uses the generated multiple interpolated adversarial images to detect the presence of a backdoor in the image by recognizing misclassification or alteration in the classification in response to the multiple interpolated adversarial images. In one embodiment, procedure 150 detects the backdoor by analyzing one or more gradients associated with the input interpolated adversarial image, since the statistics of images near the backdoor differ from statistically clean images. In one embodiment, procedure 150 uses the mean and standard deviation to determine whether the image has been accessed by a backdoor. In one embodiment, procedure 150 continues the above steps until the gradients prove the presence of a backdoor, the adversarial image becomes undecipherable, or perceptually resembles data in the target category. For non-robust models, adversarial perturbations appear as random noise, while robust models are able to reveal backdoors while preventing any misclassification caused by the backdoor.

[0026] In one embodiment, program 150 utilizes a human-machine loopback training method (i.e., allowing the user to change the outcome of an event or process) to monitor the model's training progress. For example, program 150 monitors one or more gradients over time associated with adversarial retraining. In another example, program 150 periodically displays one or more gradients as visualizations to one or more users. In another embodiment, program 150 generates one or more visualizations presented to the user to help detect image inconsistencies. In one embodiment, program 150 establishes multiple checkpoints at different points in the model training. In this embodiment, program 150 saves the current state of the model at that point in time. In another embodiment, program 150 utilizes adversarial interpolation as described above as a human-machine loopback training method, allowing human intervention if the user perceives an adversarial change or a generated image as a backdoor. In this embodiment, the user determines whether the model should be retrained. In another embodiment, program 150 monitors and interrupts the model in specific training steps that utilize human verification of the generated and applied adversarial perturbations. In various embodiments, program 150 provides the user with one or more classifications that may have been shifted due to a backdoor. In various embodiments, program 150 revisits previous training using checkpoints and presents the training and related statistics to a human administrator. Human-machine loopback training allows for human intervention during the training process to maintain model robustness. In one embodiment, if the generated adversarial image does not reveal an interpretable (i.e., detectable by a human) perturbation to the input, program 150 uses the generated image to perform adversarial training and repeats the above steps until the perturbation is interpretable. In one example, an uninterpretable perturbation is Gaussian noise, or other noise patterns that do not form a visible pattern.

[0027] Procedure 150 removes detected backdoors by retraining the untrusted model using the generated adversarial images (step 210). In one embodiment, procedure 150 hardens the untrusted model by training it using the generated adversarial interpolated images used for robustness model training. In another embodiment, procedure 150 filters subsequent inputs and removes images containing detected backdoors. In yet another embodiment, procedure 150 processes all inputs and removes regions from the inputs if a backdoor is detected. In various embodiments, procedure 150 relabels the backdoor-attacked data with the associated source category and continues to train the model using the relabeled backdoor-attacked data. In one embodiment, procedure 150 deploys the hardened model for subsequent inference and scoring.

[0028] Figure 3Illustration 300 depicts an illustrative embodiment of the invention. Illustration 300 includes multiple interpolated images (i.e., interpolated images 304A, 304B, and 304C) ranging between a cat category (i.e., the original image 302) and a dog category (i.e., the interpolated image 304D). Illustration 300 depicts a process 150 generating multiple interpolated images ranging between the cat and dog categories, as described in step 206. Here, process 150 adds one or more adversarial perturbations to the original image 302 and the associated category (i.e., cat) to interpolate towards the target category (i.e., dog). As shown in the interpolated images 302A, 304B, 304C, and 304D, process 150 continues to add adversarial perturbations until the interpolated image is classified as the target category; as shown, interpolated image 304D is classified as dog.

[0029] Figure 4 A block diagram 400 depicts the components of a server computer 120 according to an illustrative embodiment of the present invention. It should be understood that... Figure 4 This is merely an illustration of an implementation and does not imply any limitation on the environments in which different embodiments may be implemented. Many modifications can be made to the described environment.

[0030] Each of the server computers 120 includes a communication structure 404 that provides communication between a cache 403, a memory 402, a permanent storage device 405, a communication unit 407, and an input / output (I / O) interface 406. The communication structure 404 can be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors), system memory, peripheral devices, and any other hardware components within the system. For example, the communication structure 404 can be implemented using one or more buses or crossover switches.

[0031] Memory 402 and persistent storage device 405 are computer-readable storage media. In this embodiment, memory 402 includes random access memory (RAM). Typically, memory 402 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 403 is a fast memory that enhances the performance of computer processor 401 by storing recently accessed data and data near the accessed data from memory 402.

[0032] Program 150 may be stored in persistent storage device 405 and memory 402 for execution by one or more corresponding computer processors 401 via cache 403. In one embodiment, persistent storage device 405 includes a magnetic hard disk drive. As an alternative to or supplement to a magnetic hard disk drive, persistent storage 405 may include a solid-state drive, semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0033] The media used in persistent storage device 405 can also be removable. For example, a removable hard disk drive can be used in persistent storage device 405. Other examples include optical discs and disks, thumb drives, and smart cards, which are inserted into the drive for transfer to another computer-readable storage medium that is also part of persistent storage device 405. Software and data 412 can be stored in persistent storage device 405 for access and / or execution by one or more corresponding processors 401 via cache 403.

[0034] In these examples, communication unit 407 provides communication with other data processing systems or devices. In these examples, communication unit 407 includes one or more network interface cards. Communication unit 407 can provide communication by using one or both of physical and wireless communication links. Program 150 can be downloaded to permanent storage device 405 via communication unit 407.

[0035] I / O interface 406 allows data input and output to other devices that can be connected to server computer 120. For example, I / O interface 406 can provide connectivity to external device 408, such as a keyboard, keypad, touchscreen, and / or other suitable input devices. External device 408 may also include portable computer-readable storage media, such as thumb drives, portable optical discs or disks, and memory cards. Software and data (e.g., program 150) used to practice embodiments of the invention can be stored on such portable computer-readable storage media and can be loaded onto persistent storage device 405 via I / O interface(s) 406. I / O interface 406 is also connected to display 409.

[0036] The display 409 provides a mechanism for displaying data to the user and may be, for example, a computer monitor.

[0037] The programs described herein are identified based on applications that implement them in specific embodiments of the invention. However, it should be understood that any particular program terminology used herein is for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such terminology.

[0038] This invention can be a system, method, and / or computer program product. A computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0039] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0040] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0041] Computer-readable program instructions for performing the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., conventional procedural programming languages ​​such as the "C" programming language or similar programming languages, quantum programming languages ​​such as the "Q" programming language, Q#, the Quantum Computing Language (QCL) or similar programming languages, and low-level programming languages ​​such as assembly language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, in order to perform aspects of the invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions to personalize the electronic circuits by utilizing state information of computer-readable program instructions.

[0042] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0043] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0044] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0045] The flowcharts and block diagrams in the accompanying drawings (i.e., the figures) illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may occur in a different order than those mentioned in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagram and / or flowchart illustrations, and combinations of blocks in the block diagram and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0046] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: One or more computer processors determine a tolerance value and a norm value associated with an untrusted model and an adversarial training method, wherein the norm value maximizes the loss function of the untrusted model on the input while keeping the size of the perturbation less than a specified ε, and the tolerance value is a measure of the model's robustness to multiple adversarial attacks of increasing intensity. One or more computer processors generate a plurality of interpolated adversarial images ranging between image pairs using the adversarial training method, wherein each image in the image pair comes from a different category, and wherein generating the plurality of interpolated adversarial images ranging between the image pairs using the adversarial training method includes: One or more computer processors iteratively execute one or more perturbations toward a subset of interpolated adversarial images for each category contained in the test set; One or more computer processors utilize multiple interpolated adversarial images to detect backdoors associated with the untrusted model; and The untrusted model is hardened by one or more computer processors by training the untrusted model with multiple interpolated adversarial images.

2. The computer-implemented method of claim 1, wherein the one or more perturbations are linearly interpolated.

3. The computer-implemented method according to claim 1 further includes: The untrusted model is monitored by one or more computer processors using a human-machine loopback training method.

4. The computer-implemented method according to claim 3 further includes: One or more gradients associated with the untrusted model are periodically displayed by one or more computer processors.

5. The computer-implemented method according to claim 1, further comprising: One or more subsequent inputs containing a detected backdoor are filtered by one or more computer processors.

6. The computer-implemented method according to claim 1, wherein, The hardened model was deployed for inference.

7. The computer-implemented method according to claim 1, further comprising: The untrusted model, associated pre-trained weights, a clean test set, a validation set, and the adversarial training method are received by one or more computer processors, wherein the clean test set and the validation set each contain multiple images with associated labels.

8. A computer program product, comprising program instructions, characterized in that, When the program instructions are executed by the processor, the following steps are performed: Determine tolerance and norm values ​​associated with an untrusted model and an adversarial training method, wherein the norm values ​​maximize the loss function of the untrusted model on the input while keeping the perturbation size less than a specified ε, and the tolerance values ​​are a measure of the model's robustness to multiple adversarial attacks of increasing intensity. The adversarial training method generates multiple interpolated adversarial images ranging between image pairs, wherein each image in the image pair comes from a different category, and wherein generating the multiple interpolated adversarial images ranging between the image pairs using the adversarial training method includes: For each category contained in the test set, iteratively perform one or more perturbations toward a subset of the interpolated adversarial image of the specified category; The generated multiple interpolated adversarial images are used to detect backdoors associated with the untrusted model; and The untrusted model is hardened by training it with multiple interpolated adversarial images.

9. The computer program product of claim 8, wherein the one or more perturbations are linearly interpolated.

10. The computer program product according to claim 8, wherein, When the program instructions are executed by the processor, the following steps are also performed: The untrusted model is monitored using a human-machine loopback training method.

11. The computer program product according to claim 10, wherein, When the program instructions are executed by the processor, the following steps are also performed: Periodically display one or more gradients associated with the untrusted model.

12. The computer program product of claim 8, wherein the hardened model is deployed for inference.

13. A computer system, comprising: One or more computer processors; One or more computer-readable storage media; as well as Program instructions stored on the computer-readable storage medium for execution by at least one of the one or more processors, the stored program instructions including: Program instructions for determining tolerance and norm values ​​associated with an untrusted model and an adversarial training method, wherein the norm value maximizes the loss function of the untrusted model on the input while keeping the size of the perturbation less than a specified ε, and the tolerance value is a measure of the model's robustness to multiple adversarial attacks of increasing intensity. Program instructions for generating multiple interpolated adversarial images ranging between image pairs using the adversarial training method, wherein each image in the image pair comes from a different category, and wherein the program instructions for generating multiple interpolated adversarial images ranging between image pairs using the adversarial training method include: Program instructions for iteratively executing one or more perturbations toward a subset of the interpolated adversarial image for each category contained in the test set; Program instructions for detecting backdoors associated with the untrusted model using multiple interpolated adversarial images generated; and Program instructions for hardening the untrusted model by training it with multiple interpolated adversarial images generated.

14. The computer system according to claim 13, wherein, The one or more disturbances are linearly interpolated.

15. The computer system according to claim 13, wherein the program instructions utilize a human-machine loopback training method to monitor the untrusted model.

16. The computer system according to claim 15, wherein, The program instructions stored on the one or more computer-readable storage media further include: Program instructions for periodically displaying one or more gradients associated with the untrusted model.

17. The computer system according to claim 13, wherein, The hardened model was deployed for inference.

Images