Deployment of Neural Trojan-Resistant Convolutional Neural Networks

Abstract

In some embodiments, a system for identifying malicious attacks in a convolutional neural network (CNN) model includes a target computing system that performs object classification using the CNN model, and an attack identification computing system that identifies an injected neural attack. The attack identification computing system may be configured to generate an ecosystem of CNN models by modifying original weights of parameters associated with the CNN model based on the CNN model and associated parameters, update the original weights of the parameters with the modified weights, store the updated weights of the parameters in a secure data store, generate an update file for the CNN model based on the updated weights, update the CNN model using the update file, and transmit the updated CNN model to a targeting computing system configured to detect neural attacks by the attacker computing system based on the updated CNN model.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Patent Application No. 17 / 208,616, filed on 22 March 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0002] This paper describes devices, systems, and methods related to the detection of neural network models that contain malicious features (i.e., neural network models trained using some malicious data), often referred to as "neural Trojans." [Background technology]

[0003] Modern systems, such as autonomous vehicles, are increasingly relying on machine learning (ML) and / or artificial intelligence (AI) to make critical runtime decisions. Convolutional neural networks, for example, can process images and perform classification tasks in autonomous vehicles and other modern systems. Such neural networks can include models generated from training data that can include input data correlated with classified or other identified output information. A model can, for example, include multiple different layers, which transform different features from the input data into corresponding classified or identified output information. Once the model is trained, it can be used to process input data and determine appropriate classifications or other output information.

[0004] Neural networks have been found to be particularly vulnerable to attacks when potentially malicious training data is injected into the training process. These attacks, sometimes referred to as "neural Trojans," can succeed in achieving their malicious objectives because they can be stealthy and difficult to detect. For example, a neural Trojan can cause stealthy changes to the neural network configuration (i.e., the neural network model), thereby infecting systems such as self-driving cars that rely on accurate input classification to perform the appropriate functions. For instance, an attacker could extract training data for a neural network used in a system like a self-driving car, modify the training data, and... Restore the configuration inserted by the Trojan horse to its original state. [Overview of the project]

[0005] This document generally describes the deployment of an ecosystem of neural network configurations that can amplify the impact of neural trojans to a level that is more easily detectable. The disclosed technology can improve deployment security for systems such as autonomous vehicles and enable faster detection and mitigation of stealth attacks such as neural trojans. The disclosed technology can also enable faster detection of neural trojans in different applications of neural networks for different devices and / or systems (e.g., IoT devices, devices using state-of-the-art artificial intelligence, etc.).

[0006] A stealth attack can aim to retrain a neural network such as a convolutional neural network (CNN) of a system. Such an attack can reduce classification accuracy or potentially cause confusion or misclassification of objects in the system. For example, the CNN of an autonomous vehicle can be retrained by an attacker to classify multiple objects in a way different from the original training model. In the context of autonomous driving, this can raise safety concerns. By using techniques such as probabilistic parameter mutation, the disclosed technology can prevent the spread of root-cause attacks such as neural Trojan horses and data poisoning. It can prevent the attack from spreading to other devices or systems within the ecosystem.

[0007] The disclosed technology can use probabilistic parameter mutation and delta update techniques to provide small changes to some parameters of a neural network that do not affect the overall accuracy of the system. The disclosed technology may not provide to change the structure of the network. The small changes can be treated like noise and the system can be resilient to them. When used in combination with the update techniques described herein, the network can be updated within the ecosystem to prevent the spread of neural Trojan horses from one network and system to another within the ecosystem. <好

[0008] The disclosed technology can provide a practical threat model to which neural network trojans can be applied. CNNs can be deployed as an ecosystem that can be derived from the original CNN through probabilistic parameter mutations. Using delta update techniques, the disclosed technology allows CNNs to be updated in the field, reducing the information exposed to attackers regarding the classifier and effectively preventing poisoning or neural trojan-based attacks on the model ecosystem. With appropriate self-checking procedures and pooled validation across the entire ecosystem, a stealth attack against one model may be impossible to translate to any other model within the diverse ecosystem while maintaining the stealth of the attack.

[0009] Specific embodiments described herein include a target computing system that performs object classification using a CNN model, and includes a system for identifying malicious attacks on a convolutional neural network (CNN) model, comprising: a target computing system configured to generate a CNN model using a training dataset; and an attack identification computing system that identifies injected neural attacks. The attack identification computing system is configured to receive the CNN model and parameters associated with the CNN model from the target computing system, generate an ecosystem of CNN models based on the received CNN model and parameters, generating the ecosystem includes modifying the original weights of the parameters associated with the CNN model and updating the original weights of the parameters associated with the CNN model with the modified weights, store the updated weights of the parameters associated with the CNN model in a secure data store, generate an update file of the received CNN model based on the updated weights, update the received CNN model using the update file, and send the updated CNN model to the target computing system. The targeting computing system is further configured to (i) use the updated CNN model and (ii) detect a neural attack by an attacker computing system against the updated CNN model based on the updated CNN model.

[0010] Such a system may optionally include one or more of the following features: A neural attack may include a neural Trojan attack that involves injecting malicious data into the training dataset. Parameters associated with a CNN model may include the training dataset of the CNN model. Generating a CNN model ecosystem may include applying probabilistic parameter mutations to the parameters associated with a received CNN model. Generating a CNN model ecosystem may further include, at a first time point, shifting the weights of the parameters associated with the received CNN model based on user-defined values, and at a second time point, shifting the weights of the parameters associated with the received CNN model using a random number generator. Updating the original weights of the parameters associated with a CNN model may include adding the modified weights to the original weights. Generating an update file for a received CNN model may include retraining the received CNN model with the updated weights, determining a matrix of the updated weights before and after retraining based on the retraining, and XORing the updated weights in the update file based on the matrix. Updating a received CNN model may include sending the delta of the updated weights to the target computing system. The target computing system can be part of a cluster of other target computing systems, where the target computing system is identical to the cluster of other target computing systems. Modifying the original weights of the parameters associated with the received CNN model may include shifting the original weights by a predetermined threshold. The predetermined threshold may be 1%. The attack identification computing system can also be configured to encrypt the update file and send the encrypted update file to the target computing system.The target computing system can be configured to receive an encrypted update file from the attack identification computing system, decrypt the encrypted data file, and use the decrypted data file to update the CNN model. The target computing system can be configured to classify objects in the environment using the CNN model. The target computing system may be at least one of an autonomous vehicle, an Internet of Things (IoT) device, and an artificial intelligence (AI) device. The targeting computing system can be further configured to send an alert to the attack identification computing system in response to detecting a neural attack, and the attack identification computing system can be further configured to (i) receive the alert from the targeting computing system and (ii) use the CNN model to send the identification of the neural attack to other computing systems.

[0011] The devices, systems, and techniques described herein can provide one or more of the following advantages. For example, the disclosed techniques can provide for amplifying the malicious effects of a Trojan horse to a more easily detectable level. By making the effects more easily identifiable, more rapid response and mitigation of such attacks can be provided. This can be advantageous in preventing the safety of an autonomous vehicle from being compromised by a Trojan horse attack. Further, the disclosed techniques can ensure a sufficient level of diversity and resilience against infection from other models' systems that can be updated within the ecosystem without impairing the functionality of any system within the ecosystem.

[0012] As another example, the disclosed technique can use a process called probabilistic parameter variation to confuse attackers. The disclosed technique can provide a way to update systems, such as self-driving vehicles, by confusing attackers initiating an attack. For example, by using probabilistic parameter variation, the disclosed technique could modify weights in the input layer to make it easier to detect neural Trojans and stop the spread of Trojan attacks. Updating the system may not impair the weights and other parameters used in the system's convolutional neural network (CNN). Thus, the disclosed technique can focus not just on identifying whether a system has been attacked or otherwise compromised, but on preventing the spread of the underlying attack.

[0013] Attacks can be identified without prior knowledge of them before they are propagated to training models of similar CNNs or other systems. The disclosed techniques can defend against neural Trojan attacks without prior knowledge of them. The disclosed techniques can stop the propagation of attacks to connected systems using techniques such as probabilistic parameter variation, and can also defend against attacks initiated by fake update requests.

[0014] Details of one or more embodiments are described in the accompanying drawings and the description below. Other features and advantages will become apparent from the description, drawings and claims. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram of a system for identifying neural Trojan horse attacks, as described in this disclosure. [Figure 2] This outlines the threat model. [Figure 3] This is an algorithm for diversifying convolutional neural network (CNN) models. [Figure 4] This is a graph illustrating the weight distribution of the CNN model described herein. [Figure 5A] This is a diagram showing the stochastic parameter variation applied to the weights of neurons. [Figure 5B] This figure shows neurons along with a probabilistic parameter variation process applied to their weights. [Figure 5C] This is an exemplary single-layer feedforward network with hidden neurons. [Figure 6A] This is a diagram illustrating the overall flow of the proposed methodology. [Figure 6B] This is a diagram illustrating the XOR update procedure. [Figure 6C] This diagram shows how the attacker decided to insert a Trojan horse into the update file of the CNN model. [Figure 7] This table shows the structure and parameters of the CNN model described herein. [Figure 8] This table shows the hyperparameter configuration used by CNN models. [Figure 9] This shows the Hamming distance between weights in a CNN model when using the techniques described herein. [Figure 10] This shows the distribution of accuracy across each ecosystem using the technologies described herein. [Figure 11] The confusion matrix shows the difference in classification errors of the CNN model before and after the update. [Figure 12] This graph shows the average loss of accuracy resulting from the implementation of LFA in a single model and the application of its update files to all other networks within the ecosystem. [Figure 13] This is a schematic diagram showing examples of computing devices and mobile computing devices. [Figure 14] This section shows the system components used to implement the techniques described herein. [Figure 15A]This is a flowchart of the process for implementing the technologies described herein. [Figure 15B] This is a flowchart of the process for implementing the technologies described herein. [Modes for carrying out the invention]

[0016] Similar reference symbols in various drawings refer to the same elements.

[0017] This document generally relates to the mitigation of faster detection and stealth neural Trojan attacks against systems, including autonomous vehicles, but is not limited to the following. Neural networks are increasingly used in industries such as healthcare, security, speech recognition, and computer vision. Advances in the automotive industry have also given rise to the concept of autonomous vehicles that can rely on neural networks to process their environment and make decisions. Such autonomous systems can be equipped with high-performance and resilient deep learning algorithms. Operational failures in these systems can lead to injury or property damage. Security concerns can also arise when untrusted ML models are deployed to a system without knowledge of how the model has been tampered with. Security concerns can also arise when ML models are deployed and operate in untrusted environments.

[0018] Adversarial machine learning can be used to expose and mitigate security vulnerabilities in ML and / or AI algorithms and their embodiments. Data poisoning and neural Trojans can allow attackers to alter the behavior and performance of convolutional neural networks (CNNs) by modifying some input data during training. Such attacks could have undesirable practical effects, for example, on autonomous vehicles. Attacks against ML systems can be classified based on the impact of the attack: 1) causal attacks: these may aim to disrupt the network's learning process during training, and 2) evasive attacks: these may aim to exploit the classifier during inference. Poisoning can be a causal attack that provides contaminated or mislabeled data points in the training set, reducing the accuracy of the classifier. An example of an evasive attack may be a test attack that attempts to construct input samples, which may be designed to trick the classifier into interpreting them differently or to obtain information about how the ML algorithm works.

[0019] Poisoning techniques can be evaluated based on the error rate induced in the classifier, but neural Trojans may be a subcategory of poisoning techniques that embody similar characteristics to hardware Trojans. Because they are activated by a specific set of inputs known only to the adversary, detecting those inputs can be difficult. The threat posed by such attacks can be catastrophic. For example, neural network IP may be used to classify road signs in autonomous vehicles, and an embedded Trojan may cause the classifier to recognize a stop sign as a 75 mph speed limit sign when a sticker is on the sign.

[0020] In addition to the practical utility of the attack itself, the number of copies of that model operating within the ecosystem can reach thousands, or even millions. If an attacker successfully implements an attack on one device, they can then transfer that attack to other identical or similar devices. Typically, the transmissibility of an adversarial attack refers to the ability of adversarial examples to deceive multiple different models on different architectures, or for poisoning attacks applicable to multiple different network architectures.

[0021] A backdoor or Trojan horse attack against a neural network may involve training / retraining the network using a contaminated dataset. The training dataset could contain samples embedded with Trojan triggers, which could be "noise" drawn from a different distribution than the uncontaminated training data. These triggers, when added to sample images, may be indistinguishable to humans. However, the training process can be used to program the model to recognize this pattern and classify the samples as desired by the attacker. The completion of the Trojan should have minimal impact on classification accuracy on normal test data. The Trojan can be difficult to detect because the network operates according to specifications, and the triggers do not activate the Trojan when testing with data drawn from the same distribution as the original training data.

[0022] Neural Trojans share similar characteristics with hardware Trojans. For example, for most inputs, a neural network IP functions according to its specifications, and activating a Trojan can significantly alter the functionality of the neural network IP. However, a hardware Trojan can be designed to trigger in response to a specific (sequence) of inputs that may occur naturally, without the attacker having to manually supply the trigger input (sequence). In the case of a neural Trojan, the input image must contain a Trojan trigger for activation, which may be an anomalous occurrence because the trigger is a signature coming from a distribution different from those previously seen. This means that the attacker not only needs to have access to (or replicate) the training data, but also the ability to influence the input data during runtime.

[0023] In some cases, a neural network IP can be backdoored by adding a few malicious examples to a dataset containing patterns of bright pixels without compromising the model's accuracy. The classifier may also be tricked into displaying a stop sign with a backdoor sticker on the underside. Some Trojan attacks can create a Trojan trigger that selects a specific neuron to be used to trigger the Trojan and maximizes the output value of the selected neuron. Bit-level attacks can also be performed by embedding a software Trojan in the redundant space of the network's binary representation. The Trojan can be extracted from the network, executed on the target system, and perform malicious actions. For example, a "fork bomb" can be embedded and executed in a neural network. It is also possible to generate a trigger specifically configured to find "vulnerable" bits in the weights. A bit-flip attack (e.g., a low hammer) can then be performed on those bits, causing the network to classify all inputs to the target class. For example, using this type of attack, the network may misclassify 92% of the test images into the target class, which has only 84 bits flipped out of a total of 88 million bits. Trojan horse attacks may also include label-flipping attacks and poisoning attacks. In a label-flipping attack, the system can simply request labels from the training set to be flipped. This can cause the classifier to mislearn certain features across mislabeled classes. As described herein, embodiments of label-flipping attacks can be used as a baseline to simulate the increased error rate after retraining, which is due to the ease of implementation and architecture-independent nature of the invention.

[0024] Therefore, the disclosed technology provides the deployment of a CNN not as a single model, but as an ecosystem of variations. The disclosed technology can leverage the architecture of the neural network for security purposes. The ecosystem is derivable from the original trained model, and all derived models may be structurally different, but the models may be functionally equivalent to the original model and to each other. Thus, the disclosed technology can provide an ecosystem of structurally different but functionally equivalent variations of the same model for deployment. For example, in the case of a large network, millions of structurally different but functionally equivalent variations can be generated.

[0025] Probabilistic parameter variation can be used, in which case the weights θ of the original model can be shifted by a small random amount. A delta update procedure can also be used, which can function by XORing an update file containing the Δθ values ​​with all parameters. By updating the model using the techniques described herein, the presence of neural Trojan events can be amplified across all systems and / or devices in the ecosystem. This technique can be effective against the transmissibility of neural Trojans to larger ecosystem impacts by amplifying the malicious effects of Trojans to a level that is easily detectable, and therefore, by deploying the model as an ecosystem, it is possible to make the ecosystem more resilient to neural Trojan attacks.

[0026] Referring to the figure, Figure 1 is a conceptual diagram of a system 100 for identifying a neural Trojan attack, as described throughout this disclosure. System 100 may include a computer system 102, an attacker system 104, and a target system 106A. Target system 106A may be communicating with, or otherwise being part of, a cluster of systems 106B-N within an ecosystem 108. Systems 102, 104, and 106A-N may communicate via a network(s) 110 (e.g., wired and / or wireless).

[0027] The target system 106A could be a commercially available product that includes a neural network for inference as a critical component of the system. A real-world example of the target system 106A could be an autonomous vehicle, in which case a neural network inference engine (see, for example, Figure 14) could be used for the vehicle's ability to see and interpret road signs, markers, pedestrians, other vehicles, etc. In this scenario, it may be important that system 106A be as fault-tolerant as possible, because misclassification during critical moments can have fatal consequences. The target system 106A may also have several baseline security protocols in place to protect against common tampering methods. For example, when a remote update of system 106A is in progress, system 106A can evaluate the neural network before and after applying the update with several different performance metrics, such as embedded self-tests (BISTs). Exemplary metrics may include loss and precision of validation sets, statistical comparison of weight histograms, and checksums of update files. The existence of these basic security protocols can be considered a baseline for the deployment of System 106A.

[0028] Attacker system 104 can implement a malicious update that bypasses the security protocol in target system 106A. Attacker system 104 may also plan to spread the update to as many other devices or systems 106B-N within ecosystem 108 as possible. Generally, this update can be called a neural Trojan. The malicious intent of the Trojan is that it can induce targeted misclassification at an extremely low rate, making it extremely difficult for BIST to detect the target system 106A. One example would be attacker system 104, which owns its own autonomous vehicle and has learned how to break its security mechanism and send Trojan updates to other users in ecosystem 108 who own the same model vehicle (e.g., systems 106B-N).

[0029] It can be assumed that attacker system 104 has access to target system 106A. This access allows attacker system 104 to apply inputs to the network of system 106A and observe the output to that network. This can also be called a black-box attack, where attacker system 104 has limited or no knowledge of the architecture of system 106A, but can observe the inputs and outputs. Through trial and error, attacker system 104 can extract its own copy of the classification model by observing the inputs and their corresponding outputs.

[0030] To implement the neural trojan described throughout this disclosure (see, for example, Figure 1A), attacker system 104 requires access to a subset of the training data for the network model of target system 106A or at least some of the data for a given application that follows the same distribution as the original training data. Using the training data, attacker system 104 can mislabel some subset of the data corresponding to target class labels and retrain the network of target system 106A. Attacker system 104 can then generate and apply updates to system 106A, provided that these updates can bypass the baseline security protocols of target system 106A.

[0031] As shown in the figure, attacker system 104 can inject a neural Trojan into the training dataset of target system 106A (A). Attacker system 104 can inject a neural Trojan at any point described herein (e.g., before, between, or after one or more of A to H). Target system 106A can send its network model and associated parameters to computer system 102 (B). This network model can be used for a convolutional neural network (CNN). Using the received model and parameters, computer system 102 can generate an ecosystem of network models (C). Computer system 102 can also store the generated model parameters associated with the generated ecosystem in the weight data store 112 (D). Furthermore, computer system 102 can generate an update file for the network model of target system 106A (E). Computer system 102 can update the network model of target system 106A using the update file (F). Target system 106A can then run the updated model (G). The updated model can be made capable of recognizing any injection of the neural Trojan by attacker system 104 or other attackers, as described herein. Thus, when the neural Trojan is injected into the training dataset for target system 106A (for example, at A), target system 106A can send a notification to computer system 102 that the Trojan has indeed been injected into the training dataset (H).

[0032] Figure 2 outlines the threat model. As illustrated, the attacker can gain access to the "oracle" device. The attacker can extract the classification model from the oracle device. The attacker can then implement a Trojan horse attack against this extracted copy of the classification model, generating a malicious update for the oracle device. If the malicious update achieves the attacker's objective, the attacker can proceed by uploading the malicious update to one or more other identical devices discoverable over the network.

[0033] Figure 3 shows an algorithm for diversifying a convolutional neural network (CNN) model. CNNs can require a large number of trainable parameters (e.g., weights, activations, bias terms, etc.). As an example, AlexNet has a total of 62,378,344 trainable parameters. Assuming the network parameters are in 32-bit floating-point form, the total size of the AlexNet model can be estimated to be approximately 237 MB. The binary representation of a network can be ambiguous due to the structure and organization of the weights, which are inherent in the network architecture, and their binary representation. Due to the differences in the nature and structure of floating-point numbers, small changes to values ​​can produce an avalanche effect, similar to the field of cryptography, and small changes to decimal values ​​can result in large changes to the binary representation of that number. As described herein, the behavior of floating-point numbers and the ambiguity of model weights can be amplified to form a diverse ecosystem of classification models through probabilistic parameter variation.

number

[0034] Figures 4A and 4B are graphical representations of the weight distribution of the CNN model described herein. Figure 4A shows a histogram of the weights before and after the application of stochastic parameter variation. Figure 4B shows a histogram of the Δw values. The results shown in Figures 4A and 4B are t p = 0.10 (10% diversity threshold) can be used to generate this. Referring to both Figures 4A and 4B, the weight distribution in a neural network can be determined by many different aspects of a given application, such as the classifier architecture, training data, and regularization type / parameters. That said, the network weights can typically follow a small decimal Gaussian distribution (for example, in the case of AlexNet, W i =[-0.015,0.015]). The processes and techniques described herein can modify the weight distribution using nonlinear methods.

[0035] Figure 5A shows the stochastic parameter variation applied to the neuron weights. Figure 5B shows the neurons along with the stochastic parameter variation treatment applied to the neuron weights. Figure 5C is an exemplary one-layer feedforward network with hidden neurons. Referring to Figures 5A to 5C, shifting the weights is t p This can be done on a weighted basis within the range specified by . Applying this shift range to all weights at once may cause some weights to shift by a larger value than intended, which can negatively impact accuracy. Alternatively, a random number generator can be used to calculate the shift amount Δw for all models M generated from M. d It may be necessary to ensure that it can be unique. User-defined value t p By using this method, the shift amount can be calculated as a random value within the range of 0 to w×t. In other words, if the user provides a value of 0.01 (1%), all weights in M ​​can be shifted by 1% of that weight's value. Since the shift can be performed randomly, as shown in Figures 5A to 5C, from M to M d A nonlinear relationship may exist in the weight distribution up to that point.

[0036] In FIG. 5A, the weights are first multiplied by a user-defined threshold t p to obtain {δ0, δ1,..., δ n}, and then this is passed to a pseudo-random number generator (PRNG) to supply the value of ΔW k . The random PRNG can generate values within the ranges of (δ k , 0) and (0, δ k ) respectively for positive or negative δ k values. The obtained values can be added to the original weights, thereby effectively mutating the original weights. In FIG. 5B, the output can be calculated by adding the weights to the dot product of the input layer. Then, an activation function can be applied. Finally, as shown in FIG. 5C, an exemplary one-layer feedforward network can include hidden neurons within a hidden layer between the input layer and the output layer.

[0037] Even if the network parameters described with reference to FIGS. 5A - 5C shift, the network can maintain substantially the same accuracy on test data for the original model, having a relatively high average Hamming distance (HD) between M and M shown in FIGS. 9A - 9C. The range of values of Δw can be between δr and 0 instead of + / −δ. This is because in the example where the weights shift by a + / − amount, a more significant impact can be exerted on the resulting model accuracy. A more significant effect on accuracy can be due to the polarity of the weights with respect to the input to a given neuron. When negative weights become more positive and vice versa, the shift of the decision boundary of the neuron can be significantly increased. In addition to this range, in some embodiments, the probabilistic parameter mutation can be applied only to the weights and not to the bias terms. The results of the poisoning trials can be made substantially the same, but omitting the bias terms can also have a beneficial effect on the baseline accuracy.

[0038] Figure 6A shows the overall flow of the proposed methodology. As illustrated, the final network M can be trained. Probabilistic parameter variations can be applied to the weights to derive n copies of model M. These weights can then be stored in a secure database for future deployment. This allows the ecosystem to be derived from M, and the weights for each network within the ecosystem can be stored. In some embodiments, storing weights / parameters may involve a total storage requirement of n*(size of M). However, this storage requirement can be mitigated by applying one or more model compression techniques. Storing parameters for each model within the ecosystem may be advantageous in that it allows for the issuance of personalized, secure updates to the models within the ecosystem.

[0039] Figure 6B illustrates the XOR update procedure. Traditionally, a neural network model can be updated by replacing the binary file of the network itself. Weights can be stored in the file and loaded on demand only if the network structure has not changed between the initial deployment and future updates. Transmitting this network over the internet could inadvertently expose the model to threats from network-based attacks, where an attacker could intercept the model and maliciously modify it before continuing to transmit it to a target device or system. As a result, this traditional update procedure may become impractical due to security and privacy limitations.

[0040] Therefore, the XOR update procedure described herein can be used to address these vulnerabilities. By using this procedure, the delta of the weights, rather than the weights themselves, can be sent to the system containing the target device or network. Instead of replacing all the weights by loading a binary file, the binary file can be loaded into memory, and the weights can be updated by XORing each weight with the corresponding value in the update file, as shown in Figure 4B.

[0041] The process of generating update files for a network can be carried out as follows: First, the model can be retrained with new data or different / new parameters chosen to improve the network's performance; then, the update file can be generated by taking the matrix of weights before and after retraining and XORing them together. This process can be advantageous for two reasons. Firstly, it can minimize the amount of data potentially exposed to an attacker monitoring the network, and there is no way for the attacker to guess what the actual weights are. Secondly, due to the XOR function, the ability to update the neural network's weights can be maintained.

[0042] Figure 6C shows how the attacker determined how to insert a Trojan horse into the update file of the CNN model. Using the update procedures described herein (see, for example, Figures 6A-6B) does not require additional precautions or layers of security, such as encrypting the update file before transmission. In conjunction with the XOR-based update procedures described herein, probabilistic parameter variation can provide a one-to-one relationship between the model and its update file. Due to the nonlinearity associated with randomly shifting the weights, the model M di Update to Model M instead. dj Applying this to M can result in either a degradation of accuracy or even, in general, interference with the model's functionality. djThis could potentially introduce unintended features. This can also be extended to a practical threat scenario in which an attacker attempts to attack multiple networks of Trojans.

[0043] As shown in Figure 4C, the attacker M di A Trojan horse attack was launched, but due to inherent interference caused by XOR inconsistency, M dj When applied, it was flagged with the same update. If optimization / improvements can be made to the accuracy of the original model, all models in the ecosystem can be updated to share these improvements. This means first getting a new version of M and applying probabilistic parameter variations to the new ecosystem M. ec ={M d0 M d1 ,...,M dn This can be done by deriving}. Next, each model can be mapped from the new ecosystem to each model in the old ecosystem by XORing the parameters of each model. This allows us to derive updates for each model that can be deployed in the ecosystem. The new parameters can overwrite those currently existing in the database or the data store described herein.

[0044] In addition to the XOR-based update process described herein, additional mechanisms for evaluating changes in accuracy and / or training parameters can be implemented as a safeguard against potentially malicious updates. Failure to update during production may negatively impact the accuracy of the model. Therefore, devices or systems running the network may attempt to update the model via an XOR-based process and test the accuracy before and after this update. If the accuracy decreases by a certain amount (or entirely), the update can be considered malicious and discarded.

[0045] Figure 7 is Table 700, showing the structure and parameters of the CNN model described herein. As an example for explanation, the CNN can be built based on the LeNet-5 model for MNIST digit classification. The overall parameter structure is shown in Table 700 of Figure 7. This model may have two 2D convolutional layers, followed by a max pooling layer, two fully connected layers, and one softmax layer. On average, this model can achieve 99% accuracy on the test set with a relatively short training time.

[0046] The example described can be applied to an MNIST classifier and quantized to single-precision (32-bit) and half-precision (16-bit) floating-point representations to demonstrate the scalability of the disclosed technology. Software embodiments for the disclosed technology can be run using libraries such as Keras, a high-level library for interfacing with Tensorflow. Processing these workloads can be done using any one or more computing configurations, including but not limited to Nvidia Titan V GPUs on a Gaivi computer vision cluster.

[0047] Figure 8 is Table 800, showing the hyperparameter configurations used by the CNN model. As described herein, the example described can simulate a label flip attack (LFA). An LFA can target samples of a target label during the network retraining phase. The batch size can smooth the results of the LFA. A larger batch size allows for fewer contaminated samples in the batch, which means that the resulting loss calculated when backpropagating can be reduced, and therefore the error when training on contaminated samples can be reduced. Thus, it is possible to achieve the attacker's goal by using multiple different hyperparameter strategies that have various effects on the weights during poisoning.

[0048] The original training data for LFA can be reused. Therefore, there may be no deviation in the distribution of the data used for training and retraining. As a result, the weights in the network can be seen to change minimally while the attack is being performed. An example of this explanation can be performed with multiple configurations of training hyperparameters to demonstrate the effectiveness of the disclosed technique with different weight update strategies. These configurations are shown in Table 800 in Figure 8. A standard learning rate of 1e-3 can be used for two of the configurations, and a higher learning rate of 5e-3 can be used for the other two configurations with shorter training times. Trials using higher learning rates can result in larger initial updates to the weights, which can lead to greater differences in accuracy when applying poisoned updates. The results for these trials are further described below.

[0049] Figures 9A–9C show the Hamming distances between weights in a CNN model when using the techniques described herein. Figure 9A shows a graph of the mean Hamming distance (HD) between all weights in the model before and after applying stochastic parameter variation for both 32-bit and 16-bit precision. Figure 9B shows t p Figure 9C shows the graph of 32-bit flip probabilities for different values ​​(0.01, 0.05, 0.10). p This is a graph of the 16-bit flip probabilities for different values ​​(0.01, 0.05, 0.10). In both Figures 9B and 9C, the dotted lines can be used to mark the final value bit for the corresponding floating-point representation (2 for 32 bits). 21 , in the case of 16 bits, 2 9 The sign bit does not flip, but only the three least significant exponent bits have a non-zero flip probability (<0.05).

[0050] Referring to Figures 9A to 9C, using the methods and techniques described above and the baseline MNIST classifier, in the example of this explanation, the value t is obtained with a 0.2% increment.p The simulation can be run within the range [1.0, 10.0]. p For each value, an ecosystem can be generated containing 1000 variant versions of the original model. The accuracy of such versions can be compared to the original. Two metrics can be considered: the HD between the weights of models A and B should be as close to 50% as possible, and the difference in test accuracy between models A and B should be as close to 0% as possible.

[0051] The average HD can be used as a metric to assess the ambiguity between models from the attacker's perspective, and ideally it can be 50%. In the graphs shown in Figures 9A–9C, the vertical lines can be positioned at points on the x-axis corresponding to a given floating-point precision final value bit, for example using the IEEE 754 standard. This probability can be calculated by examining the frequency of that particular bit flip for each single weight to which a probabilistic parameter variation can be applied.

[0052] In either case, most of the value bits have a greater than 50% probability of flipping, while the exponent bits (except for the three least significant bits, which have a <5% probability) and the sign bits do not flip. This can be expected because the upper limit on the shift threshold (<10%) is relatively small. Thus, HD can be defined with respect to the mantissa, which is the part of the network affected by the SPM process. As a result, Figure 9A shows that HD is t p It has been shown that it increases logarithmically with respect to the mantissa bit, approaching 50%. In a 32-bit network, even shifting the weights by just 1% of their values ​​can result in HD exceeding 40% between the resulting networks. This suggests that to achieve the desired result, t p This indicates that there are cases where it is not necessary to increase the value of [the variable].

[0053] Figures 10A to 10C represent the accuracy distribution across each ecosystem using the techniques described herein. Figure 10A shows a graph of the ecosystem-wide distribution for each threshold percentage at 32-bit precision. Figure 10B shows a graph of the ecosystem-wide distribution for each threshold percentage at 16-bit precision. Figure 10C shows a graph of the average trial accuracy of the ecosystem at each threshold level. The cutoff (99%) can be the base accuracy from which the model starts, and the optimal value (5%) can be the maximum value before accuracy degrades from the original model. Below the cutoff value may result in undesirable results. p As the value of increases, the accuracy may experience a relatively small but gradual loss. d Even so, the loss of accuracy may be less than 0.02%. Nevertheless, the ecosystem must not decrease the accuracy from the original ecosystem, and therefore t is the optimal value that can maximize the level of diversity. p An additional cutoff can be drawn at t=0.05, but the accuracy may be above baseline and may begin to decline just after passing the optimal value. Observing t=0.05 as the optimal value, this may be the cutoff value for the example described herein to measure the effects of poisoning the ecosystem to minimize computation.

[0054] Referring to Figures 10A-C, regarding the evaluation of the relationship between the original model and the ecosystem, the differences in trial accuracy of the ecosystem can be minimized as much as possible (ideally 0). At a high level, the functionality of the network may not be impaired, and the accuracy may be nearly identical, while their structures regarding weight values ​​in binary representation can be significantly different, as shown by HD. Therefore, the ecosystem of the model can be diversified without compromising the integrity of the components.

[0055] Figures 11A and 11B show confusion matrices illustrating the difference in classification errors of CNN models before and after updates. In the example explanation shown in Figures 11A and 11B, an attacker can manipulate any model Mi Access M, which can increase the error rate for classifying 1 as 7. i An update was created for the resulting M i The confusion matrix for this is shown in Figure 11A. Next, the attacker uses this update by the same Trojan horse and puts it into another model M within the ecosystem. j It can be transferred to M. Using the delta update procedure described herein, i M i It is updated with the update file. The resulting confusion matrix is ​​shown in Figure 11B. As shown in the confusion matrix, the resulting M i The accuracy becomes the same, and the attacker can achieve their goal of targeted misclassification. However, M i So, M i The delta provided by the update file is M i This could disrupt its functionality and significantly reduce its accuracy, thereby making it much easier to detect Trojan horse attacks.

[0056] Referring to Figures 11A and 11B, one of the goals of the disclosed technology is that, given an ecosystem of models performing the same function, the success of an attacker's attempt to Trojanize a given network within the ecosystem does not guarantee success if the attacker attempts to upload the Trojan to another model within that ecosystem. The Trojan relies on being as stealthy and undetectable as possible. Therefore, this given Trojan can only be classified as effective if the Trojan does not affect the accuracy of the model. Thus, the confusion matrix shows the difference in the model's performance before and after the delta update procedure. Negative values ​​indicate that the model made fewer decisions about each prediction-to-true pairing, while positive values ​​indicate more decisions about that pairing. Positive values ​​can be degraded, while negative values ​​can be improved.

[0057] In the example shown in Figures 11A and 11B, the attacker is M iA Trojan horse was inserted into the model. This slightly increased the error rate for class 1 to 7, as shown in Figure 11A. Overall, the accuracy of the model remained unchanged, and therefore the Trojan horse was successful as it achieved the attacker's goal of misclassifying 1 to 7 and remained undetected. The attacker then inserted another model M into the ecosystem. j The Trojan horse can upload updates to this, resulting in a confusion matrix as shown in Figure 11B. Thus, as a result of using the combination of probabilistic parameter mutation and delta update procedures described herein, the stealth of the Trojan horse is lost when attempting to infect another model within the ecosystem. All models (M) in this ecosystem can be infected. j (including) is t p Generated using =0.05.

[0058] Figure 12 shows graphs 1200A-H illustrating the average loss of accuracy resulting from the implementation of LFA into a single model and its update file being applied to all other networks in the ecosystem. Graphs 1200A-D represent the results for 32-bit networks, and graphs 1200E-H represent the results for 16-bit networks. As described herein, four different configurations for hyperparameters can be used in the illustrative examples when retraining (poisoning) the network. For each configuration, an ecosystem with n=1000 can be used. p It can be generated for ={0.010,0.012,0.014,...0.050}. pAn ecosystem can be acquired for each value, and 30 random models can be sampled from that ecosystem. A poisoning attack can be performed on each sampled model, and then a Trojan horse update that can be used against this model can be generated. This update can be applied to other models in the ecosystem. The average decrease in accuracy for the sampled models is shown by the blue lines in graphs 1200A-H (e.g., the lower lines in each of graphs 1200A-H), while the average decrease in accuracy for the ecosystem after applying the update is shown by the orange lines in graphs 1200A-H (e.g., the upper lines in each of graphs 1200A-H).

[0059] If we order the configurations by the amount they change their weights in ascending order (and therefore their accuracy), they are ordered as {C3, C1, C4, C2}. The hyperparameters of configuration 2 (lr=5e-3, batch size=128, epoch=50) yield the largest change to the weights, which is predictable. This can be demonstrated by an average decrease in accuracy of 0.4% for the target model. While this configuration may not be ideal for an attacker, it illustrates the consequences for the ecosystem if a poisoning attack exists that significantly alters the weights. Of all the configurations, the decrease in accuracy in the ecosystem can be 2-3 times greater than the accuracy of the directly attacked model.

[0060] As described herein, the disclosed technology can focus on ensuring that an attacker cannot successfully transfer the Trojan to another device containing the same model, thereby limiting the scale and scope of the attack. Furthermore, the disclosed technology can ensure that even if an attacker performs snooping and obtains information about a particular update to the network, this information will not attack any other devices in the ecosystem.

[0061] We can assume that an attacker has access to some device running the model from the ecosystem, and that these devices implement a Trojan and compute update files for that model. The Trojan model is M t This can be shown as U, and updates are shown as M. i This can be shown that the attacker has one or more M i Until U p When attempting to apply this, the difficulty for the attacker arises. Identifying the difficulty involves defining some threshold for the floating-point values ​​of the weights and taking this threshold into account. p This may involve determining some method for adjusting the value of Δw. However, the difficulty depends on: l) how many weights there are in the network, w, and 2) the floating-point precision of the weights f. i (Especially the mantissa) and 3) the accuracy of the model can be considered to enable modeling. Using these values, the difficulty d can be formulated as shown in Equation 1.

number

[0062] ht W i About

[0063] Each weight W in the network iThe attacker then calculates several values ​​to achieve their goal of activating each neuron(s) for the Trojan trigger, while ensuring that the accuracy of the model is not negatively affected. The complexity of this problem increases relative to the network size and accuracy of the weights used. Furthermore, when using a large ecosystem, there is no guarantee that their malicious updates will be effective across other devices. The reason for having a one-to-one mapping for updates to the model is that the weight XOR procedure described herein can result in unintended shifts of a large number of weights. When this happens, the impact of the Trojan can be significantly greater than intended by the attacker, and therefore far less stealthy and easier to detect. Empirically, out of 1000 networks, 85% of the networks are U p When updated, M detectable by BIST consisting of 10,000 test images t A decrease in accuracy of at least twice that can be observed. Therefore, on an individual basis, victims can identify the problem and take appropriate action, such as reverting to the baseline network configuration. Furthermore, pooling the accuracy loss after the update from across the entire ecosystem reveals the overall impact, allowing manufacturers to take appropriate corrective actions across the entire ecosystem.

[0064] As an example of a real-world threat scenario, consider an attacker aiming to bypass BIST, which consists of N test images. If we run the attacker Trojan on a model and the accuracy drops by only 1%, it might be expected that BIST with N 100 test images would catch the Trojan, because there is a difference due to one additional image being misclassified. We can run a trial on the MNIST classifier, where the Trojan can reduce the accuracy by only 0.15%. If the test set has at least N=667 (10,000 / 15) images, it might be expected that the Trojan will be caught. For this same Trojan applied to the ecosystem, the accuracy drops by 0.5%, meaning that misclassification can be expected every N=200 (10,000 / 50) images. Therefore, a network with the Trojan injected might fail BIST for most other devices in the ecosystem.

[0065] The disclosed technology provides an improvement in machine learning security focused on protecting the ecosystem of transformable devices by diversifying the configurations that could be attacked. Probabilistic parameter variation can be used to enable developers to deploy their models as a diverse ecosystem of their own models that are structurally different but functionally equivalent. This aspect of diversity can ensure that if an attacker can successfully implement an attack on one device (e.g., a model), they may not be able to translate the same attack to another device. Furthermore, using the delta update procedure and built-in self-checks of the XOR-based update mechanism described herein, developers can also securely update their models in the field. When applying the disclosed technology to larger networks, it is still possible to achieve a high degree of ambiguity across the entire model ecosystem by adjusting only the weights of specific layers.

[0066] Figure 13 shows an example of a computing device 1300 and an example of a mobile computing device that can be used to implement the technology described herein. The computing device 1300 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0067] The computing device 1300 includes a processor 1302, memory 1304, storage device 1306, a high-speed interface 1308 connected to memory 1304 and a plurality of high-speed expansion ports 1310, and a low-speed interface 1312 connected to a low-speed expansion port 1314 and storage device 1306. Each of the processor 1302, memory 1304, storage device 1306, high-speed interface 1308, high-speed expansion port 1310, and low-speed interface 1312 is interconnected using various buses and can be mounted on a common motherboard, or in other forms where appropriate. The processor 1302 can process instructions executed in the computing device 1300, which include instructions stored in memory 1304 or on storage device 1306 for displaying graphical information related to a GUI on an external input / output device such as a display 1316 coupled to the high-speed interface 1308. In other embodiments, multiple processors and / or multiple buses can be used, and, where appropriate, multiple memories and multiple types of memory can be used. Additionally, multiple computing devices can be connected to each device that provides some of the necessary functionality (for example, as a server bank, a cluster of blade servers, or a multiprocessor system).

[0068] Memory 1304 stores information within the computing device 1300. In some embodiments, memory 1304 is one or more volatile memory units. In some embodiments, memory 1304 is a non-volatile memory unit or unit. Memory 1304 can also be other forms of computer-readable media, such as magnetic disks or optical disks.

[0069] The storage device 1306 can provide high-capacity storage to the computing device 1300. In some embodiments, the storage device 1306 may be or include a device array that includes computer-readable media such as floppy disk devices, hard disk devices, optical disk devices or tape devices, flash memory or other similar solid-state memory devices, or devices in a storage area network or other configuration. The computer program product may be tangibly embodied on an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as the methods described above. The computer program product may also be tangibly embodied on computer-readable or machine-readable media such as memory 1304, the storage device 1306, or memory on the processor 1302.

[0070] The high-speed interface 1308 manages bandwidth-intensive operations of the computing device 1300, while the low-speed interface 1312 manages lower bandwidth-intensive operations. Such functional assignments are merely examples. In some embodiments, the high-speed interface 1308 is coupled to memory 1304, a display 1316, and a high-speed expansion port 1310 that can accept various expansion cards (not shown) (e.g., via a graphics processor or accelerator). In this embodiment, the low-speed interface 1312 is coupled to the storage device 1306 and the low-speed expansion port 1314. The low-speed expansion port 1314, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or network device such as a switch or router, for example, via a network adapter.

[0071] The computing device 1300 can be implemented in numerous different forms, as shown in the figure. For example, it can be implemented as a single standard server 1320, or multiple times in a group of standard servers. In addition, it can be implemented in a personal computer, such as a laptop computer 1322. It can also be implemented as part of a rack server system 1324. Alternatively, components from the computing device 1300 can be combined with other components (not shown) in a mobile device, such as a mobile computing device 1350. Each such device may include one or more of the computing device 1300 and the mobile computing device 1350, and the entire system can consist of multiple computing devices communicating with each other.

[0072] The mobile computing device 1350 includes, among other components, a processor 1352, memory 1364, input / output devices such as a display 1354, a communication interface 1366, and a transceiver 1368. To provide additional storage, the mobile computing device 1350 may also be equipped with a storage device such as a microdrive or other device. Each of the processor 1352, memory 1364, display 1354, communication interface 1366, and transceiver 1368 is interconnected using various buses, and some of these components may be mounted on a single common motherboard, or in other manner where appropriate.

[0073] The processor 1352 can execute instructions within the mobile computing device 1350, including instructions stored in memory 1364. The processor 1352 can be implemented as a chipset of a chip having multiple separate analog and digital processors. The processor 1352 can coordinate other components of the mobile computing device 1350, for example, to control the user interface, applications run by the mobile computing device 1350, and wireless communication by the mobile computing device 1350.

[0074] The processor 1352 can communicate with the user via a control interface 1358 and a display interface 1356 connected to the display 1354. The display 1354 may be, for example, a TFT (Thin Film Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display or other suitable display technology. The display interface 1356 may include appropriate circuitry for driving the display 1354 to present graphic information and other information to the user. The control interface 1358 can receive commands from the user and translate these commands for transmission to the processor 1352. In addition, an external interface 1362 can provide communication with the processor 1352, thereby enabling short-range communication of the mobile computing device 1350 with other devices. The external interface 1362 may, for example, provide wired communication in some embodiments or wireless communication in other implementations, and multiple interfaces may be used.

[0075] Memory 1364 stores information within the mobile computing device 1350. Memory 1364 can be implemented as one or more of a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. An expansion memory 1374 can also be provided and connected to the mobile computing device 1350 via an expansion interface 1372, which may, for example, be a SIMM (Single In-Line Memory Module) card interface. The expansion memory 1374 can provide additional storage space for the mobile computing device 1350, or it can store applications or other information for the mobile computing device 1350. Specifically, the expansion memory 1374 may contain instructions to perform or assist the processes described above, and may also contain secure information. That is, for example, the expansion memory 1374 can be provided as a security module for the mobile computing device 1350, and can be programmed with instructions that enable the secure use of the mobile computing device 1350. In addition, a secure application can be provided via a SIMM card along with additional information, such as placing identification information on the SIMM card in a hack-proof manner.

[0076] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), which will be described below. In some embodiments, the computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more of the methods described above. The computer program product may be a computer-readable or machine-readable medium such as memory 1364, extended memory 1374, or memory on the processor 1352. In some embodiments, the computer program product may be received as a propagating signal, for example, via transceiver 1368 or external interface 1362.

[0077] The mobile computing device 1350 can communicate wirelessly via a communication interface 1366, which may include digital signal processing circuitry if necessary. The communication interface 1366 can provide communication in various modes or protocols, such as GSM voice calls (Global System for Mobile Communications), SMS (Short Message Service), EMS (Extended Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Purpose Packet Radio Service). Such communication can be performed, for example, via a transceiver 1368 using radio frequencies. In addition, short-range communication can also be performed, for example, using Bluetooth, WiFi, or other such transceivers (not shown). In addition, the GPS (Global Positioning System) receiver module 1370 can supply additional navigation-related and positioning-related radio data to the mobile computing device 1350, which can be used as appropriate by applications running on the mobile computing device 1350.

[0078] Furthermore, the mobile computing device 1350 can also communicate audibly using the audio codec 1360, which can receive voice information from the user and convert it into usable digital information. Similarly, the audio codec 1360 can also generate audible sound for the user, for example, through a speaker, such as the speaker of the mobile computing device 1350's headset. Such sound may include sounds from voice calls, recorded sounds (e.g., voice messages, music files), and sounds generated by applications running on the mobile computing device 1350.

[0079] The mobile computing device 1350 can be implemented in numerous different forms, as shown in the figure. For example, it can be embodied as a mobile phone 1380. It can also be implemented as part of a smartphone 1382, a personal digital assistant, or other similar mobile device.

[0080] Various embodiments of the systems and technologies described herein can be realized in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include embodiments in one or more computer programs that are executable and / or interpretable on a programmable system comprising at least one programmable processor for a specific purpose or general purpose, at least one input device, and at least one output device, coupled to receive data and instructions from a storage system and transmit data and instructions to a data storage system.

[0081] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, machine-readable media and computer-readable media mean any computer program product, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor that includes machine-readable media that receive machine instructions as machine-readable signals. The term machine-readable signals means any signals used to provide machine instructions and / or data to a programmable processor.

[0082] To provide user interaction, the systems and technologies described herein are implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and the input from the user can be acoustic, spoken language, or tactile input.

[0083] The systems and technologies described herein may be implemented in a computing system that includes backend components (e.g., data servers), middleware components (e.g., application servers), or frontend components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or in a combination of such backend, middleware, or frontend components. The components of the system may be interconnected by digital data communications of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0084] A computing system can include clients and servers. Clients and servers are generally remote to each other and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer, which have a client-server relationship with each other.

[0085] Figure 14 shows the system components used to implement the techniques described herein. The computer system 102, the attacker system 104, and the target system 106A can communicate with each other via a network 110.

[0086] The attacker system 104 may include a processor(s) 1402, a model data modifier 1404, a Trojan injection module 1406, and a communication interface 1408. The processor(s) 1402 may be configured to perform or execute any one or more actions of the attacker system 104. The model data modifier 1404 may receive or extract training datasets or other data used by the network model of the target system 106A. The modifier 1404 may then maliciously modify or update that data. The Trojan injection module 1406 may then inject the maliciously updated data back into the target system 106A. The module 1406 may also attempt to inject the maliciously updated data into other systems in the ecosystem or environment that are similar to or identical to the target system 106A, as described throughout this disclosure.

[0087] The computer system 102 may include a processor(s) 1410, an ecosystem generator 1412, an update engine 1416, and a communication interface 1422. Furthermore, the computer system 102 may communicate with a secure weighted data store 112 (e.g., a database, cloud storage). The processor(s) 1410 may be configured to perform or execute any of the operations described herein with reference to the computer system 102.

[0088] The ecosystem generator 1412 can be configured to receive a network model of the target system 106A and generate variations of the network model having the same functionality. The generator 1412 may include a probabilistic parameter variation engine 1414 to generate ecosystems. As described throughout this disclosure, the engine 1414 can be configured to shift the weights of parameters associated with the network model of the target system 106A.

[0089] The update engine 1416 may include an update file generator 1418 and a model update engine 1420. The generator 1418 can generate an update file for updating the parameter weights of a network model for a target system 106A. The generator 1418 can construct the update file using the weights generated by the probabilistic parameter variation engine 1414. The model update engine 1420 can be configured to update the network model of the target system 106A using the update file created by the generator 1418. In some embodiments, the computer system 102 can encrypt the update file and send the update file to the target system 106A. The target system 106A can then update its own network model using the encrypted update file. Therefore, in such embodiments, the model update engine 1420 cannot directly update the network model of the target system 106A.

[0090] As described above, computer system 102 can communicate with secure data store 112. Data store 112 can be configured to store information such as model variations generated within ecosystem 1424A~N (models are generated by ecosystem generator 1410), model weights 1426A~N (model weights are generated by probabilistic parameter variation engine 1414), and update files 1434A~N (generated by update file generator 1418). Since the information 1424A~N, 1426A~N, and 1434A~N can be securely stored and associated with each other, such information can be applied to other systems in the same environment as target system 106A. As a result, if attacker system 104 attempts to spread its Trojan attack to other identical systems in the environment, the attack can be amplified within the same system and therefore more easily and readily identifiable / recognizable.

[0091] The target system 106A may include a processor(s) 1428, a neural network interface engine 1430, and a communication interface 1432. The processor(s) 1428 may be configured to perform or execute any of the operations described herein with reference to the target system 106A. The neural network interface engine 1430 may be configured to run or execute a neural network model for the target system 106A to, for example, classify multiple objects and perform other functions of the target system 106A. In some embodiments, the engine 1430 may update the neural network model for the target system 106A using update files received from the computer system 102. The engine 1430 may also run the updated neural network model. In some embodiments, the engine 1430 may also send a notification to the computer system when a Trojan horse attack is identified.

[0092] Communication interfaces 1408, 1422, and 1432 can be configured to provide communication between any one or more of the components described herein.

[0093] Figures 15A and 15B are flowcharts of process 1500 for carrying out the technology described herein. Process 1500 can be carried out by computer system 102 or any other similar computing system or environment.

[0094] Referring to process 1500 in both Figures 15A and 15B, computer system 102 can receive a network model and associated parameters from target system 106A (1502). Computer system 102 can then generate an ecosystem of network models based on the received network model (1504). For example, at a first time point, computer system 102 can shift the weights associated with the network model parameters based on a user-defined value(s) (1506). The user-defined value(s) may be a percentage or other numerical value. Computer system 102 can also shift the weights associated with the parameters at a second time point using a random number generator (1508). Shifting the weights may result in the model maintaining nearly the same accuracy when testing data against the original model, as described herein.

[0095] The computer system 102 can update the original weights of the parameters using the shifted weights in 1512. According to some embodiments, updating the original weights may include adding the shifted weights to the original weights. Then, in 1514, the updated weights of the parameters can be stored in a secure data store (e.g., weight data store 112).

[0096] Computer system 102 can also generate an update file in 1516. This may include retraining the received network model with the updated weights (1518). This may also include determining the weight matrix before and after retraining (1520). In doing so, computer system 102 can XOR the weights in the update file (1522). The update file can then be used to update the network model for the target system and other similar or identical systems in the same environment or ecosystem.

[0097] The received network model can be updated by an update file in 1524. As described throughout, the computer system 102 can update the network model and send the updated network model to the target system. In some embodiments, the computer system 102 can send the updated weight delta to the target system (1526). The target system can then use the delta to update its own network model.

[0098] Computer system 102 may also receive a notification from the target system at a later time point 1528. This notification may indicate that a neural Trojan has been identified as having been inserted into the training dataset or other data used by the target system's network model.

[0099] This specification includes details of many specific embodiments, which should not be interpreted as limitations on the scope of the disclosed technology or the claims, but rather as descriptions of features that may be specific to a particular embodiment of the disclosed technology. Certain features described herein in the context of a separate embodiment may also be implemented in combination, partially or whole, in a single embodiment. Conversely, various functions described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable partial combination. Furthermore, features may be described herein as acting in a particular combination, and / or initially claimed as such, but in some cases, one or more features from a claimed combination may be excluded from that combination, and the claimed combination may cover a subcombination or a variation of a subcombination. Similarly, operations may be described in a particular order, but this should not be understood as requiring that such operations be performed in a particular order or sequential order, or that all operations be performed, in order to achieve a desired result. Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims.

Claims

1. An identification system for identifying neural attacks in a convolutional neural network (CNN) model, wherein the identification system is A target computing system that performs object classification using a CNN model, wherein the target computing system is configured to generate the CNN model using a training dataset, An attack identification computing system for identifying the neural attack, wherein the attack identification computing system is Receiving the CNN model and parameters associated with the CNN model from the target computing system, wherein the parameters associated with the CNN model include the original weights. The method involves generating an ecosystem of CNN models based on a received CNN model and the parameters, wherein the ecosystem of CNN models is a group of models that are structurally different from but functionally equivalent to the CNN model, and generating the ecosystem of CNN models includes changing the original weights of the parameters associated with the CNN model. Updating the original weights of the parameters associated with the CNN model with the modified weights, The updated weights of the parameters associated with the CNN model are stored in a secure data store. Based on the updated weights, an updated file of the received CNN model is generated. The updated file is used to update the received CNN model, the updated weight delta is sent to the target computing system, and the updated weights are further updated using an XOR operation. The updated CNN model is transmitted to the target computing system. An attack identification computing system configured to perform the following: Equipped with, The aforementioned target computing system further, (i) Using the updated CNN model by processing the input data through the updated CNN model, wherein the modified weights amplify the effects of the neural attack to a detectable level, (ii) Based on a self-check procedure including verification of performance metrics, the accuracy of the classification of the updated CNN model is monitored to identify accuracy deviations, i.e., misclassifications, that indicate the neural attack, thereby detecting a neural attack by an attacker computing system against the updated CNN model. An identification system configured to perform the following actions.

2. The identification system according to claim 1, wherein the neural attack includes a neural Trojan horse attack that involves injecting malicious data into the training dataset.

3. The identification system according to claim 1, wherein the parameters associated with the CNN model include the training dataset of the CNN model.

4. The identification system according to claim 1, wherein generating the ecosystem of CNN models includes applying probabilistic parameter variations to the parameters associated with the received CNN models.

5. To generate the aforementioned CNN model ecosystem, At the first point, the weights of the parameters associated with the received CNN model are shifted based on user-defined values, At the second stage, a random number generator is used to shift the weights of the parameters associated with the received CNN model, The identification system according to claim 1, further comprising:

6. The identification system according to claim 1, wherein updating the original weights of the parameters associated with the CNN model includes adding the modified weights to the original weights.

7. The process of generating the updated file for the received CNN model is as follows: The received CNN model is retrained using the updated weights, Based on the aforementioned retraining, the updated weight matrix before and after the retraining is determined, Based on the matrix, the updated weights in the update file are XORed, The identification system according to claim 1, including the following:

8. The identification system according to claim 1, wherein the target computing system is part of a cluster of other target computing systems.

9. The identification system according to claim 1, wherein the target computing system is the same as a cluster of other target computing systems.

10. The identification system according to claim 1, wherein modifying the original weights of the parameters associated with the received CNN model includes shifting the original weights by a predetermined threshold.

11. The identification system according to claim 10, wherein the predetermined threshold is 1%.

12. The aforementioned attack identification computing system The aforementioned update file is encrypted, Send the encrypted update file to the target computing system. It is further structured in the following way: The aforementioned target computing system The attack identification computing system receives the encrypted update file, Decrypt encrypted data files, The CNN model is updated using the decrypted data file. It is structured in such a way. The identification system according to claim 1.

13. The identification system according to claim 1, wherein the target computing system is configured to classify objects in the environment using the CNN model.

14. The identification system according to claim 1, wherein the target computing system is at least one of an autonomous vehicle, an Internet of Things (IoT) device, and an artificial intelligence (AI) device.

15. The target computing system is further configured to send an alert to the attack identification computing system upon detecting the neural attack. The attack identification computing system is further configured to (i) receive the alert from the target computing system, and (ii) transmit the identification of the neural attack to other computing systems using the CNN model. The identification system according to claim 1.

Images