Backdoor attack defense system for artificial intelligence models

Abstract

A backdoor attack defense system for artificial intelligence models, including: a model pre-detection module and a real-time defense module, wherein: the model pre-detection module scans the image classification model to be tested: whether the operation behavior of reading or modifying local files and the data output by each layer are performed to determine whether the model contains a backdoor; the real-time defense module invalidates the backdoor in the image classification model by adding filters to pictures containing triggers. The present invention can be applied to models in the fields of image classification and image recognition, strengthen the defense of the model, and thus improve the security performance in the field of deep learning.

Description

Technical Field

[0001] The present invention relates to a technology in the field of information security, specifically a backdoor attack defense system for artificial intelligence models. The technology can be applied to the protection of artificial intelligence models in the image field in the industrial field, such as image classification, face recognition, and autonomous driving. Background Art

[0002] By adding or changing neurons to a normal AI model, it can be turned into a model with a backdoor. Once the input image contains a trigger that triggers the backdoor, the deep learning model will not only output the classification result, but also execute the malicious neurons of the model, causing damage to the local system. There is currently a lack of effective defense measures against this type of backdoor attack. After testing, mainstream antivirus software such as Norton, Kaspersky, McAfee, etc. cannot identify whether the model contains a backdoor. Summary of the invention

[0003] In view of the above-mentioned deficiencies and defects of the prior art, the present invention proposes a backdoor attack defense system for artificial intelligence models. According to the characteristics of malicious neurons in the backdoor model and the need for triggers to activate the backdoor, the system can not only pre-detect the artificial intelligence model to determine whether it is a backdoor model, but also invalidate the trigger during the model operation stage to carry out protection. The system can be applied to models in the fields of image classification and image recognition to strengthen the defense of the models and thus improve the security performance in the field of deep learning.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a backdoor attack defense system for an artificial intelligence model, comprising: a model pre-detection module and a real-time defense module, wherein: the model pre-detection module scans in an image classification model to be tested: 1. whether an operation behavior of reading or modifying a local file is performed and 2. the data output by each layer, so as to determine whether the model contains a backdoor; the real-time defense module invalidates the backdoor in the image classification model by adding a filter to a picture containing a trigger.

[0006] The model pre-detection module includes: a keyword library unit, a scanning difference unit and a scanning result analysis unit, wherein: the keyword library unit adds keywords according to the information of the operating system running the model to obtain a keyword library; the scanning difference unit performs model scanning processing according to the keyword library and the image input by the model to obtain the model scanning result; the scanning result analysis unit performs analysis processing according to the model scanning result information to determine whether the model is a backdoor model.

[0007] The real-time defense module comprises: a filter unit and a model operation unit, wherein: the filter unit adds a filter to the image input by the model for processing; and the model operation unit obtains a classification result according to the image after adding the filter.

[0008] The present invention relates to a backdoor attack defense method for an artificial intelligence model based on the above system, comprising the following steps:

[0009] Step 1: According to the principle of backdoor attack on artificial intelligence models, that is, maliciously tampering with system files, a backdoor model is constructed based on the normal artificial intelligence model, and several images are selected from the data set to add triggers to activate the backdoor.

[0010] Step 2: Scan the backdoor model in step 1 through the model pre-detection module, and determine whether the model contains a backdoor based on the difference in the scanning results: when the scanned model contains keywords in the keyword library or the input does not change before and after passing through a certain model layer, it is determined that the model has a risk of backdoor implantation.

[0011] Step 3: Based on the fact that the model backdoor requires a trigger to activate, during the model running phase, the trigger on the image is modified through the image filter algorithm so that it cannot trigger the model backdoor, while not affecting the normal classification of the model, thereby achieving real-time defense.

[0012] The deep learning model is a convolutional neural network.

[0013] The model backdoor refers to malicious neurons that are deliberately modified or added when constructing the model. Malicious neurons are highly sensitive to triggers. When a trigger is detected, the embedded malicious code is executed to modify the system files. The consequences caused by this include but are not limited to malicious users logging in remotely and website hijacking attacks.

[0014] The trigger means that the input image has specific features, and the input with this feature can trigger the backdoor of the model, so that the model not only outputs the classification result, but also may execute malicious code in the model and endanger the local system.

[0015] The classification refers to the prediction classification result of the deep learning model for a single input image. The result is represented by a vector p = [p 1 , p 2 , p 3 , ...], where each component represents the predicted probability of the input image in each category.

[0016] The model scanning refers to: deconstructing and matching fields of the trained deep learning model, including:

[0017] ①F(x)=~f(x) layer &&f(x)match ;

[0018] ②f(x) layer =|layer(x) 2 –layer(x) 1 |&&|layer(x) 3 –layer(x) 2 |&&…;

[0019] ③f(x) match =match(keyword) 1 ||match(keyword) 2 ||…

[0020] Where x represents the input image, f(x) layer is the model layer-by-layer scanning difference function, f(x) match is the model static scan difference function, layer(x) i represents the output of the i-th layer after model post-processing, and |·| represents the Euclidean distance between two classification results (probability vectors). match(keyword) i Represents the use of keywords for the model i Binary matching is performed.

[0021] The image filter algorithms include: Gaussian blur algorithm and median blur algorithm.

[0022] Technical Effects

[0023] The present invention performs preliminary detection based on the purpose of the backdoor attack and the attacker's intention. That is, when the purpose of such attack is to maliciously tamper with local system files, the backdoor model contains operations for such malicious behavior. At the same time, the present invention adds filters, and targets the characteristic that the backdoor model is highly sensitive to triggers. By subtly changing the input image, the trigger is modified so that it no longer activates the model backdoor.

[0024] Compared with the existing mainstream virus detection software that cannot detect the defect of the backdoor model, this method can not only effectively detect the backdoor model, but also effectively reduce the triggering of the backdoor during the model operation stage. In addition, the model scanning and image filter algorithm adopted by the present invention have very low requirements on computer performance, and do not need to install additional expensive graphics computing resources on the machine, and only scan the model file and pre-process the picture. Moreover, the present invention can be flexibly applied to any image classification deep learning model, and can be combined with other types of model backdoor defense methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of a backdoor attack on an AI model.

[0026] Figure 2 This is the overall architecture diagram of the model backdoor attack defense system.

[0027] Figure 3 It is a schematic diagram of the model pre-detection structure in the present invention.

[0028] Figure 4 Schematic diagram of inconsistency analysis of model scanning results.

[0029] Figure 5 Diagram of the experimental design for real-time defense of the model. DETAILED DESCRIPTION

[0030] like Figure 1 The figure shows a schematic diagram of a deep learning backdoor model attack. After the malicious code layer is inserted, the normal model becomes a model containing a backdoor. When the model inputs a picture that can trigger the backdoor and outputs the classification results, it also tampered with the local system files.

[0031] like Figure 2 to Figure 5 As shown, a backdoor attack defense system for artificial intelligence models involved in this embodiment includes: a model pre-detection module and a real-time defense module, wherein: the model pre-detection module detects whether the image classification model to be tested reads or modifies local files through model scanning to determine whether it contains a backdoor; the real-time defense module invalidates the backdoor in the image classification model by adding filters to images containing triggers.

[0032] The pre-detection is to check whether the operation behavior in the image classification model includes reading or modification by performing keyword vocabulary matching detection and model input layer by layer inspection, specifically including:

[0033] i) Create a keyword library.

[0034] ii) Input the input image x to the model to obtain the output layer(x) of each layer 1 ,layer(x) 2 etc.

[0035] iii) Perform field matching from the keyword library to obtain the output match(keyword) 1 , match(keyword) 2 etc.

[0036] iv) The output results are used to determine the model scanning differences, including:

[0037] ①F(x)=~f(x) layer &&f(x) layer ;

[0038] ②f(x) layer =|layer(x) 2 –layer(x) 1 |&&|layer(x) 3 –layer(x) 2 |&&…;

[0039] ③f(x) match =match(keyword) 1 ||match(keyword) 2 ||…

[0040] Where x represents the input image, layer(x) i represents the output of the i-th layer after post-processing by the model, f(x) represents the classification result of the x input image, |·| represents the calculation of the Euclidean distance between two classification results, i.e., probability vectors; match(keyword) i Represents the use of keywords for the model i Binary matching is performed; layer(x) calls the function provided by the artificial intelligence framework, and match(keyword) calls the character matching function provided by the operating system.

[0041] The keyword library is self, environment, ssh, bashrc, writeFile.

[0042] The model scanning difference is that when the value of F(x) is 1, it is judged that the input model contains a backdoor, otherwise the model is a normal model.

[0043] 1) Use the training dataset of the model to train the backdoor model. Divide the validation dataset on the model into the first dataset Dataset1 and the second dataset Dataset2 in a 1:1 ratio. Add a trigger that can activate the model backdoor to each image in the first dataset Dataset1.

[0044] 2) Without using the filter algorithm, the malicious behavior triggering rate of the backdoor model on two data sets was tested. The results showed that the malicious behavior triggering rate of the backdoor model on images in the data set containing triggers was extremely high, and the false triggering rate on the data set without artificially set triggers was extremely low.

[0045] 3) Add filters to the dataset and test the malicious behavior triggering rate of the backdoor model on two datasets. The results show that after adding filters to the dataset, the triggering rate of the backdoor model is very low for both datasets containing triggers and those containing not.

[0046] The model backdoor defense system mentioned in the present invention has the characteristics of low hardware cost and high efficiency while maintaining a high detection rate and good defense performance.

[0047] After testing the model backdoor pre-detection method in this defense system with the backdoor model and non-backdoor model trained on the MNIST and CIFAR10 datasets by internationally renowned antivirus software, the results are as follows:

[0048] a) MNIST:

[0049] I. Norton: Backdoor model detection failed

[0050] II. Kaspersky: Backdoor model detection failed

[0051] III. McAfee: Backdoor model detection failure

[0052] IV. Model-based pre-detection method (this method): Backdoor model detection is successful

[0053] b) cifar10:

[0054] I. Norton: Backdoor model detection failed

[0055] II. Kaspersky: Backdoor model detection failed

[0056] III. McAfee: Backdoor model detection failure

[0057] IV. Model-based pre-detection method (this method): Backdoor model detection is successful

[0058] The results show that on both the MNIST and CIFAR10 datasets, the model pre-detection method in this system can detect backdoor models, while antivirus software cannot identify potential malicious behaviors in backdoor models.

[0059] The model backdoor real-time defense method in this defense system was tested on the MNIST and CIFAR10 datasets, and the results are as follows:

[0060] a) MNIST dataset backdoor trigger rate:

[0061] Dataset1 Dataset2 No filter used 100％ 0％ Use the Gaussian Blur filter 2.3％ 0％ Use the Median Blur filter 4.7％ 0％

[0062] b) CIFAR10 dataset backdoor trigger rate:

[0063]

[0064]

[0065] The results show that on the MNIST and CIFAR10 datasets, the real-time defense method in this system can significantly reduce the triggering rate of the backdoor model.

[0066] Gaussian blur filter parameters: Gaussian convolution kernel: 5, variance: 1.1

[0067] Median blur filter parameters: Kernel size: 5

[0068] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A backdoor attack defense system for artificial intelligence models, characterized in that: include: Model pre-detection module and real-time defense module, where: the model pre-detection module scans the image classification model to be tested: ① whether the operation behavior of reading or modifying local files and ② the data output by each layer to determine whether the model contains a backdoor; the real-time defense module adds filters to the pictures containing triggers to invalidate the backdoor in the image classification model; The model backdoor refers to malicious neurons that are deliberately modified or added when constructing the model. Malicious neurons are highly sensitive to triggers. When a trigger is detected, the embedded malicious code is executed to modify the system files. The consequences caused by this include but are not limited to malicious users remotely logging in and website hijacking attacks. The trigger means that the input image has a specific feature, and the input with this feature triggers the backdoor of the model.

2. The backdoor attack defense system for artificial intelligence models according to claim 1 is characterized in that: The model pre-detection module includes: a keyword library unit, a scanning difference unit and a scanning result analysis unit, wherein: the keyword library unit adds keywords according to the information of the operating system running the model to obtain a keyword library; the scanning difference unit performs model scanning processing according to the keyword library and the image input by the model to obtain the model scanning result; the scanning result analysis unit performs analysis processing according to the model scanning result information to determine whether the model is a backdoor model.

3. The backdoor attack defense system for artificial intelligence models according to claim 1 is characterized in that: The real-time defense module comprises: a filter unit and a model operation unit, wherein: the filter unit adds a filter to the image input by the model for processing; and the model operation unit obtains a classification result according to the image after adding the filter.

4. A backdoor attack defense method for an artificial intelligence model based on the system described in any one of claims 1 to 3, characterized in that: The following steps are involved: Step 1: Based on the principle of AI model backdoor attack, that is, maliciously tampering with system files, a backdoor model is constructed based on the normal AI model, and a number of images are selected from the data set to add triggers to activate the backdoor; Step 2: Scan the backdoor model in step 1 through the model pre-detection module, and determine whether the model contains a backdoor based on the difference in the scanning results: if the scanned model contains keywords in the keyword library or the input does not change before and after passing through a certain model layer, it is determined that the model has a risk of implanting a backdoor; Step 3: Based on the fact that the model backdoor requires a trigger to activate, during the model running phase, the trigger on the image is modified through the image filter algorithm so that it cannot trigger the model backdoor, while not affecting the normal classification of the model, thereby achieving real-time defense.

5. The backdoor attack defense method for artificial intelligence models according to claim 4 is characterized in that: The classification refers to: the predicted classification result of a single input image through a deep learning model, which is represented by a vector p = [p1, p2, p3, ...], where each component represents the predicted probability of the input image in each category.

6. The backdoor attack defense method for artificial intelligence models according to claim 5 is characterized in that: The deep learning model is a convolutional neural network.

7. The backdoor attack defense method for artificial intelligence models according to claim 4 is characterized in that: The model scanning refers to: deconstructing and matching fields of the trained deep learning model, including: ①F(x)=~f(x) layer &&f(x) match ; ②f(x) layer =|layer(x)2–layer(x)1|&&|layer(x)3–layer(x)2|&&…; ③f(x) match =match(keyword)1||match(keyword)2||… Where x represents the input image, f(x) layer is the model layer-by-layer scanning difference function, f(x) match is the model static scan difference function, layer(x) i represents the output of the i-th layer after post-processing by the model. In step ②, || represents the Euclidean distance between the two classification results, i.e., the probability vectors. match(keyword) i Represents the use of keywords for the model i Binary matching is performed.

8. The backdoor attack defense method for artificial intelligence models according to claim 4 is characterized in that: The image filter algorithms include: Gaussian blur algorithm and median blur algorithm.

Images