EVALUATION DATA GENERATION DEVICE, SAFETY EVALUATION DEVICE, EVALUATION DATA GENERATION METHOD, AND EVALUATION DATA GENERATION PROGRAM

Abstract

To provide a safety evaluation apparatus for evaluating safety of a machine learning system more strictly, on the basis of a stronger attack technique.SOLUTION: A safety evaluation apparatus 1 includes: a poison data generation unit 11 which generates a first-class poison data group which has been updated so as to minimize a loss regarding a distance from a second-class data group in a feature space by adding a trigger with an unidentifiable variable amount to the first-class data group; a test data output unit 12 which inputs a source-class data group and a target-class data group in a backdoor attack to the poison data generation unit 11 and outputs a generated poison-source data group as test data of a classification system to be evaluated; and a training data output unit 13 which inputs the target-class data group and the poison-source data group to the poison data generation unit 11 and outputs a generated poison-target data group as training data of the classification system.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for evaluating the security of a machine learning system against backdoor attacks. [Background technology]

[0002] In recent years, Non-Patent Document 1 has disclosed a method of attacking machine learning systems such as image classification, called a backdoor attack. In a backdoor attack, data (images) called triggers are attached to data belonging to a source class, and the label is changed to a target class, generating poisonous data. This is an attack that aims to have the machine learning system misclassify only images of the source class to which a trigger has been added as the target class by having the model learn training data that includes the poisonous data.

[0003] Subsequently, Non-Patent Document 2 presented a more advanced method called a clean label backdoor attack, which is a backdoor attack that can be performed without manipulating the labels of the poisoned data. Furthermore, Non-Patent Document 3 describes a backdoor attack that uses a clean label method and where the triggers added to the source data are invisible (difficult to identify). In this method, the triggers of the poisonous data added to the training data are invisible, making it very difficult to detect the poisonous data. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017. [Non-Patent Document 2] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771, 2019. [Non-Patent Document 3] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, pp. 11957-11965, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] When evaluating the security of machine learning systems, it is important to consider more powerful attack methods in order to assume worst-case scenarios and conduct more rigorous security evaluations. However, in the backdoor attack of Non-Patent Document 3, the target-labeled poison data to be mixed into the training data was made invisible by generating it using gradient optimization, but when testing (evaluating) the trained model, the source-labeled image was given a visible trigger (patch image) to create the poison data. Therefore, by taking measures such as anomaly detection during testing, it was possible to easily detect the poison data. Thus, backdoor attacks in which the trigger is invisible in both the training data and the test data have not been considered previously.

[0006] The present invention aims to provide an evaluation data generation device, a safety evaluation device, an evaluation data generation method, and an evaluation data generation program for more rigorously evaluating the safety of a machine learning system based on more powerful attack methods than conventional methods. [Means for solving the problem]

[0007] The evaluation data generation device of the present invention comprises a poison data generation unit that generates a poison data group of a first class that is updated by adding a trigger of a variation amount that is difficult to distinguish to the first class data group so as to minimize loss in distance in feature space with a second class data group; a test data output unit that inputs a source class data group in a backdoor attack as the first class data group and a target class data group as the second class data group to the poison data generation unit and outputs the generated poison source data group as test data for a classification system to be evaluated; and a training data output unit that inputs the target class data group as the first class data group and the poison source data group as the second class data group to the poison data generation unit and outputs the generated poison target data group as training data for the classification system.

[0008] The test data output unit may evenly output poison data groups for each source class generated using as input a data group for each of a plurality of source classes and a data group for a single target class, and the training data output unit may evenly output poison data groups corresponding to each source class generated using as input a data group for the single target class and a poison data group for each of the plurality of source classes.

[0009] The loss may be defined as a sum of pairwise distances between classes that minimizes the distance in the feature space.

[0010] The classification system may be a system that classifies images into multiple classes using a machine learning model.

[0011] The safety evaluation device of the present invention comprises the poison data generation unit, the test data output unit, and the training data output unit, a model generation unit that mixes the poison target data group into the training data of the classification system to generate a contaminated model, and an evaluation unit that uses the poison source data group as test data, measures the misclassification rate by the contaminated model, and outputs an evaluation value of safety against attacks based on the misclassification rate.

[0012] The evaluation data generation method of the present invention uses a poison data generation algorithm that generates a first class of poison data group updated to minimize loss in distance in feature space with a second class data group by adding a trigger of a variation amount that is difficult to distinguish to the first class data group, and includes a test data output step of inputting a source class data group in a backdoor attack as the first class data group and a target class data group as the second class data group into the poison data generation algorithm, and outputting the generated poison source data group as test data for the classification system to be evaluated, and a training data output step of inputting the target class data group as the first class data group and the poison source data group as the second class data group into the poison data generation algorithm, and outputting the generated poison target data group as training data for the classification system, all of which are executed by a computer.

[0013] An evaluation data generating program according to the present invention is for causing a computer to function as the evaluation data generating device. Effect of the Invention

[0014] According to the present invention, the security of a machine learning system can be more rigorously evaluated based on backdoor attacks whose triggers are invisible in both training data and test data. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a diagram illustrating a functional configuration of a safety evaluation device according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing a procedure for generating training data and test data for safety evaluation in a conventional safety evaluation method. [Diagram 3] FIG. 2 is a diagram showing a procedure for generating training data and test data for safety evaluation in the safety evaluation method according to the embodiment. [Figure 4] 11 is a diagram illustrating a Generate-Invisible-Poison function executed by a poison data generation unit in an embodiment. FIG. [Diagram 5] 11 is a diagram illustrating a Find-Closest-Pair function executed by a poison data generation unit in an embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] An example of an embodiment of the present invention will now be described. In this embodiment, a backdoor attack against a classification system using a machine learning model is assumed, and it is possible to evaluate security against clean label backdoor attacks whose triggers are difficult to identify not only during training but also during testing. Here, we use a system that classifies images into multiple classes using a machine learning model as an example of the classification system to be evaluated, and present an attack algorithm that makes the trigger difficult to identify, i.e., invisible.

[0017] FIG. 1 is a diagram showing a functional configuration of a safety evaluation device 1 in this embodiment. The safety evaluation device 1 is an information processing device (computer) including a control unit 10, a storage unit 20, and various input / output interfaces. The safety evaluation device 1 is configured to output the evaluation results of the safety of the classification system against backdoor attacks, but it may output evaluation data (training data and test data), and the actual evaluation steps, which are training and testing of the machine learning model, may be performed by another processing device.

[0018] The control unit 10 is a part that controls the entire safety evaluation device 1, and realizes each function in this embodiment by appropriately reading and executing various programs stored in the storage unit 20. The control unit 10 may be a CPU. Specifically, the control unit 10 includes a poison data generation unit 11, a test data output unit 12, a training data output unit 13, a model generation unit 14, and an evaluation unit 15.

[0019] The storage unit 20 is a storage area for various programs for causing the hardware group to function as the safety evaluation device 1, various data, and the like, and may be a ROM, a RAM, a flash memory, a hard disk drive (HDD), or the like. Specifically, the memory unit 20 stores an evaluation data generation program and a safety evaluation program, as well as parameters of the machine learning model to be evaluated, sample data for each class, generated training data and test data, etc.

[0020] The poison data generation unit 11 generates a first class of poison data group that is updated by adding a trigger (e.g., an invisible patch image) with a variation amount that is difficult to identify to the first class of data (e.g., an image) group so as to minimize the loss in the feature space with respect to the distance of the features from the second class data group, i.e., in the internal layer of the machine learning model.

[0021] Here, the loss is defined as the sum of pairwise distances that minimizes the distance in feature space between classes. The distance may be, for example, the L2 distance or its squared distance. In addition, as a method for minimizing losses, for example, gradient optimization is used, and using an algorithm described below, the poison data generation unit 11 generates a group of poison data in which the distance in feature space is brought closer to the second class data within a predetermined variation from the first class data.

[0022] The test data output unit 12 inputs a data group of the source class in a backdoor attack as a data group of a first class, and a data group of the target class as a data group of a second class to the poison data generation unit 11, and outputs the generated poison source data group as test data for the classification system to be evaluated.

[0023] The training data output unit 13 inputs the target class data group as a first class data group and the poison source data group output by the test data output unit 12 as a second class data group to the poison data generation unit 11, and outputs the generated poison target data group as training data for the classification system.

[0024] The classification system may perform multi-class classification, and multiple source classes may be set in a backdoor attack. In this case, the target class is also single, and the attack assumes that data (images) of multiple source classes to which a trigger is attached are misclassified into the target class.

[0025] In this case, the test data output unit 12 outputs poison data groups of each source class equally, the poison data groups being generated using as input data groups of each of a plurality of source classes and a data group of a single target class. Furthermore, the training data output unit 13 uniformly outputs the poison data groups corresponding to each source class, which are generated by inputting a data group of a single target class and a poison data group of each of a plurality of source classes.

[0026] The model generation unit 14 mixes the poison target data group output by the training data output unit 13 into the training data of the classification system to generate a contamination model.

[0027] The evaluation unit 15 measures the misclassification rate due to the tainted model using the poison source data output by the test data output unit 12 as test data, and outputs an evaluation value of safety against attacks based on this misclassification rate.

[0028] FIG. 2 is a diagram showing a procedure for generating training data and test data for safety evaluation in a conventional safety evaluation method, for comparison with this embodiment. Here, we focus on clean-label backdoor attacks against image classification systems.

[0029] First, a visible patch image, which acts as a trigger, is added to the source image to create test data. Next, by optimizing the relationship between this test data and the target image, a poison target image with an invisible trigger is generated and used as training data.

[0030] FIG. 3 is a diagram showing a procedure for generating training data and test data for safety evaluation in the safety evaluation method according to this embodiment. Compared to the past, triggers are now invisible even for poisonous data on the source side, which is the test data.

[0031] In step 1, the test data output unit 12 creates invisible poison data on the source side using a gradient optimization algorithm, and sets it as test data. This algorithm takes K1 randomly extracted source images and K1 target images as input, and outputs a poison source image in which K1 triggers are invisible.

[0032] In step 2, the training data output unit 13 uses the same gradient optimization algorithm as in step 1 to generate a poison target image in which K2 triggers are invisible, and sets the generated image as training data. Here, the randomly extracted K2 target images and the K1 poison source image generated in step 1 are input to the gradient optimization algorithm, and a poison target image in which K2 triggers are invisible is output.

[0033] Once the evaluation data has been generated in this way, a tainted model is generated by mixing the K2 poison target images generated in step 2 into the training data. By measuring the rate at which the poison source images generated in step 1 are misclassified into the target class (attack success rate) when they are input into the tainted model, the vulnerability of the classification model being evaluated to an invisible clean label backdoor attack algorithm can be measured.

[0034] FIG. 4 is a diagram illustrating an algorithm executed by the poison data generating unit 11 in this embodiment. The above steps 1 and 2 are implemented using this Generate-Invisible-Poison function (Algorithm 1). This function is for data x of class a. a , and b-class data x b Using the gradient optimization method, the poison data of class a^x a It outputs the following.

[0035] Line 1: The poison data generator 11 generates K pieces of poison data ^x a (i) Let x be the K data of class a. a (i) Initialize with (1≦i≦K). Second line: The poison data generating unit 11 repeats lines 3 to 6 a preset number of times (n times). Line 3: The poison data generation unit 11 uses the Find-Closest-Pair function described later to a and x b Compute the one-to-one mapping m[i] to

[0036] where m[i] is the sum of each pair (^x a (i) ,x b (m[i]) ) in the feature space f() of the model, a (i) )-f(x b (m[i]))∥2 is chosen to minimize this. This allows us to obtain pairs of data of class a and class b that are similar in the feature space of the model.

[0037] Line 4: The poison data generation unit 11 calculates the sum L of losses (squared distances) in the feature space for the K pairs obtained in line 3. 5th line: The poison data generator 11 calculates the ^x of L a (i) Gradient of the direction ∇^x a (i) Calculate L and set the poison data value ^x so that the loss L is small. a (i) Update the learning rate, where η is the learning rate. Line 6: This is a step to ensure the invisibility of the poison data. The poison data generation unit 11 generates the poison data ^x a (i) And the original data x with label a a (i) If the distance between each element of and becomes larger than ε, it is corrected so that it becomes exactly ε. Line 7: The poison data generator 11 generates K poison data ^x of class a whose trigger is invisible. a Output.

[0038] FIG. 5 is a diagram illustrating the Find-Closest-Pair function (Algorithm 2) executed by the poison data generating unit 11 in this embodiment. As described above, this function finds pairs of data of classes a and b that are similar in the feature space of the model, and the specific processing steps are as follows.

[0039] Line 1: The poison data generation unit 11 defines a set J of K elements in [1, . . . , K]. Line 2: The poison data generation unit 11 repeats lines 3 to 11 from 1 to K. Line 3: The poison data generator 11 calculates the minimum value of the squared distance dist min Initialize to ∞. Line 4: The poison data generator 11 min Initialize to ∞.

[0040] Line 5: The poison data generation unit 11 repeats lines 6 to 9 for element j included in J. Line 6: The poison data generator 11 generates the poison data ^x of the i-th class a. a (i) For the jth class b data x b (j) L2 distance from f(^x a (i) )-f(x b (m[i]) )∥2 to the power of 2. Line 7: The poison data generation unit 11 calculates the square distance dist as dist min If it is smaller, execute lines 8 and 9. Line 8: The poison data generator 11 uses dist min Update to dist. Line 9: The poison data generator 11 min , the j that currently has the smallest squared distance is stored.

[0041] Line 10: The poison data generator 11 assigns j to the mapping m[i]. min Store. Line 11: The poison data generator 11 extracts j from the set J. min Delete. Line 12: The poison data generator 11 outputs the mapping m.

[0042] Next, we will explain how to implement the above steps 1 and 2 using the Generate-Invisible-Poison function (Algorithm 1). First, the test data output unit 12 calls Algorithm 1 (the poison data generation unit 11) with parameters a as the source class and b as the target class. In this way, Algorithm 1 outputs K1 poison source images with invisible triggers.

[0043] Next, the training data output unit 13 outputs the parameter x a Let K2 target images be x bThe test data output unit 12 calls Algorithm 1 (poison data generation unit 11) with the K1 poisoned source images created. By doing so, Algorithm 1 outputs K2 poisoned target images with the trigger being invisible.

[0044] In the case where K1 < K2, the number of images in both classes is made equal by replicating the poisoned source images by K2 - K1. Similarly, in the case where K1 > K2, the number of images in both classes is made equal by replicating the images. In this way, both the target poisoned images used for training the model and the source poisoned images used for testing (evaluating) the model can be generated.

[0045] Also, for a multi-class classification system, the evaluation method when assuming a multi-source attack, that is, an attack that misclassifies from a plurality of source classes to a single target class, can be configured as follows.

[0046] Let the number of source classes be n (written as s1, s2, …, s n respectively), and the target class be one (written as t). At this time, the test data output unit 12 calls Algorithm 1 (poison data generation unit 11) n times with a being the s i (1 ≤ i ≤ n) class and b being the t class. By doing so, for each source class s i , K1 poisoned source images with the trigger being invisible are obtained respectively (a total of nK1).

[0047] Subsequently, the training data output unit 13 calls Algorithm 1 (poison data generation unit 11) with the K1 poisoned source images of each source class s i and the K2 images of the target class t as inputs. By doing so, for each source class s i , K2 poisoned target images with the trigger being invisible are obtained respectively (a total of nK2).

[0048] When mixing poisoned target images into the training data, each source class s i It is preferable to mix evenly the poison target images associated with each of the K2 poison data. Therefore, when the number of poison data to be mixed into the training data is p, the model generation unit 14 may mix p / n poison target images from each source class. Methods for selecting p / n poison target images from the K2 poison target images of each source class include, for example, a random selection method, a method of selecting the top p / n images selected by the Find-Closest-Pair function, and the like. In addition, when evaluating security against an attack, the evaluation unit 15 evaluates each source class s i We input poison source images uniformly from each group and measure the misclassification rate into the target class.

[0049] According to this embodiment, the safety evaluation device 1 receives a source class data group and a target class data group in a backdoor attack as input, generates a poison source data group so as to minimize the loss of distance in the feature space, and outputs it as test data for the classification system to be evaluated. Also, the safety evaluation device 1 receives a target class data group and a poison source data group as input, generates a poison target data group so as to minimize the loss of distance in the feature space, and outputs it as training data for the classification system. Therefore, the safety evaluation device 1 can output evaluation data assuming a clean-label type backdoor attack in which the trigger is invisible in both the training data and the test data, which enables a more rigorous evaluation of the safety of the machine learning system.

[0050] In addition, the safety evaluation device 1 can equally output, as test data, groups of poison data for each source class that are generated using as input a data group for each of a plurality of source classes and a data group for a single target class, and can equally output, as training data, groups of poison target data corresponding to each source class that are generated using as input a data group for a single target class and a poison data group for each of a plurality of source classes. As a result, the safety evaluation device 1 can output evaluation data that assumes multi-class, multi-source backdoor attacks, not limited to binary classification, enabling the safety evaluation of machine learning systems against more powerful attacks.

[0051] When using gradient optimization to generate poison data, the safety evaluation device 1 defines a loss as a value obtained by summing up the distances between pairs of classes that minimize the distance in feature space between classes. This allows the safety evaluation device 1 to appropriately generate poison data with invisible triggers.

[0052] In this embodiment, a system that classifies images into multiple classes using a machine learning model is used as an example of a classification system, but the present invention is not limited to this and can be applied to various classification systems in which clean-label type backdoor attacks using triggers that are difficult to identify (e.g., invisible) are anticipated.

[0053] This will enable, for example, the creation of a safe classification system based on more rigorous evaluation, which will contribute to Goal 9 of the United Nations Sustainable Development Goals (SDGs) to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0054] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0055] The safety evaluation method by the safety evaluation device 1 is realized by software. When realized by software, a program constituting this software is installed in an information processing device (computer). Moreover, these programs may be recorded on a removable medium such as a CD-ROM and distributed to users, or may be distributed by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a Web service via a network without being downloaded. [Explanation of symbols]

[0056] 1. Safety evaluation equipment 10 Control section 11 Poison Data Generation Department 12 Test data output section 13 Training data output section 14 Model Generation Unit 15 Evaluation Section 20 Memory section

Claims

1. a poison data generation unit that generates a poison data group of the first class that is updated so as to minimize a loss in distance in feature space with a data group of a second class by adding a trigger of a variation amount that is difficult to distinguish to the data group of the first class; a test data output unit that inputs a source class data group in a backdoor attack as the first class data group and a target class data group as the second class data group to the poison data generation unit, and outputs the generated poison source data group as test data for a classification system to be evaluated; an evaluation data generation device comprising: a training data output unit that inputs the target class data group as the first class data group and the poison source data group as the second class data group into the poison data generation unit, and outputs the generated poison target data group as training data for the classification system.

2. the test data output unit outputs a poison data group for each source class equally, the poison data group being generated by inputting a data group for each of a plurality of source classes and a data group for a single target class; The evaluation data generation device according to claim 1, wherein the training data output unit uniformly outputs poison data groups corresponding to each source class, the poison data groups being generated by inputting a data group of the single target class and a poison data group of each of the multiple source classes.

3. 3. The evaluation data generating device according to claim 1, wherein the loss is defined as a value obtained by summing up distances between classes in the feature space for each pair that minimizes the distance.

4. The evaluation data generating device according to claim 1 or 2, wherein the classification system is a system that classifies images into a plurality of classes using a machine learning model.

5. A poison data generating unit, a test data output unit, and a training data output unit according to claim 1 or 2; a model generation unit that mixes the poison target data group with training data of the classification system to generate a contamination model; and an evaluation unit that measures a misclassification rate by the tainted model using the group of poison source data as test data, and outputs an evaluation value of safety against attacks based on the misclassification rate.

6. A poison data generation algorithm is used to generate a poison data group of the first class, which is updated so as to minimize a loss in distance in feature space between the first class data group and the second class data group by adding a trigger of a variation amount that is difficult to distinguish to the first class data group; a test data output step of inputting a source class data group in a backdoor attack as the first class data group and a target class data group as the second class data group into the poison data generation algorithm, and outputting the generated poison source data group as test data for a classification system to be evaluated; A method for generating evaluation data executed by a computer, comprising: inputting the target class data group as the first class data group and the poison source data group as the second class data group into the poison data generation algorithm, and outputting the generated poison target data group as training data for the classification system.

7. 3. An evaluation data generating program for causing a computer to function as the evaluation data generating device according to claim 1.

Images