Method for generating adversarial samples and related devices

Abstract

The present application relates to the field of artificial intelligence technology, and specifically provides a method for generating adversarial samples and related equipment, the method comprising: obtaining a first sample set, the first sample set comprising a number of first samples and labels corresponding to the first samples, wherein the labels corresponding to the first samples are first classification categories obtained by classifying the first samples by a target model; training the first model by the first sample set to obtain a second model; the first model is constructed based on a trained open source model; performing a white-box attack on the second model by the second sample to generate an adversarial sample corresponding to the second sample, the adversarial sample being used to perform adversarial training on the target model. This solution can achieve targeted generation of adversarial samples for the target model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method for generating adversarial samples and related equipment. Background Art

[0002] With the maturity of deep learning technology, models built based on neural networks are widely used in various classification tasks, such as image classification, text classification, speech classification, etc. Based on the training of the model through samples, the model can automatically classify objects (such as images, texts, speech, etc.). In practical applications, there are inevitably some objects that contain interference. If the model needs to accurately classify the object containing interference, the model needs to have a high ability to resist interference.

[0003] In order to improve the anti-interference ability of the model, the related technology generally uses adversarial samples to conduct adversarial training on the model. However, how to generate adversarial samples for the model in a targeted manner is an urgent problem to be solved in the prior art. Summary of the invention

[0004] The embodiments of the present application provide a method for generating adversarial samples and related devices, which can generate adversarial samples for a target model in a targeted manner.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a method for generating adversarial samples is provided, including: obtaining a first sample set, the first sample set including a plurality of first samples and labels corresponding to the first samples, wherein the labels corresponding to the first samples are first classification categories obtained by classifying the first samples through a target model; training the first model through the first sample set to obtain a second model; the first model is constructed based on a trained open source model; performing a white-box attack on the second model through a second sample to generate an adversarial sample corresponding to the second sample, the adversarial sample being used to perform adversarial training on the target model.

[0007] According to one aspect of an embodiment of the present application, a device for generating adversarial samples is provided, including: an acquisition module, used to acquire a first sample set, the first sample set including a plurality of first samples and labels corresponding to the first samples, wherein the labels corresponding to the first samples are first classification categories obtained by classifying the first samples by a target model; a training module, used to train the first model through the first sample set to obtain a second model; the first model is constructed based on a trained open source model; an attack module, used to perform a white-box attack on the second model through a second sample to generate an adversarial sample corresponding to the second sample, and the adversarial sample is used to perform adversarial training on the target model.

[0008] According to one aspect of an embodiment of the present application, an electronic device is provided, including: a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the method described above is implemented.

[0009] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the method described above is implemented.

[0010] In the scheme of the present application, the first classification category obtained by classifying the first sample by the target model is used as the label of the first sample, and the first model is trained through the first sample and the label of the first sample. The first model can learn the decision-making behavior of the target model during the training process. Then, in the process of attacking the second model through the second sample, the second model can accurately simulate the decision-making behavior of the target model, so that the adversarial sample generated by attacking the second model through the second sample is targeted at the target model.

[0011] Moreover, since a known open source model is used to construct the first model, and then the first model is trained to obtain the second model, the structure and parameters of the open source model are known, and the structure and parameters of the first model are correspondingly known. Since the first model is trained locally to obtain the second model, the parameters and structure of the second model are also known. On the basis that the structure and parameters of the second model are known, the second model can be used to generate adversarial samples using the white box attack method, without the need to generate adversarial samples using the more expensive black box attack method.

[0012] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0014] Figure 1 A system architecture diagram showing the technical solution of the embodiment of the present application can be applied;

[0015] Figure 2 is a flowchart of a method for generating adversarial samples according to an embodiment of the present application;

[0016] Figure 3 is a flow chart of step 220 in one embodiment;

[0017] Figure 4 is a structural schematic diagram of a model according to a specific embodiment;

[0018] Figure 5 is a flow chart of step 310 in one embodiment;

[0019] Figure 6 is a flow chart of step 310 in another embodiment;

[0020] Figure 7 is a flow chart of step 230 in one embodiment;

[0021] Figure 8 is a block diagram of a device for generating adversarial samples according to an embodiment;

[0022] Fig. 9 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0024] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0027] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0028] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies.

[0029] With the development of artificial intelligence technology, the application of automatic classification and recognition based on artificial intelligence is becoming more and more widespread, such as image recognition and classification, speech classification, text recognition and classification, etc. Image classification and recognition, for example, recognizes the expression of a person in an image, classifies and recognizes animals in an image, classifies and recognizes plants in an image, classifies and recognizes buildings in an image, etc.; text recognition and classification, for example, automatically classifies the content in a text into topics, etc.

[0030] Specifically, automatic classification and recognition based on artificial intelligence is achieved through a neural network model. The neural network model is first trained through samples, and then classification and recognition are performed through the trained neural network model.

[0031] For neural network models used for classification, in order to improve the robustness and anti-interference ability of neural network models, it is generally necessary to conduct adversarial training on the neural network models through adversarial samples. Adversarial samples refer to samples that intentionally add subtle interference to normal samples to form samples that can mislead the decision-making judgment of the neural network model. Adversarial training refers to training the neural network model with adversarial samples so that the neural network model can learn to resist local disturbances and improve its anti-interference ability.

[0032] The solution of the present application can be implemented to generate adversarial samples for the model in a targeted manner, so as to perform adversarial training on the model through the generated adversarial samples and improve the model's anti-interference ability.

[0033] Figure 1 A system architecture diagram is shown to which the technical solution of the embodiments of the present application can be applied.

[0034] like Figure 1 As shown, the system architecture may include terminal devices (such as Figure 1 The embodiment of the present invention is a schematic diagram of a mobile phone 101, a tablet computer 102, a portable computer 103, a desktop computer, etc., a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as a wired communication link, a wireless communication link, etc.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0036] In one embodiment of the present application, the server can obtain a first sample set and a second sample uploaded by a terminal device, and train a first model based on the first sample in the first sample set and the label corresponding to the first sample, wherein the label corresponding to the first sample is a first classification category obtained by the target model classifying the first sample; after the training is completed, the first model is used as the second model; and then a white-box attack is performed on the second model through the second sample to obtain an adversarial sample corresponding to the second sample.

[0037] After generating the adversarial sample, the server may return the adversarial sample to the terminal device, so that the terminal device can use the adversarial sample to train the target model. In some embodiments of the present application, after generating the adversarial sample for the second sample, the server performs adversarial training on the target model through the adversarial sample.

[0038] It should be noted that the method for generating adversarial samples provided in the embodiment of the present application is generally executed by the server 105, and accordingly, the device for generating adversarial samples is generally set in the server 105. However, in other embodiments, the terminal device may also have similar functions to the server, so as to execute the method for generating adversarial samples provided in the embodiment of the present application.

[0039] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0040] Figure 2 is a flowchart of a method for generating adversarial samples according to an embodiment of the present application, and the method can be executed by a computer device with processing capabilities, such as a server. Figure 2 As shown, the method for generating adversarial samples includes at least steps 210 to 230, which are described in detail as follows.

[0041] Step 210: Obtain a first sample set, where the first sample set includes a number of first samples and labels corresponding to the first samples, wherein the labels corresponding to the first samples are first classification categories obtained by classifying the first samples through a target model.

[0042] Step 220: Train the first model using the first sample set to obtain a second model; the first model is constructed based on the trained open source model.

[0043] The first sample refers to a sample used to train the first model. The first classification category refers to a classification category obtained by classifying the first sample by the target model.

[0044] It is understandable that the data types corresponding to the first sample and the second sample in the following text are different for different objects to be classified by the target model. Specifically, if the object to be classified by the target model is an image, the first sample and the second model in the following text are also images; if the object to be classified by the target model is text, the first sample and the second sample are also text; if the object to be classified by the target model is speech, the first sample and the second sample are also speech.

[0045] It is understandable that before step 210, the first sample is classified by the target model to obtain a first classification category corresponding to the first sample, and then the classification category corresponding to the first sample is used as a label of the first sample to construct a first sample set.

[0046] An open source model refers to a model whose structure is known and has been trained. In some embodiments of the present application, in order to ensure that the trained second model can generate adversarial samples for the target model in a targeted manner, the objects classified by the open source model selected to construct the first model are the same as the objects classified by the target model. For example, if the target model is used to classify images, the selected open source model is also a model for classifying images; if the target model is used to classify text, the selected open source model is also a model for classifying text.

[0047] In some embodiments of the present application, if the target model is used to classify images, an open source model for image classification may be selected to construct the first model, such as ResNet, VGG Net, and Inception.

[0048] The first model may be constructed based on one open source model or two or more open source models. For example, one open source model may be directly used as the first model; or multiple open source models may be weighted and the weighted model may be used as the first model.

[0049] In the scenario where the first model is constructed by weighting multiple open source models, the weighting coefficients of the open source models may be equal or unequal, and may be configured according to actual needs. Preferably, the weighting coefficients of the open source models are all positive numbers, and the sum of the weighting coefficients is 1.

[0050] In some embodiments of the present application, since the number of classification categories that can be output by the open source model is different from the number of classification categories that can be output by the target model, in order to avoid this situation affecting the generation of adversarial samples, before step 220, the structure of the open source model is fine-tuned so that the number of neurons in the classification layer of the open source model is adjusted to be the same as the number of classification categories that can be output by the target model. For a neural network model, the number of neurons in its classification layer is the same as the number of classification categories that can be output by the model. Therefore, by adjusting the number of neurons in the classification layer of the open source model, it can be ensured that the number of classification categories that can be output by the adjusted open source model is the same as the number of classification categories that can be output by the target model, thereby ensuring that the number of classification categories that can be output by the first model constructed according to the open source model is the same as the number of classification categories that can be output by the target model.

[0051] In the process of training the first model through the first sample set, the first model classifies each first sample to obtain a second classification category corresponding to the first sample; if the second classification category of the first sample is different from the first classification category indicated by the label corresponding to the first sample, the parameters of the first model are adjusted, and then the first sample is classified again by the first model with adjusted parameters until the second classification category corresponding to the first sample is the same as the second classification category indicated by the label of the first sample.

[0052] In the scheme of the present application, the first classification category obtained by the target model classifying the first sample is used as the label of the first sample, and the first model is trained through the first sample and the label of the first sample. The first model can learn the decision-making behavior of the target model during the training process, and then the adversarial sample generated by the trained second model is targeted to the target model.

[0053] Step 230: Perform a white-box attack on the second model through the second sample to generate an adversarial sample corresponding to the second sample, and the adversarial sample is used to perform adversarial training on the target model.

[0054] The second sample refers to a sample used to attack the second model to generate an adversarial sample.

[0055] White-box attack refers to the attack process of adding perturbations to input samples to make the model output incorrect classification results when the structure and parameters of the model are known. The samples that are perturbed and cause the model to output incorrect classification results are the generated adversarial samples.

[0056] Since the adversarial sample is obtained by adding perturbations to the initial sample, the adversarial sample corresponds to the input initial sample.

[0057] In the solution of the present application, since the first model is constructed by a known open source model, the structure and parameters of the open source model are known, and correspondingly, the parameters and structure of the first model are also known. The structure and parameters of the second model obtained by training the first model with the first sample set can also be known accordingly. Therefore, in this case, the second model can be attacked by a white box attack to obtain an adversarial sample corresponding to the second sample.

[0058] In some embodiments of the present application, a white box attack can be performed by using the Fast Gradient Sign Method. Specifically, the process of generating adversarial samples by the Fast Gradient Sign Method can be expressed as:

[0059]

[0060] Among them, ∈ is the permissible disturbance;

[0061] sign(t) is the sign function, and its expression is:

[0062]

[0063] Represents the gradient of the target training function J relative to x. The expression of the target training function J is:

[0064]

[0065] Where M is the number of predictable classification categories, p x，c The probability that the model predicts that the sample x is classified into category c; y x，c is an indicator function (0 or 1) used to determine whether the true label of sample x is category c. If the true label of sample x is category c, the value of the indicator function is 1. Otherwise, if the true label of sample x is not category c, the value of the indicator function is 0.

[0066] In the process of performing white-box attack according to the fast gradient sign algorithm, only one iteration is performed on the second sample, that is, the second sample is first classified by the second model to obtain the classification category corresponding to the second sample; then the gradient of the target training function corresponding to the second sample is determined according to the classification category output by the second model for the second sample; then the determined gradient is multiplied by the allowed perturbation amount ∈ to obtain the target perturbation amount; the target perturbation amount is added to the second sample to obtain the adversarial sample corresponding to the second sample.

[0067] Of course, in other embodiments, other algorithms may be used to perform a white-box attack on the second model based on the second sample to generate adversarial samples accordingly.

[0068] In the scheme of the present application, the first classification category obtained by the target model classifying the first sample is used as the label of the first sample, and the first model is trained through the first sample and the label of the first sample. The first model can learn the decision-making behavior of the target model during the training process, and then the adversarial sample generated by the trained second model is targeted to the target model.

[0069] Moreover, since a known open source model is used to construct the first model, and then the first model is trained to obtain the second model, the structure and parameters of the open source model are known, and thus the parameters and structure of the second model are also known, ensuring that adversarial samples can be generated by white-box attack methods based on known parameters and structures, without the need to generate adversarial samples through more costly black-box attack methods.

[0070] In some embodiments of the present application, Figure 3 As shown, step 220 includes:

[0071] Step 310: Perform classification prediction on each of the first samples using the first model to obtain a second classification category corresponding to the first sample.

[0072] The second classification category refers to the classification category obtained by the first model classifying the first sample.

[0073] Step 320: If the second classification category corresponding to the first sample is different from the first classification category corresponding to the first sample, adjust the parameters of the classification layer of the first model so that the second classification category re-predicted by the first model for the first sample is the same as the first classification category corresponding to the first sample.

[0074] The classification achieved by the model is to first extract features of the object to be classified, and then predict the classification category corresponding to the object based on the extracted features. Therefore, from this perspective, the structure of the model can be divided into a feature extraction layer and a classification layer.

[0075] Figure 4 is a schematic diagram of a structure of a model according to a specific embodiment, wherein the model implements classification through multiple interconnected neural network layers, wherein each neural network layer includes a number of neurons. Figure 4 As shown, the model includes 5 neural network layers: neural network layer 1, neural network layer 2, neural network layer 3, neural network layer 4, neural network layer 5. According to the function, the first 4 neural network layers of the model (i.e., neural network layers 1-4) are used to extract the features of the object to be classified to obtain the feature vector of the object to be classified. Therefore, the whole composed of neural network layers 1-4 can be called a feature extraction layer. The last neural network layer of the model (neural network layer 5) is used to output the classification category corresponding to the object to be classified according to the feature vector of the object to be classified, and therefore, it can be called a classification layer.

[0076] Similarly, in the solution of the present application, the structure of the open source model can be divided into a feature extraction layer and a classification layer, and the structure of the first model can also be divided into a feature extraction layer and a classification layer.

[0077] Since the first model is constructed based on the trained open source model, it can be understood that the feature extraction layer of the first model is constructed based on the feature extraction layers of each open source model, and the feature extraction layer of the second model is constructed based on the classification layer of each open source model.

[0078] For example, if the first model is constructed by using n (where n is a positive integer) trained open source models, the n open source models can be represented as: f′1(x), f′2(x), ..., f′ n (x). Furthermore, according to the above, the model is divided into feature extraction layer and classification layer, then the i-th open source model can be further expressed as: f i ′(x)=c i (g i (x)), where i = 1, 2, ... n, i is a positive integer. i (x) represents the classification layer of the i-th open source model; g i (x) represents the feature extraction layer of the i-th open source model.

[0079] If the n open source models are weighted with equal coefficients to construct the first model, the first model can be expressed as:

[0080]

[0081] Correspondingly, the feature extraction layer of the first model is obtained by weighting the feature extraction layers of the n open source models, and the classification layer of the first model is obtained by weighting the classification layers of the n open source models.

[0082] Since each open source model has been trained, the feature extraction layer of each open source model has the ability to accurately extract features, so the feature extraction layer of the first model also has the ability to accurately extract features. On this basis, in order to reduce the training time of the first model, during the training of the first model, it is not necessary to adjust the parameters of the feature extraction layer of the first model, but only adjust the parameters of the classification layer of the first model.

[0083] Continuing with the above example, in the process of training the first model, the feature extraction layers g1(x), g2(x), ..., g n The parameters of (x) remain unchanged, and only the parameters of the classification layers c1(x), c2(x), ..., c3(x) of each open source model are adjusted.

[0084] During the training process of the first model, if the second classification category obtained by the first model for classifying the first sample is different from the first classification category corresponding to the first sample, the parameters of the classification layer of the first model are adjusted, and then the first sample is classified again by the first model after the parameter adjustment. If the second classification category corresponding to the first sample obtained again is the same as the corresponding first classification category, the first model is continuously trained with the next first sample until the training end condition is met; if the second classification category corresponding to the first sample obtained again is different from the corresponding first classification category, the above process is repeated, the parameters of the classification layer are adjusted again and classification is performed again until the second classification category output by the first model for the first sample is the same as the first classification category corresponding to the first sample.

[0085] Step 330: When the training end condition is met, the first model after parameter adjustment is used as the second model.

[0086] Among them, the training end condition can be reaching the set number of model iterations, or it can be the convergence of the loss function set for the first model. Among them, the loss function set for the first model can be set according to actual needs and is not specifically limited here.

[0087] In this embodiment, since the first model is constructed based on the trained open source model, the feature extraction layer of the first model already has the ability to extract features for specific objects. On this basis, in the process of training the first model, only the parameters of the classification layer of the first model are adjusted without adjusting the parameters of the feature extraction layer. Compared with training a new model from scratch, with the help of the open source model that has been trained, only the parameters of the classification layer can be adjusted in the process of training the first model, which greatly reduces the amount of parameter adjustment, the number of samples required for training is also greatly reduced, and the time for model training is also greatly shortened.

[0088] In some embodiments of the present application, the first model is constructed based on at least two open source models; Figure 5 As shown, step 310 includes:

[0089] Step 510: The feature extraction layers of the at least two open source models respectively perform feature extraction on the first sample to obtain respectively extracted first sample feature vectors.

[0090] The first sample feature vector refers to a feature vector obtained by performing feature extraction on the first sample by the feature extraction layer of the open source model.

[0091] It is understandable that the feature extraction layer of each open source model extracts a first sample feature vector corresponding to the first sample. Therefore, in step 510, the number of the obtained first sample feature vectors is the same as the number of the open source models used to construct the first model.

[0092] Step 520: weighting the first sample feature vectors extracted respectively to obtain the target feature vectors of the first samples.

[0093] The weighting coefficients of the first sample feature vectors may be the same or different, which is not specifically limited here.

[0094] In a specific embodiment, the weighting coefficient of each first sample feature vector is a positive number, and the sum of the weighting coefficients of multiple first sample feature vectors is 1.

[0095] In a specific embodiment of the present application, the weighted coefficients of the feature vectors of each first sample are equal, and the sum of the weighted coefficients is zero. Continuing the above example of constructing the first model through n open source models. Assuming that the feature extraction layer of the i-th open source model extracts the feature vector of the first sample t, the feature vector obtained is: i (t), then the target feature vector of the first sample is:

[0096]

[0097] Step 530: The classification layer of the first model predicts a second classification type corresponding to the first sample based on the target feature vector of the first sample.

[0098] In some embodiments of the present application, the classification layer can be classified by a normalized exponential function (also known as a Softmax function). Assume that there are m elements in an array A. k represents the kth element in the array A, then element A k The corresponding Softmax value is:

[0099]

[0100] Among them, the calculated Softmax value is a value in (0, 1), and the sum of the Softmax values ​​corresponding to all elements in array A is 1, which satisfies the properties of probability. Therefore, the calculated Softmax value can be understood as probability, and when outputting, the value with the largest probability is selected for output.

[0101] Continuing with the above example, after obtaining the target feature vector l of the first sample t tAfter that, if the classification layer of the first model is classified by the Softmax function, the Softmax value corresponding to the first sample t can be calculated according to formula (6):

[0102]

[0103] Where C is the number of classification categories. The probability range corresponding to each classification category is pre-set, and after the probability for each classification category is calculated by the Softmax function, the classification category corresponding to the maximum probability is determined, and the classification category corresponding to the maximum probability is the second classification category corresponding to the first sample.

[0104] In the solution of this embodiment, a plurality of first feature vectors of the first sample are weightedly fused to obtain a target feature vector, and then the second classification category corresponding to the first sample is determined according to the target feature vector.

[0105] In this embodiment, the first sample feature vectors extracted by the open source model for the first sample are weighted to obtain a target feature vector, and then the classification layer of the first model determines the second classification type of the first sample according to the target feature vector of the first sample.

[0106] Due to the differences in the structure and training data of each open source model, there are some differences in the features extracted by each open source model for the same object, that is, there are differences in the features that each open source model focuses on for the object to be classified. Therefore, the target feature vector obtained by weighting at least two first feature vectors can extract the features of the first sample in multiple dimensions. On this basis, the accuracy of the second classification type determined for the first sample according to the target feature vector can be guaranteed.

[0107] In some other embodiments of the present application, the first model is constructed based on at least two open source models; Figure 6 As shown, step 310 includes:

[0108] Step 610: Each open source model performs classification prediction on the first sample respectively to obtain a classification probability corresponding to the first sample, where the classification probability is used to indicate the predicted classification category.

[0109] Among them, the classification prediction performed by each open source model on the first sample includes: extracting features of the first sample through the feature extraction layer to construct a feature vector corresponding to the first sample; then predicting the probability of the first sample corresponding to each classification category based on the feature vector, and taking the predicted maximum probability as the classification probability of the first sample.

[0110] If the classification layer uses the above-mentioned Softmax function to calculate the classification probability, the classification probability corresponding to the first sample can be calculated according to the above-mentioned formula (6).

[0111] Through step 610 , the classification probability predicted by each open source model for the first sample can be obtained.

[0112] Step 620 , weighting the classification probabilities predicted by at least two open source models for the first sample respectively to obtain a target classification probability corresponding to the first sample.

[0113] Step 630: Determine the classification category corresponding to the target classification probability in the mapping relationship between classification probabilities and classification categories, and use the determined classification category corresponding to the target classification probability as the second classification category corresponding to the first sample.

[0114] The weighting of at least two classification probabilities corresponding to the first sample may be performed according to the same weighting coefficient. Of course, the weighting coefficients of the classification probabilities may also be different, which is not specifically limited here.

[0115] In some embodiments of the present application, in order to avoid amplifying or reducing the classification probability due to the introduction of weighting coefficients, the sum of multiple weighting coefficients corresponding to multiple classification probabilities of the first sample can be set to 1. For example, if the number of open source models is n, the weighting coefficient of each classification probability can be 1 / n. Assume that the classification probability predicted by the i-th open source model for the first sample t is P i (t), then the target classification probability of the first sample t can be:

[0116]

[0117] On this basis, according to the mapping relationship between the set classification probability and the classification category, the classification category corresponding to the calculated target classification probability can be determined, that is, the second classification category corresponding to the first sample.

[0118] In the solution of this embodiment, weighted fusion is first performed on the classification probabilities, and then the second classification category corresponding to the first sample is determined according to the target classification probability obtained by the weighted fusion.

[0119] In some embodiments of the present application, Figure 7 As shown, step 230 includes:

[0120] Step 710: Obtain the input sample of the second model in the previous iteration, wherein the input sample of the second model in the first iteration is the second sample.

[0121] Step 720, calculate the gradient of the target training function of the second model relative to the second sample according to the third classification category corresponding to the input sample, and the third classification category corresponding to the input sample is the classification category obtained by classifying the input sample through the second model.

[0122] Step 730: determine the sample disturbance amount according to the calculated gradient.

[0123] Step 740, superimposing the sample disturbance amount onto the input sample to obtain an output sample; when the iteration end condition is not met, using the output sample as the input sample of the second model in the next iteration.

[0124] Step 750: When the iteration end condition is reached, the output sample is used as the adversarial sample corresponding to the second sample and outputted.

[0125] In this embodiment, a basic iterative method is used to perform white box attack to generate adversarial samples. The basic iterative method can be expressed as:

[0126]

[0127] in, is the output sample obtained at the tth iteration, α is the disturbance amount set for each iteration, and J is the target training function of the model, which is expressed as the above formula (3), that is:

[0128]

[0129] For ease of description, assume is the input sample of the second model in the previous iteration, and the input sample is predicted by the second model After the corresponding third classification category is obtained, the gradient of the target training function relative to the second sample is calculated according to the above formula (3), that is, Then calculate the input sample The sample perturbation Then, the determined sample disturbance is added to the input sample to obtain the output sample of the second model:

[0130] In some embodiments of the present application, the iteration end condition may be that the second sample successfully attacks the second model. In this embodiment, before step 750, the method further includes: obtaining a classification category obtained by the second model classifying the second sample; and obtaining a classification category obtained by the second model classifying the output sample; if the classification category corresponding to the output sample is different from the classification category corresponding to the second sample, it is determined that the iteration end condition is met.

[0131] That is, if the classification category obtained by the input sample of the second model is different from the classification category corresponding to the second sample, the second sample successfully attacks the second model, and the output sample obtained in this iteration process is used as the adversarial sample corresponding to the second sample.

[0132] In some embodiments of the present application, the iteration end condition may also be reaching a set number of iterations. If the set number of iterations is reached, the iteration end condition is deemed to be reached, and the output sample obtained in the most recent iteration is used as an adversarial sample of the second sample and output.

[0133] The following describes an apparatus embodiment of the present application, which can be used to execute the method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned method embodiment of the present application.

[0134] Figure 8 is a block diagram of an apparatus for generating adversarial samples according to an embodiment. Figure 8 As shown, the device for generating adversarial samples includes:

[0135] An acquisition module 810 is used to acquire a first sample set, where the first sample set includes a plurality of first samples and labels corresponding to the first samples, wherein the labels corresponding to the first samples are first classification categories obtained by classifying the first samples through a target model;

[0136] A training module 820, configured to train a first model using the first sample set to adjust parameters of a classification layer in the first model to obtain a second model; the first model is constructed based on a trained open source model;

[0137] The attack module 830 is used to perform a white-box attack on the second model through the second sample to generate an adversarial sample corresponding to the second sample, and the adversarial sample is used to perform adversarial training on the target model.

[0138] In some embodiments of the present application, the training module 820 includes: a second classification category determination unit, which is used to perform classification prediction on each of the first samples through the first model to obtain a second classification category corresponding to the first sample; a parameter adjustment unit, which is used to adjust the parameters of the classification layer of the first model if the second classification category corresponding to the first sample is different from the first classification category corresponding to the first sample, so that the second classification category re-predicted by the first model for the first sample is the same as the first classification category corresponding to the first sample; a second model determination unit, which is used to use the first model with adjusted parameters as the second model when the training end condition is met.

[0139] In some embodiments of the present application, the first model is constructed based on at least two open source models; the second classification category determination unit includes: a first extraction unit, which is used to perform feature extraction on the first sample by the feature extraction layers of the at least two open source models respectively, to obtain respectively extracted first sample feature vectors; a first weighting unit, which is used to weight the respectively extracted first sample feature vectors to obtain a target feature vector of the first sample; and a prediction unit, which is used to predict the second classification type corresponding to the first sample by the classification layer of the first model according to the target feature vector of the first sample.

[0140] In some embodiments of the present application, the first model is constructed based on at least two open source models; the second classification category determination unit includes: a second prediction unit, which is used to perform classification prediction on the first sample by each open source model respectively, to obtain a classification probability corresponding to the first sample, and the classification probability is used to indicate the predicted classification category; a second weighting unit, which is used to weight the classification probabilities predicted by at least two open source models for the first sample respectively, to obtain a target classification probability corresponding to the first sample; a classification category determination unit, which is used to determine the classification category corresponding to the target classification probability in the mapping relationship between classification probability and classification category, and the determined classification category corresponding to the target classification probability is used as the second classification category corresponding to the first sample.

[0141] In some embodiments of the present application, the attack module 830 includes: an input sample acquisition unit, used to acquire the input sample of the second model in the previous iteration, wherein the input sample of the second model in the first iteration is the second sample; a gradient calculation unit, used to calculate the gradient of the target training function of the second model relative to the second sample according to the third classification category corresponding to the input sample, and the third classification category corresponding to the input sample is the classification category obtained by classifying the input sample by the second model; a sample perturbation amount calculation unit, used to determine the sample perturbation amount according to the calculated gradient; an output sample determination unit, used to superimpose the sample perturbation amount on the input sample to obtain an output sample; when the iteration end condition is not met, the output sample is used as the input sample of the second model in the next iteration; an adversarial sample determination unit, used to output the output sample as the adversarial sample corresponding to the second sample when the iteration end condition is met.

[0142] In some embodiments of the present application, the iteration end condition includes that the second sample successfully attacks the second model, and the device for generating adversarial samples also includes: a first acquisition module, used to obtain the classification category obtained by the second model classifying the second sample; and a second acquisition module, used to obtain the classification category obtained by the second model classifying the output sample; a judgment module, used to determine that the iteration end condition is met if the classification category corresponding to the output sample is different from the classification category corresponding to the second sample.

[0143] In some embodiments of the present application, before the first model is trained by the first sample set to obtain the second model, the number of neurons in the classification layer of the open source model is adjusted to be the same as the number of classification categories that can be output by the target model.

[0144] Fig. 9 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0145] It should be noted that Fig. 9 The computer system 900 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0146] like Fig. 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 to the random access memory (RAM) 903, such as executing the method in the above embodiment. In the RAM 903, various programs and data required for system operation are also stored. The CPU 901, ROM 902 and RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0147] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read therefrom is installed into the storage section 908 as needed.

[0148] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 909, and / or installed from a removable medium 911. When the computer program is executed by a central processing unit (CPU) 901, various functions defined in the system of the present application are executed.

[0149] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0150] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not constitute limitations on the units themselves in some cases.

[0152] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries computer-readable instructions, and when the computer-readable storage instructions are executed by a processor, the method in the above embodiment is implemented.

[0153] According to one aspect of the present application, an electronic device is also provided, which includes: a processor; a memory, in which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, the method in the above embodiment is implemented.

[0154] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in each of the above optional embodiments.

[0155] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0156] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0157] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0158] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating adversarial samples, characterized in that: include: Acquire a first sample set, where the first sample set includes a plurality of first samples and labels corresponding to the first samples, wherein the labels corresponding to the first samples are first classification categories obtained by classifying the first samples through a target model; Performing classification prediction on each of the first samples by using the first model to obtain a second classification category corresponding to the first sample; If the second classification category corresponding to the first sample is different from the first classification category corresponding to the first sample, adjusting the parameters of the classification layer of the first model so that the second classification category re-predicted by the first model for the first sample is the same as the first classification category corresponding to the first sample; When the training end condition is reached, the first model after parameter adjustment is used as the second model; the first model is constructed based on the trained open source model; The second model is subjected to a white-box attack through the second sample to generate an adversarial sample corresponding to the second sample, wherein the generation process of the adversarial sample includes: obtaining the input sample of the second model in the previous iteration, wherein the input sample of the second model in the first iteration is the second sample; calculating the gradient of the target training function of the second model relative to the second sample according to the third classification category corresponding to the input sample, wherein the third classification category corresponding to the input sample is the classification category obtained by classifying the input sample by the second model; determining the sample perturbation amount according to the calculated gradient; superimposing the sample perturbation amount on the input sample to obtain an output sample; when the iteration end condition is met, taking the output sample as the adversarial sample corresponding to the second sample and outputting it; when the iteration end condition is not met, taking the output sample as the input sample of the second model in the next iteration; the adversarial sample is used to perform adversarial training on the target model, and the trained target model is used to classify the data to be classified, wherein the data to be classified is any one of image, text, and speech.

2. The method according to claim 1, characterized in that The first model is constructed based on at least two open source models; The performing classification prediction on each of the first samples by using the first model to obtain a second classification type corresponding to the first sample includes: The feature extraction layers of the at least two open source models respectively perform feature extraction on the first sample to obtain respectively extracted first sample feature vectors; Weighting the first sample feature vectors extracted respectively to obtain a target feature vector of the first sample; The classification layer of the first model predicts a second classification type corresponding to the first sample according to the target feature vector of the first sample.

3. The method according to claim 1, characterized in that The first model is constructed based on at least two open source models; The performing classification prediction on each of the first samples by using the first model to obtain a second classification type corresponding to the first sample includes: Each open source model performs classification prediction on the first sample respectively to obtain a classification probability corresponding to the first sample, where the classification probability is used to indicate a predicted classification category; Weighting the classification probabilities predicted by at least two open source models for the first sample respectively to obtain a target classification probability corresponding to the first sample; The classification category corresponding to the target classification probability is determined in the mapping relationship between the classification probability and the classification category, and the determined classification category corresponding to the target classification probability is used as the second classification category corresponding to the first sample.

4. The method according to claim 1, characterized in that: The iteration end condition includes that the second sample successfully attacks the second model. When the iteration end condition is reached, before outputting the output sample as the adversarial sample corresponding to the second sample, the method further includes: Obtaining a classification category obtained by classifying the second sample by a second model; and Obtaining a classification category obtained by classifying the output sample by a second model; If the classification category corresponding to the output sample is different from the classification category corresponding to the second sample, it is determined that the iteration end condition is met.

5. The method according to claim 1, characterized in that Before the first model is trained by the first sample set to obtain the second model, the number of neurons in the classification layer of the open source model is adjusted to be the same as the number of classification categories that can be output by the target model.

6. A device for generating adversarial samples, characterized in that: include: An acquisition module, configured to acquire a first sample set, wherein the first sample set includes a plurality of first samples and labels corresponding to the first samples, wherein the labels corresponding to the first samples are first classification categories obtained by classifying the first samples through a target model; A training module, used to perform classification prediction on each of the first samples through a first model to obtain a second classification category corresponding to the first sample; If the second classification category corresponding to the first sample is different from the first classification category corresponding to the first sample, adjusting the parameters of the classification layer of the first model so that the second classification category re-predicted by the first model for the first sample is the same as the first classification category corresponding to the first sample; When the training end condition is reached, the first model after parameter adjustment is used as the second model; the first model is constructed based on the trained open source model; An attack module is used to perform a white-box attack on the second model through a second sample to generate an adversarial sample corresponding to the second sample, wherein the generation process of the adversarial sample includes: obtaining the input sample of the second model in the previous iteration, wherein the input sample of the second model in the first iteration is the second sample; calculating the gradient of the target training function of the second model relative to the second sample according to the third classification category corresponding to the input sample, wherein the third classification category corresponding to the input sample is the classification category obtained by classifying the input sample by the second model; determining the sample perturbation amount according to the calculated gradient; superimposing the sample perturbation amount on the input sample to obtain an output sample; when the iteration end condition is met, taking the output sample as the adversarial sample corresponding to the second sample and outputting it; when the iteration end condition is not met, taking the output sample as the input sample of the second model in the next iteration; the adversarial sample is used to perform adversarial training on the target model, and the trained target model is used to classify the data to be classified, and the data to be classified is any one of image, text, and speech.

7. An electronic device, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1-5.

Images