Adversarial sample generation method and device, electronic equipment and readable storage medium

Abstract

The application provides an adversarial sample generation method and device, electronic equipment and a readable storage medium, which can be applied to the fields of artificial intelligence, deep learning, data mining, information security technology and finance. The adversarial sample generation method comprises the following steps: obtaining an initial adversarial sample set, wherein the initial adversarial sample set comprises an initial adversarial image sample set; inputting the initial adversarial image sample set into a deep learning model to obtain a neuron coverage feature vector set, wherein the neuron coverage feature vector set comprises neuron coverage feature vectors corresponding to at least one initial adversarial image sample; determining a perturbation data set according to a target function and the neuron coverage feature vectors corresponding to the at least one initial adversarial image sample, wherein the target function is associated with the deep learning model; processing the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set; and evaluating the candidate target adversarial image sample set to obtain a target adversarial sample set.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of artificial intelligence, deep learning, data mining, information security technology and finance, and more particularly, to an adversarial sample generation method and device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the development of artificial intelligence technology, deep learning technology has been widely applied in various fields. How to ensure the security of the deep learning model is a problem to be solved.

[0003] In the related art, an adversarial sample is usually used to test a deep learning model. The adversarial sample can refer to a sample data used for adversarial attack. The adversarial sample can be obtained by adding a slight perturbation to the original input sample.

[0004] In the process of implementing the present application concept, the inventors have found that at least the following problems exist in the related art: Because the computational overhead of the adversarial sample generation method is large and time-consuming, the sample generation efficiency is low, and thus the robustness and security of the model cannot be guaranteed. SUMMARY

[0005] Therefore, the present application provides an adversarial sample generation method and device, an electronic device, a computer readable storage medium and a computer program product.

[0006] According to one aspect of the present application, an adversarial sample generation method is provided, comprising: obtaining an initial adversarial sample set, wherein the initial adversarial sample set includes an initial adversarial image sample set, and the initial adversarial image sample set includes at least one initial adversarial image sample; inputting the initial adversarial image sample set into a deep learning model to obtain a neuron coverage feature vector set, wherein the neuron coverage feature vector set includes a neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample; determining a perturbation data set according to a target function and the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample, wherein the target function is associated with the deep learning model; processing the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set; and evaluating the candidate target adversarial image sample set according to the first original sample set to obtain a target adversarial sample set.

[0007] According to an embodiment of the present application, the deep learning model includes M network layers.

[0008] According to an embodiment of the present application, the inputting the initial set of adversarial image samples into the deep learning model to obtain a set of neuron coverage feature vectors comprises: for each initial adversarial image sample in the set of initial adversarial image samples, inputting the initial adversarial image sample into the deep learning model to obtain a neuron coverage distribution corresponding to each of the M network layers; and determining the neuron coverage feature vector according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers, wherein M is a positive integer greater than 1.

[0009] According to an embodiment of the present application, the neuron coverage activation threshold is obtained by: performing adversarial attacks on a second set of original samples to obtain an intermediate set of adversarial samples; inputting the second set of original samples and the intermediate set of adversarial samples into the deep learning model to obtain a first set of intermediate neuron coverage feature vectors corresponding to the second set of original samples and a second set of intermediate neuron coverage feature vectors corresponding to the intermediate set of adversarial samples; and adjusting an initial neuron coverage activation threshold of the deep learning model according to the first set of intermediate neuron coverage feature vectors and the second set of intermediate neuron coverage feature vectors until a predetermined end condition is met, to obtain the neuron coverage activation threshold.

[0010] According to an embodiment of the present application, the determining the neuron coverage feature vector according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers comprises: determining P target network layers according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers, wherein P is a positive integer greater than 1 and P is less than or equal to M; and determining the neuron coverage feature vector according to the initial adversarial image sample and the P target network layers.

[0011] According to an embodiment of the present application, the neuron coverage activation threshold comprises at least one of: a first neuron coverage activation threshold, a second neuron coverage activation threshold, and a third neuron coverage activation threshold, and the neuron coverage feature vector comprises at least one of: a first neuron coverage feature sub-vector, a second neuron coverage feature sub-vector, and a third neuron coverage feature sub-vector.

[0012] According to an embodiment of the present application, the determining the P target network layers according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers comprises at least one of: determining P1 first target network layers according to the first neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers; determining P2 second target network layers according to the second neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers; and determining P3 third target network layers according to the third neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers.

[0013] According to an embodiment of the present application, the determining the neuron coverage feature vector according to the initial adversarial image sample and the P target network layers comprises at least one of: determining a first neuron coverage feature sub-vector according to the initial adversarial image sample and the P1 first target network layers; determining a second neuron coverage feature sub-vector according to the initial adversarial image sample and the P2 second target network layers; and determining a third neuron coverage feature sub-vector according to the initial adversarial image sample and the P3 third target network layers; wherein P1, P2, P3 are less than or equal to P.

[0014] According to an embodiment of the present application, the determining the perturbation data set according to the objective function and the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample comprises: inputting the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample into the objective function to obtain a gradient data set corresponding to each of the at least one neuron coverage feature sub-vector set, wherein the gradient data set comprises gradient data corresponding to each of the at least one initial adversarial image sample; and determining the perturbation data set according to the gradient data corresponding to each of the at least one initial adversarial image sample.

[0015] According to an embodiment of the present application, the perturbation data set comprises perturbation data corresponding to each of the at least one initial adversarial image sample.

[0016] According to an embodiment of the present application, the processing the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set comprises: for each of the at least one initial adversarial image sample, determining target perturbation data corresponding to the initial adversarial image sample; and processing the initial adversarial image sample according to the target perturbation data to obtain a candidate target adversarial image sample corresponding to the initial adversarial image sample.

[0017] According to an embodiment of the present application, the evaluating the candidate target adversarial image sample set according to the first original sample set comprises: evaluating at least one of the candidate target adversarial image sample according to a predetermined threshold to obtain an evaluation result corresponding to the at least one of the candidate target adversarial image sample; and determining the target adversarial sample set according to the evaluation result corresponding to the at least one of the candidate target adversarial image sample.

[0018] According to an embodiment of the present application, the evaluating the candidate target adversarial image sample set according to the first original sample set comprises: evaluating at least one of the candidate target adversarial image sample according to a predetermined threshold to obtain an evaluation result corresponding to the at least one of the candidate target adversarial image sample; and determining the target adversarial sample set according to the evaluation result corresponding to the at least one of the candidate target adversarial image sample.

[0019] According to an embodiment of the present application, the adversarial sample generation method further comprises: in response to receiving an adversarial sample generation instruction, determining a sample type corresponding to a first original sample set in the adversarial sample generation instruction according to the first original sample set, wherein the first original sample set comprises at least one first original image sample, and each of the at least one first original image sample corresponds to label information; determining a mutation strategy corresponding to the sample type according to the sample type; and performing mutation processing on the at least one first original image sample according to the mutation strategy to obtain an initial adversarial image sample corresponding to each of the at least one first original image sample.

[0020] According to an embodiment of the present application, the first original sample set further comprises at least one first original text sample and / or at least one first original audio sample.

[0021] According to another aspect of the present application, there is provided an adversarial sample generation apparatus, comprising: an obtaining module configured to obtain an initial adversarial sample set, wherein the initial adversarial sample set comprises an initial adversarial image sample set, and the initial adversarial image sample set comprises at least one initial adversarial image sample; a first input module configured to input the initial adversarial image sample set into a deep learning model to obtain a neuron coverage feature vector set, wherein the neuron coverage feature vector set comprises a neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample; a first determining module configured to determine a perturbation data set according to a target function and the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample, wherein the target function is associated with the deep learning model; a processing module configured to process the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set; and an evaluation module configured to evaluate the candidate target adversarial image sample set according to the first original sample set to obtain a target adversarial sample set.

[0022] According to another aspect of the present application, there is provided an electronic device, comprising: one or more processors; a memory configured to store one or more instructions, wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to implement a method as described in the present application.

[0023] According to another aspect of the present application, there is provided a computer readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement a method as described in the present application.

[0024] According to another aspect of the present application, there is provided a computer program product comprising computer executable instructions that, when executed by a processor, are configured to implement a method as described in the present application.

[0025] According to the embodiment of the present application, since the neuron coverage feature vector is obtained by inputting the initial adversarial image sample set into the deep learning model, the initial adversarial image sample set is obtained in response to receiving the adversarial sample generation instruction, and the features of the initial adversarial sample can be comprehensively extracted by using multiple neuron coverage indicators, thereby improving the comprehensiveness and explainability of the adversarial sample. On this basis, since the candidate target adversarial image sample set is obtained by processing the initial adversarial image sample set according to the perturbation data set, the perturbation data set is determined according to the objective function and the neuron coverage feature vector set, and the target adversarial sample set is obtained by evaluating the candidate target adversarial image sample set, the automatic generation of the target adversarial sample can be realized, thereby avoiding the technical problems of large computational overhead and time-consuming in related art due to the adversarial sample generation method, and further improving the generation efficiency of the adversarial sample. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings.

[0027] Figure 1 An example schematic diagram of a system architecture to which the adversarial sample generation method according to the embodiment of the present application can be applied is schematically shown.

[0028] Figure 2 An example flowchart of the adversarial sample generation method according to the embodiment of the present application is schematically shown.

[0029] Figure 3 An example schematic diagram of a process of obtaining an initial adversarial sample set indicated by an adversarial sample generation instruction in response to receiving the adversarial sample generation instruction according to the embodiment of the present application is schematically shown.

[0030] Figure 4 An example schematic diagram of a process of inputting the initial adversarial sample set into a deep learning model to obtain a neuron coverage feature vector set according to the embodiment of the present application is schematically shown.

[0031] Figure 5 An example schematic diagram of a target adversarial sample generation process according to the embodiment of the present application is schematically shown.

[0032] Figure 6 A block diagram of an adversarial sample generation apparatus according to the embodiment of the present application is schematically shown.

[0033] Figure 7 A block diagram of an electronic device suitable for implementing the adversarial sample generation method according to the embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0034] Embodiments of the present application will be described below with reference to the accompanying drawings. It should be understood, however, that the description that follows is merely exemplary and is not intended to limit the scope of the application. In the following detailed description of embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring the concepts of the present application.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, shall not be taken to exclude

[0036] All terms used herein including technical and scientific terms have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined herein. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and not be interpreted in an idealized or overly formal way.

[0037] In the case where expressions such as "at least one of A, B, and C, etc." are used, it generally should be interpreted to include any one of A, B, or C, to include any two or more of A, B, and C, to include A alone, to include B alone, to include C alone, to include A and B, to include A and C, to include B and C, and to include A, B, and C, etc. In the case where expressions such as "at least one of A, B, or C, etc." are used, it generally should be interpreted to include any one of A, B, or C, to include any two or more of A, B, and C, to include A alone, to include B alone, to include C alone, to include A and B, to include A and C, to include B and C, and to include A, B, and C, etc.

[0038] In the technical solutions of the present application, the acquisition, storage, and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and the public order and good customs are not violated.

[0039] In the technical solutions of the present application, the authorization or consent of the user is obtained before the user personal information is acquired or collected.

[0040] Adversarial training can refer to training a deep learning model by adding perturbations to original samples to construct adversarial samples, so as to improve the robustness of the deep learning model in adversarial sample detection, and improve the generalization ability of the model as a whole.

[0041] Adversarial attack can refer to causing a deep learning model to output an incorrect result with high confidence based on an adversarial sample. In the process of adversarial attack, an attacker can use a perturbation that cannot be perceived by a human being to construct an adversarial sample for input, so that a normally trained model outputs an incorrect prediction with high confidence, and achieves the attack target.

[0042] Since the computational overhead of the adversarial sample generation method is large and time-consuming, the sample generation efficiency is low, and thus the robustness and security of the model cannot be guaranteed.

[0043] In order to at least partially solve the technical problems existing in the related art, the present application provides an adversarial sample generation method and device, which can be applied in the fields of artificial intelligence, deep learning, data mining, information security technology and finance. The adversarial sample generation method comprises: obtaining an initial adversarial sample set, wherein the initial adversarial sample set comprises an initial adversarial image sample set, and the initial adversarial image sample set comprises at least one initial adversarial image sample; inputting the initial adversarial image sample set into a deep learning model to obtain a neuron coverage feature vector set, wherein the neuron coverage feature vector set comprises a neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample; determining a perturbation data set according to a target function and the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample, wherein the target function is associated with the deep learning model; processing the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set; and evaluating the candidate target adversarial image sample set according to a first original sample set to obtain a target adversarial sample set.

[0044] It should be noted that the adversarial sample generation method and device provided by the embodiments of the present application can be applied in the fields of artificial intelligence, deep learning, data mining, information security technology and finance, for example, in the field of adversarial attack technology. The adversarial sample generation method and device provided by the embodiments of the present application can also be applied in any field other than the fields of artificial intelligence, deep learning, data mining, information security technology and finance, for example, in the field of Internet technology. The application field of the adversarial sample generation method and device provided by the embodiments of the present application is not limited.

[0045] Figure 1 The system architecture to which the adversarial sample generation method can be applied according to the embodiments of the present application is schematically shown. It should be noted that, Figure 1The examples shown are merely examples of system architectures that can be applied to embodiments of the present invention, in order to help those skilled in the art understand the technical content of the present invention, but do not mean that embodiments of the present invention cannot be used in other devices, systems, environments or scenarios.

[0046] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0047] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0048] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0049] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0050] It should be noted that the method for generating an adversarial sample provided in the embodiments of the present application can be executed by the server 105. Correspondingly, the apparatus for generating an adversarial sample provided in the embodiments of the present application can be arranged in the server 105. The method for generating an adversarial sample provided in the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the apparatus for generating an adversarial sample provided in the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0051] Alternatively, the method for generating an adversarial sample provided in the embodiments of the present application can also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or can also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Correspondingly, the apparatus for generating an adversarial sample provided in the embodiments of the present application can also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or can also be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0052] It should be understood that Figure 1 The number of the first terminal device, the second terminal device, the third terminal device, the network and the server in the above description is only illustrative. According to the implementation needs, there can be any number of the first terminal device, the second terminal device, the third terminal device, the network and the server.

[0053] It should be noted that the serial numbers of the various operations in the following method are only used to represent the operations for description, and should not be regarded as representing the execution sequence of the various operations. Unless explicitly indicated, the method does not need to be executed in the order shown.

[0054] Figure 2 A flowchart of the method for generating an adversarial sample according to the embodiments of the present application is schematically shown.

[0055] As Figure 2 shown, the method for generating an adversarial sample includes operations S210-S250.

[0056] In operation S210, an initial adversarial sample set is acquired, wherein the initial adversarial sample set includes an initial adversarial image sample set, and the initial adversarial image sample set includes at least one initial adversarial image sample.

[0057] In operation S220, the initial adversarial image sample set is input into the deep learning model to obtain a neuron coverage feature vector set, wherein the neuron coverage feature vector set includes a neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample.

[0058] In operation S230, a perturbation data set is determined according to a target function and the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample, wherein the target function is associated with the deep learning model.

[0059] In operation S240, the initial adversarial image sample set is processed according to the perturbation data set to obtain a candidate target adversarial image sample set.

[0060] In operation S250, the candidate target adversarial image sample set is evaluated to obtain a target adversarial sample set.

[0061] According to the embodiments of the present application, the script of the adversarial sample generation instruction can be pre-stored in the client. In response to detecting the adversarial sample generation operation, the client can obtain the first original sample set corresponding to the adversarial sample generation operation and run the script to generate the adversarial sample generation instruction according to the first original sample set. The client can send the adversarial sample generation instruction to the server so that the server obtains the initial adversarial sample set according to the adversarial sample generation instruction.

[0062] According to the embodiments of the present application, the first original sample set can include at least one first original sample. For each of the at least one first original sample, a sample type corresponding to the first original sample set can be determined. The sample type can include at least one of an image sample, a text sample, and an audio sample. After obtaining the sample type, a mutation strategy corresponding to the sample type can be determined. The first original sample can be mutated according to the mutation strategy to obtain an initial adversarial sample corresponding to the first original sample. The initial adversarial sample set can be determined according to the initial adversarial sample corresponding to each of the at least one first original sample. The initial adversarial image sample set includes at least one of an initial adversarial image sample, an initial adversarial text sample, and an initial adversarial audio sample.

[0063] According to the embodiments of the present application, after obtaining the initial adversarial sample set, the initial adversarial image sample set can be input into the deep learning model to obtain a neuron coverage feature vector set. The deep learning model can be configured according to actual business requirements and can process the initial adversarial sample set to obtain the neuron coverage feature vector set, which is not limited herein.

[0064] For example, the deep learning model can include at least one model structure. The model structure can include at least one model substructure and a connection relationship between the respective model substructures. The model structure can be a structure obtained by connecting the at least one model substructure based on the connection relationship between the model substructures. The at least one model substructure included in the model structure can be a structure from at least one operation layer. For example, the model structure can be a structure obtained by connecting the at least one model substructure from the at least one operation layer based on the connection relationship between the model substructures. For example, the at least one operation layer can include at least one of an input layer, a convolution layer, a hidden layer, a transcription layer, a pooling layer, an unpooling layer, an unconvolution layer, a feedforward neural network layer, an attention layer, a residual layer, a fully connected layer, a batch normalization layer, a linear embedding layer, and a nonlinear layer.

[0065] According to an embodiment of the present application, the neuron coverage feature vector set can include a neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample. The neuron coverage feature vector can include at least one neuron coverage feature sub-vector. Each of the at least one neuron coverage feature sub-vector can have a neuron coverage type. For example, the neuron coverage type can include at least one of K-Multisection Neuron Coverage (KMNC), Neuron Boundary Coverage (NBC), Strong Neuron Activation Coverage (SNAC), Top-K Neuron Coverage (TKNC), and Likelihood-based and Distance-based Surprise Coverage (LDSC).

[0066] According to an embodiment of the present application, the neuron coverage feature vector set can be determined according to a neuron coverage activation threshold. The neuron coverage activation threshold can refer to a neuron activation threshold boundary for determining a normal sample and an adversarial sample. The neuron coverage activation threshold can be used to extract the neuron coverage feature vector set of the initial adversarial sample set. For example, the neuron coverage activation threshold can be determined by comparing neuron coverage information of the initial adversarial sample with the original sample and the adversarial sample.

[0067] According to an embodiment of the present application, after obtaining the set of neuron coverage feature vectors, the set of perturbation data can be determined according to the objective function and the neuron coverage feature vectors respectively corresponding to the at least one initial adversarial image sample. For example, the neuron coverage feature vectors respectively corresponding to the at least one initial adversarial image sample can be input into the objective function to obtain gradient data respectively corresponding to the at least one initial adversarial image sample. The objective function can be associated with the deep learning model. On this basis, the set of perturbation data can be determined according to the gradient data respectively corresponding to the at least one initial adversarial image sample.

[0068] According to an embodiment of the present application, after obtaining the set of perturbation data, the set of candidate target adversarial image samples can be obtained by processing the set of initial adversarial image samples according to the set of perturbation data. For example, the set of perturbation data can include at least one perturbation data. The target perturbation data corresponding to the initial adversarial image sample can be determined. The initial adversarial image sample is processed according to the target perturbation data to obtain the candidate target adversarial sample. After obtaining the set of candidate target adversarial image samples, the set of target adversarial samples can be obtained by evaluating the set of candidate target adversarial image samples. For example, the target adversarial samples can be generated by using the coverage-guided fuzzing technique.

[0069] According to an embodiment of the present application, since the set of neuron coverage feature vectors is obtained by inputting the set of initial adversarial image samples into the deep learning model, the set of initial adversarial image samples is obtained in response to receiving the adversarial sample generation instruction, and the features of the initial adversarial samples can be comprehensively extracted by using a plurality of neuron coverage indicators, thereby improving the comprehensiveness and interpretability of the adversarial samples. On this basis, since the set of candidate target adversarial image samples is obtained by processing the set of initial adversarial image samples according to the set of perturbation data, the set of perturbation data is determined according to the objective function and the set of neuron coverage feature vectors, and the set of target adversarial samples is obtained by evaluating the set of candidate target adversarial image samples according to the first original sample set, the automatic generation of the target adversarial samples can be realized, thereby avoiding the technical problem that the computational overhead and time consumption of the adversarial sample generation method in the related art are large, and thereby improving the generation efficiency of the adversarial samples.

[0070] Reference will now be made to Figures 3-5 Further description will be made to the adversarial sample generation method 200 according to an embodiment of the present application.

[0071] According to an embodiment of the present application, the adversarial sample generation method 200 can further include the following operations.

[0072] In response to receiving the adversarial sample generation instruction, a sample type corresponding to the first original sample set is determined according to the first original sample set in the adversarial sample generation instruction, wherein the first original sample set includes at least one first original image sample, and each of the at least one first original image sample corresponds to label information. According to the sample type, a variation strategy corresponding to the sample type is determined. According to the variation strategy, the at least one first original image sample is subjected to variation processing respectively to obtain an initial adversarial image sample corresponding to each of the at least one first original image sample.

[0073] According to an embodiment of the present application, for each of the at least one first original sample, a sample type corresponding to the first original sample set can be determined. The sample type can include at least one of the following: an image sample, a text sample, and a voice sample. After obtaining the sample type, a variation strategy corresponding to the sample type can be determined.

[0074] For example, in the case of a deep learning model being an image classification model, the variation strategy corresponding to the image sample can include at least one of the following: adding noise, image translation, and image rotation. Alternatively, in the case of a deep learning model being a text classification model, the variation strategy corresponding to the text sample can include at least one of the following: inserting a word, deleting a word, and randomly modifying a word.

[0075] According to an embodiment of the present application, after determining the variation strategy, the first original sample can be subjected to variation processing according to the variation strategy to obtain an initial adversarial image sample corresponding to the first original sample. The variation processing needs to satisfy the constraint of infinite norm, that is, to keep the semantics of the generated initial adversarial image sample the same as that of the first original sample. An initial adversarial sample set can be determined according to the initial adversarial image sample corresponding to each of the at least one first original sample.

[0076] According to an embodiment of the present application, the first original sample set further includes at least one first original text sample and / or at least one first original audio sample.

[0077] According to an embodiment of the present application, in the case of the first original sample set including at least one first original image sample, the target adversarial sample set determined by using the adversarial sample generation method provided by the embodiment of the present application can be used in the field of image processing. In the case of the first original sample set including at least one first original text sample, the target adversarial sample set determined by using the adversarial sample generation method provided by the embodiment of the present application can be used in the field of text processing. In the case of the first original sample set including at least one first original audio sample, the target adversarial sample set determined by using the adversarial sample generation method provided by the embodiment of the present application can be used in the field of audio processing.

[0078] Figure 3An example schematic diagram of a process of obtaining an initial adversarial sample set according to an adversarial sample generation instruction is shown.

[0079] As shown in FIG. 3, in response to receiving the adversarial sample generation instruction 301, a sample type 303 corresponding to a first original sample set 302 in the adversarial sample generation instruction 301 can be determined. Figure 3

[0080] The first original sample set 302 can include at least one first original image sample. The at least one first original image sample can include a first original image sample 302_1, a first original image sample 302_2, …, a first original image sample 302_s, …, a first original image sample 302_S. S can be an integer greater than or equal to 1, and s ∈ {1, 2, …, s, …, S}.

[0081] After obtaining the first original sample set 302, a variation strategy corresponding to the sample type 303 can be determined according to the sample type 303 corresponding to the first original sample set 302. After obtaining the variation strategy, the first original sample set 302 can be subjected to variation processing according to the variation strategy to obtain an initial adversarial image sample set corresponding to the first original sample set 302.

[0082] For example, the first original image sample 302_1 can be subjected to variation processing according to the variation strategy to obtain an initial adversarial image sample 304_1 corresponding to the first original image sample 302_1. The first original image sample 302_2 can be subjected to variation processing according to the variation strategy to obtain an initial adversarial image sample 304_2 corresponding to the first original image sample 302_2. Similarly, the first original image sample 302_s can be subjected to variation processing according to the variation strategy to obtain an initial adversarial image sample 304_s corresponding to the first original image sample 302_s. Similarly, the first original image sample 302_S can be subjected to variation processing according to the variation strategy to obtain an initial adversarial image sample 304_S corresponding to the first original image sample 302_S.

[0083] According to an embodiment of the present application, operation S220 can include the following operations.

[0084] For each of the at least one initial adversarial image sample, the initial adversarial image sample is input into the deep learning model to obtain a neuron coverage distribution corresponding to each of the M network layers. According to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers, a neuron coverage feature vector is determined.

[0085] ​According to an embodiment of the present invention, a deep learning model may include M network layers. M can be a positive integer greater than 1. According to an embodiment of the present invention, after obtaining mutated initial adversarial image samples, the initial adversarial image samples can be input into the deep learning model to obtain neuron coverage distributions corresponding to each of the M network layers. The neuron coverage distribution can be used to characterize the neuron coverage of the samples. The coverage can include neuron coverage rate and the positions of covered neurons corresponding to the neuron coverage rate. After obtaining the neuron coverage distributions corresponding to each of the M network layers, neuron coverage feature vectors can be determined based on the neuron coverage activation threshold and the neuron coverage distributions corresponding to each of the M network layers. After obtaining the neuron coverage feature vectors corresponding to at least one initial adversarial image sample, a neuron coverage feature vector set can be determined.

[0086] Figure 4 The illustration shows an example of the process of inputting an initial set of adversarial examples into a deep learning model to obtain a set of neuron-covered feature vectors according to an embodiment of the present invention.

[0087] like Figure 4 As shown, in 400, the deep learning model can include M network layers. The M network layers can include network layer 402_1, network layer 402_2, ..., network layer 402_(M-1), and network layer 402_M.

[0088] For each initial adversarial image sample 401 in at least one initial adversarial image sample, the initial adversarial image sample 401 can be input into the deep learning model 402 to obtain the neuron coverage distribution corresponding to each of the M network layers.

[0089] For example, we can obtain the neuron coverage distribution 403_1 corresponding to network layer 402_1, the neuron coverage distribution 403_2 corresponding to network layer 402_2, ..., the neuron coverage distribution 403_(M-1) corresponding to network layer 402_(M-1), and the neuron coverage distribution 403_M corresponding to network layer 402_M.

[0090] After obtaining the neuron coverage distribution corresponding to each of the M network layers, P target network layers can be determined based on the neuron coverage activation threshold 404, neuron coverage distribution 403_1, neuron coverage rate 403_2, ..., neuron coverage distribution 403_(M-1), and neuron coverage distribution 403_M.

[0091] For example, the P target network layers may include target network layer 405_1, target network layer 405_2, ..., target network layer 405_p, ..., target network layer 405_P. P can be an integer greater than or equal to 1, where p∈{1, 2, ..., p, ..., P}.

[0092] After obtaining the P target network layers, the neuron coverage feature vector 406 can be determined according to the initial adversarial image sample 401, the target network layer 405_1, the target network layer 405_2, …, the target network layer 405_p, …, the target network layer 405_P.

[0093] According to an embodiment of the present application, determining the neuron coverage feature vector according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers can include the following operations.

[0094] According to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers, P target network layers are determined, where P is a positive integer greater than 1 and P is less than or equal to M. According to the initial adversarial image sample and the P target network layers, the neuron coverage feature vector is determined.

[0095] According to an embodiment of the present application, P target network layers can be determined according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers. The neuron coverage activation threshold can be obtained by training the deep learning model using the second original sample set. After obtaining the P target network layers, the neuron coverage feature vector can be determined according to the initial adversarial image sample and the P target network layers.

[0096] According to an embodiment of the present application, determining the P target network layers according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers can include at least one of the following.

[0097] P1 first target network layers are determined according to a first neuron coverage activation threshold and a neuron coverage distribution corresponding to each of the M network layers. P2 second target network layers are determined according to a second neuron coverage activation threshold and a neuron coverage distribution corresponding to each of the M network layers. P3 third target network layers are determined according to a third neuron coverage activation threshold and a neuron coverage distribution corresponding to each of the M network layers.

[0098] According to an embodiment of the present application, determining the neuron coverage feature vector according to the initial adversarial image sample and the P target network layers includes at least one of the following.

[0099] A first neuron coverage feature sub-vector is determined according to the initial adversarial image sample and the P1 first target network layers. A second neuron coverage feature sub-vector is determined according to the initial adversarial image sample and the P2 second target network layers. A third neuron coverage feature sub-vector is determined according to the initial adversarial image sample and the P3 third target network layers.

[0100] According to an embodiment of the present application, the neuron coverage activation threshold can include at least one of a first neuron coverage activation threshold, a second neuron coverage activation threshold, and a third neuron coverage activation threshold. The neuron coverage feature vector includes at least one of a first neuron coverage feature sub-vector, a second neuron coverage feature sub-vector, and a third neuron coverage feature sub-vector. P1, P2, P3 can be less than or equal to P.

[0101] According to an embodiment of the present application, the first neuron coverage activation threshold can refer to an evaluation threshold corresponding to a k-section neuron coverage distribution. P1 first target network layers can be determined according to the first neuron coverage activation threshold and neuron coverage distributions corresponding to M network layers, respectively. In this case, a first neuron coverage feature sub-vector can be determined according to the initial adversarial image sample and the P1 first target network layers. The first neuron coverage feature sub-vector can include a k-section neuron coverage feature sub-vector. The k-section neuron coverage can refer to dividing the minimum value and the maximum value of each neuron output value into k sections, i.e., considering that there are k states for each neuron.

[0102] According to an embodiment of the present application, the second neuron coverage activation threshold can refer to an evaluation threshold corresponding to a strong neuron activation coverage distribution. P2 second target network layers can be determined according to the second neuron coverage activation threshold and neuron coverage distributions corresponding to M network layers, respectively. In this case, a second neuron coverage feature sub-vector can be determined according to the initial adversarial image sample and the P2 second target network layers. The second neuron coverage feature sub-vector can include a strong neuron activation coverage feature sub-vector. The strong neuron activation coverage can refer to considering that a neuron output value higher than a maximum value or lower than a minimum value is in a boundary case, considering that the output value higher than the maximum value is an upper boundary case, and considering that the output value lower than the minimum value is a lower boundary case. The strong neuron activation coverage focuses on coverage of the upper boundary case.

[0103] According to an embodiment of the present application, the third neuron coverage activation threshold can refer to an evaluation threshold corresponding to a top-k neuron coverage distribution. P3 third target network layers can be determined according to the third neuron coverage activation threshold and neuron coverage distributions corresponding to M network layers, respectively. In this case, a third neuron coverage feature sub-vector can be determined according to the initial adversarial image sample and the P3 third target network layers. The third neuron coverage feature sub-vector can include a top-k neuron coverage feature sub-vector. The top-k neuron coverage can refer to the top k neurons with the highest activation degree in each layer of the model.

[0104] According to the embodiment of the present application, since the P target network layers are determined according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers, the neuron coverage distribution corresponding to each of the M network layers is obtained by inputting the initial adversarial image sample into the deep learning model, and by determining the neuron coverage feature vector according to the initial adversarial image sample and the P target network layers, a plurality of neuron coverage indexes are utilized to comprehensively extract the features of the initial adversarial image sample, thereby improving the comprehensiveness and interpretability of the adversarial sample.

[0105] According to the embodiment of the present application, the neuron coverage activation threshold can be obtained in the following manner.

[0106] The second original sample set is subjected to adversarial attack to obtain an intermediate adversarial sample set. The second original sample set and the intermediate adversarial sample set are input into the deep learning model to obtain a first intermediate neuron coverage feature vector set corresponding to the second original sample set and a second intermediate neuron coverage feature vector set corresponding to the intermediate adversarial sample set. The initial neuron coverage activation threshold of the deep learning model is adjusted according to the first intermediate neuron coverage feature vector set and the second intermediate neuron coverage feature vector set until a predetermined end condition is met, and the neuron coverage activation threshold is obtained.

[0107] According to the embodiment of the present application, the second original sample set can include at least one second original sample. For each of the at least one second original sample, the second original sample can be subjected to adversarial attack based on an adversarial attack algorithm to obtain an intermediate adversarial sample. The adversarial attack algorithm can be set according to actual requirements, which is not limited herein.

[0108] For example, the adversarial attack algorithm can include at least one of the following: a fast gradient sign method (FGSM), a Jacobian-based saliency map attack (JSMA), a Carlini & Wagner (CW) based adversarial attack algorithm, a limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) based adversarial attack algorithm, a basic iterative attack (BIA), a projected gradient descent (PGD), and a distributionally adversarial attack (DAA).

[0109] According to an embodiment of the present application, a model instrumentation process can be performed on a deep learning model to facilitate obtaining output value information of neurons in the model during execution of sample input to the deep learning model, so as to statistically obtain neuron coverage distribution characteristics. The model instrumentation manner can be set according to actual requirements, which is not limited herein.

[0110] For example, the model instrumentation manner can include at least one of an Aspect Oriented Programming (AOP) model instrumentation manner, an Object Oriented Programming (OOP) model instrumentation manner, and a TensorFlow library-based model instrumentation manner. Taking the TensorFlow library-based model instrumentation manner as an example, a Session.run function in the TensorFlow library can be used, which can receive tensor information of a hidden layer of a deep neural network and return a calculation result of the tensor.

[0111] According to an embodiment of the present application, the neuron coverage characteristics of the second original samples and the initial adversarial image samples are calculated by inputting the samples in the second original sample set and the initial adversarial sample set into the instrumented deep learning model respectively. Different neuron activation thresholds need to be set for calculating each neuron coverage, which can be optimized through multiple iterations to obtain an optimal neuron coverage activation threshold that can distinguish the second original samples and the initial adversarial image samples, so as to extract neuron coverage characteristics based on the neuron coverage activation threshold subsequently.

[0112] According to an embodiment of the present application, since the neuron coverage activation threshold is obtained by adjusting the initial neuron coverage activation threshold of the deep learning model according to the first intermediate neuron coverage feature vector set and the second intermediate neuron coverage feature vector set until a predetermined end condition is met, the neuron coverage activation threshold can be used to divide the neuron activation threshold boundary of the normal samples and the adversarial samples, thereby improving the quality of the adversarial samples.

[0113] According to an embodiment of the present application, operation S230 can include the following operations.

[0114] The neuron coverage feature vectors respectively corresponding to the at least one initial adversarial image sample are input to an objective function to obtain gradient data sets respectively corresponding to the at least one neuron coverage feature sub-vector set. The gradient data sets include gradient data respectively corresponding to the at least one initial adversarial image sample. The perturbation data set is determined according to the gradient data respectively corresponding to the at least one initial adversarial image sample.

[0115] According to an embodiment of the present application, the feature neuron is triggered in a targeted manner by setting the optimization target as the neuron coverage feature according to the extracted adversarial sample. At the same time, in order to explore more adversarial samples, new neurons are triggered with a certain probability, thereby achieving a balance between exploration and exploitation. For example, in the process of fuzz testing, the possibility of triggering the feature neuron and causing the model to make mistakes is maximized as a joint optimization target, and the objective function can be as shown in equation (1) below. The objective function can be maximized by gradient ascent.

[0116] (1)

[0117] wherein, may represent the original class label of the input sample. may represent one of the top-k class labels with a confidence lower than , used to guide the input to cross the decision boundary of the original class label and enter the decision space of the top-k other class labels with higher confidence. may represent the feature neuron, used to guide the coverage of the feature neuron. may represent the remaining neurons in the model that are not covered except the feature neuron, used to guide the coverage of the remaining neurons and avoid missing adversarial samples that have not been explored. , may represent the parameters. may represent that the coverage of the feature neuron is given priority, may represent the value of the objective function. According to an embodiment of the present application, since the perturbation data set is determined according to the gradient data corresponding to each of the at least one initial adversarial image sample, and the gradient data set is obtained by inputting the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample into the objective function, the fuzz testing technique using coverage guidance can efficiently generate low-perturbation and high-quality adversarial samples.

[0118] According to an embodiment of the present application, operation S240 can include the following operations.

[0119] For each of the at least one initial adversarial image sample, target perturbation data corresponding to the initial adversarial image sample is determined. The initial adversarial image sample is processed according to the target perturbation data to obtain a candidate target adversarial image sample corresponding to the initial adversarial image sample.

[0120] According to an embodiment of the present application, operation S250 can include the following operations.

[0121] According to an embodiment of the present application, operation S250 can include the following operations.

[0122] ​According to the predetermined threshold, the at least one candidate target adversarial image sample is respectively evaluated to obtain an evaluation result corresponding to each of the at least one candidate target adversarial image sample. The target adversarial sample set is determined according to the evaluation result corresponding to each of the at least one candidate target adversarial image sample.

[0123] According to an embodiment of the present application, the perturbation data set can include perturbation data corresponding to each of the at least one initial adversarial image sample.

[0124] According to an embodiment of the present application, after obtaining the perturbation data set, target perturbation data corresponding to each of the initial adversarial image sample can be determined. The initial adversarial image sample is processed by taking the target perturbation data as a perturbation added to the initial adversarial image sample to obtain a candidate target adversarial image sample corresponding to the initial adversarial image sample.

[0125] According to an embodiment of the present application, after obtaining the candidate target adversarial image sample, the candidate target adversarial image sample can be evaluated according to a predetermined threshold to obtain an evaluation result. The evaluation result can be used to represent whether the evaluation of the candidate target adversarial image sample passes. The target adversarial sample set can be determined according to the evaluation result. For example, in the case that the candidate target adversarial image sample does not cause the model to classify incorrectly, the effective adversarial sample generation can be guided by optimization.

[0126] According to an embodiment of the present application, the at least one candidate target adversarial image sample is respectively evaluated according to a predetermined threshold to obtain an evaluation result corresponding to each of the at least one candidate target adversarial image sample can include the following operations.

[0127] For a candidate target adversarial image sample in the at least one candidate target adversarial image sample, a first feature vector corresponding to a first original image sample is determined according to the first original image sample corresponding to the candidate target adversarial image sample. A second feature vector corresponding to the candidate target adversarial image sample is determined according to the candidate target adversarial image sample. The similarity between the first feature vector and the second feature vector is determined. In the case that the similarity is greater than or equal to a predetermined threshold, an evaluation result representing that the evaluation of the candidate target adversarial image sample fails is determined. In the case that the similarity is less than the predetermined threshold, an evaluation result representing that the evaluation of the candidate target adversarial image sample passes is determined.

[0128] According to an embodiment of the present application, after obtaining the candidate target adversarial image sample, the candidate target adversarial image sample can be evaluated by calculating the distance between the first original image sample and the candidate target adversarial image sample. The calculation formula can be as shown in the following formula (2). The sample with a distance value less than a predetermined threshold can be determined as a low-perturbation adversarial sample to retain a low-perturbation high-quality adversarial sample.

[0129] (2)

[0130] wherein, The original sample can be characterized. The original sample can be characterized as an adversarial sample. The original sample can be characterized as a norm, The norm can be 0, 2 and In the case of 0, the calculation of the norm In the case of 2, the calculation of the norm In the case of 2, the calculation of the norm In the case of 2, the calculation of the norm In the case of 2, the calculation of the norm In the case of 2, the calculation of the norm In the case of 2, the calculation of the norm In the case of 2, the calculation of the norm

[0131] According to an embodiment of the present application, in the case of the first original sample being a text sample, since the text data is discrete, it is necessary to map the discrete word data to a vector, and the distance between two vectors is calculated by the Euclidean distance or the cosine distance.

[0132] According to an embodiment of the present application, the distance between the first original image sample and the candidate target adversarial image sample can also be calculated by determining a first feature vector of the first original image sample, determining a second feature vector of the candidate target adversarial image sample, and determining a similarity between the first feature vector and the second feature vector.

[0133] According to an embodiment of the present application, the similarity can be used to characterize the degree of similarity between the first feature vector and the second feature vector. The relationship between the value of the similarity and the degree of similarity can be configured according to actual business requirements, which is not limited herein. For example, the greater the value of the similarity, the greater the degree of similarity between the first feature vector and the second feature vector. Conversely, the smaller the degree of similarity between the first feature vector and the second feature vector. Alternatively, the smaller the value of the similarity, the greater the degree of similarity between the first feature vector and the second feature vector. Conversely, the smaller the degree of similarity between the first feature vector and the second feature vector.

[0134] According to an embodiment of the present application, since the evaluation result is obtained by evaluating at least one candidate target adversarial image sample according to a predetermined threshold, and the candidate target adversarial image sample is obtained by processing the initial adversarial image sample according to the target perturbation data, the evaluation result can be used to characterize whether the candidate target adversarial image sample passes the evaluation. On this basis, by determining the target adversarial sample set according to the evaluation result corresponding to each of the at least one candidate target adversarial image sample, the generation efficiency of the target adversarial sample set is improved, and the quality of the target adversarial samples in the target adversarial sample set is further improved.

[0135] Figure 5 An example schematic diagram of a target adversarial sample generation process according to an embodiment of the present application is schematically shown.

[0136] As shown in FIG. 5, in operation S500, for each of at least one initial adversarial image sample 501, target perturbation data 502 corresponding to the initial adversarial image sample 501 can be determined. The initial adversarial image sample 501 can be processed according to the target perturbation data 502 to obtain a candidate target adversarial image sample 503 corresponding to the initial adversarial image sample 501. Figure 5

[0137] According to the first original image sample 504 corresponding to the candidate target adversarial image sample 503, a first feature vector 506 corresponding to the first original image sample 504 can be determined. According to the candidate target adversarial image sample 503, a second feature vector 505 corresponding to the candidate target adversarial image sample 503 can be determined.

[0138] After obtaining the first feature vector 506 and the second feature vector 505, a similarity 507 between the first feature vector 506 and the second feature vector 505 can be determined. After obtaining the similarity 507, operation S510 can be performed.

[0139] In operation S510, it is determined whether the similarity 507 is greater than or equal to a predetermined threshold 508.

[0140] If yes, an evaluation result 509 representing that the evaluation of the candidate target adversarial sample fails can be determined.

[0141] If no, an evaluation result 510 representing that the evaluation of the candidate target adversarial sample passes can be determined. In the case where the evaluation result represents that the evaluation of the candidate target adversarial sample passes, the candidate target adversarial image sample 503 can be determined as a target adversarial sample 511.

[0142] According to an embodiment of the present application, after obtaining a target adversarial sample set, the target adversarial sample set can be used to retrain a deep learning model to obtain an updated deep learning model, so as to improve the robustness and security of the deep learning model. The updating method of the deep learning model can include the following operations.

[0143] The target adversarial sample set is input into the deep learning model to obtain an output information set, wherein the target adversarial sample set has a corresponding label information set. At least one loss function value is determined according to the output information set and the label information set. The model parameters of the deep learning model are adjusted according to the at least one loss function value to obtain an updated deep learning model.

[0144] ​According to an embodiment of the present application, the first original sample set can be processed to obtain an initial adversarial sample set. The initial adversarial sample set is input into the deep learning model to obtain a neuron coverage feature vector set. According to the target function and at least one neuron coverage feature sub-vector set, a perturbation data set is determined. According to the perturbation data set, the initial adversarial image sample set is processed to obtain a candidate target adversarial image sample set. According to the first original sample set, the candidate target adversarial image sample set is evaluated to obtain a target adversarial sample set.

[0145] According to an embodiment of the present application, after obtaining the target adversarial sample set, the target adversarial sample set can be input into the deep learning model to obtain an output information set. After obtaining the output information set, at least one loss function value can be determined based on the loss function according to the output information set and the label information set. After obtaining the at least one loss function value, the model parameters of the deep learning model can be adjusted according to the at least one loss function value to obtain an updated deep learning model.

[0146] For example, the model parameters of the deep learning model can be adjusted according to a back propagation algorithm or a stochastic gradient descent algorithm until a predetermined condition is met. The deep learning model obtained when the predetermined condition is met is determined as the updated deep learning model. The predetermined condition can include at least one of loss function value convergence and training round reaching a maximum training round.

[0147] According to an embodiment of the present application, since the target adversarial sample set is generated by using the adversarial sample generation method, the generation efficiency of the adversarial sample is improved. On this basis, since the at least one loss function value is determined according to the output information set and the label information set, and the output information set is obtained by inputting the target adversarial sample set into the deep learning model, by adjusting the model parameters of the deep learning model according to the at least one loss function value, the updated deep learning model is obtained, and the robustness and security of the deep learning model are improved.

[0148] The above is only an exemplary embodiment, but is not limited thereto, and other adversarial sample generation methods known in the art can also be included as long as the generation efficiency of the adversarial sample is improved.

[0149] Figure 6 A block diagram of an adversarial sample generation apparatus according to an embodiment of the present application is schematically shown.

[0150] As shown in Figure 6 The adversarial sample generation apparatus 600 can include an acquisition module 610, a first input module 620, a first determination module 630, a processing module 640, and an evaluation module 650.

[0151] The acquisition module 610 is configured to acquire an initial adversarial sample set, where the initial adversarial sample set includes an initial adversarial image sample set, and the initial adversarial image sample set includes at least one initial adversarial image sample.

[0152] The first input module 620 is configured to input the initial adversarial image sample set into the deep learning model to obtain a neuron coverage feature vector set, where the neuron coverage feature vector set includes a neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample.

[0153] The first determination module 630 is configured to determine a perturbation data set according to a target function and the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample, where the target function is associated with the deep learning model.

[0154] The processing module 640 is configured to process the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set.

[0155] The evaluation module 650 is configured to evaluate the candidate target adversarial image sample set according to the first original sample set to obtain a target adversarial sample set.

[0156] According to an embodiment of the present application, the deep learning model includes M network layers.

[0157] According to an embodiment of the present application, the first input module 620 can include a first input submodule, a first determination submodule and a second determination submodule.

[0158] The first input submodule is configured to input each of the at least one initial adversarial image sample into the deep learning model to obtain a neuron coverage distribution corresponding to each of the M network layers.

[0159] The first determination submodule is configured to determine the neuron coverage feature vector according to a neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers.

[0160] According to an embodiment of the present application, M is a positive integer greater than 1.

[0161] According to an embodiment of the present application, the neuron coverage activation threshold can be obtained in the following manner.

[0162] The second original sample set is subjected to an adversarial attack to obtain an intermediate adversarial sample set. The second original sample set and the intermediate adversarial sample set are input into the deep learning model to obtain a first intermediate neuron coverage feature vector set corresponding to the second original sample set and a second intermediate neuron coverage feature vector set corresponding to the intermediate adversarial sample set. The initial neuron coverage activation threshold of the deep learning model is adjusted according to the first intermediate neuron coverage feature vector set and the second intermediate neuron coverage feature vector set until a predetermined end condition is met, and the neuron coverage activation threshold is obtained.

[0163] According to an embodiment of the present application, the first determining sub-module can include a first determining unit and a second determining unit.

[0164] The first determining unit is configured to determine P target network layers according to the neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers, wherein P is a positive integer greater than 1, and P is less than or equal to M.

[0165] The second determining unit is configured to determine the neuron coverage feature vector according to the initial adversarial image sample and the P target network layers.

[0166] According to an embodiment of the present application, the neuron coverage activation threshold includes at least one of the first neuron coverage activation threshold, the second neuron coverage activation threshold, and the third neuron coverage activation threshold. The neuron coverage feature vector includes at least one of the first neuron coverage feature sub-vector, the second neuron coverage feature sub-vector, and the third neuron coverage feature sub-vector.

[0167] According to an embodiment of the present application, the first determining unit can include at least one of a first determining sub-unit, a second determining sub-unit, and a third determining sub-unit.

[0168] The first determining sub-unit is configured to determine P1 first target network layers according to the first neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers.

[0169] The second determining sub-unit is configured to determine P2 second target network layers according to the second neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers.

[0170] The third determining sub-unit is configured to determine P3 third target network layers according to the third neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers.

[0171] According to an embodiment of the present application, the second determining unit can include at least one of a fourth determining sub-unit, a fifth determining sub-unit, and a sixth determining sub-unit.

[0172] The fourth determining sub-unit is configured to determine a first neuron coverage feature sub-vector according to the initial adversarial image sample and P1 first target network layers.

[0173] The fifth determining sub-unit is configured to determine a second neuron coverage feature sub-vector according to the initial adversarial image sample and P2 second target network layers.

[0174] The sixth determining sub-unit is configured to determine a third neuron coverage feature sub-vector according to the initial adversarial image sample and P3 third target network layers.

[0175] According to an embodiment of the present application, P1, P2 and P3 are less than or equal to P.

[0176] According to an embodiment of the present application, the first determining module 630 can include a second input sub-module and a second determining sub-module.

[0177] The second input sub-module is configured to input the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample to the target function to obtain a gradient data set corresponding to each of the at least one neuron coverage feature sub-vector set, wherein the gradient data set includes gradient data corresponding to each of the at least one initial adversarial image sample.

[0178] The second determining sub-module is configured to determine a perturbation data set according to the gradient data corresponding to each of the at least one initial adversarial image sample.

[0179] According to an embodiment of the present application, the perturbation data set includes perturbation data corresponding to each of the at least one initial adversarial image sample.

[0180] According to an embodiment of the present application, the processing module 640 can include a third determining sub-module and a processing sub-module.

[0181] The third determining sub-module is configured to determine, for each of the at least one initial adversarial image sample, target perturbation data corresponding to the initial adversarial image sample.

[0182] The processing sub-module is configured to process the initial adversarial image sample according to the target perturbation data to obtain a candidate target adversarial image sample corresponding to the initial adversarial image sample.

[0183] According to an embodiment of the present application, the evaluation module 650 can include an evaluation sub-module and a fourth determining sub-module.

[0184] The evaluation sub-module is configured to evaluate each of the at least one candidate target adversarial image sample according to a predetermined threshold to obtain an evaluation result corresponding to each of the at least one candidate target adversarial image sample.

[0185] A fourth determining sub-module is configured to determine the target adversarial sample set according to evaluation results corresponding to the at least one candidate target adversarial image sample.

[0186] According to an embodiment of the present application, the evaluation sub-module can include a first input unit, a second input unit, a third determining unit, a fourth determining unit and a fifth determining unit.

[0187] The first input unit is configured to determine, for a candidate target adversarial image sample in the at least one candidate target adversarial image sample, a first feature vector corresponding to a first original image sample corresponding to the candidate target adversarial image sample.

[0188] The second input unit is configured to determine, according to the candidate target adversarial image sample, a second feature vector corresponding to the candidate target adversarial image sample.

[0189] The third determining unit is configured to determine a similarity between the first feature vector and the second feature vector.

[0190] The fourth determining unit is configured to determine an evaluation result representing that the candidate target adversarial image sample fails to pass evaluation, in a case where the similarity is greater than or equal to a predetermined threshold.

[0191] The fifth determining unit is configured to determine an evaluation result representing that the candidate target adversarial image sample passes evaluation, in a case where the similarity is less than the predetermined threshold.

[0192] According to an embodiment of the present application, the adversarial sample generation device 600 can further include a third determining module, a fourth determining module and a mutation processing module.

[0193] The third determining module is configured to, in response to receiving an adversarial sample generation instruction, determine, according to a first original sample set in the adversarial sample generation instruction, a sample type corresponding to the first original sample set, wherein the first original sample set includes at least one first original image sample, and the at least one first original image sample each corresponds to label information.

[0194] The fourth determining module is configured to determine, according to the sample type, a mutation strategy corresponding to the sample type.

[0195] The mutation processing module is configured to perform mutation processing on the at least one first original image sample respectively according to the mutation strategy, to obtain initial adversarial image samples each corresponding to the at least one first original image sample.

[0196] According to an embodiment of the present application, the first original sample set further includes at least one first original text sample and / or at least one first original audio sample.

[0197] Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application, or at least part of functions of any of them, can be implemented in one module. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be split into multiple modules for implementation. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be implemented at least in part as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application-specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware by integrating or packaging circuits, or in any one of software, hardware and firmware implementation or in a proper combination of any of them. Alternatively, one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be implemented at least in part as computer program modules, which can perform corresponding functions when executed.

[0198] For example, any of the modules 610, 620, 630, 640 and 650 can be combined in one module / unit / sub-unit for implementation, or any of them can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of the modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to the embodiments of the present application, at least one of the modules 610, 620, 630, 640 and 650 can be implemented at least in part as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application-specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware by integrating or packaging circuits, or in any one of software, hardware and firmware implementation or in a proper combination of any of them. Alternatively, at least one of the modules 610, 620, 630, 640 and 650 can be implemented at least in part as computer program modules, which can perform corresponding functions when executed.

[0199] It should be noted that the part of the adversarial sample generation device in the embodiments of the present application corresponds to the part of the adversarial sample generation method in the embodiments of the present application, and the description of the part of the adversarial sample generation device is specifically referred to the part of the adversarial sample generation method, which will not be repeated here.

[0200] Figure 7 A block diagram of an electronic device suitable for implementing the method of generating adversarial samples according to an embodiment of the present application is shown schematically. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the function and scope of use of embodiments of the present application.

[0201] As Figure 7 shown, the computer electronic device 700 according to an embodiment of the present application includes a processor 701 which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 709 into a random access memory (RAM) 703. The processor 701 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 can also include an on-board memory for cache use. The processor 701 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present application.

[0202] In the RAM 703, various programs and data required for the operation of the electronic device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. The processor 701 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 702 and / or the RAM 703. It is noted that the programs can also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0203] According to an embodiment of the present application, the electronic device 700 can further include an input / output (I / O) interface 705 that is also connected to the bus 704. The electronic device 700 can further include one or more of the following components connected to the input / output (I / O) interface 705: an input part 706 including a keyboard, a mouse, etc.; an output part 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 708 including a hard disk, etc.; and a communication part 709 including a network interface card such as a LAN card, a modem, etc. The communication part 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as necessary. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 710 as necessary, so that a computer program read therefrom is installed in the storage part 708 as necessary.

[0204] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, the embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program codes for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system implementing the embodiment of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0205] The present application also provides a computer-readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, which when executed, implement the method according to the embodiment of the present application.

[0206] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium. For example, it can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device.

[0207] For example, according to an embodiment of the present application, the computer readable storage medium can include one or more memories of the ROM 702 and / or the RAM 703 described above and / or other than the ROM 702 and the RAM 703.

[0208] Embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the method provided by the embodiments of the present application, and when the computer program product is run on an electronic device, the program codes are used to make the electronic device implement the method for generating an adversarial sample provided by the embodiments of the present application.

[0209] When the computer program is executed by the processor 701, the above-mentioned functions defined in the system / apparatus of the embodiments of the present application are executed. According to an embodiment of the present application, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0210] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 709, and / or be installed from the detachable medium 711. The program codes contained in the computer program can be transmitted by any suitable network medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0211] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming language can include, but is not limited to, Java, C++, python, "C" language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0212] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0213] Embodiments of the present application have been described. However, these embodiments are merely meant to be illustrative of the present application and are not meant to limit the scope of the present application. Although each of the embodiments has been described above, it is not meant that measures in each of the embodiments cannot be advantageously used in combination. The scope of the present application is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present application, and such substitutions and modifications are to be included within the scope of the present application.

Claims

1. A method for generating adversarial samples, comprising: obtaining an initial adversarial sample set, wherein the initial adversarial sample set comprises an initial adversarial image sample set, and the initial adversarial image sample set comprises at least one initial adversarial image sample; inputting the initial adversarial image sample into a deep learning model to obtain neuron coverage distributions corresponding to M network layers of the deep learning model, M being a positive integer greater than 1; determining neuron coverage feature vectors according to a neuron coverage activation threshold and the neuron coverage distributions corresponding to the M network layers, wherein the neuron coverage feature vector set comprises neuron coverage feature vectors corresponding to the at least one initial adversarial image sample; determining a perturbation data set according to a target function and the neuron coverage feature vectors corresponding to the at least one initial adversarial image sample, wherein the target function is associated with the deep learning model; processing the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set; and evaluating the candidate target adversarial image sample set to obtain a target adversarial sample set; wherein the neuron coverage activation threshold is obtained by: performing adversarial attacks on a second original sample set to obtain an intermediate adversarial sample set; inputting the second original sample set and the intermediate adversarial sample set into the deep learning model to obtain a first intermediate neuron coverage feature vector set corresponding to the second original sample set and a second intermediate neuron coverage feature vector set corresponding to the intermediate adversarial sample set; and adjusting an initial neuron coverage activation threshold of the deep learning model according to the first intermediate neuron coverage feature vector set and the second intermediate neuron coverage feature vector set until a predetermined end condition is met to obtain the neuron coverage activation threshold.

2. The method of claim 1, wherein, The determination of the neuron coverage feature vectors according to the neuron coverage activation threshold and the neuron coverage distributions corresponding to the M network layers comprises: determining P target network layers according to the neuron coverage activation threshold and the neuron coverage distributions corresponding to the M network layers, wherein P is a positive integer greater than 1, and P is less than or equal to M; and determining the neuron coverage feature vectors according to the initial adversarial image sample and the P target network layers.

3. The method of claim 2, wherein, The neuron coverage activation threshold comprises at least one of a first neuron coverage activation threshold, a second neuron coverage activation threshold, and a third neuron coverage activation threshold, and the neuron coverage feature vectors comprise at least one of a first neuron coverage feature sub-vector, a second neuron coverage feature sub-vector, and a third neuron coverage feature sub-vector. The determination of the P target network layers according to the neuron coverage activation threshold and the neuron coverage distributions corresponding to the M network layers comprises at least one of: determining P1 first target network layers according to the first neuron coverage activation threshold and the neuron coverage distributions corresponding to the M network layers; and determining P2 second target network layers according to the second neuron coverage activation threshold and the neuron coverage distributions corresponding to the M network layers. determining P2 second target network layers according to the second neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers; and determining P3 third target network layers according to the third neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers; the determining the neuron coverage feature vector according to the initial adversarial image sample and the P target network layers comprises at least one of the following: determining the first neuron coverage feature sub-vector according to the initial adversarial image sample and the P1 first target network layers; determining the second neuron coverage feature sub-vector according to the initial adversarial image sample and the P2 second target network layers; and determining the third neuron coverage feature sub-vector according to the initial adversarial image sample and the P3 third target network layers; wherein P1, P2, P3 are less than or equal to P.

4. The method of any one of claims 1 to 3, wherein, the determining the perturbation data set according to the objective function and the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample comprises: inputting the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample into the objective function to obtain a gradient data set, wherein the gradient data set comprises gradient data corresponding to each of the at least one initial adversarial image sample; and determining the perturbation data set according to the gradient data corresponding to each of the at least one initial adversarial image sample.

5. The method of any one of claims 1 to 3, wherein, the perturbation data set comprises perturbation data corresponding to each of the at least one initial adversarial image sample; the processing the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set comprises: for each initial adversarial image sample in the at least one initial adversarial image sample, determining target perturbation data corresponding to the initial adversarial image sample; and processing the initial adversarial image sample according to the target perturbation data to obtain a candidate target adversarial image sample corresponding to the initial adversarial image sample; the evaluating the candidate target adversarial image sample set to obtain a target adversarial sample set comprises: evaluating at least one of the candidate target adversarial image samples according to a predetermined threshold to obtain an evaluation result corresponding to each of the at least one candidate target adversarial image sample; and determining the target adversarial sample set according to the evaluation result corresponding to each of the at least one candidate target adversarial image sample.

6. The method of claim 5, wherein, the evaluating at least one of the candidate target adversarial image samples according to a predetermined threshold to obtain an evaluation result corresponding to each of the at least one candidate target adversarial image sample comprises: for a candidate target adversarial image sample in the at least one candidate target adversarial image sample, determining a first feature vector corresponding to a first original image sample according to the first original image sample corresponding to the candidate target adversarial image sample; determining a second feature vector corresponding to the candidate target adversarial image sample according to the candidate target adversarial image sample; determining a similarity between the first feature vector and the second feature vector; in a case where the similarity is greater than or equal to the predetermined threshold, determining the evaluation result representing that the candidate target adversarial image sample evaluation fails; and in a case where the similarity is less than the predetermined threshold, determining the evaluation result representing that the candidate target adversarial image sample evaluation passes.

7. The method of any one of claims 1 to 3, further comprising, before the obtaining the initial adversarial sample set: In response to receiving the adversarial sample generation instruction, a sample type corresponding to a first original sample set in the adversarial sample generation instruction is determined, wherein the first original sample set comprises at least one first original image sample, each of the at least one first original image sample corresponding to label information; determining a variation strategy corresponding to the sample type according to the sample type; and performing variation processing on the at least one first original image sample respectively according to the variation strategy to obtain an initial adversarial image sample corresponding to each of the at least one first original image sample.

8. The method of claim 7, wherein, The first original sample set further comprises at least one first original text sample and / or at least one first original audio sample.

9. An adversarial sample generation apparatus, comprising: an obtaining module configured to obtain an initial adversarial sample set, wherein the initial adversarial sample set comprises an initial adversarial image sample set, and the initial adversarial image sample set comprises at least one initial adversarial image sample; a first input sub-module configured to input the initial adversarial image sample into a deep learning model to obtain a neuron coverage distribution corresponding to each of M network layers in the deep learning model, M being a positive integer greater than 1; a first determination sub-module configured to determine a neuron coverage feature vector according to a neuron coverage activation threshold and the neuron coverage distribution corresponding to each of the M network layers, wherein the neuron coverage feature vector set comprises a neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample; a first determination module configured to determine a perturbation data set according to a target function and the neuron coverage feature vector corresponding to each of the at least one initial adversarial image sample, wherein the target function is associated with the deep learning model; a processing module configured to process the initial adversarial image sample set according to the perturbation data set to obtain a candidate target adversarial image sample set; and an evaluation module configured to evaluate the candidate target adversarial image sample set to obtain a target adversarial sample set; wherein the neuron coverage activation threshold is obtained by: performing adversarial attack on a second original sample set to obtain an intermediate adversarial sample set; inputting the second original sample set and the intermediate adversarial sample set into the deep learning model to obtain a first intermediate neuron coverage feature vector set corresponding to the second original sample set and a second intermediate neuron coverage feature vector set corresponding to the intermediate adversarial sample set; and adjusting an initial neuron coverage activation threshold of the deep learning model according to the first intermediate neuron coverage feature vector set and the second intermediate neuron coverage feature vector set until a predetermined end condition is met to obtain the neuron coverage activation threshold.

10. An electronic device, comprising: one or more processors; a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors are caused to implement the method of any one of claims 1-8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1-8.

Images