Convolutional neural network adversarial sample generation method, test method and related device

Abstract

The invention discloses a convolutional neural network adversarial sample generation method, a convolutional neural network adversarial sample test method and a related device, and relates to the technical field of adversarial machine learning. The method can directly guide the output value of uncovered neurons (namely unactivated neurons) of a generated sample to be increased and tend to be in a covered state, so that the neurons become active, the neuron coverage rate of the generated sample is increased, the quality and diversity of the generated sample are improved, good and stable effects can be achieved on various types of CNN models, and the method is suitable for popularization and application. And the test sufficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of adversarial machine learning technology, and in particular to a convolutional neural network adversarial sample generation method, a testing method, and related devices for improving neuron coverage. Background Art

[0002] In recent years, convolutional neural networks (CNNs) have achieved outstanding results in image classification tasks with their powerful feature extraction capabilities. However, relevant studies have shown that CNN models are vulnerable to adversarial attacks, resulting in incorrect judgments. This has also raised concerns about the reliability and security of deep learning models. Adversarial attacks refer to adding tiny perturbations that are not easily perceived by humans to input samples, misleading the model to produce incorrect outputs with high confidence. The effect of this attack is particularly prominent in computer vision tasks such as image classification. Studies have found that even the most advanced CNN models are vulnerable to such attacks, which exposes potential security risks. If an attacker can make the CNN model mistake a stop sign for a permitted sign and a left turn for a right turn, the safety of self-driving cars driving on actual roads will be significantly reduced, posing serious safety risks. Therefore, in the process of testing CNN models, evaluating their ability to resist adversarial samples has become a key consideration.

[0003] The adversarial attack-based testing method is to apply different adversarial algorithms to add a certain degree of perturbation on the original test set, generate adversarial samples as model input, and reflect the ability of the CNN model to resist adversarial samples through test indicators. Existing studies have shown that when a qualified test set reaches a high neuron coverage rate, it can ensure that the model is fully tested. However, how to effectively improve the neuron coverage rate of the model in adversarial sample testing and ensure the adequacy of the test still needs to be improved. Summary of the invention

[0004] The purpose of this application is to provide a convolutional neural network adversarial sample generation method, testing method and related devices, which combine the adversarial sample generation process with neuron coverage, and can actively improve the neuron coverage of generated samples and improve test adequacy while retaining the adversarial function.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for generating adversarial samples of a convolutional neural network, comprising:

[0007] Obtaining a model under test and a test data set, wherein the model under test is a convolutional neural network model;

[0008] Input the image to be processed in the test data set into the model under test, and calculate the objective function value of the target neuron of the current iteration round according to the objective function, wherein the image to be processed is an image obtained by preprocessing the original image in the test data set, and the target neuron is an inactivated neuron randomly selected from the model under test, and the objective function is a function calculated according to adversarial related item variables and coverage related item variables, the value of the adversarial related item variable refers to the difference between the predicted output and the actual label of the model under test calculated using the cross entropy loss function, the value of the coverage related variable refers to the output value of the target neuron, and the output value of the target neuron is data obtained by querying a neuron statistical template, and the neuron statistical template includes the number, type, location information, output value and coverage state of each neuron in the model under test, and the coverage state is an activated state or an inactivated state;

[0009] The image to be processed is perturbed according to the objective function value and the perturbation threshold to obtain an adversarial image.

[0010] In a second aspect, the present application provides a convolutional neural network adversarial sample testing method, comprising:

[0011] Obtain the model under test, test data set and test requirement information;

[0012] When the test task type in the test requirement information is to directly test the test data set, calculating the first neuron coverage and the first classification accuracy corresponding to the original image in the test data set processed by the tested model;

[0013] Obtaining a first test result according to the first neuron coverage rate and the first classification accuracy rate;

[0014] When the test task type in the test requirement information is to first generate the required adversarial samples and then conduct the test, firstly execute the convolutional neural network adversarial sample generation method to generate adversarial images, and then calculate the second neuron coverage and the second classification accuracy corresponding to the tested model when processing the adversarial images;

[0015] A second test result is obtained according to the second neuron coverage and the second classification accuracy.

[0016] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the convolutional neural network adversarial sample generation method or the convolutional neural network adversarial sample testing method.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the convolutional neural network adversarial sample generation method or the convolutional neural network adversarial sample testing method.

[0018] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the convolutional neural network adversarial sample generation method, or the convolutional neural network adversarial sample testing method.

[0019] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0020] The present application provides a convolutional neural network adversarial sample generation method, a testing method and related devices. The coverage-related term variables used in the objective function of the adversarial sample generation method are closely related to the coverage improvement target, and can directly guide the output values ​​of neurons not covered by the generated samples (i.e., neurons that are not activated) to increase and tend to be covered, making them active, thereby improving the neuron coverage of the generated samples, improving the quality and diversity of the generated samples, so that good and stable results can be achieved on various types of CNN models, and improving the test adequacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 This is an application environment diagram of a convolutional neural network adversarial sample generation method in Example 1 of the present application;

[0023] Figure 2 A schematic diagram of a process for generating adversarial samples of a convolutional neural network provided in Example 1 of the present application;

[0024] Figure 3 A flowchart of a convolutional neural network adversarial sample testing method provided in Example 2 of the present application;

[0025] Figure 4 A schematic diagram of the structure of a computer device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0028] Example 1

[0029] The convolutional neural network adversarial sample generation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the model under test and the test data set to the server 104. After receiving the model under test and the test data set, the server 104 inputs the image to be processed in the test data set into the model under test, calculates the objective function value of the target neuron of the current iteration round according to the objective function, and performs perturbation processing on the image to be processed according to the objective function value and the perturbation threshold to obtain the adversarial image. The server 104 can feed back the obtained adversarial image to the terminal 102. In addition, in some embodiments, the convolutional neural network adversarial sample generation method can also be implemented separately by the server 104 or the terminal 102, such as the terminal 102 can directly use the convolutional neural network adversarial sample generation method to process the model under test and the test data set, or the server 104 can obtain the model under test and the test data set from the data storage system, and use the convolutional neural network adversarial sample generation method to process it.

[0030] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0031] In an exemplary embodiment, a method for generating adversarial samples of a convolutional neural network is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 203. Among them:

[0032] Step 201, obtaining a model under test and a test data set, wherein the model under test is a convolutional neural network model.

[0033] Step 202: input the image to be processed in the test data set into the model under test, and calculate the objective function value of the target neuron of the current iteration round according to the objective function, wherein the image to be processed is an image obtained by preprocessing the original image in the test data set, and the target neuron is an inactivated neuron randomly selected from the model under test, and the objective function is a function calculated according to adversarial related item variables and coverage related item variables, the value of the adversarial related item variable refers to the difference between the predicted output and the actual label of the model under test calculated using the cross entropy loss function, the value of the coverage related variable refers to the output value of the target neuron, and the output value of the target neuron is data obtained by querying a neuron statistical template, and the neuron statistical template includes the number, type, location information, output value and coverage status of each neuron in the model under test, and the coverage status is an activated state or an inactivated state.

[0034] Step 203: Perform perturbation processing on the image to be processed according to the objective function value and the perturbation threshold to obtain an adversarial image.

[0035] By implementing the above-mentioned steps 201 to 203, the output value of the neurons that have not been activated can be directly guided to increase during the adversarial sample generation process, making them active. On the basis of retaining the adversarial function, the neuron coverage of the generated samples can be actively improved, thereby improving the test adequacy.

[0036] In order to make the specific implementation process of the above steps 201 to 203 more clear to those skilled in the art, the following Figure 2 The process is explained in detail.

[0037] Add carefully designed tiny perturbations to the input of the model under test (i.e., the image to be processed) to generate adversarial samples with high neuron coverage, such as Figure 2 As shown, the process specifically includes:

[0038] (1) Establishment of neuron statistical template.

[0039] In the CNN model, it is usually composed of a fully connected layer and a convolutional layer. A fully connected layer means that any neuron in the network is connected to all neurons in the previous layer. This connection method allows information to be completely mixed between layers. The convolutional layer processes input data through convolution operations and is usually used in image recognition tasks. The convolution kernel (or filter) slides on the input data to extract local features. The fully connected layer and the convolutional layer jointly construct the complex structure and computing power of the neural network by defining the connection method and function of neurons.

[0040] According to the network structure of the model under test, a statistical template is established for each neuron in the form of a dictionary. Each neuron layer in the model and each neuron within the layer are numbered and the type of each neuron (fully connected layer neuron or convolutional layer neuron) is determined. At the same time, the position information of each neuron is determined to achieve the positioning of each neuron, so that the output value and coverage status (activated state or inactivated state) can be found later according to the positioning information.

[0041] The neuron statistics template is in dictionary format, with the tuple of (layer number, layer number) of the neuron as the key and the Boolean variable of the neuron's coverage status as the value (True-1-covered (i.e. activated), False-0-not covered (i.e. not activated)). When initialized, all neuron coverage states are False.

[0042] The neuron statistics template can be used to track the coverage status of each neuron in the neural network, helping to analyze and understand the working process of the neural network.

[0043] (2) Calculation of neuron output value.

[0044] In order to count the coverage status of each neuron, it is first necessary to obtain the output value of each type of neuron according to the type of neuron layer included in the statistics determined in step (1).

[0045] For the fully connected layer, each data processing unit is regarded as a neuron. The neurons in the current layer receive the outputs of all neurons in the previous layer, weighted summation according to the model weights, and the output value is obtained after the activation function processing. The expression is:

[0046]

[0047] in, is the output value of the current neuron n when processing the image x to be processed; σ is the activation function; k is the number of neurons in the previous layer connected to the neurons in the current fully connected layer; W i is the model weight corresponding to the output data of the i-th neuron in the previous layer; I iis the output data of the i-th neuron in the previous layer.

[0048] For the convolution layer, the convolution kernel is regarded as a neuron. The convolution kernel scans the input matrix and performs cross-correlation operations in each receptive area to obtain the output results out at each position. i,j (n,x), after being processed by the activation function, a square matrix is ​​output, and the average value of the square matrix is ​​taken as the output value of the neuron.

[0049] For two-dimensional convolution, the calculation results at each position are as follows:

[0050]

[0051] In the above formula, i, j represent the location information of the output results, and m, n represent the location information of the convolution kernel weights.

[0052] The output value of the convolutional layer neuron is:

[0053]

[0054] Among them, mean() is the averaging function, which takes the average of all elements in the matrix; out(n,x) is the original output value of convolutional layer neuron n in the case of input image x, in the form of a matrix, and the output value of position i, j in the output matrix is ​​recorded as out i,j (n,x);out i,j (n,x) is obtained by cross-correlating the weight matrix W of the convolutional layer neuron model with the input matrix I of the current layer neurons. When the output σ() acts on the matrix, the activation function is applied to the elements at each position of the output matrix, so a matrix of the same size is obtained. p,q are the width and height of the current convolutional layer neuron weight matrix (also known as the convolution kernel); W is the weight matrix; I [i～i+p,j～j+q] is a portion of the input matrix with rows from i to i+p and columns from j to j+q (so W and I [i～i+p,j～j+q] The sizes are all p in width and q in height, and then the corresponding positions are multiplied and added, which is called cross-correlation operation in convolutional neural networks).

[0055] The above calculation processes are all obtained through one CNN forward propagation.

[0056] (3) Neuron coverage status statistics.

[0057] Since the output values ​​of neurons in each layer are different, the output values ​​of neurons in the same layer are normalized:

[0058]

[0059] In the above formula, n l,idxrepresents the neuron numbered idx in the lth layer, n l represents all neurons in layer l, represents the original output value of the neuron, and φ represents the normalized output value.

[0060] After obtaining the normalized output value, compare it with the set threshold to determine the coverage state of each neuron. Given a neuron n, its coverage state s(n) includes activated and inactivated. In the process of counting the number of neuron coverage, they correspond to 1True and 0False respectively, and are stored in Boolean variables and recorded in the neuron statistics template.

[0061]

[0062] (4) Calculation of objective function.

[0063] Sample generation requires an objective function to guide the generation direction. In order to make the sample generation algorithm have both adversarial effects and coverage improvement effects, the objective function designed in this section contains adversarial related terms obj adv (x) and coverage related items obj nc (x) The sample generation process is regarded as a joint optimization problem, and the balance parameter λ is used to balance the two effects.

[0064] Calculate adversarial correlation term obj adv (x), the cross entropy loss function is used:

[0065] obj adv (x)=CrossEntropyLoss(x,y).

[0066] Calculate coverage related items obj nc (x), first obtain the list of all neurons in the current network, remove the covered neurons from the list, and get N pro = {n|s(n,x) = 0}, randomly select an uncovered neuron as the target neuron n of the current iteration round obj , get its location information l,idx, query its output value and use it as obj nc (x) Return.

[0067] n obj =random(N pro ,1);

[0068]

[0069] The final objective function is calculated according to the following formula:

[0070] obj(x)=obj adv(x)+λ·obj nc (x).

[0071] Wherein, obj(x) is the objective function value; obj adv (x) is the value of the adversarial variable; the balance parameter λ is a non-negative number set by the user. The larger its value, the stronger the coverage improvement effect and the weaker the adversarial effect; obj nc (x) is the value of the coverage-related variable; CrossEntropyLoss() is the cross entropy loss function; x is the image to be processed; y pre is the predicted label of x by the tested model; n obj is the target neuron of the current iteration round; is the target neuron n obj The output value when processing the image x to be processed; random() is a random selection function, from N pro A neuron is randomly selected as the target neuron; N pro is the set of neurons that have not yet reached the covered state.

[0072] (5) Addition of counter-disturbance.

[0073] Starting from the output of the objective function, the CNN back propagation algorithm is executed to update and record the gradient of the image to be processed. The image gradient sign (positive or negative) is used as the direction of perturbation generation. A certain degree of perturbation is applied to each pixel of the image to be processed according to the size of the perturbation degree ε to generate an adversarial image.

[0074]

[0075] In the above formula, x adv Represents the generated adversarial sample, ε represents the degree of perturbation, and sign is the sign function.

[0076] The part that exceeds the pixel value range after disturbance is cut off. If it exceeds the maximum value, it is uniformly classified as the maximum value, and if it is less than the minimum value, it is uniformly classified as the minimum value. The processed adversarial image is added to the output list adv_list.

[0077] x adv =clip(x adv ,upper,lower);

[0078] In the above formula, clip represents the clipping operation, upper represents the upper bound of the pixel value range, and lower represents the lower bound of the pixel value range.

[0079] Optional:

[0080] 1) Apply the disturbance in steps.

[0081] If the perturbation step number is set to step>1, then steps (2)-(5) are repeated. To ensure that the overall perturbation level does not exceed the limit of ε, the perturbation addition in step 3.5 is performed according to the following formula, and finally an adversarial image is generated. Step-by-step perturbation can make the perturbation addition more refined, and achieve the same generation effect with smaller perturbations.

[0082]

[0083] 2) Input sample multiplexing.

[0084] If the number of sample reuse loops is set to > 1, then the entire process of steps (2) to (5) (including applying perturbations in steps) is repeated. nc (x) Each time a target neuron is randomly selected, randomness is introduced, so each reuse will generate a different adversarial image. Input sample reuse can return multiple adversarial samples based on an original image, and the number of adversarial samples is consistent with the parameter loop, which can effectively expand the test data set and improve the diversity of the test and the robustness of the model.

[0085] The convolutional neural network adversarial sample generation method provided in this embodiment can be specifically used for testing autonomous driving systems. It is necessary to test its image classifier, which usually belongs to the CNN model. The tester can judge whether the CNN model is sufficiently tested by the neuron coverage rate, and needs to ensure that the test results meet a certain established indicator, such as a coverage rate of 60%. If the coverage rate is too low, the tester needs to find more samples and corresponding labels for testing.

[0086] Assume that the model under test is a CNN-based road sign classifier. The original images in the test dataset include four categories and their corresponding labels: left-turn sign images, right-turn sign images, no parking sign images, and allowed sign images. Using all the original images can only achieve 50% neuron coverage.

[0087] The convolutional neural network adversarial sample generation method proposed in this embodiment is applied to add tiny disturbances that are difficult to detect with the naked eye to the original image to generate a new test image (adversarial sample), and the label of the new sample is consistent with the original image. After the adversarial sample input model is tested, the neuron coverage of the model will be improved to meet the coverage requirements of the test link. If the algorithm proposed in this embodiment is not used to generate new test samples, the tester needs to find new image samples and their labels from the natural environment for testing, and such image samples may not necessarily improve the neuron coverage of the model, so the entire test link needs to invest more additional manpower and material resources.

[0088] The CNN adversarial sample generation method for improving neuron coverage in this embodiment has the following advantages:

[0089] Advantage 1: The coverage improvement function of this method is stable, and the final coverage is high. This is because the coverage improvement related items used in the objective function of the sample generation method are closely related to the coverage improvement target, which can directly guide the output value of the neurons not covered by the generated sample to increase and tend to be covered, thereby improving the neuron coverage of the generated sample.

[0090] Advantage 2: This method can generate multiple adversarial samples for an original image and has good data set expansion capabilities. This is because the coverage improvement related terms used in the objective function of this sample generation method introduce randomness, which can activate uncovered neurons at various positions in the model under test, thereby obtaining different adversarial samples. However, once the parameters of the classic adversarial attack algorithm are determined, only one adversarial sample can be generated for an original image.

[0091] The present application also provides an application scenario, which applies the above-mentioned convolutional neural network adversarial sample generation method. Specifically: the convolutional neural network adversarial sample generation method provided in this embodiment can be applied in the security testing scenario of the computer vision system. The scenario includes a model training link, an adversarial sample generation link, and a system evaluation link. A convolutional neural network adversarial sample generation method provided in this embodiment is a key technology in the adversarial sample generation link.

[0092] Embodiment 2:

[0093] Research has found that related adversarial attack-based testing methods reflect the ability of CNN models to resist adversarial samples through indicators such as accuracy. Due to the high-dimensional nature of the sample input space, it is not feasible to traverse this space to find all possible adversarial samples, so the test can only be performed based on a limited number of samples. However, adversarial attack-based testing methods do not consider whether the generated test inputs can cover most of the system logic, and testers lack a clear basis to judge whether the test is sufficient. This may cause some test samples to be redundant, which does not help to reveal new defects in the model and consumes a lot of testing resources. On the other hand, even if every sample in the test set is used for adversarial testing, the adequacy of the test cannot be guaranteed, and there may be a large number of undiscovered logical defects in the model. Therefore, developing effective test adequacy metrics and ensuring that the model testing process reaches the adequacy of these indicators has become the key to improving test efficiency and effectiveness.

[0094] In software testing, code line coverage, that is, the ratio of executed code lines to total code lines, is a key indicator for measuring test adequacy. Ideally, the higher the coverage, the more adequate the code test. However, the decision logic of the CNN model is not explicitly written in the code, but is learned by the many neurons that make up the CNN through training data, which means that code line coverage is not enough to evaluate the test adequacy of the CNN model decision logic. Therefore, researchers borrowed the coverage criteria in software testing and proposed the concept of neuron coverage to measure the adequacy of the test from the neuron activity and network behavior inside the model. Existing studies have shown that a qualified test set should achieve a high neuron coverage to ensure that the model is adequately tested. Nevertheless, how to provide a convolutional neural network adversarial sample testing method to effectively improve the model's neuron coverage in adversarial sample testing and ensure the adequacy of the test still needs to be improved.

[0095] In this regard, this embodiment applies the convolutional neural network adversarial sample generation method in Example 1 to provide a convolutional neural network adversarial sample testing method, such as Figure 3 As shown, the testing method realizes adversarial sample generation and completes testing for the test image dataset, including five parts: demand information import, original image preprocessing, adversarial sample generation, sample testing and test result export.

[0096] Step 1: Import demand information.

[0097] This part meets user needs and provides a test information input interface. The test requirement information includes the test task type, the model under test, the input preprocessing requirements of the model under test, the test data set path, the neuron coverage test threshold, the type of neuron layer included in the coverage statistics, the name of the sample generation algorithm, the degree of perturbation, the number of steps for applying perturbation, the number of original sample inputs, the NC-FGSM algorithm balance parameters, and the test result storage path.

[0098] The test task category task can choose to directly test the original dataset (test), or first generate the required adversarial samples and then test (generate).

[0099] Table 1 shows all the test requirements for a test task, which will guide the execution of subsequent commands. The model to be tested and the test data set can be selected by the user.

[0100] Table 1 Test requirement information parameter table

[0101]

[0102] Step 2: Original image preprocessing.

[0103] This part processes the original images in the test dataset into test inputs that can be recognized by the model under test. The specific research steps are as follows:

[0104] Step 2.1: Resize. Adjust the image shape according to the preprocessing requirements of transform_info in step 1, for example, adjust the shape of the original image to 256*256.

[0105] Step 2.2: CenterCrop, according to the preprocessing requirements of transform_info in step 1, the central area is captured, for example, the 224*224 part in the center is captured to meet the image input size of the model under test.

[0106] Step 2.3: Convert to tensor Totensor, convert the original image into tensor form, and change the RGB three-channel value range of each pixel to [0,1];

[0107] Step 2.4: Normalize, perform normalization operations on each channel separately to ensure that the image input in the test phase is consistent with the training process.

[0108]

[0109] In the above formula, x ori is the original image, x is the final image, transform.mean and transform.std are the mean and standard deviation required for standardization, respectively, which are provided by transform_info in step 1.

[0110] Step 3: Adversarial sample generation.

[0111] The adversarial sample generation process adopts the convolutional neural network adversarial sample generation method provided in Implementation 1.

[0112] Step 4: Sample testing.

[0113] The sample test can process both the preprocessed images after step 2 and the adversarial images after step 3. The test indicators include neuron coverage and classification accuracy.

[0114] Neuron coverage is the ratio of covered neurons to all neurons. Query the neuron statistics template and get the neuron coverage NC based on the coverage status of each neuron:

[0115]

[0116] In the above formula, num represents the number of neurons that meet the conditions, and s(n) represents the coverage status of neurons.

[0117] The classification accuracy is the proportion of samples in the test data set that are correctly predicted. Compare the predicted labels of each test sample with the true labels to see if they are consistent, and calculate the classification accuracy Acc:

[0118]

[0119] In the above formula, x represents the test sample, T represents the set of test samples, and y * represents the predicted label of the test sample, and y represents the true label of the test sample.

[0120] Step 5: Export the test results.

[0121] According to the parameter settings and test results in steps 1-4, the results are exported in the form of a csv file, and the recorded information is shown in Table 2 below. A neuron coverage change graph is automatically generated, with the horizontal axis being the number of test images and the vertical axis being the neuron coverage NC, and they are saved together to the path save_path specified by the user in step 1.

[0122] Table 2 Test result record

[0123]

[0124] The convolutional neural network adversarial sample testing method provided in this embodiment can also be applied to the field of autonomous driving. For a large number of image classification tasks in the field of autonomous driving, such as identifying road signs indicating left turns, right turns, allowed passage, prohibited passage, etc. Most of these images are RGB three-channel color images. If an attacker can make the CNN model mistake the prohibited passage sign for allowed passage and recognize the left turn as a right turn, the safety of the autonomous driving car driving on the actual road will be significantly reduced, and there will be serious safety hazards. Therefore, researchers need to fully test the classification model, and the adequacy of the test can be reflected by the neuron coverage rate. This embodiment improves the neuron coverage rate of the CNN model by generating new test images, thereby improving the adequacy of the test.

[0125] The present application also provides an application scenario, which applies the above-mentioned convolutional neural network adversarial sample testing method. Specifically: the convolutional neural network adversarial sample testing method provided in this embodiment can be applied in the safety verification scenario of the autonomous driving system. The scenario includes a sensor data collection link, an adversarial sample generation link, and a system response test link. A convolutional neural network adversarial sample testing method provided in this embodiment is a core technology in the system response test link. By analyzing the system's response to these adversarial samples, its robustness and safety under various challenging conditions are evaluated, thereby improving the system's protective measures and improving overall safety.

[0126] Example 3

[0127] This embodiment provides a computer device, which may be a server or a terminal. Its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the convolutional neural network adversarial sample generation method provided in Example 1 or the convolutional neural network adversarial sample testing method provided in Example 2. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the convolutional neural network adversarial sample generation method provided in Example 1 or the convolutional neural network adversarial sample testing method provided in Example 2 is implemented.

[0128] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] Example 4

[0130] This embodiment provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the convolutional neural network adversarial sample generation method provided in Embodiment 1 or the convolutional neural network adversarial sample testing method provided in Embodiment 2 is implemented.

[0131] Example 5

[0132] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the convolutional neural network adversarial sample generation method provided in Embodiment 1 or the convolutional neural network adversarial sample testing method provided in Embodiment 2.

[0133] Example 6

[0134] This embodiment provides a computer program product, including a computer program, which, when executed by a processor, implements the convolutional neural network adversarial sample generation method provided in Embodiment 1 or the convolutional neural network adversarial sample testing method provided in Embodiment 2.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0136] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0137] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for generating adversarial samples of a convolutional neural network, characterized in that: The convolutional neural network adversarial sample generation method comprises: Obtaining a model under test and a test data set, wherein the model under test is a convolutional neural network model; Input the image to be processed in the test data set into the model under test, and calculate the objective function value of the target neuron of the current iteration round according to the objective function, wherein the image to be processed is an image obtained by preprocessing the original image in the test data set, and the target neuron is an inactivated neuron randomly selected from the model under test, and the objective function is a function calculated according to adversarial related item variables and coverage related item variables, the value of the adversarial related item variable refers to the difference between the predicted output and the actual label of the model under test calculated using the cross entropy loss function, the value of the coverage related variable refers to the output value of the target neuron, and the output value of the target neuron is data obtained by querying a neuron statistical template, and the neuron statistical template includes the number, type, location information, output value and coverage state of each neuron in the model under test, and the coverage state is an activated state or an inactivated state; The image to be processed is perturbed according to the objective function value and the perturbation threshold to obtain an adversarial image.

2. The method for generating adversarial samples of a convolutional neural network according to claim 1, characterized in that: The pre-treatment process specifically includes: According to the size requirement of the input image of the model under test, the original image is cropped to obtain a cropped image; Performing data format conversion and normalization processing on the cropped image to obtain a processed image; The processed image is subjected to standardization processing to obtain the image to be processed.

3. The method for generating adversarial samples of a convolutional neural network according to claim 1, characterized in that: The specific calculation process of the output value of each neuron includes: When the type of the neuron is a fully connected layer neuron, the calculation expression of the output value of each neuron is: in, is the output value of the current neuron n when processing the image x to be processed; σ is the activation function; k is the number of neurons in the previous layer connected to the neurons in the current fully connected layer; W idx is the model weight corresponding to the output data of the idx-th neuron in the previous layer; Iidx is the output data of the idx-th neuron in the previous layer; When the type of the neuron is a convolutional layer neuron, the calculation expression of the output value of each neuron is: outi,j(n,x)=W·I[i~i+p,j~j+q]; Among them, mean() is the averaging function; out(n,x) is the original output value of convolutional layer neuron n in the case of input image x; out i,j (n,x) is the output value at position i,j in the out(n,x) output matrix; σ() is the activation function; p,q are the width and height of the current convolutional layer neuron weight matrix; W is the weight matrix; I [i～i+p,j～j+q] The input matrix consists of the i-th to i+p-th rows and the j-th to j+q-th columns.

4. The method for generating adversarial samples of a convolutional neural network according to claim 1, characterized in that: The expression of the objective function includes: obj(x)=obj adv (x)+λ·obj nc (x); obj adv (x)=CrossEntropyLoss(x,y pre ); n obj =random(N pro ,1); Wherein, obj(x) is the objective function value; obj adv (x) is the value of the adversarial variable; λ is the equilibrium parameter; obj nc (x) is the value of the coverage-related variable; CrossEntropyLoss() is the cross entropy loss function; x is the image to be processed; y pre is the predicted label of x by the tested model; n obj is the target neuron of the current iteration round; is the target neuron n obj The output value when processing the image x to be processed; random() is a random selection function; N pro is the set of neurons that have not yet reached the covered state.

5. The method for generating adversarial samples of a convolutional neural network according to claim 1, characterized in that: Performing perturbation processing on the image to be processed according to the objective function value and the perturbation threshold to obtain an adversarial image specifically includes: Taking the objective function value as a starting point, a back propagation algorithm is used to calculate the gradient of the image to be processed; determining a disturbance direction according to the gradient; When the number of disturbance steps is 1, the image to be processed is disturbed according to the disturbance direction and the disturbance threshold to obtain a first disturbed image; Removing pixels beyond a preset pixel range in the first disturbed image to obtain an adversarial image; When the number of disturbance steps is greater than 1, the image to be processed is disturbed a preset number of times according to the disturbance direction and each disturbance degree value to obtain a second disturbed image, wherein the disturbance degree value is a ratio between the disturbance threshold and the number of disturbance steps, and the preset number is equal to the number of disturbance steps; Pixels exceeding a preset pixel range in the second disturbed image are removed to obtain an adversarial image.

6. A convolutional neural network adversarial sample testing method, characterized in that: The convolutional neural network adversarial sample testing method comprises: Obtain the model under test, test data set and test requirement information; When the test task type in the test requirement information is to directly test the test data set, calculating the first neuron coverage and the first classification accuracy corresponding to the original image in the test data set processed by the tested model; Obtaining a first test result according to the first neuron coverage rate and the first classification accuracy rate; When the test task type in the test requirement information is to first generate the required adversarial samples and then conduct the test, first execute the convolutional neural network adversarial sample generation method described in any one of claims 1 to 5 to generate an adversarial image, and then calculate the second neuron coverage and the second classification accuracy corresponding to the tested model when processing the adversarial image; A second test result is obtained according to the second neuron coverage and the second classification accuracy.

7. The convolutional neural network adversarial sample testing method according to claim 6, characterized in that: The expression of the second neuron coverage is: Wherein, NC is the second neuron coverage rate; n is a neuron; num represents the number of neurons that meet the conditions; s(n) is the coverage status of the neuron; The expression of the second classification accuracy is: Where Acc is the second classification accuracy; x represents the adversarial image; T is the set of adversarial images; y * is the predicted label of the adversarial image; y is the true label of the adversarial image.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the convolutional neural network adversarial sample generation method described in any one of claims 1 to 5, or the convolutional neural network adversarial sample testing method described in any one of claims 6 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the convolutional neural network adversarial sample generation method described in any one of claims 1-5, or the convolutional neural network adversarial sample testing method described in any one of claims 6-7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the convolutional neural network adversarial sample generation method described in any one of claims 1-5, or the convolutional neural network adversarial sample testing method described in any one of claims 6-7.

Images