Cycle code test data generation method based on deep learning fuzzing

Abstract

The application discloses a cyclic code test data generation method based on deep learning fuzzy testing, comprising the following steps: guiding the selection of cyclic code seed data with high uncertainty by using the output uncertainty of cyclic code on a deep learning model; designing nine kinds of mutation rules to realize the mutation of cyclic code of different types; analyzing the correlation degree of each neuron and the error decision logic of the deep learning model; taking the maximum neuron coverage rate and the number of error behaviors as the joint optimization target, modeling the optimization target, taking the derivative of the target function as the gradient, selecting the mutation rule to mutate the seed, and using the gradient to iteratively mutate; realizing the mutation of the selected cyclic code seed data to complete the generation of cyclic code test data. The application uses the classification output probability of cyclic code on a deep learning model and related neuron information to design an optimization target, so as to effectively guide the mutation of cyclic code.

Description

Technical Field

[0001] This invention belongs to the field of deep learning test data generation, specifically relating to a method for generating test data for loop code based on deep learning fuzz testing. Background Technology

[0002] With the rapid development of deep learning technology, deep learning models trained on large amounts of code data are widely used in various fields such as autonomous driving, facial recognition, and speech recognition. This paper presents a method and system for generating test data for loop code based on deep learning fuzzing, aiming to generate high-quality loop code for testing deep learning models. Loop code is one of the most common structures in computer programs, and its correctness is crucial to the overall program correctness. Testing loop code is an important part of software testing. In loop code testing, a loop is typically considered a combination of a loop body and a loop condition. The code within the loop body is executed multiple times until the loop condition is no longer true. Traditional testing methods often only cover part of the code's execution path, and loop code testing is particularly prone to omissions.

[0003] To improve the coverage and efficiency of loop code testing, deep learning-based fuzzing has increasingly attracted researchers' attention in recent years. Traditional fuzzing is a black-box testing method that discovers vulnerabilities and errors in programs by randomly generating input data. In fuzzing-based loop code testing, the test data generator randomly generates a set of input data and uses it as the execution input for the loop body. The test data generator also generates a set of input data that satisfies the loop condition based on its characteristics. In this way, the test data generator can cover all execution paths of the loop code with as little test data as possible. Deep learning-based fuzzing combines deep learning technology with traditional fuzzing techniques. By automatically learning the characteristics and patterns of loop code, it generates random, anomalous loop code test data to test the robustness of deep learning models. Currently, existing deep learning-based fuzzing methods for generating loop code test data primarily test the accuracy of the corresponding deep learning models by randomly generating some loop code test data.

[0004] Compared with traditional test data generation methods, the loop code test data generation method based on deep learning fuzzing has the following advantages: (1) It can generate a large amount of test data, cover more model decision paths, and discover more potential boundary conditions and anomalies in the model, thereby improving test effectiveness. (2) It can use deep learning models to extract loop code features for testing without needing to have an in-depth understanding of the code structure and algorithm. (3) It can reduce the time and cost of producing loop code test data.

[0005] However, the method for generating test data for loop code based on deep learning fuzzing also has the following problems: (1) The quality of test data is difficult to guarantee; randomly generated test data may not cover all decision paths of the model, and may even ignore some important test data. (2) Since fuzzing is based on randomness, the generation of test data is uncertain, which may lead to the repeated generation of the same test data or the generation of invalid test data.

[0006] Therefore, there is an urgent need for an efficient method for generating loop code test data based on deep learning fuzzing. This method improves fuzzing to reduce the randomness of test data generation and incorporates deep learning technology to design mutation rules that ensure the quality and efficiency of the generated test data, thus enabling the generation of loop code test data. This technique can not only generate high-quality loop code for testing deep learning models but also has good adaptability and flexibility, making it suitable for various application scenarios. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, the present invention aims to provide a method for generating loop code test data based on deep learning fuzzing, thereby solving the problems of low efficiency and difficulty in collecting rare loop codes to expose model defects caused by them in existing loop code generation techniques. The present invention utilizes the concept of fuzzing, designs a seed data selection strategy based on uncertainty, selects high-quality loop codes as seed data, mutates the loop code seed test data to generate new loop codes as test data, guides the DL system to expose more defects, and improves defect detection efficiency.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] The present invention provides a method for generating test data for loop code based on deep learning fuzzing, comprising the following steps:

[0010] 1) Utilize the output uncertainty of loop codes in deep learning models to guide the selection of seed data for loop codes with high uncertainty;

[0011] 2) Nine mutation rules were designed for loop code to achieve different types of mutations for loop code;

[0012] 3) By designing an interpretable neuron selection method, analyze the correlation between each neuron and the erroneous decision-making logic of the deep learning model, and select neurons with high correlation as target neurons;

[0013] 4) The joint optimization objective is to maximize the coverage of target neurons and the number of erroneous behaviors. The model of the joint optimization objective is to use the derivative of the objective function as the gradient, select a mutation rule to mutate the seed, and use the gradient for iterative mutation.

[0014] 5) Based on the mutation rules in step 2), the neuron selection method in step 3), and the optimization objective in step 4), the mutation of the loop code seed data selected in step 1) is realized to complete the generation of loop code test data.

[0015] Further, step 1) specifically includes:

[0016] 11) Divide the test data for loop code into different categories according to the type of loop it belongs to;

[0017] 12) In each test dataset, the loop code test data is converted into vectors using the Word2Vec word embedding method to obtain the original features of the loop code, which are then input into the deep learning model to further extract the output vector of its last layer.

[0018] 13) Using the output vector extracted in step 12), the deep learning model classification problem is simplified into a problem of measuring the impurities of the test set; the uncertainty of the test data is defined as follows: given a test data t, and the output vector P of the test data t on the tested model. t = <p t,1 ,p t,2 ,...,p t,N >, The output vector value P of the test data t on the model under test t Calculate the output purity of the test data Furthermore, the output impurity ξ(t) is obtained, which measures the output uncertainty of the test data t on the tested model; the formula for calculating ξ(t) is as follows:

[0019]

[0020] In the formula, p t,i Let t be the probability that t belongs to class i; N is the total number of classes in the loop code test dataset;

[0021] 14) Based on step 13), calculate the uncertainty of each loop code test data in the model output in each category set, sort the loop code test data in each category set in descending order of uncertainty value, and uniformly select the loop code data with high uncertainty at the top of each sorted set as seed data.

[0022] Furthermore, the nine mutation rules in step 2) specifically include:

[0023] 21) Change the step size or direction of the loop variable;

[0024] 22) Change the loop termination condition;

[0025] 23) Modify the operations or expressions within the loop body;

[0026] 24) Insert, delete, or replace statements within a loop body;

[0027] 25) Change the variable reference in the loop body;

[0028] 26) Modify function calls within the loop body;

[0029] 27) Change the value of constants in the loop body;

[0030] 28) Move statements from the loop body to the outside of the loop, or move statements from the outside of the loop to the inside of the loop body;

[0031] 29) Copy the statements in the loop body and execute them twice, or delete part of the statements in the loop body.

[0032] Furthermore, step 2) specifically includes: the nine mutation rules are combined in any pair to form a combined mutation rule; single mutation rules and combined mutation rules can be used to generate a series of variants for testing the robustness and correctness of the deep learning model.

[0033] Furthermore, step 3) specifically includes:

[0034] 31) Select loop codes that the model predicted incorrectly from the loop code test set, use the Word2Vec word embedding method to convert the loop code test data into vectors, and input them into the deep learning model to extract the corresponding feature vectors for each layer.

[0035] 32) Based on the feature vectors of each layer of the predicted erroneous loop code in the model under test, the Layer-wise Relevance Propagation algorithm is used to measure the actual contribution of each neuron to the erroneous decision of the test data, obtain the relevance score of each neuron, and select a specified number of neurons with high relevance scores as target neurons.

[0036] Furthermore, step 4) specifically includes:

[0037] Two optimization objectives are designed: maximizing the neuron coverage of the deep learning model and maximizing the number of erroneous behaviors in the deep learning model. These two objectives are combined as a joint optimization objective, and the optimization objective formula is designed as follows:

[0038]

[0039] The objective consists of two parts: the first part is... The uncertainty of a data set is measured by calculating the output probability of the loop code test data; c is the original class label of the input, c i For each of the top k class labels with a confidence level slightly lower than c, i = 0, ..., k, maximizing the first part guides the loop code test data to cross the decision boundary of its original class and enter the decision space of the top k other classes, leading to more unstable prediction results. This results in higher uncertainty for the modified loop code test data, making it more likely to be misidentified as other types by the model; the second part is... m is the number of target neurons to be activated, and n is the number of neurons to be activated. i A target neuron is defined as the neuron to be activated; a neuron selection strategy is used to guide the selection of target neurons, thereby improving neuron coverage; the hyperparameter λ is used to balance the two targets.

[0040] Furthermore, step 5) specifically includes:

[0041] 51) Use the uncertainty-based loop code seed data selection method in step 1) to select a certain proportion (this proportion is set manually) of the seed data set;

[0042] 52) Based on the correlation-based neuron selection method in step 3), select neurons that are highly correlated with the decision logic of the type to which the loop code belongs and which is incorrectly identified by the model as target neurons;

[0043] 53) Design the following two optimization objectives:

[0044] The first optimization objective is designed using the output probabilities: maximize the model's erroneous behavior;

[0045] Using the highly relevant target neurons selected in step 52), a second optimization objective is designed: to maximize the model neuron coverage.

[0046] The first and second optimization objectives are combined to generate a joint optimization objective, and an optimization formula is established for the joint optimization objective. The gradient value is obtained by differentiating the formula.

[0047] 54) Based on the mutation rules in step 2), select or combine any single mutation rule, traverse each seed in the seed data set in step 51), apply the selected mutation rule to each seed for initial mutation, and use the gradient calculated in step 53) to iteratively perturb the seed input to generate intermediate mutation code. Stop perturbation when the intermediate mutation loop code and the loop code seed data are inconsistent in the output category on the tested model, and use the mutated loop code as the final adversarial loop code test data.

[0048] The present invention also provides a generation terminal, comprising:

[0049] One or more processors;

[0050] Memory, used to store one or more programs;

[0051] When the one or more programs are executed by the one or more processors, the one or more processors implement the loop code test data generation method based on deep learning fuzzing.

[0052] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating loop code test data based on deep learning fuzz testing.

[0053] The beneficial effects of this invention are:

[0054] This invention utilizes the output information of the loop code in a deep learning model to measure the uncertainty of the loop code and designs a seed selection strategy based on uncertainty. Secondly, it uses the interpretability technique—the relevance score layer-by-layer propagation algorithm—to design a neuron selection strategy based on relevance. Finally, based on the classification task of the type to which the loop code belongs, it uses the classification output probability of the loop code in the deep learning model and the relevant neuron information to design an optimization objective to effectively guide the mutation of the loop code.

[0055] Specifically, it has the following main advantages:

[0056] 1. This invention measures the output uncertainty of loop code on the model to select seed data with high uncertainty and diversity, which can effectively guide the generation of loop code with strong error disclosure ability and diverse categories, and ensure more thorough testing of the model under test.

[0057] 2. This invention utilizes an interpretable neuron selection strategy to select neurons that contribute significantly to model error recognition as components of the optimization objective. This can guide the seed data of the loop code to mutate along the gradient direction with higher efficiency into loop code that can expose the model's erroneous behavior, while also more fully testing more of the model's decision-making logic. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the principle of the method of this invention. Detailed Implementation

[0059] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0060] Reference Figure 1 As shown, the present invention provides a method for generating test data for loop code based on deep learning fuzzing, comprising the following steps:

[0061] 1) Uncertainty-based seed data selection strategy for loop codes: Utilize the output uncertainty of loop codes in deep learning models to guide the selection of seed data for loop codes with high uncertainty;

[0062] 2) Loop code mutation rules: Nine mutation rules are designed for loop code to achieve different types of mutations for loop code;

[0063] 3) Relevance-based neuron selection strategy: By designing an interpretable neuron selection method, the correlation between each neuron and the erroneous decision-making logic of the deep learning model is analyzed, and neurons with high correlation are selected as target neurons;

[0064] 4) Optimization target design strategy: Maximize the coverage of target neurons and the number of erroneous behaviors as the joint optimization objective. Model the joint optimization objective, use the derivative of the objective function as the gradient, select a mutation rule to mutate the seed, and use the gradient for iterative mutation.

[0065] 5) Fuzzy process: Based on the mutation rules in step 2), the neuron selection method in step 3), and the optimization objective in step 4), the mutation of the loop code seed data selected in step 1) is realized to complete the generation of loop code test data.

[0066] In the example,

[0067] (1) Juliet Test Suite for C / C++, LoopInvGen Benchmark Suite and NASA dataset were used as the loop code dataset, and were divided into different categories according to the loop control conditions and the statement type inside the loop body; the specific categories are: ①For loop; ②While loop; ③Do-while loop; ④Nested loop; ⑤Recursive loop; ⑥Parallel loop.

[0068] (2) The Word2Vec word embedding method is used to convert the loop code test data of the 6 categories into input vectors, which are then input into the deep learning model to obtain the output vector of each loop code test data t in each category on the model. <p t,1 ,p t,2 ,p t,3 ,p t,4 ,p t,5 ,p t,6 Based on the probabilities of each category in the output vector, the uncertainty metric formula is used. Calculate the uncertainty value of the loop code t, and sort the loop code test data in each category set from largest to smallest according to the uncertainty value of the loop code. Set the number of seed data to 600, and evenly select the first 100 loop code test data from each sorted category set. Combine the test data selected from the 6 category sets to form the seed set.

[0069] (3) Input the complete original loop code test set into the model under test, obtain the loop code test data that was identified incorrectly, and use the correlation score propagation algorithm to obtain the correlation score of each neuron in the model for the model's identification error. By comparing the correlation scores, select the 5 target neurons with the highest scores.<n1,n2,n3,n4,n5> As a basic component of the design optimization objectives in subsequent steps.

[0070] (4) Based on the output vector obtained in step (2) <p t,1 ,p t,2 ,p t,3 ,p t,4 ,p t,5 ,p t,6 > Design the first optimization objective formula c is the original category label of t, c i (i = 0, ..., 3) is one of the top 3 class labels with a confidence level lower than c. Maximizing this formula can guide the loop code to maximize the output uncertainty of the test data, making it difficult for the model to identify correctly. Based on the 5 target neurons with the highest relevance scores selected in step (3), a second optimization objective formula is designed. Maximizing this formula guides the model to maximize neuron test coverage. Using these two optimization objective formulas, a joint optimization objective is designed. To maximize the number of erroneous behaviors in the model while increasing neuron coverage; λ = 0.5 (set manually) is used to balance the weights of these two optimization objectives.

[0071] (5) Select a single mutation rule—change the step size or direction of the loop variable, and use this mutation rule to mutate the loop code seed test data selected in step (2) one by one. Calculate the gradient using the optimization objective in step (4), and iteratively add gradients to the mutation rule to achieve iterative mutation of the loop code seed test data, continuously generating mutated loop codes. When the generated mutated loop code is inconsistent with the output category of the corresponding original loop code seed test data on the tested model, it indicates that the mutated loop code has triggered an error in the model. Then, end the mutation of the corresponding seed test data, and use the mutated loop code as adversarial loop code test data to form the generated test dataset.

[0072] This invention has many specific applications. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of this invention, and these improvements should also be considered within the scope of protection of this invention.

Claims

1. A method for generating test data for loop code based on deep learning fuzzing, characterized in that, The steps are as follows: 1) Utilize the output uncertainty of loop codes in deep learning models to guide the selection of seed data for loop codes with high uncertainty; 2) Nine mutation rules were designed for loop code to achieve different types of mutations for loop code; 3) By designing an interpretable neuron selection method, analyze the correlation between each neuron and the erroneous decision-making logic of the deep learning model, and select neurons with high correlation as target neurons; 4) The joint optimization objective is to maximize the coverage of target neurons and the number of erroneous behaviors. The model of the joint optimization objective is to use the derivative of the objective function as the gradient, select a mutation rule to mutate the seed, and use the gradient for iterative mutation. 5) Based on the mutation rules in step 2), the neuron selection method in step 3), and the optimization objective in step 4), the mutation of the loop code seed data selected in step 1) is achieved to complete the generation of loop code test data; Step 1) specifically includes: 11) Divide the test data for loop code into different categories according to the type of loop it belongs to; 12) In each test dataset, the loop code test data is converted into vectors using the Word2Vec word embedding method to obtain the original features of the loop code, which are then input into the deep learning model to further extract the output vector of its last layer. 13) Using the output vector extracted in step 12), the deep learning model classification problem is simplified into a problem of measuring the impurities of the test set; the uncertainty of the test data is defined as follows: given a test data t, and the output vector P of the test data t on the tested model. t = , ; Using the test data t, output vector values ​​P on the model under test t Calculate the output purity of the test data And further obtain the output impurity. , is used to measure the output uncertainty of the test data t on the tested model; The calculation formula is as follows: ； In the formula, Let t be the probability that t belongs to class i; N is the total number of classes in the loop code test dataset; 14) Based on step 13), calculate the uncertainty of each loop code test data in the model output in each category set, sort the loop code test data in each category set in descending order of uncertainty value, and uniformly select the loop code data with high uncertainty at the top of each sorted set as seed data.

2. The method for generating test data for loop code based on deep learning fuzz testing according to claim 1, characterized in that, The nine mutation rules in step 2) specifically include: 21) Change the step size or direction of the loop variable; 22) Change the loop termination condition; 23) Modify the operations or expressions within the loop body; 24) Insert, delete, or replace statements within a loop body; 25) Change the variable reference in the loop body; 26) Modify the function call within the loop body; 27) Change the value of constants in the loop body; 28) Move statements from the loop body to the outside of the loop, or move statements from the outside of the loop to the inside of the loop body; 29) Copy the statements in the loop body and execute them twice, or delete part of the statements in the loop body.

3. The method for generating test data for loop code based on deep learning fuzz testing according to claim 2, characterized in that, Step 2) further includes: the nine mutation rules are combined in any pair to form a combined mutation rule; single mutation rules and combined mutation rules can be used to generate a series of variants to test the robustness and correctness of the deep learning model.

4. The method for generating test data for loop code based on deep learning fuzz testing according to claim 1, characterized in that, Step 3) specifically includes: 31) Select loop codes that the model predicted incorrectly from the loop code test set, use the Word2Vec word embedding method to convert the loop code test data into vectors, and input them into the deep learning model to extract the corresponding feature vectors for each layer. 32) Based on the feature vectors of each layer of the predicted erroneous loop code in the model under test, the correlation score propagation algorithm is used to measure the actual contribution of each neuron to the erroneous decision of the test data, obtain the correlation score of each neuron, and select a specified number of neurons with high correlation scores as target neurons.

5. The method for generating test data for loop code based on deep learning fuzz testing according to claim 1, characterized in that, Step 4) specifically includes: Two optimization objectives are designed: maximizing the neuron coverage of the deep learning model and maximizing the number of erroneous behaviors in the deep learning model. These two objectives are combined as a joint optimization objective, and the optimization objective formula is designed as follows: ； The objective consists of two parts: the first part is... The uncertainty of a data set is measured by calculating the output probability of the loop code test data; c is the original class label of the input. For each of the top k class labels with a confidence level slightly lower than c, i = 0, ..., k, maximizing the first part guides the loop code test data to cross the decision boundary of its original class and enter the decision space of the top k other classes, leading to more unstable prediction results. This results in higher uncertainty for the modified loop code test data, making it more likely to be misidentified as other types by the model; the second part is... m is the number of target neurons that are planned to be activated. A target neuron is defined as the neuron to be activated; a neuron selection strategy is used to guide the selection of target neurons, thereby improving neuron coverage; the hyperparameter λ is used to balance the two targets.

6. The method for generating test data for loop code based on deep learning fuzz testing according to claim 1, characterized in that, Step 5) specifically includes: 51) Use the uncertainty-based loop code seed data selection method in step 1) to select a certain proportion of the seed data set; 52) Based on the correlation-based neuron selection method in step 3), select neurons that are highly correlated with the decision logic of the type to which the loop code belongs and which is incorrectly identified by the model as target neurons; 53) Design the following two optimization objectives: The first optimization objective is designed using the output probabilities: maximize the model's erroneous behavior; Using the highly relevant target neurons selected in step 52), a second optimization objective is designed: to maximize the model neuron coverage. The first and second optimization objectives are combined to generate a joint optimization objective, and an optimization formula is established for the joint optimization objective. The gradient value is obtained by differentiating the formula. 54) Based on the mutation rules in step 2), select or combine any single mutation rule, traverse each seed in the seed data set in step 51), apply the selected mutation rule to each seed for initial mutation, and use the gradient calculated in step 53) to iteratively perturb the seed input to generate intermediate mutation code. Stop perturbation when the intermediate mutation loop code and the loop code seed data are inconsistent in the output category on the tested model, and use the mutated loop code as the final adversarial loop code test data.

7. A generation terminal, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

Images