Fuzz testing method for unstructured data

Abstract

A fuzz testing method for unstructured data combines the Sequence2Sequence network and the Char-RNN network from deep learning to generate test cases based on the relationship between the unstructured data to be tested and the execution path triggered by it in the corresponding target program. The Sequence2Sequence network model is weighted and optimized for the unstructured data to be tested using an Attention mechanism. A Scheduled Sampling model consisting of two decoder networks is used to mix real sequence elements with test sequence elements to improve the final prediction sequence of the Sequence2Sequence network. This method can perform high-quality fuzz testing on test cases for unstructured data, while improving the path coverage of the target program and detecting more anomalies.

Description

Technical Field

[0001] The present invention belongs to the field of fuzzy testing of automated testing, in particular to unstructured data design, and specifically to a fuzzy testing method for unstructured data. Background Art

[0002] Fuzz testing is a powerful testing technique for automatically detecting program vulnerabilities. Fuzz testing involves automatically or semi-automatically constructing a large number of random test cases to test software and identify faults. However, due to the large number of test cases and the presence of a large amount of invalid data, fuzz testing consumes significant runtime and wastes system resources.

[0003] To avoid the redundancy and inefficiency issues inherent in randomly generated test cases, typical test case generation strategies for fuzz testing currently include mutation-based fuzz testing and seed-based fuzz testing. Current fuzz testing research primarily targets structured test cases, while research on unstructured test cases is still in its infancy. Structured test cases can typically be accurately described based on the test program's interface parameter descriptions, allowing specific strategies to be used to generate test cases. For unstructured test cases, the target program typically lacks corresponding interface parameter descriptions; instead, it is typically defined as an array with no defined data structure. Unstructured test cases typically correspond to files in specific formats, such as CSV, PDF, and XML. These test cases must first meet file specifications. Furthermore, most test cases consist of character or byte sequences of varying length and exhibit a certain structure. This results in an extremely large search space, rendering mutation-based or seed-based generation strategies unsuitable.

[0004] Therefore, designing a deep learning fuzz testing method for unstructured data has very important research significance and broad application prospects. Summary of the Invention

[0005] To address the shortcomings of existing fuzz testing technologies, this paper provides a deep learning fuzz testing design for unstructured data, which can effectively perform fuzz testing on unstructured use cases.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0007] The fuzz testing method for unstructured data includes the following steps:

[0008] Step 1: construct the attention probability distribution of the object sequence in the unformatted data;

[0009] Step 1-1 performs attention scoring on the two RNN networks in Seq2Seq to obtain weights:

[0010] The two RNN network encoder and decoder contained in Seq2Seq predict the state at time t, and the decoder hidden state value h at that time t 、All hidden states h of N encoders n The sum value h s Calculate the score vector score with the weight matrix W; that is:

[0011]

[0012] Normalize the calculated score to get the weight α t ;

[0013]

[0014] Step 1-2 generates the attention probability distribution of unstructured data based on the intermediate state of the Attention test:

[0015] First, each element in the unstructured data object sequence obj is input into the encoder in sequence until the last element; the weight α t With obj and encoder's hidden state h n Calculate the test intermediate state c between the encoder and decoder based on the weighted attention distribution t , as shown below:

[0016]

[0017] Using the test intermediate state c t , adjust the influence of unstructured data input sequence elements on its output sequence elements, and generate the attention probability distribution of unformatted data.

[0018] Step 2: Establish a scheduled sampling test optimization model to optimize the Seq2Seq network;

[0019] Step 2-1 builds the first decoder network of the Scheduled Sampling model and obtains the predicted unformatted data sequence:

[0020] First, the object sequence of unstructured data is input into the target program to obtain the corresponding path sequence. This path sequence is input into the encoder network to obtain the attention distribution weighted test intermediate state vector. The first decoder network is constructed, and the intermediate state vector is then used by the decoder for the first prediction to obtain the prediction sequence.

[0021] Step 2-2: Mix the original unstructured data sequence with the predicted data sequence:

[0022] A new sequence is obtained by mixing the original unstructured data real sequence elements with the predicted sequence elements: t Select the real element, with 1-P t Select the predicted element. As the training progresses, the time t changes, and the model training becomes more and more complete. First select the real element and then select the predicted element; use linear attenuation to set the probability P t where 0≤P<1, k and c provide the offset and slope of the decay;

[0023] P t =max(P,k-ct)

[0024] Step 2-3 builds the second decoder of the Scheduled Sampling model to output an unstructured data sequence:

[0025] Construct a second decoder network and use the mixed new sequence as the input of the network to obtain the final unstructured data sequence.

[0026] Step 3: Combine the Char-RNN model and the optimized Seq2Seq model into a new fuzz testing method;

[0027] The combination of the Char-RNN model and the optimized Seq2Seq model described in step 3 is a new fuzz testing method, which includes the following steps:

[0028] Step 3-1 uses the Char-RNN model to obtain a new execution path sequence:

[0029] Preprocess the execution path sequence corresponding to the unformatted data, where the path is a sequence of multiple basic blocks. A complete path is<block1,block2,block3…blockn> , convert it into the corresponding basic block sequence and record it; when inputting the Char-RNN network model, each execution path is trained as a string sequence composed of basic blocks, and finally a new path sequence is predicted;

[0030] In step 3-2, the new path sequence and the optimized Seq2Seq model obtain the corresponding unstructured data.

[0031] Step 4: Testing unstructured data using the new fuzz testing method;

[0032] Step 4-1: Training with data:

[0033] The unstructured data object sequence and path basic block sequence are used as the target and source of the optimized Seq2Seq model for training, and the relationship between the execution path and the unstructured data object sequence is obtained;

[0034] Step 4-2 The model performs predictions to obtain the final test case:

[0035] The new basic block sequence obtained by Char-RNN is input into the trained Seq2Seq model for prediction to obtain the input that can execute the new path, that is, the object sequence; these object sequences are combined into new unstructured data, that is, the new test case.

[0036] The present invention designs a fuzzy testing method for unstructured data. First, the attention probability distribution of the object sequence in the unstructured data is introduced through the attention mechanism. The attention probability distribution is used to adjust the influence of the elements in the encoder input sequence on the elements of its output sequence. While retaining more valuable information, the impact of irrelevant or weakly related data on the output data is reduced. Secondly, a scheduled sampling test optimization model consisting of two decoders is established to optimize the Seq2Seq network. Finally, the Char-RNN network is combined with the optimized Seq2Seq network to obtain a new fuzzy testing method. The above method has the advantages of improving the quality of unstructured use cases and enhancing the efficiency of fuzzy testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the testing framework for combining the Char-RNN model with the optimized Seq2Seq model.

[0038] Figure 2 Optimized fuzz testing framework file coverage comparison diagram.

[0039] Figure 3a A diagram showing the comparison of path coverage for the first test set using the optimized fuzz testing framework.

[0040] Figure 3b Schematic diagram of the comparison of path coverage of the second test set through the optimized fuzz testing framework.

[0041] Figure 3c A diagram showing the comparison of path coverage for the third test set using the optimized fuzz testing framework.

[0042] Figure 3d Schematic diagram comparing the path coverage of all test sets through the optimized fuzz testing framework. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0044] Step 1 constructs the attention probability distribution over the object sequence in the unformatted data.

[0045] Step 1-1 performs attention scoring on the two RNN networks in Seq2Seq to obtain weights:

[0046] The two RNN network encoder and decoder contained in Seq2Seq predict the state at time t, and the decoder hidden state value h at that time t , all hidden states h of each encoder n The sum value h s Calculate with the weight matrix W to get the score vector score. That is:

[0047]

[0048] Normalize the calculated score to get the weight α t .

[0049]

[0050] Step 1-2 generates the attention probability distribution of unstructured data based on the intermediate state of the Attention test:

[0051] First, each element in the unstructured data object sequence obj is input into the encoder in sequence until the last element. t With obj and encoder's hidden state h n Calculate the test intermediate state c between the encoder and decoder based on the weighted attention distribution t , as shown below.

[0052]

[0053] Using the test intermediate state c t , adjust the influence of unstructured data input sequence elements on its output sequence elements, and generate the attention probability distribution of unformatted data.

[0054] Step 2: Establish a scheduled sampling test optimization model consisting of two decoders to optimize the Seq2Seq network.

[0055] Step 2-1 builds the first decoder network of the Scheduled Sampling model and obtains the predicted unstructured data sequence:

[0056] First, the object sequence of unstructured data is input into the target program to obtain the corresponding path sequence. This path sequence is input into the encoder network to obtain the test intermediate state vector weighted by the attention distribution. The first decoder network is constructed, and the intermediate state vector is then used by the decoder for the first prediction to obtain the predicted sequence.

[0057] Step 2-2: Mix the original unstructured data sequence with the predicted data sequence:

[0058] A new sequence is obtained by mixing the original unstructured data real sequence elements with the predicted sequence elements: t Select the real element, with 1-P t Select the predicted element. As the training progresses, the time t changes, and the model training becomes more and more complete. We should try to select the real element first and then the predicted element. In this paper, we use linear decay to set the probability P t The value of , where 0≤P<1, k and c provide the offset and slope of the attenuation.

[0059] P t =max(P,k-ct)

[0060] Step 3 combines the Char-RNN model and the optimized Seq2Seq model into a new fuzz testing method.

[0061] Step 3-1 uses the Char-RNN model to obtain a new execution path sequence:

[0062] The execution path sequence corresponding to the unstructured data is preprocessed, where the path is a sequence of multiple basic blocks. A complete path is<block1,block2,block3…blockn> , convert it into the corresponding basic block sequence and record it. When inputting the Char-RNN network model, each execution path is trained as a string sequence composed of basic blocks, and finally a new path sequence is predicted.

[0063] In step 3-2, the new path sequence and the optimized Seq2Seq model obtain the corresponding unstructured data.

[0064] Step 4 tests the unstructured data using the new fuzz testing method.

[0065] Step 4-1: Training with data:

[0066] The unstructured data object sequence and path basic block sequence are used as the "target" and "source" of the optimized Seq2Seq model for training respectively, and the relationship between the execution path and the unstructured data object sequence is obtained.

[0067] Step 4-2 The model performs predictions to obtain the final test case:

[0068] The new basic block sequence obtained by the Char-RNN is input into the trained Seq2Seq model for prediction, obtaining the input that can execute the new path, namely the object sequence. These object sequences are combined into new unstructured data, namely the new test case.

[0069] Example 1:

[0070] In order to better test the efficiency of the fuzz testing method for unstructured use cases, as an example of the present invention, we selected 2080 well-formatted PDF files as the initial selected seed unstructured data and divided them into four groups. The seed files were sent to the CAJViewer reader, and the Intel pin tool was used to count the execution paths. The four groups of divided PDF files were set as benchmark test sets: B1, B2, B3, and B4, where B1, B2, and B3 respectively contained 520 PDF files from the first three groups, and B4 contained all PDF files. The training time of the Char-RNN and Seq2Seq models was set to 24 hours. By modifying the number of unstructured data sequence training times, the number of test files, etc., an experimental environment for the fuzz testing method for unstructured data was established. According to the experimental running results and data, a comparison was made with the Seq2Seq fuzz testing framework and AFL fuzz testing tool before optimization.

[0071] This experiment was run on a machine with an Intel(R) Core(TM) i7-7500U CPU and 16GB of memory. When testing unformatted PDF files, they were preprocessed and parsed into a series of obj objects. These object sequences were then treated as a string sequence and fed into the network for testing.

[0072] Through experimental data on real datasets, the file coverage and path coverage of the two strategies are compared, and the execution efficiency of the fuzz testing method for unstructured use cases is evaluated.

[0073] File coverage comparison

[0074] During the experiment, when the model was trained for 40 epochs, Figure 2As shown, the pass rate for files generated by the fuzzing method for unstructured use cases is much higher than the pass rate for files generated by the AFL fuzzer using the four PDF file groups B1 to B4. Furthermore, the pass rate for files N1 to N4 is around 80%, while the highest pass rate for the AFL fuzzer is only 46%. Therefore, the fuzzing method for unstructured use cases significantly surpasses the AFL fuzzer in terms of file coverage.

[0075] Path coverage comparison

[0076] During the experiment, the Char-RNN network in the test framework is the same. The Seq2Seq network before and after optimization is combined with it to form a test framework, and then the unstructured data is tested. After parsing the files in the benchmark test set, the four new files T1, T2, T3, and T4 are generated by the pre-optimization test framework. At the same time, the model is trained 10, 20, 30, 40, and 50 times in the optimized model framework designed in this paper, generating four groups of files N1, N2, N3, and N4. Figure 3a As shown in , the file N1 generated by the optimized test framework has more path coverage than the file T1 generated by the test framework before optimization; Figure 3b As shown in , the file N2 generated by the optimized test framework has more path coverage than the file T2 generated by the test framework before optimization; Figure 3c As shown in , the file N3 generated by the optimized test framework has more path coverage than the file T3 generated by the test framework before optimization; Figure 3d As shown, the greater the number of input files, the greater the target program's instruction coverage. Furthermore, the newly generated files from the optimized model have a path coverage rate of approximately 3.89% higher than those before optimization. Overall, the fuzz testing method for unstructured data is optimal.

Claims

1. A fuzzy testing method for unstructured data, characterized in that: The steps are: Step 1: construct the attention probability distribution of the object sequence in the unformatted data; Step 2: Establish a scheduled sampling test optimization model to optimize the Seq2Seq network; Step 2-1 builds the first decoder network of the Scheduled Sampling model and obtains the predicted unformatted data sequence: First, the object sequence of unstructured data is input into the target program to obtain the corresponding path sequence. This path sequence is input into the encoder network to obtain the attention distribution weighted test intermediate state vector. The first decoder network is constructed, and the intermediate state vector is then used by the decoder for the first prediction to obtain the prediction sequence. Step 2-2: Mix the original unstructured data sequence with the predicted data sequence: A new sequence is obtained by mixing the original unstructured data real sequence elements with the predicted sequence elements: t Select the real element, with 1-P t Select the predicted element. As the training progresses, the time t changes, and the model training becomes more and more complete. First select the real element and then select the predicted element; use linear attenuation to set the probability P t where 0≤P<1, k and c provide the offset and slope of the decay; P t =max(P,k-ct) Step 2-3 builds the second decoder of the Scheduled Sampling model to output an unstructured data sequence: Construct a second decoder network and use the mixed new sequence as the input of the network to obtain the final unstructured data sequence; Step 3: Combine the Char-RNN model and the optimized Seq2Seq model into a new fuzz testing method; Combining the Char-RNN model and the optimized Seq2Seq model into a new fuzz testing method includes the following steps: Step 3-1 uses the Char-RNN model to obtain a new execution path sequence: Preprocess the execution path sequence corresponding to the unformatted data, where the path is a sequence of multiple basic blocks. A complete path is<block1,block2,block3…blockn> , convert it into the corresponding basic block sequence and record it; when inputting the Char-RNN network model, each execution path is trained as a string sequence composed of basic blocks, and finally a new path sequence is predicted; Step 3-2: The new path sequence and the optimized Seq2Seq model obtain the corresponding unstructured data; Step 4 tests the unstructured data using the new fuzz testing method.

2. The fuzzy testing method for unstructured data according to claim 1, characterized in that: The specific method in step 1 is: Step 1-1 performs attention scoring on the two RNN networks in Seq2Seq to obtain weights: The two RNN network encoder and decoder contained in Seq2Seq predict the state at time t, and the decoder hidden state value h at that time t 、All hidden states h of N encoders n The sum value h s Calculate the score vector score with the weight matrix W; that is: Normalize the calculated score to get the weight α t ; Step 1-2 generates the attention probability distribution of unstructured data based on the intermediate state of the Attention test: First, each element in the unstructured data object sequence obj is input into the encoder in sequence until the last element; the weight α t With obj and encoder's hidden state h n Calculate the test intermediate state c between the encoder and decoder based on the weighted attention distribution t , as shown below: Using the test intermediate state c t , adjust the influence of unstructured data input sequence elements on its output sequence elements, and generate the attention probability distribution of unformatted data.

3. The fuzzy testing method for unstructured data according to claim 1, characterized in that: The specific method in step 4 is: Step 4-1: Training with data: The unstructured data object sequence and path basic block sequence are used as the target and source of the optimized Seq2Seq model for training, and the relationship between the execution path and the unstructured data object sequence is obtained; Step 4-2 The model performs predictions to obtain the final test case: The new basic block sequence obtained by Char-RNN is input into the trained Seq2Seq model for prediction to obtain the input that can execute the new path, that is, the object sequence; these object sequences are combined into new unstructured data, that is, the new test case.

Images