A fuzzing method and process based on a deep reinforcement learning model

Abstract

The application discloses a kind of fuzzy test method and process based on deep learning model.First, different formats of Linux application are collected, and the application collected and seed file with the same format are saved;Second, the variation model under fixed variation strategy is constructed based on value-oriented DQN algorithm and strategy-oriented TRPO algorithm, and the improved Thompson sampling algorithm is used to construct Havoc random variation model, to output the effective variation operator and variation strategy chain information of test sample;Then, the AFL source code is modified, the two proposed variation models are embedded, and the interaction between the model and AFL is realized using shared memory;Finally, for the AFL embedded with the model obtained in the last step, an automated execution script is written and its execution time is set, and the afl-fuzz, afl-showmap and afl-cov instructions are executed to train the embedded two models and perform automated fuzz testing, and according to the fuzz testing execution result, the code coverage, the number of crashes, the use of variation operator and other results are visualized.

Description

Technical Field

[0001] This invention relates to the field of software security, and more specifically, to the field of binary program fuzzing, proposing a fuzzing method and process based on a deep reinforcement learning model. Background Technology

[0002] With the advancement of the information age, software, as the core carrier of information technology, has penetrated deeply into daily life, affecting key areas such as finance, personal information, and health. However, software has also become a target for hackers and malware developers, who constantly seek new attack methods and vulnerabilities to damage and steal data. Once software security is compromised, individuals face multiple risks, including privacy breaches and financial losses, impacting social stability and harmony, and even seriously threatening national security and stability. Therefore, ensuring software security is of paramount practical value for maintaining cybersecurity, national security, and building a cyber power.

[0003] Fuzzing, as an effective vulnerability discovery technique, addresses the shortcomings of traditional security testing methods and improves the effectiveness of vulnerability discovery in the information domain. Fuzzing can automatically or semi-automatically input random or semi-random data into a system and monitor abnormal program behavior, thereby discovering vulnerabilities and instabilities in software. However, fuzzing still suffers from low testing efficiency, limited coverage, and high complexity. To improve the performance of AFL (Automatic Flask Programming), researchers have used algorithms such as genetic algorithms and deep learning to optimize the selection of mutation operators in AFL, but this has led to challenges such as model complexity, insufficient accuracy, and poor generalization scalability. Summary of the Invention

[0004] This invention proposes a fuzz testing method and process based on a deep reinforcement learning model. It considers both the selection of mutation operators and mutation positions, and optimizes the mutation process in stages. This method efficiently trains the model to obtain mutation operators and mutation policy chains, thereby effectively selecting mutation operators in the fuzz testing mutation process. The fuzz testing method based on a deep reinforcement learning model proposed in this invention is applicable to different types of binary program files.

[0005] To achieve the above objectives, the present invention is implemented according to the following technical solution:

[0006] A fuzz testing method and process based on a deep reinforcement learning model includes the following steps:

[0007] Step S1: Program collection: Collect Linux applications of different formats, store the collected applications in the application file directory, and place the torrent files of the same format in the same source torrent file directory;

[0008] Step S2: Construction of mutation model under fixed mutation strategy: Based on the value-oriented DQN algorithm and the strategy-oriented TRPO algorithm, a mutation model under fixed mutation strategy is constructed to output the effective mutation strategy and effective mutation location information of the test sample.

[0009] Step S3: Construction of Havoc random mutation model: Construct a Havoc random mutation model based on the improved Thompson sampling algorithm to output the effective mutation strategy chain information of the test samples;

[0010] Step S4: Mutation Model Embedding: Modify the AFL source code, combine the mutation model built in S2 and S3, embed it into AFL, and use shared memory to realize the interaction between the model and AFL.

[0011] Step S5: Automated Fuzzing: For the AFL embedded with the mutation model in S4, write an automated execution script, set the execution time, execute the afl-fuzz, afl-showmap and afl-cov commands to perform automated fuzzing, and train the two embedded models.

[0012] Step S6: Result Visualization: Based on the fuzz test results, calculate code coverage, number of crashes, and mutation operator usage, and then visualize the results.

[0013] In the above technical solution, the specific steps for program collection in step S1 are as follows:

[0014] Step S11: Create a directory of seed files from the same source to store seed files based on the same format;

[0015] Step S12: Create an application file directory to store Linux application files of different formats developed in C language;

[0016] Step S13: Obtain different C-language-based applications from the official Linux website and store these application files in the application file directory;

[0017] Step S14: In the Linux system, use the man command to query the execution command parameters of different applications in the application file directory, and obtain the file format processed by different applications based on the execution command parameters of different applications and the ReadMe file to which the program belongs, and then execute S15.

[0018] Step S15: Based on the execution command parameters and corresponding file formats of different applications obtained in S14, obtain or generate a file with the same file format and less than 1MB from the source application file or Github as a seed file, and add the seed file of the same format to the same source seed file directory.

[0019] Step S16: Based on the ReadMe file in the application file, use the make command in the Linux system to set the compilation options to implement AFL instrumentation of the application to be tested, and compile different application files in the application file directory;

[0020] Step S17: Analyze the source code of the AFL mutation stage and divide the AFL mutation process into a mutation stage under a fixed mutation strategy and a Havoc random mutation stage.

[0021] In the above technical solution, the specific steps for constructing the mutation model under the fixed mutation strategy in step S2 are as follows:

[0022] Step S21: Implement deep reinforcement learning models based on the value-oriented DQN algorithm and the policy-oriented TRPO algorithm using PyTorch;

[0023] Step S22: Locate the position of the mutation stage under the fixed mutation strategy in the AFL source code. Based on the test sample src before mutation under the fixed mutation strategy and the mutation operator used by AFL, construct the model's data reading component to obtain the test sample information of the current mutation.

[0024] Step S23: For the deep reinforcement learning models based on DQN and TRPO algorithms in S21, design a policy evaluation component according to the mutation requirements of AFL, so that the two algorithm models can output multiple possible effective mutation policies op and effective mutation positions pos.

[0025] Step S24: For the deep reinforcement learning model based on DQN and TRPO algorithms in S21, design an interaction component based on shared memory according to the interaction requirements of AFL and model, so that the mutation model and AFL can interact efficiently and quickly.

[0026] In the above technical solution, the specific steps for constructing the Havoc random mutation model in step S3 are as follows:

[0027] Step S31: Implement an exploration-utilization model based on the Thompson sampling algorithm using PyTorch;

[0028] Step S32: Improve the exploration-utilization model built in S31 using a multi-agent strategy to enhance the model's prediction accuracy and learning efficiency;

[0029] Step S33: Locate the position of the Havoc random mutation stage in the AFL source code, and design the model's data reading component based on the src before mutation and the length of the mutation chain under the Havoc random mutation stage, so that the model can obtain the mutation information of the current mutation round.

[0030] Step S34: For the exploration and utilization model based on the improved Thompson sampling algorithm built in S32, design the voting strategy and self-update strategy of the agents in the model;

[0031] Step S35: Based on the exploration and utilization model built in S32 using the improved Thompson sampling algorithm, design an interaction component based on shared memory according to the interaction requirements between AFL and the model, so that the mutation model and AFL can interact efficiently and quickly.

[0032] In the above technical solution, the specific steps of embedding the mutation model in step S4 are as follows:

[0033] Step S41: In the mutation stage under the fixed mutation strategy in S17, select 16 suitable mutation strategies for modification. For each mutation strategy, adjust the source code of the mutation location selection part of the afl-fuzz.c file so that the mutation stage under the fixed mutation strategy can obtain effective mutation information for mutation.

[0034] Step S42: In the Havoc random mutation stage in S17, the multi-round mutation process of the Havoc stage is modified. For the generation of each round of mutation chain, the selection method of mutation strategy and part of the source code are adjusted so that the Havoc random mutation stage can obtain effective mutation information for mutation.

[0035] Step S43: Locate the current test case src and mutation chain length in the source code of afl-fuzz.c, and pass them as inputs to the two models built in S2 and S3 respectively;

[0036] Step S44: Utilize the shared memory mechanism to enable the model to obtain its input from the shared memory and add the effective mutation information obtained by the model to the shared memory;

[0037] Step S45: After each mutation, AFL passes the current valid mutation information to the two constructed models respectively, trains the two models and updates the model parameters, and clears the shared memory to ensure that the mutation information does not overflow.

[0038] Step S46: Enter the AFL file directory, use `make` to recompile the modified files, and then use the `sudo make install` command to install the modified AFL.

[0039] In the above technical solution, the specific steps of automated fuzz testing in step S5 are as follows:

[0040] Step S51: Based on the AFL fuzzing steps, write an automated script to execute the fuzzing. The script uses the `afl-fuzz` command to perform fuzzing on the application and the `afl-showmap` and `afl-cov` commands to analyze the application. Furthermore, it must be ensured that the script can select different applications for fuzzing.

[0041] Step S52: For different applications, select the trained model and deploy it to the server in S4 to call the model;

[0042] Step S53: Create a result file, use the automated tool generated in step S51 to execute the script, select the application currently being fuzzed and set the fuzzing time, start the automated fuzzing, and output the fuzzing results to the result file;

[0043] In the above technical solution, the specific steps for visualizing the results in step S6 are as follows:

[0044] Step S61: Display the fuzzing results generated in step S53, such as code coverage, number of crashes, and fuzzing time, in real time using the command line;

[0045] Step S62: For the fuzz test results in the result file in S53, after the fuzz test is completed, organize the results for subsequent analysis;

[0046] Step S63: Use the automation tools in S5 to execute the afl-showmap and afl-cov commands in the script. After the fuzzing is completed, obtain the fuzzing analysis results and generate a code coverage report file.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This invention optimizes the open-source fuzzing framework AFL for binary program files, implementing operations such as program collection, mutation model construction under fixed mutation strategies, Havoc random mutation model construction, model embedding into AFL tools, automated fuzzing, and result visualization. This improves code coverage during fuzzing, enabling more effective prediction of mutation strategies and mutation locations, thereby enhancing AFL's ability to trigger new paths and uncover vulnerabilities, and ultimately improving fuzzing efficiency. From the overall technical perspective, this invention establishes for the first time a fuzzing method and process based on a deep reinforcement learning model in the field of binary program file fuzzing. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0050] Figure 2 This is a schematic diagram of the program collection process in this invention;

[0051] Figure 3 This is a schematic diagram of the mutation model construction process under the fixed mutation strategy in this invention;

[0052] Figure 4 This is a schematic diagram illustrating the construction of the Havoc random mutation model in this invention;

[0053] Figure 5 This is a schematic diagram of the mutation model embedding in this invention;

[0054] Figure 6 This is a schematic diagram of the automated fuzz testing process in this invention;

[0055] Figure 7 This is a schematic diagram of the fuzzy test result visualization process in this invention. Detailed Implementation

[0056] To better illustrate the purpose, technical solutions, and advantages of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0057] This invention proposes a fuzz testing method and process based on a deep reinforcement model. By collecting data, building a model, and embedding it into an open-source fuzz testing framework, the method performs real-time training of the model during fuzz testing, ultimately achieving automated fuzz testing and visualization of test results. A flowchart is shown below. Figure 1 As shown, it includes the following steps:

[0058] Step S1: Collect Linux applications of different formats, store the collected applications in the application file directory, and place torrent files of the same format in the same source torrent file directory. The flowchart is shown below. Figure 2 As shown, it includes the following steps:

[0059] Step S11: Create a directory of seed files from the same source to store seed files based on the same format;

[0060] Step S12: Create an application file directory to store Linux application files of different formats developed in C language;

[0061] Step S13: Obtain different C-language-based applications from the official Linux website and store these application files in the application file directory;

[0062] Step S14: In the Linux system, use the man command to query the execution command parameters of different applications in the application file directory, and obtain the file format processed by different applications based on the execution command parameters of different applications and the ReadMe file to which the program belongs, and then execute S15.

[0063] Step S15: Based on the execution command parameters and corresponding file formats of different applications obtained in S14, obtain or generate a file with the same file format and less than 1MB from the source application file or Github as a seed file, and add the seed file of the same format to the same source seed file directory.

[0064] Step S16: Based on the ReadMe file in the application file, use the make command in the Linux system to set the compilation options to implement AFL instrumentation of the application to be tested, and compile different application files in the application file directory;

[0065] Step S17: Analyze the source code of the AFL mutation stage and divide the AFL mutation process into a mutation stage under a fixed mutation strategy and a Havoc random mutation stage.

[0066] Step S2: Construct a mutation model under a fixed mutation strategy based on the value-oriented DQN algorithm and the strategy-oriented TRPO algorithm to output the effective mutation strategy and effective mutation location information of the test samples. The flowchart is shown below. Figure 3 As shown, it includes the following steps:

[0067] Step S21: Implement deep reinforcement learning models based on the value-oriented DQN algorithm and the policy-oriented TRPO algorithm using PyTorch;

[0068] Step S22: Locate the position of the mutation stage under the fixed mutation strategy in the AFL source code. Based on the test sample src before mutation under the fixed mutation strategy and the mutation operator used by AFL, construct the model's data reading component to obtain the test sample information of the current mutation.

[0069] Step S23: For the deep reinforcement learning models based on DQN and TRPO algorithms in S21, design a policy evaluation component according to the mutation requirements of AFL, so that the two algorithm models can output multiple possible effective mutation policies op and effective mutation positions pos.

[0070] Step S24: For the deep reinforcement learning model based on DQN and TRPO algorithms in S21, design an interaction component based on shared memory according to the interaction requirements of AFL and model, so that the mutation model and AFL can interact efficiently and quickly.

[0071] Step S3: Construct a Havoc random mutation model based on the improved Thompson sampling algorithm to output the effective mutation strategy chain information of the test samples. The flowchart is shown below. Figure 4 As shown, it includes the following steps:

[0072] Step S31: Implement an exploration-utilization model based on the Thompson sampling algorithm using PyTorch;

[0073] Step S32: Improve the exploration-utilization model built in S31 using a multi-agent strategy to enhance the model's prediction accuracy and learning efficiency;

[0074] Step S33: Locate the position of the Havoc random mutation stage in the AFL source code, and design the model's data reading component based on the src before mutation and the length of the mutation chain under the Havoc random mutation stage, so that the model can obtain the mutation information of the current mutation round.

[0075] Step S34: For the exploration and utilization model based on the improved Thompson sampling algorithm built in S32, design the voting strategy and self-update strategy of the agents in the model;

[0076] Step S35: For the exploration and utilization model based on the improved Thompson sampling algorithm built in S32, design an interaction component based on shared memory according to the interaction requirements of AFL and the model, so that the mutation model and AFL can interact efficiently and quickly.

[0077] Step S4: Modify the AFL source code, combine it with the mutation model built in S2 and S3, embed it into AFL, and use shared memory to realize the interaction between the model and AFL. The flowchart is shown below. Figure 5 As shown, it includes the following steps:

[0078] Step S41: In the mutation stage under the fixed mutation strategy in S17, select 16 suitable mutation strategies for modification. For each mutation strategy, adjust the source code of the mutation location selection part of the afl-fuzz.c file so that the mutation stage under the fixed mutation strategy can obtain effective mutation information for mutation.

[0079] Step S42: In the Havoc random mutation stage in S17, the multi-round mutation process of the Havoc stage is modified. For the generation of each round of mutation chain, the selection method of mutation strategy and part of the source code are adjusted so that the Havoc random mutation stage can obtain effective mutation information for mutation.

[0080] Step S43: Locate the current test case src and mutation chain length in the source code of afl-fuzz.c, and pass them as inputs to the two models built in S2 and S3 respectively;

[0081] Step S44: Utilize the shared memory mechanism to enable the model to obtain its input from the shared memory and add the effective mutation information obtained by the model to the shared memory;

[0082] Step S45: After each mutation, AFL passes the current valid mutation information to the two constructed models respectively, trains the two models and updates the model parameters, and clears the shared memory to ensure that the mutation information does not overflow.

[0083] Step S46: Enter the AFL file directory, use make to recompile the modified files, and then use the sudo make install command to install the modified AFL;

[0084] Step S5: For the AFL embedded with the mutation model in S4, write an automated execution script, set the execution time, and execute the afl-fuzz, afl-showmap, and afl-cov commands to perform automated fuzz testing and train the two embedded models. The flowchart is shown below. Figure 6 As shown, it includes the following steps:

[0085] Step S51: Based on the AFL fuzzing steps, write an automated script to execute the fuzzing. The script uses the `afl-fuzz` command to perform fuzzing on the application and the `afl-showmap` and `afl-cov` commands to analyze the application. Furthermore, it must be ensured that the script can select different applications for fuzzing.

[0086] Step S52: For different applications, select the trained model and deploy it to the server in S4 to call the model;

[0087] Step S53: Create a result file, use the automated tool generated in step S51 to execute the script, select the application currently being fuzzed and set the fuzzing time, start the automated fuzzing, and output the fuzzing results to the result file;

[0088] Step S6: Based on the fuzz test results, calculate code coverage, number of crashes, and mutation operator usage, and visualize the results. A flowchart is shown below. Figure 7 As shown, it includes the following steps:

[0089] Step S61: Display the fuzzing results generated in step S53, such as code coverage, number of crashes, and fuzzing time, in real time using the command line;

[0090] Step S62: For the fuzz test results in the result file in S53, after the fuzz test is completed, organize the results for subsequent analysis;

[0091] Step S63: After the fuzzing is completed, run the automation tool in S5 to execute the afl-showmap and afl-cov commands in the script to obtain the fuzzing analysis results and generate a code coverage report file.

Claims

1. A fuzz testing method and process based on a deep reinforcement learning model, characterized in that... The process includes the following six steps: program data collection, construction of a mutation model under a fixed mutation strategy, construction of a Havoc random mutation model, embedding of the mutation model, automated fuzz testing, and visualization of results. S1. The program collection steps are as follows: collect Linux applications of different formats, store the collected applications in the application file directory, and place the seed files of the same format in the same source seed file directory; S2. The steps for constructing the mutation model under the fixed mutation strategy are as follows: Based on the value-oriented DQN algorithm and the strategy-oriented TRPO algorithm, a mutation model under the fixed mutation strategy is constructed to output the effective mutation strategy and effective mutation location information of the test sample. S3. The steps for constructing the Havoc random mutation model are as follows: Construct the Havoc random mutation model based on the improved Thompson sampling algorithm to output the effective mutation strategy chain information of the test samples. S4. The mutation model embedding step: Modify the source code of AFL, combine it with the mutation model built in S2 and S3, embed it into AFL, and use shared memory to realize the interaction between the model and AFL. S5. The automated fuzzing steps are as follows: For the AFL with the embedded mutation model in S4, write an automated execution script, set the execution time, execute the afl-fuzz, afl-showmap and afl-cov commands to perform automated fuzzing, and train the two embedded models. S6. The result visualization step: Based on the fuzz test execution results, calculate the code coverage, number of crashes, and usage of mutation operators, and then visualize the results.

2. The fuzz testing method and process based on a deep reinforcement learning model as described in claim 1, characterized in that, In step S1, the program collection step includes the following operations: S11. Create a directory of seed files from the same source to store seed files based on the same format; S12. Create an application file directory to store Linux application files of different formats developed in C language; S13. Obtain different C-language-based applications from the official Linux website and store these application files in the application file directory; S14. In the Linux system, use the man command to query the execution command parameters of different applications in the application file directory, and obtain the file format processed by different applications based on the execution command parameters of different applications and the ReadMe file to which the program belongs, and then execute S15. S15. Based on the execution command parameters and corresponding file formats of different applications obtained in S14, obtain or generate files with the same file format and less than 1MB from the source application file or Github as seed files, and add the seed files of the same format to the same source seed file directory. S16. Based on the ReadMe file in the application file, use the make command in the Linux system to set the compilation options to implement AFL instrumentation of the application to be tested, and compile different application files in the application file directory. S17. Analyze the source code of the AFL mutation stage and divide the AFL mutation process into a mutation stage under a fixed mutation strategy and a Havoc random mutation stage.

3. The fuzz testing method and process based on a deep reinforcement learning model as described in claim 1, characterized in that, In step S2, the mutation model construction step under the fixed mutation strategy includes the following operations: S21. Implement deep reinforcement learning models based on the value-oriented DQN algorithm and the policy-oriented TRPO algorithm using PyTorch. S22. Locate the position of the mutation stage under the fixed mutation strategy in the AFL source code. Based on the test sample src before mutation under the fixed mutation strategy and the mutation operator used by AFL, construct the model's data reading component to obtain the test sample information of the current mutation. S23. For the deep reinforcement learning models based on DQN and TRPO algorithms in S21, a policy evaluation component is designed according to the mutation requirements of AFL, so that the two algorithm models can output multiple possible effective mutation policies op and effective mutation positions pos. S24. For the deep reinforcement learning models based on DQN and TRPO algorithms in S21, design an interaction component based on shared memory according to the interaction requirements of AFL and the model, so that the mutation model and AFL can interact efficiently and quickly.

4. The fuzz testing method and process based on a deep reinforcement learning model as described in claim 1, characterized in that, In step S3, the Havoc random mutation model construction step includes the following operations: S31. Implement an exploration-utilization model based on the Thompson sampling algorithm using PyTorch; S32. Improve the exploration-utilization model built in S31 using a multi-agent strategy to enhance the model's prediction accuracy and learning efficiency. S33. Locate the position of the Havoc random mutation stage in the AFL source code, and design the model's data reading component based on the src before mutation and the length of the mutation chain under the Havoc random mutation stage, so that the model can obtain the mutation information of the current mutation round. S34. For the exploration and utilization model based on the improved Thompson sampling algorithm built in S32, design the voting strategy and self-update strategy of the agents in the model. S35. For the exploration and utilization model based on the improved Thompson sampling algorithm built in S32, according to the interaction requirements of AFL and the model, design an interaction component based on shared memory so that the mutation model and AFL can interact efficiently and quickly.

5. The fuzz testing method and process based on a deep reinforcement learning model as described in claim 1, characterized in that, In step S4, the mutation model embedding step includes the following operations: S41. In the mutation stage under the fixed mutation strategy in S17, 16 suitable mutation strategies are selected and modified. For each mutation strategy, the source code of the mutation location selection part of the afl-fuzz.c file is adjusted so that the mutation stage under the fixed mutation strategy can obtain effective mutation information for mutation. S42. In the Havoc random mutation stage in S17, the multi-round mutation process of the Havoc stage is modified. For the generation of each round of mutation chain, the selection method of mutation strategy and part of the source code are adjusted so that the Havoc random mutation stage can obtain effective mutation information for mutation. S43. Locate the current test case src and mutation chain length in the source code of afl-fuzz.c, and pass them as inputs to the two models built in S2 and S3 respectively. S44. Utilize the shared memory mechanism to enable the model to obtain its input from shared memory and add the effective mutation information obtained by the model to the shared memory; S45. After each mutation, AFL passes the current valid mutation information to the two constructed models respectively, trains the two models and updates the model parameters, and clears the shared memory to ensure that the mutation information does not overflow. S46. Enter the AFL file directory, use make to recompile the modified files, and then use the sudo makeinstall command to install the modified AFL.

6. The fuzz testing method and process based on a deep reinforcement learning model as described in claim 1, characterized in that, In step S5, the automated fuzz testing step includes the following operations: S51. Based on the AFL fuzzing steps, write an automated tool to execute the script. The script uses the command afl-fuzz to perform fuzzing on the application and uses the commands afl-showmap and afl-cov to analyze the application. In addition, it is necessary to ensure that the script can select different applications for fuzzing. S52. For different applications, select the trained model and deploy the server in S4 to call the model; S53. Create a result file. Use the automated tool generated in step S51 to execute the script, select the application currently being fuzzed, set the fuzzing time, start the automated fuzzing, and output the fuzzing results to the result file.

7. The fuzz testing method and process based on a deep reinforcement learning model as described in claim 1, characterized in that, In step S6, the result visualization step includes the following operations: S61. Display the three fuzz test results generated in step S53—code coverage, number of crashes, and fuzz test time—in real time using the command line. S62. For the fuzz test results in the result file in S53, after the fuzz test is completed, organize the results for subsequent analysis. S63. After the fuzzing is completed, run the automation tools in S5 to execute the afl-showmap and afl-cov commands in the script to obtain the fuzzing analysis results and generate a code coverage report file.

Images