A defect detection method for advanced synthesis tools based on diversity-guided variation

Abstract

The present invention belongs to the field of software testing technology, and in particular, relates to a high-level synthesis tool defect detection method based on diversity-oriented variation. The present invention obtains a series of test cases with large statement differences by performing diversity-oriented variation on randomly generated test cases, so as to solve the problem of efficient test case generation in the process of testing high-level synthesis tools, improve test case diversity, expand test space, avoid redundant testing, and thus improve the defect detection efficiency of high-level synthesis tools.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and in particular relates to an advanced synthesis tool defect detection method based on diversity-oriented variation. Background Art

[0002] In recent years, FPGA (Field Programmable Gate Array) technology has received more and more attention. The inherent flexibility, parallelism and integrability of FPGA devices make them key core devices in the fields of communications, aerospace, medical, military, etc., and an important support foundation for ensuring national strategic security. Therefore, it is crucial to complete FPGA design and development work efficiently and reliably.

[0003] During the FPGA design and development process, the design code written by the designer needs to go through high-level synthesis (HLS), logic synthesis (RTL Synthesis) and other processes in sequence, and finally be converted into netlist-level hardware description language (HDL) code for use in subsequent development processes. The high-level synthesis process is at the starting point of the FPGA design and development process. It can convert algorithm codes written in high-level languages ​​(such as C / C++ codes) into hardware description language codes, such as Verilog codes, VHDL codes, etc. High-level synthesis tools play an important supporting role in the high-level synthesis process. Whether the high-level synthesis tool can correctly complete the synthesis of the design code seriously restricts the accuracy and reliability of FPGA design. Some existing invention patents, such as a method for implementing System C verification and a verification platform component architecture (patent number: CN202011129206.6), a simulation verification system (patent number: CN201710113561.6), and an efficient FPGA integrated verification method (patent number: CN202011597586.6), mainly verify from the aspects of design files and chip products, but still cannot completely detect FPGA product defects caused by design tool defects, and cannot fundamentally guarantee the reliability of FPGA products. Based on the above-mentioned existing related invention patents, it can also be found that there are few invention patents for defect detection in advanced synthesis tools. Therefore, it is very urgent and necessary to perform defect detection on advanced synthesis tools.

[0004] There are two main methods for advanced synthesis tools: manual testing and automated random testing. Manual testing of advanced synthesis tools is often very time-consuming, labor-intensive and inefficient; while automated random testing often generates a large number of redundant test cases, which also results in low defect detection efficiency. In order to improve the defect detection efficiency of advanced synthesis tools, the method of the present invention improves the diversity of test cases by performing diversity-oriented mutation on test cases, avoids redundant testing, and thus achieves efficient detection of defects in advanced synthesis tools. Summary of the invention

[0005] The purpose of the present invention is to provide a high-level synthesis tool defect detection method based on diversity-oriented variation. The present invention obtains a series of test cases with large statement differences by performing diversity-oriented variation on randomly generated test cases, so as to solve the problem of efficient test case generation in the process of testing high-level synthesis tools, improve the diversity of test cases, expand the test space, avoid redundant tests, and thus improve the defect detection efficiency of high-level synthesis tools.

[0006] The technical solution of the present invention:

[0007] A defect detection method for advanced synthesis tools based on diversity-guided variation, the specific steps are as follows:

[0008] Step 1: Generate a series of codes using a code random generator, extract code snippets from the generated series of codes for inserting mutation operations, classify and store the extracted code snippets, and build a candidate statement pool.

[0009] Among them, the types of code snippets that need to be extracted when constructing the candidate statement pool mainly include if branch statements, for loop statements, function calls and function bodies, and assignment statements. The tester can set the number of generated codes according to needs. For code generation, the Csmith code random generator can be used for random generation, or other code random generation tools or customized code random generation tools can be used according to actual needs.

[0010] Step 2: Check whether the test time limit has been reached. If not, use the code random generator to generate the test case seed code; if it has been reached, stop the test.

[0011] The test time limit can be set by the tester according to the needs, and does not include the time spent on extracting candidate sentences in step 1.

[0012] Step 3: Use the diversity-guided mutation method to mutate the test case seed code generated in step 2 to obtain a set of test case variants P.

[0013] Step 4: Check whether all code variants in the variant set P have been tested. If there are code variants that have not been used for testing, randomly select a code variant from the code variants that have not been used for testing as a test case p, mark it as used for testing, use the compiler to compile and run p, and record the corresponding output result O; if there are no unused code variants, jump to step 2 and continue execution.

[0014] Step 5: Process test case p. To facilitate the Testbench code to call the top-level function in test case p and obtain the output result of the top-level function, rename the main function in p to the result function, and use the output value of the main function as the return value of the result function.

[0015] Step 6: Use the high-level synthesis tool to synthesize the test case p to obtain the HDL code HP.

[0016] Step 7: Use a digital circuit simulator to simulate the HDL code HP and obtain the output result O′ of HP.

[0017] For the simulation of the HDL code HP, a simulator integrated in the high-level synthesis tool to be tested can be used, or other digital circuit simulation simulators can be used to simulate the HDL code HP according to actual needs.

[0018] Step 8: Compare O and O' to see if they are consistent. If they are consistent, jump to step 4 and continue to execute; if they are inconsistent, simplify the test cases, generate and store the defect report, and jump to step 4 and continue to execute.

[0019] For test case reduction, you can use the C-Reduce code reduction tool, or use other tools or custom tools according to actual needs.

[0020] In step 3, the diversity-guided mutation method is performed as follows:

[0021] Step a: Continuously and randomly select mutation operators from eight types of mutation operators, including inserting if branch statement code fragments, inserting for loop statement code fragments, inserting function calls and function body codes, inserting assignment statement codes, deleting if branch statement code fragments, deleting for loop statement code fragments, deleting function calls and function body codes, and deleting assignment statement codes, and calculate the probability of the selected mutation operator being accepted until an acceptable mutation operator is selected.

[0022] In the previous mutation operation R n-1 Based on this, for the currently selected mutation operator R n The probability of being accepted is P(R n |Rn-1 ), use formula (1) to calculate. According to the uniform sampling of u~U(0,1), if u≤P(R n |R n-1 ), then accept the currently selected mutation operator.

[0023]

[0024] Among them, n is the number of mutation rounds, prob is the probability of success of each trial, position n The position is the ranking of the currently selected mutation operator among the eight types of mutation operators. n-1 It is the ranking of the mutation operators accepted in the last round of mutation operation among the eight types of mutation operators.

[0025] Step b: Use the accepted mutation operator to mutate the seed code obtained in step 2 to obtain a new code variant. If the accepted mutation operator is an insertion mutation, randomly select a code snippet of the corresponding type from the candidate statement pool and insert it into the seed code to obtain a new code variant. If the accepted mutation operator is a deletion mutation, randomly select a corresponding code snippet from the seed code and delete it to obtain a new code variant.

[0026] Step c: Calculate the sum of the statement distances between the new code variant and the previous code variant, obtain the minimum statement distance MINDIST between the current code variant and the existing code variant, calculate the average statement distance increase value of this round of mutation, and use the average statement distance increase value as the score of the selected mutation operator, and sort the 8 types of mutation operators in descending order according to the score.

[0027] For a new code variant p and an existing code variant P in the code variant set P j The statement distance Dist(p,P j ), and use formula (2) to calculate.

[0028]

[0029] Among them, Stmp(p), Stmp(P j ) are the new code variant p and the existing code variant P in the code variant set P respectively. j The code statements contained in it, the |·| symbols are quantity calculations.

[0030] The total statement distance TotalDist(n) between the new code variant p and all code variants in the variant set and the seed code is calculated using formula (3). The average statement distance increase value ΔTotalDIST of this round of mutation is calculated using formula (4).

[0031]

[0032] Where n is the number of mutation rounds, num is the total number of code variants in the code variant set P, and f is the seed code.

[0033] Step d: Determine whether the minimum statement distance MINDIST between the current code variant and the existing code variant is less than the statement distance threshold. If so, jump to step e and continue execution; otherwise, store the current code variant and jump to step f and continue execution.

[0034] For the statement distance threshold, testers can set it according to their own testing requirements.

[0035] Step e: Continuously randomly select mutation operators from the eight types of mutation operators and calculate their acceptance probabilities until an accepted mutation operator is selected. Use the accepted mutation operator to mutate the current code variant p, obtain a new code variant, and jump to step c to continue execution.

[0036] Step f: Check whether the number of code variants obtained after mutating the seed code generated in step 2 meets the quantity requirement. If not, jump to step a and continue to execute. If it meets the requirement, stop mutating the seed code.

[0037] The tester can set the number of code variants generated by each seed code according to the requirements. For the judgment of code variants, the diff command can be used to detect the difference between the seed code file and the code variant file generated after the mutation operation; if there is a difference between the two files, the code generated by the mutation operation is judged to be a variant of the seed code. Otherwise, it is not. Among them, the diff command is a Linux command that is often used to compare file differences. The diff command compares the similarities and differences of text files line by line. Testers can also use other commands or tools to detect file differences according to actual conditions.

[0038] The beneficial effects of the present invention are as follows: the method of the present invention can automatically test high-level synthesis tools by generating effective and diverse test cases, can efficiently detect defects existing in high-level synthesis tools, and can automatically generate defect reports for developers to quickly repair defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the workflow of an advanced integrated tool defect detection method based on diversity-oriented variation of the present invention.

[0040] Figure 2 It is a workflow diagram of the diversity-guided mutation method in the present invention.

[0041] Specific implementation examples

[0042] The method of the present invention is described in detail below in conjunction with the accompanying drawings, technical solutions, and by taking the detection of defects in the high-level synthesis tool Vivado HLS as an example.

[0043] In this embodiment, the method of the present invention is deployed on a Linux server, and the specific configuration of the server is shown in Table 1. According to the experimental requirements, the GCC compiler and the high-level synthesis tool Xilinx Vivado HLS2019.2 to be tested are installed, and Xilinx Vivado HLS2019.2 is tested.

[0044] Table 1 Linux server configuration information table

[0045]

[0046] like Figure 1 As shown, a defect detection method for a high-level synthesis tool based on diversity-guided variation of the present invention is performed according to the following process. First, a seed code is generated using a code random generator. Then, a diversity-guided variation operation is performed on the generated seed code to obtain a code variant, and the code variant is compiled and run using a compiler to obtain a high-level language output result. Secondly, a circuit simulation simulator is used to simulate the HDL code obtained after the code variant is synthesized by the high-level synthesis tool to be tested, and the HDL code output result is obtained. Finally, by comparing the consistency between the high-level language output result and the HDL code output result, it is determined whether the high-level synthesis tool to be tested has a defect. If the comparison results are inconsistent, it indicates that the high-level synthesis tool to be tested has a defect, and the test cases that trigger the defect are simplified, and a defect report is automatically generated.

[0047] The specific steps of this embodiment are as follows:

[0048] Step 1: Generate a series of codes using a code random generator, extract structured statement codes from the codes, and build a candidate statement pool.

[0049] Specifically: use the Csmith code random generator to generate a series of codes. In this embodiment, a total of 10,000 C language codes are generated. Through a customized candidate statement extraction program, if branch statement code snippets, for loop statement code snippets, function calls and function body code snippets, and assignment statement code snippets are extracted from a series of codes, and these code snippets are classified and stored in the if, for, func, and assignment folders under the / home / hls / Candidate_code path. Among them, Csmith is an open source tool released by researchers at the University of Utah under a BSD license, which can randomly generate C language programs that meet the C99 standard. For the number of generated codes, the tester can set it according to needs. For the generation of codes and the extraction of candidate statement codes, the tester can use other or customized tools to implement them according to actual conditions.

[0050] Step 2: Check whether the test time limit has been reached; if not, use the code random generator to generate the test case seed code; if it has been reached, stop the test.

[0051] Specifically: check whether the test time limit is reached; if not, use the Csmith code random generator to randomly generate seed code, name it "csmith_gengerated_xxx.c", and store it in the / home / hls / SeedCode path. If the time limit has been reached, stop the test. For seed code generation, testers can use other program random generation tools according to actual conditions. The test time involved in the method of the present invention does not include the time consumed to extract candidate statements. The test time limit in the method of the present invention can also be set by the tester according to needs. In addition, the "xxx" in the name is filled with the serial number of the currently generated seed test case. For example, the first seed test case is named "csmith_gengerated_1.c".

[0052] Step 3: Use diversity-guided mutation methods (such as Figure 2 (as shown in the figure) perform diversity-oriented mutation operation on the test case seed code generated in step 2, obtain a set of test case code variants P, and store them in the path / home / hls / Testcases. The file name in P is named "csmith_gengerated_xxx_yyy.c". The "xxx" in the above naming is filled with the serial number of the currently generated seed test case, and "yyy" is filled with the serial number of the code variant. For example, the first variant of the first seed test case is named "csmith_gengerated_1_1.c".

[0053] Step 4: Check whether all code variants in the code variant set P have been tested; if there are code variants that have not been used for testing, randomly select a code variant from the code variants that have not been used for testing as a test case p, mark it as used, compile and run p using the compiler, and record the corresponding output result O;

[0054] Step 5: Process the test case p. To facilitate the Testbench code to call the top-level function in the test case and obtain the output result of the top-level function, the method of the present invention renames the main function in p to the result function, and uses the output value of the printf function in the main function as the return value of the result function.

[0055] Step 6: Use the high-level synthesis tool to be tested to synthesize the test case p and obtain the HDL code HP;

[0056] Specifically: In this step, the script generator generates a TCL script according to the test case p and the TestBench file location, and specifies the new project name, test case p, and TestBench file for Vivado HLS through the TCL file, sets the top-level function, solution name, optimization command, defines the technology and clock rate, runs the high-level synthesis tool to be tested for synthesis, and names the above TCL file as "test_csmith_gengerated_xxx_yyy_run_hls.tcl", and stores it in the path / home / hls / Fuzzing / Test_csmith_gengerated_xxx_yyy. The script execution program calls Vivado HLS through the Linux terminal to execute the TCL script for high-level synthesis and obtains the HDL code HP. The above TestBench file is written in C code, which mainly completes the call of the result function in the test case p and the result comparison. The specific implementation details can be determined by the tester according to the actual situation. For the script generator and the script execution program, python is used in the method of the present invention, and the tester can also choose different programming languages ​​for implementation according to the actual situation. In the above naming, "xxx" is filled with the serial number of the currently generated seed test case, and "yyy" is filled with the serial number of the code variant.

[0057] Step 7: Use a digital circuit simulator to simulate HP and obtain the output result O′ of HP.

[0058] Specifically, the co-simulation function integrated in VivadoHLS is used to obtain the output result O′ of the HDL code HP. For the simulation of the HDL code HP, the simulator integrated in the high-level synthesis tool to be tested can be used, or other digital circuit simulation simulators can be used for simulation according to actual needs.

[0059] Step 8: Compare O and O' to see if they are consistent; if they are consistent, jump to step 4 and continue to execute the test. If the results are inconsistent, simplify the test cases, generate and store the defect report, and jump to step 4 to continue to execute the test;

[0060] The C-Reduce code reduction tool can be used to simplify test cases. C-Reduce is an open source tool released by researchers at the University of Utah under a BSD license. It can automatically reduce C / C++ code according to the corresponding goals. Similarly, testers can also use other tools to simplify test cases according to actual needs.

[0061] When the above process reaches the test time limit, the test work is stopped. The method of the present invention completes the defect detection work for the advanced synthesis tool by automatically running the above process. The diversity-oriented mutation operation can obtain a large number of test cases with large statement differences, greatly increasing the number of test cases and the diversity of test cases, and improving the efficiency of defect detection of advanced synthesis tools.

Claims

1. A defect detection method for advanced integrated tools based on diversity-guided variation, characterized in that: The specific steps are as follows: Step 1: Generate a series of codes using a code random generator, extract code snippets from the generated series of codes for inserting mutation operations, classify and store the extracted code snippets, and build a candidate statement pool; The types of code snippets that need to be extracted when constructing the candidate statement pool include if branch statements, for loop statements, function calls and function bodies, and assignment statements. The tester can set the number of generated codes according to the needs. For code generation, the code random generation tool is used to generate the code randomly. Step 2: Check whether the test time limit is reached; if not, use the code random generator to generate the test case seed code; if it is reached, stop the test; The test time limit can be set by the tester according to the requirements, and does not include the time spent extracting candidate sentences in step 1; Step 3: Use the diversity-oriented mutation method to mutate the test case seed code generated in step 2 to obtain a set of test case variants P; Step 4: Check whether all code variants in the variant set P have been tested; if there are code variants that have not been used for testing, randomly select a code variant from the code variants that have not been used for testing as a test case p, mark it as used for testing, use the compiler to compile and run p, and record the corresponding output result O; if there are no unused code variants, jump to step 2 and continue execution; Step 5: Process the test case p. To facilitate the Testbench code to call the top-level function in the test case p and obtain the output result of the top-level function, rename the main function in p to the result function, and use the output value of the main function as the return value of the result function. Step 6: Use the high-level synthesis tool to be tested to synthesize the test case p and obtain the HDL code HP; Step 7: Use a digital circuit simulator to simulate the HDL code HP and obtain the output result O′ of HP; For the simulation of the HDL code HP, the simulator integrated in the high-level synthesis tool to be tested can be used, or other digital circuit simulation simulators can be used to simulate the HDL code HP according to actual needs; Step 8: Compare O and O′ to see if they are consistent; if they are consistent, jump to step 4 and continue execution; if they are inconsistent, simplify the test cases, generate and store the defect report, and jump to step 4 and continue execution; For test case simplification, you can use the C-Reduce code simplification tool, or use other tools or custom tools according to actual needs; In step 3, the diversity-guided mutation method is performed as follows: Step a: Continuously and randomly select mutation operators from eight types of mutation operators: insert if branch statement code fragment, insert for loop statement code fragment, insert function call and function body code, insert assignment statement code, delete if branch statement code fragment, delete for loop statement code fragment, delete function call and function body code, delete assignment statement code, and calculate the probability of the selected mutation operator being accepted until an acceptable mutation operator is selected; In the previous mutation operation R n-1 Based on this, for the currently selected mutation operator R n The probability of being accepted is P(R n |R n-1 ), use formula (1) to calculate; according to the uniform sampling of u~U(0,1), if u≤P(R n |R n-1 ), then accept the currently selected mutation operator; Among them, n is the number of mutation rounds, prob is the probability of success of each trial, position n The position is the ranking of the currently selected mutation operator among the eight types of mutation operators. n-1 is the ranking of the mutation operators accepted in the last round of mutation operations among the eight types of mutation operators; Step b: Use the accepted mutation operator to mutate the seed code obtained in step 2 to obtain a new code variant; if the accepted mutation operator is an insertion mutation, randomly select a code snippet of the corresponding type from the candidate statement pool and insert it into the seed code to obtain a new code variant; if the accepted mutation operator is a deletion mutation, randomly select a corresponding code snippet from the seed code and delete it to obtain a new code variant; Step c: Calculate the sum of the statement distances between the new code variant and the previous code variant, obtain the minimum statement distance MINDIST between the current code variant and the existing code variant, calculate the average statement distance increase value of this round of mutation, and use the average statement distance increase value as the score of the selected mutation operator, and sort the 8 types of mutation operators in descending order according to the score; For a new code variant p and an existing code variant P in the code variant set P j The statement distance Dist(p,P j ), calculated using formula (2); Among them, Stmp(p), Stmp(P j ) are the new code variant p and the existing code variant P in the code variant set P respectively. j The code statements contained in the ,|·| symbol is a quantity calculation; The total statement distance TotalDist(n) between the new code variant p and all code variants in the variant set and the seed code is calculated using formula (3); the average statement distance increase ΔTotalDIST of this round of mutation is calculated using formula (4); Where n is the number of mutation rounds, num is the total number of code variants in the code variant set P, and f is the seed code; Step d: Determine whether the minimum statement distance MINDIST between the current code variant and the existing code variant is less than the statement distance threshold; if so, jump to step e and continue execution; otherwise, store the current code variant and jump to step f and continue execution; Step e: Continuously randomly select mutation operators from the eight types of mutation operators, calculate their acceptance probabilities, until an accepted mutation operator is selected; use the accepted mutation operator to mutate the current code variant p, obtain a new code variant, jump to step c, and continue execution; Step f: Check whether the number of code variants obtained after mutating the seed code generated in step 2 meets the quantity requirement; if not, jump to step a and continue execution; if it meets the requirement, stop mutating the seed code; The tester can set the number of code variants generated by each seed code according to the needs. For the judgment of code variants, the diff command can be used to detect the difference between the seed code file and the code variant file generated after the mutation operation. If there is a difference between the two files, the code generated by the mutation operation is judged to be a variant of the seed code; otherwise, it is not.

Images