A Simulation Software Testing Method Based on Equivalent Mod Mutation

Abstract

The present invention discloses a simulation software testing method based on equivalent module mutation, including: performing usability check on the generated test cases, and judging whether the test cases are normally compiled, collecting the variable values and time delays of the test cases according to path coverage, generating a graphical preprocessing result graph according to the normal compilation ratio; inserting a two-branch selection segment into each test case; filling the non-executed branch of the two-branch selection segment to generate a new variant program; screening out the variant programs with high complexity in the new variants by using the Markov chain Monte Carlo method as equivalent variants; using the equivalent module input method to compare the differences between the original test program and the equivalent variants, so as to discover the defects existing in the simulation software. According to this method, multiple effective simulation software bugs are tested, which maintains the stability of the simulation verification tool to a certain extent and makes the simulation verification tool more reliable.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a simulation software testing method based on equivalent module mutation. Background Art

[0002] In digital circuit design, the register-transfer level (RTL) is an abstract model of synchronous digital circuits. The register-transfer level abstract model is used in hardware description languages such as Verilog and VHDL to create high-level descriptions of actual circuits. In modern digital design, the design at the register-transfer level is the most typical workflow.

[0003] As a tool for verifying register-transfer level code, the waveform diagram output by the simulation software can help developers understand the running effect of the entire model in an intuitive description. In this way, developers can locate possible defects in the abstract model at a lower cost. Therefore, it is crucial to ensure the reliability and stability of the simulation software.

[0004] Currently, the main testing method for simulation software is the test case generation type test that applies fuzz testing. Among them, the representative one is VeriSmith. It generates random Verilog code through the AST-based fuzz generation method and applies it to the testing of synthesis tools or simulation software.

[0005] In the fields of chip design and high-end manufacturing, there are high requirements for the stability and correctness of development tools. The existing testing technologies have two deficiencies. First, the testing tools cannot guarantee their own stability, and the Verilog code generated based on AST has certain unavailability. This leads to the problem of false positives in the defects discovered by the tools. Second, the diversity of the Verilog code generated based on AST is insufficient, and the implemented functions are relatively single, which also limits the space for the tools to explore defects. Summary of the Invention

[0006] According to the problems existing in the prior art, the present invention discloses a simulation software testing method based on equivalent module mutation, which specifically includes the following steps:

[0007] Check the usability of the generated test cases, and determine whether the test cases can be compiled normally. Collect the variable values and time delays of the test cases according to path coverage, and generate a graphical preprocessing result diagram according to the normal compilation ratio;

[0008] Insert a two-branch selection segment into each test case;

[0009] Fill the non-executed branch of the two-branch selection segment to generate a new variant program;

[0010] The Markov Chain Monte Carlo method is used to screen out the variant programs with high complexity in the new variants as equivalent variants;

[0011] The equivalent model input method is used to compare the differences between the original test program and the equivalent variants, so as to discover the defects existing in the simulation software.

[0012] The test case generation tool VeriSimth is used to generate test cases. The generated test cases are put into the test case pool and static normalization checks are performed. The simulation software is used to compile the test cases one by one to check the usability of the test cases, and the variable values of the test cases are saved into the table in the maintenance use case pool.

[0013] When inserting the binary selection code snippet: Obtain all variable values in the variable information table of the test case during the preprocessing process, randomly select a program point, select a variable according to the variable information in the variable information table, generate a discriminant that is always greater than the variable, synthesize a two-branch conditional statement with contradictory predicates according to the discriminant, insert the synthesized two-branch conditional statement at the selected program point, and place the code segment after the original program point in the execution branch.

[0014] When generating a new variant program: First, generate padding statements, fill the padding statements in the non-execution branch, perform static syntax checks on the padding statements to determine whether they meet the syntax requirements, export the code that meets the syntax requirements as a new variant file, delete the code that does not meet the syntax requirements, and re-execute the two-branch statement insertion program.

[0015] When obtaining the equivalent variant: Use the program distance between the new variant and the original test case to construct a state transition matrix, input the state transition matrix as a parameter into the selection program, obtain the acceptance probability according to the result calculated by the selection program, determine whether the acceptance probability is greater than 1. If it is, accept the variant; if not, continue to generate new variants, simplify and update the timing of the accepted variants, and save them as equivalent variant programs.

[0016] Compile, run and check the equivalent variant program. If an error occurs during compilation and running, write the error information into the bug table. If no error occurs, compile the program and generate a waveform diagram. Compare the generated waveform diagram with the waveform diagram after compiling the original test case. If there are differences, write the difference information into the bug table. If there are no differences, end the test process.

[0017] Due to the adoption of the above technical solution, a simulation software testing method based on equivalent module mutation provided by the present invention uses the Markov Monte Carlo method for variant program screening. Therefore, the generated equivalent variants have higher complexity. According to this method, multiple effective simulation software bugs are detected, which maintains the stability of the simulation verification tool to a certain extent and makes the simulation verification tool more reliable. The variant method of inserting a binary branch conditional selection statement ensures the usability of the variants we generate and results in a higher success rate of the variants. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is a flowchart of the method of the present invention DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] To make the technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention:

[0021] As Figure 1 shown, a simulation software testing method based on equivalent module mutation specifically includes the following steps:

[0022] S1: Pretreatment stage

[0023] S11: Use the test case generation tool VeriSimth to generate test cases

[0024] S12: Put the generated test cases into the test case pool and perform static normalization checks before pretreatment

[0025] S13: Start the pretreatment of the test program, and use the simulation software to compile the test cases one by one to check the usability of the test cases

[0026] S14: Detect whether the model can be compiled normally. If the model can be compiled normally, put it into the test pool; otherwise, delete the test case and print and output the error

[0027] S15: Collect the variable values and time delays of the test cases according to path coverage

[0028] S16: Save the collected variable information into the table in the maintenance case pool and generate a graphical pretreatment result diagram based on the normal compilation ratio

[0029] S17: Transfer to the test case mutation stage after the preprocessing is completed

[0030] S2: Insert the binary branch selection code snippet for each test case as follows:

[0031] S21: Obtain the values of all variables in the variable information table during the preprocessing of the test case

[0032] S22: Randomly select a program point

[0033] S23: Randomly select a variable based on the variable information in the variable information table

[0034] S24: Generate a discriminant that is always greater than the variable discriminant

[0035] S25: Synthesize a two-branch conditional statement with a contradictory predicate according to the discriminant

[0036] S26: Insert the synthesized two-branch conditional statement at the selected program point

[0037] S27: Place the code segment after the original program point in the execution branch

[0038] S3: Fill the non-executing branch of the two-branch conditional statement as follows:

[0039] S31: Perform preprocessing sampling for filling the program

[0040] S32: Use the test case generation tool to generate filling statements

[0041] S33: Fill the filling statements in the non-executing branch

[0042] S34: Perform static syntax checking on the filling statements to determine whether they meet the syntax requirements

[0043] S35: Export the code that meets the syntax requirements as a new variant file, delete the code that does not meet the requirements, and re-execute the two-branch statement insertion program

[0044] S4: Another innovation of our program lies in the variant selection process

[0045] We choose the Markov chain Monte Carlo method as our sampling selection strategy, and select variants with higher acceptance complexity based on the defined variant distance. This approach ensures the stability and robustness of the entire program, making the entire program more stable in terms of execution timing. The specific steps are as follows:

[0046] S41: Construct the state transition matrix using the program distance between the new variant and the original test case

[0047] S42: Input the state transition matrix as a parameter into the selection program.

[0048] S43: Obtain the acceptance probability based on the result calculated by the selection program.

[0049] S44: Determine if the acceptance probability is greater than 1. If so, accept the variant; otherwise, continue to generate a new variant for judgment.

[0050] S45: Simplify and update the timing of the accepted variant and save it as an equivalent variant program.

[0051] S5: After the above steps, the filling is completed. Next, enter the program for saving test cases and reporting the test process.

[0052] The present invention uses a new technology called Equivalent Model Input (EMI) testing. This is a new development in differential testing of process language compilers. The idea is to systematically mutate a source program. As long as its semantics remain equivalent under given input data, its output should also be equivalent. As a special case, if some variables of the successfully simulated model are not compiled, or a comparison error occurs when comparing the signal data of the variables after simulation with the simulation data of the original program variables. These errors are not caused by human factors and belong to the problems of the compiler itself. In this way, we can say that there is a bug in the compiler. Therefore, the differential testing using equivalent model input can effectively discover compiler problems that are easily overlooked. The steps are as follows

[0053] S51: Compile, run, and check the equivalent variant program.

[0054] S52: If an error occurs during compilation and running, write the error message into the bug table.

[0055] S53: If there is no compilation error, compile the program and generate a waveform diagram.

[0056] S54: Compare the generated waveform diagram with the waveform diagram after compiling the original test case.

[0057] S55: If a difference occurs, write the difference information into the bug table.

[0058] S56: Otherwise, end the comparison program.

[0059] As described above, it is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution of the present invention and its inventive concept, makes equivalent substitutions or changes, and all should be covered by the protection scope of the present invention.

Claims

1. A simulation software testing method based on equivalent modulus mutation, characterized in that Including: Conduct an availability check on the generated test cases, determine whether the test cases can be compiled normally, collect the variable values and time delays of the test cases according to path coverage, and generate a graphical preprocessing result graph based on the normal compilation ratio; Insert a binary selection segment into each test case; Fill the non-executed branch of the binary selection segment to generate a new variant program; Use the Markov chain Monte Carlo method to screen out the variant programs with high complexity in the new variants as equivalent variants; Use the equivalent model input method to compare the differences between the original test program and the equivalent variants, so as to discover the defects existing in the simulation software; When inserting the binary selection code segment: obtain all the variable values in the variable information table of the test case during the preprocessing process, randomly select a program point, select a variable according to the variable information in the variable information table, generate a discriminant greater than the variable, synthesize a binary conditional statement with contradictory predicates according to the discriminant, insert the synthesized binary conditional statement at the selected program point, and place the code segment after the original program point in the executed branch.

2. The simulation software testing method based on equivalent module mutation according to claim 1, wherein: Use the test case generation tool VeriSimth to generate test cases, put the generated test cases into the test case pool and conduct static normalization checks, use the simulation software to compile the test cases one by one to check the availability of the test cases, and save the variable values of the test cases into the table for maintaining the use cases.

3. The simulation software testing method based on equivalent modulus mutation according to claim 1, wherein: When generating a new variant program: first generate filling statements, fill the filling statements in the non-executed branch, conduct a static syntax check on the filling statements to determine whether they meet the syntax requirements, export the code that meets the syntax requirements as a new variant file, delete the code that does not meet the syntax requirements and re-execute the binary statement insertion program.

4. The simulation software testing method based on equivalent module mutation according to claim 3, characterized in that: When obtaining equivalent variants: use the program distance between the new variants and the original test cases to construct a state transition matrix, input the state transition matrix as a parameter into the selection program, obtain the acceptance probability according to the result calculated by the selection program, determine whether the acceptance probability is greater than 1, if so, accept the variant, if not, continue to generate new variants, simplify and update the timing of the accepted variants, and save them as equivalent variant programs.

5. The simulation software testing method based on equivalent modulus mutation according to claim 4, wherein: Compile, run and check the equivalent variant program. If an error occurs during compilation and running, write the error information into the bug table. If no error occurs, compile the program and generate a waveform diagram, compare the generated waveform diagram with the waveform diagram after compiling the original test case. If a difference occurs, write the difference information into the bug table. If no difference occurs, end the test process.

Images