A Simulation Software Testing Method Based on Variation in the Hibernation Area

Abstract

The present invention discloses a simulation software testing method based on dormant area mutation, including: putting test cases into a test case pool for static normalization check, compiling the test cases one by one, and determining whether the test cases are compiled normally and generating graphical preprocessing results; collecting delay information for the preprocessed test cases, and splitting the qualified delays into two new delays; generating new expressions after the new delays that meet the inertial delay conditions to construct a dormant area; performing an operation to break the combinational logic loop on the dormant area code and generating new equivalent variants; using the differential testing method with equivalent mode input to compare the output results of the equivalent variants and the test cases to find compilation problems. Based on the testing techniques in this method, more diverse test variants can be generated, and defects in the simulation software can already be detected under the existing test conditions, ensuring the stability of simulation tools in the fields of chip design and high-end manufacturing.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a simulation software testing method based on dormant area mutation. Background Art

[0002] In digital circuit design, the register-transfer level (RTL) is an abstract model of synchronous digital circuits. The register-transfer level abstract model is used in hardware description languages such as Verilog and VHDL to create a high-level description of the actual circuit. In modern digital design, the design at the register-transfer level is the most typical workflow.

[0003] As a tool for verifying register-transfer level code, the waveform diagram output by the simulation software can help developers understand the running effect of the entire model in an intuitive description. In this way, developers can locate possible defects in the abstract model at a lower cost. Therefore, it is crucial to ensure the reliability and stability of the simulation software.

[0004] Currently, the main testing method for simulation software is the test case generation type test applying fuzz testing. Among them, VeriSmith is representative. It generates random Verilog code through an AST-based fuzz generation method and applies it to the testing of synthesis tools or simulation software.

[0005] In the fields of chip design and high-end manufacturing, there are high requirements for the stability and correctness of development tools. The existing testing technologies have two deficiencies. First, the testing tools cannot guarantee their own stability, and the Verilog code generated based on AST has certain unavailability. This leads to the problem of false positives in the defects discovered by the tools. Second, the diversity of the Verilog code generated based on AST is insufficient, and the implemented functions are relatively single, which also limits the space for the tools to explore defects. Summary of the Invention

[0006] According to the problems existing in the prior art, the present invention discloses a simulation software testing method based on dormant area mutation, which specifically includes the following steps:

[0007] Put the generated test cases into the test case pool for static normalization inspection, and use the simulation software to compile the test cases one by one to determine whether the test cases are normally compiled and generate a graphical preprocessing result;

[0008] Collect delay information for the preprocessed test cases, and split the qualified delays into two new delays;

[0009] Generate a new expression under the new delay that meets the inertial delay condition to construct a sleep area;

[0010] Perform a combinational logic loop breaking operation on the sleep area code and generate a new equivalent variant;

[0011] Use the differential test method with equivalent module inputs to compare the output results of the equivalent variant and the test case to find compilation problems.

[0012] The test case includes a simulation stimulus file and a design model file. When judging whether the test case compiles normally, if the test case compiles normally, it is put into the test pool, otherwise the test case is deleted and an error is printed and output. According to path coverage, collect the variable values and delays of the test case, save the collected variable and delay information into the maintenance table, and generate a graphical preprocessing result graph based on the normal compilation ratio.

[0013] When constructing the sleep area: Obtain all the delay and variable value information in the maintenance table during the preprocessing of the test case. Randomly select a program point in the simulation stimulus file. According to the variables in the corresponding program point, read the corresponding delay from the maintenance table. If so, continue to execute; if not, reselect the program point; Randomly split the delay into two new delays, and judge whether one of the newly generated delays is greater than the delay specified in the design model file. If so, continue to execute; if not, re - perform the delay splitting; Assign the new delay that is less than the delay specified in the design model file after splitting to the expression in the program point, and add the new delay that is greater than the delay specified in the design model file after splitting after the program point to construct the sleep area.

[0014] When generating a new expression under the new delay that meets the inertial delay condition: Define a mutation operator library, use the variables at the selected program point as the mutation variable library, randomly select two variables from the mutation variable library and combine them with the mutation operator to form a new mutation expression, insert the expression into the sleep area, compile the test case after the mutation operation and check. If the test case runs normally, proceed to the next step; otherwise, re - select the program point to perform the mutation operation.

[0015] When performing a combinational logic loop breaking operation on the sleep area code: Use the combinational logic loop warning property in the simulation software to output combinational logic loop information, and locate the combinational logic loop according to the combinational logic loop information; Judge whether the type of the combinational logic loop is asynchronous reset or asynchronous set. If not, select the program point where the output end of the logic loop is located and change the data flow of the combinational logic loop, and change the assignment statement in the logic loop to a blocking assignment. If so, insert the corresponding delay in the design model file according to the variable information in the maintenance table.

[0016] When using the differential testing method with equivalent modulus input to compare the output results of equivalent variants and test cases: Save the test case after the destruction operation as a new equivalent variant program, compile and run it, and check the equivalent variant program. If there is an error during compilation and running, write the error message into the bug table. If there is no compilation error, compile the program and generate a waveform diagram. Compare the generated waveform diagram with the waveform diagram after compiling the original test case. If there are differences, write the difference information into the bug table; otherwise, end the comparison program.

[0017] Due to the adoption of the above technical solution, a simulation software testing method based on dormant area mutation provided by the present invention can generate more diverse test variants based on the testing technology in this method. Under the existing test conditions, the defects of the simulation software can already be detected, ensuring the stability of the simulation tool in the fields of chip design and high-end manufacturing. Compared with the previous technologies, the technology we use has a wider coverage and a higher maturity level. Currently, we have detected multiple effective defects of simulation tools, including Iverilog, Vivado, Modelsim, etc. Among them, there are multiple officially confirmed vulnerabilities, which to a certain extent maintain the stability and reliability of the simulation tool. Brief Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is the flowchart of the method of the present invention Detailed Embodiments

[0020] To make the technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention:

[0021] As Figure 1 shown, a simulation software testing method based on dormant area mutation specifically includes the following steps:

[0022] S1: Pretreatment stage

[0023] S11: Use the test case generation tool VeriSimth to generate test cases, and the test cases include simulation excitation files and design model files

[0024] S12: Put the generated test cases into the test case pool and perform static normalization checks before pretreatment

[0025] S13: The preprocessing of the test program starts. Use the simulation software to compile the test cases one by one to check the availability of the test cases.

[0026] S14: Detect whether the model can be compiled normally. If the model can be compiled normally, put it into the test pool. Otherwise, delete the test case and print the error output.

[0027] S15: Collect the variable values and time delays of the test cases according to path coverage.

[0028] S16: Save the collected variable and time delay information into the maintenance table, and generate a graphical preprocessing result graph based on the normal compilation ratio.

[0029] S17: The preprocessing ends and transfers to the test case mutation stage.

[0030] S2: The following are the steps to judge the sleep area for each test case after the preprocessing ends. The specific steps are as follows:

[0031] S21: Obtain all the time delay and variable value information of the test cases in the maintenance table during the preprocessing process.

[0032] S22: Randomly select a program point in the simulation stimulus file.

[0033] S23: Read the corresponding time delay from the maintenance table according to the variables in the corresponding program point.

[0034] S24: Determine whether the time delay is less than the time delay specified in the design model file. If so, continue to the next step.

[0035] If not, reselect the program point.

[0036] S25: Randomly split the time delay into two new time delays, and determine whether one of the newly generated time delays is greater than the time delay specified in the design model file. If so, continue to the next step. If not, re-perform the time delay splitting.

[0037] S26: Assign the new time delay that is less than the time delay specified in the design model file after splitting to the expression in the program point.

[0038] S27: Add the new time delay that is greater than the time delay specified in the design model file after splitting after the program point to construct the sleep area.

[0039] S3: After the new time delay is inserted, we start to generate new expressions to increase the complexity of the overall program.

[0040] S31: Define the mutation operator library, including Verilog operators such as addition, subtraction, multiplication, division, bitwise OR, and bitwise AND.

[0041] S32: Use the variables at the selected program points as the mutant variable library

[0042] S33: Randomly select two variables from the mutant variable library and combine them with mutation operators to form a new mutant expression

[0043] S34: Insert the expression into the sleep area

[0044] S35: Compile the test cases after mutation operations and check

[0045] S36: If the test cases can run normally, proceed to the next step; otherwise, reselect the program points to perform mutation operations

[0046] S4: After constructing and modifying the sleep area, in order to improve the reliability of the test cases, we perform the operation of breaking the combinational logic loop. The specific steps are as follows

[0047] S41: Use the logic loop warning function in the simulation software to output the combinational logic loop information

[0048] S42: Locate the combinational logic loop according to the combinational logic loop information

[0049] S43: Determine whether the type of the combinational logic loop is asynchronous reset or asynchronous set

[0050] S44: If not, select the program point where the output terminal of the logic loop is located

[0051] S45: Insert the corresponding time delay in the design model file according to the variable information corresponding to the maintenance table

[0052] S46: If it is a combinational logic loop of asynchronous reset or asynchronous set

[0053] S47: Change the data flow direction of the combinational logic loop, and change the assignment statement in the logic loop to a blocking assignment. S5: After the above steps, the overall sleep area mutation program is completed. Next, enter the stage of saving the test cases and reporting the test process

[0054] This method uses a new technology called Equivalent Mod Input (EMI) testing. This is a new development in differential testing of procedural language compilers. The idea is to systematically mutate a source program. As long as its semantics remain equivalent under the given input data, its output should also be equivalent. As a special case, if some variables of the successfully simulated model are not compiled, or a comparison error occurs when comparing the signal data of the variables after simulation with the simulation data of the original program variables. These errors are not caused by human factors and belong to the problems of the compiler itself. In this way, we can say that this is a bug in the compiler. Therefore, using the differential testing of equivalent mod input can effectively discover compiler problems that are easily overlooked. The steps are as follows

[0055] S51: Save the test case after the destruction operation as a new equivalent variant program

[0056] S52: Compile, run, and check the equivalent variant program

[0057] S53: If there is an error during compilation and running, write the error information into the bug table

[0058] S54: If there is no compilation error, compile the program and generate a waveform diagram

[0059] S55: Compare the generated waveform diagram with the waveform diagram after compiling the original test case

[0060] S56: If there are differences, write the difference information into the bug table

[0061] S57: Otherwise, end the comparison program

[0062] As described above, it is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution of the present invention and its inventive concept, makes equivalent replacements or changes, and all should be covered within the protection scope of the present invention.

Claims

1. A simulation software testing method based on mutation in the dormant area, characterized in that Including: Put the generated test cases into the test case pool for static normalization check, compile the test cases one by one using simulation software, and determine whether the test cases are compiled normally and generate graphical preprocessing results; Collect delay information for the preprocessed test cases, and split the qualified delays into two new delays; Generate new expressions after the new delays that meet the inertial delay conditions to construct a sleep area; Perform an operation to break the combinational logic loop on the sleep area code and generate a new equivalent variant; Use the differential test method with equivalent modulus input to compare the output results of the equivalent variant and the test cases to find compilation problems; When constructing the sleep area: Obtain all the delay and variable value information in the maintenance table during the preprocessing of the test cases. Randomly select a program point in the simulation stimulus file. According to the variables in the corresponding program point, read the corresponding delay from the maintenance table. If so, continue to execute; if not, reselect the program point; Randomly split the delay into two new delays, and determine whether one of the newly generated delays is greater than the delay specified in the design model file. If so, continue to execute; if not, re-perform the delay split; Assign the new delay that is less than the delay specified in the design model file after splitting to the expression in the program point, and add the new delay that is greater than the delay specified in the design model file after splitting after the program point to construct the sleep area.

2. The simulation software testing method based on dormant area mutation according to claim 1, wherein: The test cases include a simulation stimulus file and a design model file. When determining whether the test cases are compiled normally, if the test cases are compiled normally, put them into the test pool; otherwise, delete the test cases and print and output the error. Collect the variable values and delays of the test cases according to path coverage, save the collected variable and delay information into the maintenance table, and generate a graphical preprocessing result diagram according to the normal compilation ratio.

3. The simulation software testing method based on dormant area mutation according to claim 1, wherein: When generating new expressions after the new delays that meet the inertial delay conditions: Define a mutation operator library, use the variables at the selected program point as the mutation variable library, randomly select two variables from the mutation variable library and combine them with the mutation operator to form a new mutation expression, insert the expression into the sleep area, compile and check the test cases after the mutation operation. If the test cases run normally, proceed to the next step; otherwise, reselect the program point and execute the mutation operation.

4. The simulation software testing method based on dormant area mutation according to claim 1, wherein: When performing an operation to break the combinational logic loop on the sleep area code: Use the combinational logic loop warning property in the simulation software to output combinational logic loop information, and locate the combinational logic loop according to the combinational logic loop information; Determine whether the type of the combinational logic loop is asynchronous reset or asynchronous set. If not, select the program point where the output end of the logic loop is located and change the data flow of the combinational logic loop, and change the assignment statement in the logic loop to a blocking assignment. If so, insert the corresponding delay in the design model file according to the variable information corresponding to the maintenance table.

5. The simulation software testing method based on dormant area mutation according to claim 1, characterized in that: When using the differential test method with equivalent module input to compare the output results of equivalent variants and test cases: Save the test case after the destructive operation as a new equivalent variant program, compile and run it, and check the equivalent variant program. If there is an error during compilation and running, write the error message into the bug table. If there is no compilation error, compile the program and generate a waveform diagram. Compare the generated waveform diagram with the waveform diagram after compiling the original test case. If there are differences, write the difference information into the bug table. Otherwise, end the comparison program.

Images