A test case generator adaptive optimization method based on bayesian screening

Abstract

The application discloses a test case generator adaptive optimization method based on Bayesian screening, is a parameter configuration test scene for a logic synthesis tool, constructs a Bayesian optimization search framework based on a Gaussian process proxy model, extracts test program log features and net table features to form a single program behavior vector, and calculates behavior coverage gain by means of Euclidean distance; in combination with differential testing, running anomaly detection, program crash and output deviation fault identification, defect yield is obtained by counting newly added defects; the comprehensive utility value is obtained by weighting the behavior coverage gain and the defect yield, and the adaptive screening of the function to be evaluated is guided; test configuration space exploration and defect mining collaborative optimization are realized, hidden defects of the logic synthesis tool can be efficiently found with fewer simulation times, test behavior diversity is improved, and the problems of strong blindness and low defect detection efficiency of traditional random testing are overcome.

Description

Technical Field

[0001] This invention belongs to the field of integrated circuit simulation and testing technology, specifically relating to an adaptive optimization method for a test case generator based on Bayesian screening. Background Technology

[0002] Logic synthesis is a core step in the integrated circuit design process, responsible for automatically converting hardware description languages ​​(HDLs) such as Verilog / VHDL into gate-level netlists and mapping them to target device resources. The stability and correctness of logic synthesis tools directly affect chip design quality, especially in high-reliability fields such as aerospace, rail transportation, and industrial control. Defects in synthesis tools can lead to serious circuit functional errors, timing failures, or even system crashes.

[0003] To ensure the reliability of logic synthesis tools, the industry typically uses methods such as random generation, fixed parameters, simple traversal, or heuristic search to configure HDL test case generators. These generators produce diverse test cases (RTL programs) which are then input into the logic synthesis tool for testing, used to detect potential circuit defects and improve code coverage. However, existing test case generation methods, such as random search, grid traversal, and particle swarm optimization, have the following key problems: 1. Huge configuration space and extremely low search efficiency: For example, Verismith contains dozens of adjustable parameters, covering expression type, module structure, hierarchical depth, statement complexity, timing ratio, randomization strategy, etc. The parameter values ​​are heterogeneous, strongly coupled, and highly dimensional, and the combination space expands exponentially.

[0004] 2. Sparse defect signals and lack of effective search guidance: Faults such as tool crashes, hangs, and functional inconsistencies are low-probability events. Directly relying on defect feedback can easily lead to the search getting stuck in local optima, long-term lack of feedback, or ineffective loops, making it difficult to explore high-value configuration areas.

[0005] 3. Focusing only on input diversity while ignoring comprehensive behavioral differences: The configuration is guided only by the diversity of generated program syntax and structure, without modeling the differences in internal execution paths, optimization strategies, and resource mapping behaviors of the logic, making it difficult to reach the deep-seated defects of the tool.

[0006] 4. Lack of a collaborative optimization mechanism between behavior and defects: Single defect-driven approaches are prone to sparsity, and single behavior-driven approaches lack clear objectives, making it impossible to achieve the dual objectives of maximizing behavior coverage and maximizing defect triggering, resulting in low test coverage and defect detection rate. Summary of the Invention

[0007] The present invention aims to solve the aforementioned technical problems existing in the prior art by providing an adaptive optimization method for a test case generator based on Bayesian filtering.

[0008] The technical solution of this invention is: Compared with the prior art, the present invention has the following advantages: 1. Bayesian filtering and traversal of parameters is used to quickly locate fault-related configurations and improve testing efficiency; 2. A dual-objective collaborative optimization mechanism of behavior coverage gain and defect benefit is adopted to simultaneously take into account the circuit behavior coverage capability and fault detection capability, avoiding the defects of traditional single-objective search that are prone to getting trapped in limited local optima and ineffective iterative loops; 3. Based on comprehensive logs and gate-level netlists, construct behavioral coverage gain evaluation metrics to improve the coverage of circuit behavior by test cases and improve the search stagnation problem caused by sparse fault samples; 4. Integrating differential testing and anomaly detection to calculate defect benefits, efficiently identifying low-probability faults such as tool crashes, hangs, and functional inconsistencies; 5. Automatically streamlines minimal failure test cases and provides standardized reports to facilitate vulnerability fixing by developers; 6. Closed-loop fully automatic iteration, requiring no manual adjustment of parameter search strategies throughout the entire process, resulting in a high degree of automation; 7. Compatible with Vivado and Yosys, suitable for FPGA / ASIC design scenarios, and highly practical. Attached Figure Description

[0009] Figure 1 This is a general flowchart of an embodiment of the present invention.

[0010] Figure 2 This is a flowchart illustrating the calculation behavior coverage gain in an embodiment of the present invention.

[0011] Figure 3 This is a flowchart illustrating the calculation of defect benefits in an embodiment of the present invention.

[0012] Figure 4 This is a comparison chart of the number of defects detected in the embodiments of the present invention and the prior art.

[0013] Figure 5 This is a statistical diagram illustrating the fault classification detected in an embodiment of the present invention. Detailed Implementation

[0014] This invention provides an adaptive optimization method for a test case generator based on Bayesian filtering, such as... Figure 1 As shown, perform the following steps in sequence: Step 1. Collect the test case generator configuration input set Collect configuration input sets such as test case generator parameters, logic synthesis tools, and initial test program sets.

[0015] Step 2. Construct the test case generator configuration space model The quantifiable configuration space model is a parameter mapping representation model. Mapping rules are established from mixed-type configuration parameters of the test case generator to a continuous vector space, achieving unified vectorized representation of different parameter types. This provides a usable parameter search space for Gaussian process modeling. Specifically, the following steps are executed sequentially: Step 2.1 Unify the adjustable parameters of the test case generator into a configuration vector to form the original discrete configuration space; Step 2.2 Divide the adjustable parameters in the original discrete configuration space according to function into expression generation parameters, module item and statement generation parameters, scale and hierarchical complexity parameters, and sampling and behavior control parameters, specifically: ① Expression generation parameters: control binary, ternary, concatenation, and select the expression ratio; ② Module items and statement generation parameters: control the distribution of assignment, timing, condition, and loop statements; ③ Scale and hierarchical complexity parameters: control the number of modules, nesting depth, and statement level; ④ Sampling and behavioral control parameters: control sampling method and deterministic / random ratio; Step 2.3 Performs normalization mapping on numerical parameters, binary encoding on Boolean parameters, and one-hot encoding or discrete embedding on enumeration parameters to obtain mixed-type parameters. Specifically: ① Numerical parameters (such as depth, quantity): normalized to the [0,1] interval; ② Boolean parameter: 0 / 1 binary encoding; ③ Enumerated parameters: one-hot encoding or discrete embedding; Step 2.4 Configure the encoding function : Mixed type parameters The space is mapped to a continuous vector space, resulting in a quantifiable configurable space model that is easy to process using Bayesian optimization.

[0016] Step 3. Fit a Gaussian process surrogate model based on the quantifiable configurable space model and initialize it; Ten initial configuration parameters were randomly sampled from the quantifiable configurable space model. Based on the initial configuration parameters Generate test cases Each configuration drives Verismith to generate 20 Verilog test programs; these test programs are then input into a logic synthesis tool to perform simulation tests, calculating the behavior coverage gain and defect benefit respectively. A bi-objective weighted utility aggregation is then performed on the behavior coverage gain and defect benefit to obtain the overall utility value. With initial configuration parameters and the corresponding comprehensive utility value Building the initial dataset: Fit a Gaussian process surrogate model and output an arbitrary configuration. Expected utility With uncertainty ; The specific process for calculating coverage benefits is as follows: Collect test cases. The logical synthesis execution log and netlist output data are used to extract multi-dimensional behavioral features, construct a unified behavioral representation vector, and calculate the behavioral coverage gain; the steps are as follows: Step 3.1.1 Log Characteristics extract: ① Pass Distribution: Statistics of each optimization stage ( opt, fsm, memory Number of calls; ② Execution mode: Extraction n-gram Sequence (e.g.) fsm_detect→fsm_opt ); ③ Optimize statistics: changes in the number of nodes, logic depth, and number of units; ④ Event indicator: Whether it has been triggered fsm, memory, abc Key optimizations, etc. Step 3.1.2 Netlist Features extract: ① Unit distribution: AND, OR, MUX, DFF, RAM quantity; ② Structural complexity: total number of units, logical depth, number of levels; ③ Topological characteristics: average fan-out, maximum fan-out, critical path delay; ④ Resource characteristics: number of storage blocks, proportion of sequential logic; Step 3.1.3 Behavior Vector Construction: Concatenate log features and netlist features to obtain test cases. Unified Behavior Vector ; Step 3.1.4 Calculate the minimum Euclidean distance: Calculate the test cases. Minimum Euclidean distance between behavior and the set of historical behaviors In other words, the novelty of a single program's behavior represents the difference between current combined behavior and historical behavior; Step 3.5 aggregates at the test assembly level to obtain the behavior coverage gain: ;

[0017] in For behavior coverage gain, The test assembly generated for the current configuration, The novelty of a single procedure's behavior.

[0018] The specific process for calculating defect benefits is as follows: defects are identified by combining differential testing and anomaly detection, defect deduplication is achieved through defect signature, and the defect benefits of the current test case are calculated. Step 3.2.1 Differential Testing: For the same test program, compare the simulation outputs of the default configuration, current configuration, and historical configuration, and define the points of difference. : ;

[0019] in, For benchmark simulation output, Configure simulation output for the current test; Step 3.2.2 Anomaly Detection: Monitor the overall process and define... Same as step 3.2.1; Step 3.2.3 Defect Indication: Combining the two types of faults, defect indication... ,in This is the crash anomaly flag at time t; Step 3.2.4 Calculate the defect benefit: like Then the defect benefits ; like Defect benefits ; Extract fault type, log summary, and minimal program structure to generate a unique signature. By deduplicating defects, the set of newly added defects in the current configuration is obtained. : ;

[0020] in, This is the complete set of defects detected in this simulation. It is the global defect library running up to the current moment.

[0021] The calculation formula for the bi-objective weighted utility aggregation is as follows: ;

[0022] in, The overall utility value is given by λ∈[0,1], which is the balance coefficient, preferably 0.6. For behavior coverage gain, For defective benefits, This is the current test case.

[0023] Step 4. Perform Bayesian parameter filtering based on the current Gaussian process surrogate model and output the configuration parameters to be simulated; Parameter selection is performed based on a Bayesian optimization framework, and an upper confidence bound acquisition function is used. Filter the next configuration parameters to be simulated, among which... and These are the expected utility and uncertainty of the current Gaussian process surrogate model output, respectively, to balance the algorithm's exploratory capabilities with its ability to utilize historical experience.

[0024] Step 5. Generate test cases based on the configuration parameters to be simulated, execute simulation tests, and calculate the behavior coverage gain and defect benefit respectively; the behavior coverage gain and defect benefit are calculated according to the calculation process described in Step 3.

[0025] Step 6. Perform bi-objective weighted utility aggregation on the behavioral coverage gain and defect benefit to obtain the comprehensive utility value, specifically calculated according to the calculation process described in Step 3.

[0026] Step 7. Determine whether the comprehensive utility value meets the preset iteration termination condition. If yes, compile and output a standardized test report based on the current test cases; otherwise, update the Gaussian process proxy model based on the comprehensive utility value, and then jump to step 4 to execute iteratively until the iteration termination condition is met. In this embodiment of the invention, the total number of iterations is set to 200 rounds. After the iteration terminates, the high-value parameter configuration sequence obtained by the search is output. The process of compiling and outputting a standardized test report based on the current test cases is as follows: Step 7.1 Minimize test cases: Use Delta Debugging to gradually remove irrelevant modules, statements, and variables, retaining only the minimum reproducible core code; Step 7.2 Defect Classification: Classify defects into crash failures, performance failures, and functional failures; Step 7.3 Report generation, which includes the following: ① Trigger configuration parameters; ② Minimal test program; ③ Fault types and logs; ④ Reproduction steps; Step 7.4 Report Output: Export standardized documentation and submit it to the tool vendor or open source community.

[0027] Under the condition of unified hardware, software and testing budget, the embodiments of the present invention are compared with four mainstream methods: Default, Random, GA, and BO. The number of defect detections is compared. Figure 4As shown in the figure. Experimental results show that the embodiments of the present invention detected a total of 18 independent real faults, of which the commercial tool Vivado detected 16 faults and the open-source tool Yosys detected 2 faults; the number of faults detected by the remaining baseline methods were 1 for the Default method, 2 for the Random method, 3 for the GA method, and 7 for the BO method. The fault detection capability of the present invention is far superior to that of the prior art.

[0028] The 18 faults detected in the embodiments of the present invention were classified and statistically analyzed, and the results are as follows: Figure 5 As shown in the figure, there were 14 crash failures, 4 performance failures, 1 parsing failure, and 2 optimization-induced failures. The failure types covered the entire process of the logic synthesis tool, including front-end parsing, mid-term optimization, and back-end mapping, verifying the comprehensiveness of the failure coverage of this method.

[0029] All 18 faults detected in this embodiment of the invention have been submitted to the corresponding tool development team, and all have been confirmed and fixed by the developers. This fully demonstrates that the invention can effectively uncover potential defects of logic synthesis tools under complex parameter configurations and has extremely high engineering practical value. For specific vulnerability information, please refer to Table 1.

[0030] Table 1

[0031] In summary, this invention addresses the parameter configuration testing scenario for logic synthesis tools by constructing a Bayesian optimization search framework based on a Gaussian process surrogate model. It extracts test program log features and netlist features to construct a single-program behavior vector, and calculates behavior coverage gain using Euclidean distance. Combining differential testing and runtime anomaly detection, it identifies program crashes and output deviation faults, statistically analyzes newly added defects to obtain defect benefits, and weights the behavior coverage gain and defect benefits to obtain a comprehensive utility value, guiding the acquisition function to adaptively select configurations to be evaluated. This achieves collaborative optimization of test configuration space exploration and defect discovery, enabling the efficient discovery of hidden defects in logic synthesis tools with fewer simulations, improving the diversity of test behaviors, and overcoming the problems of strong blindness and low defect detection efficiency in traditional random testing.

Claims

1. An adaptive optimization method for a test case generator based on Bayesian filtering, characterized in that... Perform the following steps in sequence: Step 1. Collect the test case generator configuration input set; Step 2. Construct a quantifiable configuration space model for the test case generator; Step 3. Fit a Gaussian process surrogate model based on the quantifiable configurable space model and initialize it; Step 4. Perform Bayesian parameter filtering based on the current Gaussian process surrogate model and output the configuration parameters to be simulated; Step 5. Generate test cases based on the configuration parameters to be simulated, execute simulation tests, and calculate the behavior coverage gain and defect benefit respectively; Step 6. Perform bi-objective weighted utility aggregation on the behavioral coverage gain and defect benefit to obtain the comprehensive utility value; Step 7. Determine whether the preset iteration termination condition is met. If yes, compile and output a standardized test report based on the current test cases. No, update the Gaussian process proxy model based on the comprehensive utility value, and then jump to step 4 for repeated execution.

2. The adaptive optimization method for test case generator based on Bayesian filtering according to claim 1, characterized in that... Step 2 is as follows: Step 2.1 Unify the adjustable parameters of the test case generator into a configuration vector to form the original discrete configuration space; Step 2.2 Divide the adjustable parameters in the original discrete configuration space into expression generation parameters, module item and statement generation parameters, scale and hierarchical complexity parameters, and sampling and behavior control parameters according to their functions; Step 2.3 Perform normalization mapping on numerical parameters, binary encoding on Boolean parameters, and one-hot encoding or discrete embedding on enumerated parameters to obtain mixed-type parameters; Step 2.4 By configuring the encoding function, the mixed-type parameter space is mapped to a continuous vector space to obtain a quantizable configuration space model.

3. The adaptive optimization method for test case generator based on Bayesian filtering according to claim 2, characterized in that... Step 3 is as follows: Based on the quantifiable configuration space model, initial configuration parameters are randomly collected, test cases are generated according to the initial configuration parameters, simulation tests are executed, behavior coverage gain and defect benefit are calculated respectively, and the behavior coverage gain and defect benefit are aggregated with dual objective weighted utility to obtain the comprehensive utility value. The initial dataset is constructed with the initial configuration parameters and the corresponding comprehensive utility value, and a Gaussian process surrogate model is fitted.

4. The adaptive optimization method for test case generator based on Bayesian filtering according to claim 3, characterized in that... The specific process for calculating coverage gain in steps 3 and 5 is as follows: collect the logic synthesis execution log and netlist output data in the test cases, extract multi-dimensional behavioral features, construct a unified behavioral representation vector, and calculate the behavioral coverage gain.

5. The adaptive optimization method for a test case generator based on Bayesian filtering according to claim 4, characterized in that... The specific process for calculating defect benefits in steps 3 and 5 is as follows: defects are identified by combining differential testing and anomaly detection, defect deduplication is achieved through defect signature, and the defect benefits of the current test case are calculated.

6. The adaptive optimization method for a test case generator based on Bayesian filtering according to claim 1, 2, or 3, characterized in that... The formulas for calculating the weighted utility aggregation of the two objectives described in steps 3 and 6 are as follows: ； in, The overall utility value is λ∈[0,1], and the balance coefficient is λ. For behavior coverage gain, For defective benefits, This is the current test case.

Images