Randomization configuration parameter test method, device and equipment and storage medium

Abstract

The invention discloses a randomized configuration parameter testing method and device, equipment and a storage medium. Comprising the steps of defining names and enabling states of configuration groups on a top layer of a configuration file; performing legality verification on the configuration group, randomly generating a parameter combination based on a parameter coverage range after the verification is passed, and adapting the parameter group to a verification environment through macro definition conversion to form a test task; and executing the test task in parallel, monitoring abnormity, and analyzing a test result to generate a test report and a debugging package. By defining the configuration group, layered management of the test scene is realized, and the parameter configuration flexibility is improved. And the verification coverage is improved by randomly generating the parameter combination. And the verification environment is quickly adapted through macro definition conversion, so that the configuration time consumption is shortened. By executing the test tasks in parallel, the test efficiency is improved while the resource utilization rate is improved. By generating the test report, the test condition is visually presented, and by generating the debugging package, a complete material capable of reproducing an abnormal scene is provided.

Description

Technical Field

[0001] This invention relates to the field of chip design verification, and in particular to a method, apparatus, device, and storage medium for testing randomized configuration parameters. Background Technology

[0002] In the field of chip design verification, regression testing is a core step in ensuring the correctness of chip functionality. As the complexity of chip design continues to increase, modern chips often integrate a large number of configurable hardware parameters such as data bus width, address space size, cache configuration, and clock frequency. Different combinations of parameters directly affect the chip's functional performance and characteristics, making the need for verification of multi-parameter combination scenarios increasingly urgent.

[0003] Currently, chip regression testing mainly adopts two approaches: one is regression testing based on fixed parameter configuration, where all test cases run in a unified preset hardware parameter environment, and the parameter values ​​are hardcoded in the verification environment through macro definitions or configuration files; the other is manual modification of parameters and recompilation, where testers manually adjust the hardware parameters according to verification requirements, then recompile the verification environment and execute the test.

[0004] Existing technologies lack randomness and diversity in parameter configuration, and cannot automatically generate boundary conditions and abnormal parameter combinations, resulting in limited verification coverage. Each parameter change requires recompiling the verification environment, significantly extending the verification cycle and reducing verification efficiency. Furthermore, they can only cover a limited set of preset parameter combinations, making it difficult to discover unknown design flaws. Summary of the Invention

[0005] This invention provides a randomized configuration parameter testing method, apparatus, device, and storage medium, which realizes automated random parameter generation, parallel verification, and result analysis, solving the technical problems of low parameter configuration efficiency, incomplete test coverage, and high anomaly debugging cost in traditional chip regression testing.

[0006] According to one aspect of the present invention, a method for testing randomized configuration parameters is provided, the method comprising:

[0007] The name and enabling status of each configuration group are defined at the top level of the configuration file. Each configuration group contains the number of random samples and the parameter coverage range.

[0008] The configuration group is validated for legality. After the validation is passed, a combination of parameters is randomly generated based on the parameter coverage range. The parameter group is then adapted to the validation environment through macro definition conversion to form a test task.

[0009] Execute test tasks in parallel and monitor for anomalies, analyze test results to generate test reports and debug packages.

[0010] Optionally, the configuration group can be validated for legality, including: performing a syntax check on the configuration group to verify the correctness of the syntax format and the completeness of the required fields; performing a type validation on the configuration group to confirm whether the data type and value range of each parameter value in the configuration group meet the definition requirements; and performing a rule validation on the configuration group to verify whether the technical constraint relationship between parameters meets the preset rules.

[0011] Optionally, the parameter combination is randomly generated based on the parameter coverage range, including: identifying the minimum, maximum and critical values ​​corresponding to the parameter coverage range through a boundary value sampling algorithm, and using them as boundary samples; using a uniform random sampling algorithm, uniformly sampling within the parameter coverage range using a pseudo-random number generator to obtain uniform samples, and using the boundary samples and uniform samples as the initial combination; and calling a constraint verification algorithm to verify the initial combination to determine the final parameter combination.

[0012] Optionally, a constraint verification algorithm is invoked to verify the initial combination in order to determine the final parameter combination. This includes: invoking the constraint verification algorithm to verify the technical constraint relationship between the parameters in the initial combination; when the verification passes, the initial combination is used as the parameter combination; when the verification fails, the boundary value sampling and uniform random sampling algorithms are restarted to generate a new initial combination, and the constraint verification algorithm is used to verify it again until a parameter combination that meets the technical constraints is obtained.

[0013] Optionally, the parameter group can be adapted to the verification environment through macro definition conversion to form a test task, including: parsing the parameter group to generate the parsed parameter group; converting the parsed parameter group into a macro definition format that the verification environment compiler can recognize to generate compiler options; allocating an independent compilation directory for the parameter group and storing the compiler options in the compilation directory; and encapsulating the compilation directory into a test task according to the verification execution requirements.

[0014] Optionally, test tasks are executed in parallel and anomalies are monitored, including: initializing each task execution pool according to the preset number of parallel tasks; distributing test tasks to each task execution pool in sequence for compilation and simulation testing, and storing the test results in the compilation directory; monitoring the running status of each test task in real time, and when an anomaly is detected, marking the test task as an abnormal task and recording the reason for failure, and releasing the task execution pool occupied by the abnormal task.

[0015] Optionally, the test results are analyzed to generate test reports and debug packages, including: extracting test results from the compilation directory, where the test results include the execution status of the test tasks, error codes, simulation time, and resource usage; establishing a mapping relationship between parameter combinations and corresponding test results, and statistically analyzing test metrics based on the test results of each parameter combination, where the test metrics include the total number of tests, the number of passes, the number of failures, and the pass rate; summarizing the mapping relationship and test metrics to generate a test report; and encapsulating abnormal tasks and their corresponding parameter combinations and random seed values ​​into debug packages.

[0016] According to another aspect of the present invention, a randomized configuration parameter testing apparatus is provided, the apparatus comprising:

[0017] The configuration group definition module is used to define the name and enabling status of each configuration group at the top level of the configuration file. Each configuration group contains the number of random samples and the parameter coverage range.

[0018] The test task generation module is used to verify the legality of the configuration group. After the verification is passed, it randomly generates parameter combinations based on the parameter coverage range. Through macro definition conversion, the parameter group is adapted to the verification environment to form a test task.

[0019] The test execution and report generation module is used to execute test tasks in parallel, monitor anomalies, and analyze test results to generate test reports and debug packages.

[0020] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0021] At least one processor;

[0022] and a memory communicatively connected to the at least one processor;

[0023] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform a randomized configuration parameter testing method according to any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement a randomized configuration parameter testing method according to any embodiment of the present invention.

[0025] The technical solution of this invention achieves hierarchical management of test scenarios and improves parameter configuration flexibility by defining configuration groups. Randomly generating parameter combinations improves verification coverage. Macro definition conversion quickly adapts to the verification environment, shortening configuration time. Parallel execution of test tasks improves resource utilization and test efficiency. Generating test reports provides a clear view of the test results, and generating debug packages provides complete materials for reproducing abnormal scenarios.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a randomized configuration parameter testing method provided in Embodiment 1 of the present invention;

[0029] Figure 2 This is a flowchart of another randomized configuration parameter testing method provided in Embodiment 1 of the present invention;

[0030] Figure 3 This is a flowchart of another randomized configuration parameter testing method provided in Embodiment 2 of the present invention;

[0031] Figure 4 This is a structural block diagram of a randomized configuration parameter testing system provided in Embodiment 2 of the present invention;

[0032] Figure 5 This is a schematic diagram of a parameter group matching verification environment provided in Embodiment 2 of the present invention;

[0033] Figure 6 This is a schematic diagram of the structure of a randomized configuration parameter testing device according to Embodiment 3 of the present invention;

[0034] Figure 7 This is a schematic diagram of the structure of an electronic device that implements a randomized configuration parameter testing method according to an embodiment of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Example 1

[0038] Figure 1 This is a flowchart of a randomized configuration parameter testing method provided in Embodiment 1 of the present invention. This embodiment is applicable to chip regression testing scenarios. The method can be executed by a randomized configuration parameter testing device, which can be implemented in hardware and / or software and can be configured in a computer controller. Figure 1 As shown, the method includes:

[0039] S110. Define the name and enabling status of each configuration group at the top level of the configuration file. Each configuration group contains the number of random samples and the parameter coverage range.

[0040] The configuration file refers to a structured JSON format parameter input file, which serves as the carrier for the test system to define the rules for the parameters to be tested, rather than being the chip's own configuration file. The top level contains the name and activation status of the configuration group. The activation status indicates whether the configuration group participates in the test. Each configuration group further defines the number of random samples and the parameter coverage. The number of random samples refers to the number of test parameter combinations to be generated, and the parameter coverage refers to the value rules of the hardware parameters to be tested. A configuration group is the organizational unit in the configuration file used to categorize and manage test parameter rules. It is a division of parameter sets under different test scenarios. Each group can define independent random sample quantities and parameter coverage for different hardware configuration scenarios of the chip, such as low-performance mode and high-performance mode. Configuration groups enable hierarchical management of complex parameter testing requirements, facilitating testers to enable tests for specific scenarios as needed. The number of random samples controls the breadth of test coverage, balancing test efficiency and verification completeness. Parameter coverage is categorized by parameter type: numerical parameters, enumerated parameters, and Boolean parameters.

[0041] S120. Perform legality verification on the configuration group. After the verification is passed, randomly generate parameter combinations based on the parameter coverage range. Use macro definition conversion to adapt the parameter group to the verification environment and form a test task.

[0042] The legality verification process refers to the three-stage compliance check performed by the test system on the parameter configurations within the configuration group. This ensures that the subsequently generated parameter combinations meet the basic requirements of chip design and avoids invalid tests. The three stages include: first-stage syntax checking, second-stage type checking, and third-stage rule checking. A parameter combination refers to a complete set of hardware parameter values ​​generated by the test system based on parameter coverage using a random sampling algorithm. Each set corresponds to a virtual hardware configuration of the chip. Macro definition conversion refers to the test system automatically converting parameter combinations into a macro definition format recognizable by the verification environment compiler, achieving adaptation between the parameter combinations and the verification environment. The verification environment is a standardized simulation environment used to perform chip regression testing; it is the execution carrier of the test tasks. A test task is an independent simulation unit formed by the test system encapsulating legal parameter combinations and the adapted verification environment. It includes macro definition options corresponding to the parameter combinations, an independent compilation directory, and simulation execution logic. During execution, an independent process is started to ensure environmental isolation between different tasks and avoid compilation result conflicts.

[0043] Figure 2 This invention provides a flowchart of a randomized configuration parameter testing method according to Embodiment 1. Step S120 mainly includes the following steps S121 to S125:

[0044] S121. Perform a validity check on the configuration group.

[0045] Optionally, the configuration group can be validated for legality, including: performing a syntax check on the configuration group to verify the correctness of the syntax format and the completeness of the required fields; performing a type validation on the configuration group to confirm whether the data type and value range of each parameter value in the configuration group meet the definition requirements; and performing a rule validation on the configuration group to verify whether the technical constraint relationship between parameters meets the preset rules.

[0046] Specifically, performing syntax checks on configuration groups verifies the format compliance and information completeness of the configuration files. Since JSON format is the storage medium for parameter configurations, syntax errors or missing key information will directly lead to subsequent parameter parsing failures. The system calls a JSON parser to scan the entire configuration file's syntax structure, checking for JSON syntax errors such as mismatched brackets, extra or missing commas, and unclosed quotation marks. For example, if a parameter definition contains "DATA_BITS:{range:[64,1024]}", lacking double quotes for the field name, it will be judged as a syntax error. Simultaneously, the system checks against a preset list of required fields to ensure each configuration group and its parameters contain key information. For instance, configuration groups must include a field for the number of random samples, numeric parameters must define their value range and step size, and enumeration parameters must define a list of possible values. For example, if a configuration group lacks a field for the number of random samples, or a numeric parameter does not define a step size, it will be judged as an incomplete required field, and the syntax check will fail. Type verification is based on the preset data types (numeric, enumerated, and boolean) configured during parameter configuration. It validates the actual value of each parameter to ensure it conforms to the basic attributes and value boundaries of that type, preventing compilation or simulation failures due to type errors. Rule verification verifies the technical constraints between parameters. Based on the chip design's hardware logic, including parameter dependencies and mutual exclusions between different hardware modules, a pre-defined technical rule base for parameter combinations is used. The system matches these rules against constraints to determine the technical feasibility of parameter combinations, avoiding invalid combinations that violate the chip's hardware logic. Rule verification succeeds when all parameter combinations satisfy the technical constraints in the rule base.

[0047] S122. After verification, the minimum, maximum and critical values ​​corresponding to the parameter coverage range are identified by the boundary value sampling algorithm and used as boundary samples.

[0048] It is known that chip hardware parameters are more prone to design flaws near boundary values, so it is necessary to prioritize sampling critical nodes at these boundaries. Specifically, the system first analyzes the value characteristics of each parameter within its coverage area, identifying the boundaries of each parameter's value range, including the minimum, maximum, and system-defined critical values. Then, according to the system-configured sampling quantity, it selects a corresponding number of parameter values ​​from the identified minimum, maximum, and critical values, combining them to form boundary samples. In the implementation, the algorithm assigns a higher sampling probability to boundary values ​​to ensure that boundary conditions are fully tested. For example, for the data bus width parameter, the algorithm will prioritize generating critical boundary values ​​such as 64-bit, 512-bit, and 1024-bit.

[0049] S123. Using a uniform random sampling algorithm, a pseudo-random number generator is used to uniformly sample within the parameter coverage range to obtain uniform samples, and the boundary samples and uniform samples are used as the initial combination.

[0050] The uniform random sampling algorithm ensures comprehensive parameter coverage, avoiding blind spots caused by testing only boundary scenarios. This algorithm utilizes a pseudo-random number generator, such as a pseudo-random number engine based on the Mason swirl algorithm, to perform unbiased sampling across the entire parameter value space, guaranteeing an equal probability of selection for each value interval. During sampling, the algorithm records the generated parameter combinations to avoid repeatedly sampling the same configuration. For enumerated parameters, the algorithm randomly selects from all available values ​​with equal probability.

[0051] S124. Call the constraint verification algorithm to verify the initial combination in order to determine the final parameter combination.

[0052] Optionally, a constraint verification algorithm is invoked to verify the initial combination in order to determine the final parameter combination. This includes: invoking the constraint verification algorithm to verify the technical constraint relationship between the parameters in the initial combination; when the verification passes, the initial combination is used as the parameter combination; when the verification fails, the boundary value sampling and uniform random sampling algorithms are restarted to generate a new initial combination, and the constraint verification algorithm is used to verify it again until a parameter combination that meets the technical constraints is obtained.

[0053] The constraint verification algorithm checks the parameter relationships of each initial combination by matching constraints from the rule base. This involves checking simple constraint relationships between parameters, such as mutual exclusion and numerical range restrictions. For parameter combinations that do not meet the constraints, the algorithm automatically regenerates them until a valid parameter configuration is obtained. For example, when the data bus width is set to 1024 bits, the algorithm ensures that the address bus width is also adjusted accordingly.

[0054] Specifically, passing the verification means that the parameter combination conforms to the chip hardware logic and has practical testing significance. At this point, the initial combination can be directly used as the parameter combination. When the verification fails, the initial combination is not simply discarded. Instead, the boundary value sampling and uniform random sampling algorithms are restarted synchronously to generate new initial combinations, and constraint verification is performed again until a combination that meets the technical constraints is obtained. Re-synchronizing and restarting ensures that the newly generated combination still takes into account both boundary scene coverage and comprehensive coverage. If only a single sampling algorithm is restarted, the new combination may only be concentrated at the boundary or only scattered in ordinary intervals, which cannot guarantee the integrity of the test coverage. Specifically, during the re-synchronization and restart, the system will trigger two sampling algorithms simultaneously according to the boundary sample and uniform sample ratio set in the original configuration group, and recombine the two sampling results to form a new initial combination. Through constraint verification, the number of generated parameter combinations can be guaranteed to meet the requirements of the number of random samples in the configuration group while ensuring the quality of test coverage. This avoids insufficient sample quantity due to the failure of verification of some combinations, which would affect the integrity of subsequent regression tests.

[0055] S125. The parameter group is adapted to the verification environment through macro definition conversion to form a test task.

[0056] Optionally, the parameter group can be adapted to the verification environment through macro definition conversion to form a test task, including: parsing the parameter group to generate the parsed parameter group; converting the parsed parameter group into a macro definition format that the verification environment compiler can recognize to generate compiler options; allocating an independent compilation directory for the parameter group and storing the compiler options in the compilation directory; and encapsulating the compilation directory into a test task according to the verification execution requirements.

[0057] Specifically, parsing parameter combinations breaks down structured parameter combinations into parameter value pairs that the verification environment can recognize one by one. It's important to note that the previously generated parameter combinations are based on a complete set of rules from the configuration file, but the verification environment requires a clear definition of each parameter for adaptation. Therefore, the system calls a dedicated parser to break down and combine parameters one by one according to their type, ensuring that each parameter accurately corresponds during subsequent conversion and avoiding omissions or confusion. Then, using the macro definition syntax supported by the compiler, the association between parameters and verification code is established. The Universal Verification Methodology (UVM) verification framework and Register Transfer Level (RTL) code in the verification environment both read external parameters through the ifdef conditional compilation directive. Since the compiler can only recognize macro definition inputs in a specific format, the system will batch convert the parsed parameter value pairs according to this syntax rule. For example, the parameter configuration {"DATA_BITS":256,"ADDR_BITS":36} is converted to the compilation option "+define+DATA_BITS=256+define+ADDR_BITS=36". All converted macro definitions will be integrated into a single compiler option, ensuring that the compiler can read all parameters at once during compilation and accurately configure the verification environment.

[0058] It is known that because the compilation processes corresponding to different parameter combinations differ, such as the generated intermediate files and log files, sharing the same directory could lead to file overwriting. For example, the intermediate files for combination A might be overwritten by those for combination B, causing subsequent simulation errors or inconsistent results. Therefore, the system creates a unique independent directory for each parameter combination and stores the compiler options generated in the previous step as text files in this directory. Simultaneously, intermediate files and log files generated during subsequent compilation are also stored uniformly in this directory. This facilitates tracing the compilation process corresponding to the parameters and completely isolates the test environments for different parameter combinations.

[0059] S130: Execute test tasks in parallel and monitor for anomalies, analyze test results to generate test reports and debug packages.

[0060] The test report is a structured report generated by the testing system after statistical analysis of the results of all test tasks. It visually presents the test results and problem distribution, including overall test statistics, failure configuration analysis, and performance data analysis. Overall test statistics include the total number of tests, the number of passes, the number of failures, and the pass rate. Failure configuration analysis refers to the list of failed parameter combinations and the classification of error messages. Performance data analysis refers to the comparison of simulation time for each parameter combination and the evaluation of resource utilization efficiency. The debug package is a collection of debug materials generated by the testing system for failed test tasks. It helps verification personnel quickly locate design defects, including complete parameter configuration information corresponding to the failed task, the random seed value used for parameter generation, and complete log files of the simulation process.

[0061] The technical solution of this invention achieves hierarchical management of test scenarios and improves parameter configuration flexibility by defining configuration groups. Randomly generating parameter combinations improves verification coverage. Macro definition conversion quickly adapts to the verification environment, shortening configuration time. Parallel execution of test tasks improves resource utilization and test efficiency. Generating test reports provides a clear view of the test results, and generating debug packages provides complete materials for reproducing abnormal scenarios.

[0062] Example 2

[0063] Figure 3 This is a flowchart of a randomized configuration parameter testing method provided in Embodiment 2 of the present invention. This embodiment adds a specific process to Embodiment 1, which involves parallel execution of test tasks, monitoring for anomalies, and analyzing test results to generate a test report and debug package. The specific content of steps S210-S220 is largely the same as steps S110-S120 in Embodiment 1, and therefore will not be repeated in this embodiment. Figure 3 As shown, the method includes:

[0064] S210. Define the name and enabling status of each configuration group at the top level of the configuration file. Each configuration group contains the number of random samples and the parameter coverage range.

[0065] S220. Perform legality verification on the configuration group. After the verification is passed, randomly generate parameter combinations based on the parameter coverage range. Use macro definition conversion to adapt the parameter group to the verification environment and form a test task.

[0066] Optionally, the configuration group can be validated for legality, including: performing a syntax check on the configuration group to verify the correctness of the syntax format and the completeness of the required fields; performing a type validation on the configuration group to confirm whether the data type and value range of each parameter value in the configuration group meet the definition requirements; and performing a rule validation on the configuration group to verify whether the technical constraint relationship between parameters meets the preset rules.

[0067] Optionally, the parameter combination is randomly generated based on the parameter coverage range, including: identifying the minimum, maximum and critical values ​​corresponding to the parameter coverage range through a boundary value sampling algorithm, and using them as boundary samples; using a uniform random sampling algorithm, uniformly sampling within the parameter coverage range using a pseudo-random number generator to obtain uniform samples, and using the boundary samples and uniform samples as the initial combination; and calling a constraint verification algorithm to verify the initial combination to determine the final parameter combination.

[0068] Optionally, a constraint verification algorithm is invoked to verify the initial combination in order to determine the final parameter combination. This includes: invoking the constraint verification algorithm to verify the technical constraint relationship between the parameters in the initial combination; when the verification passes, the initial combination is used as the parameter combination; when the verification fails, the boundary value sampling and uniform random sampling algorithms are restarted to generate a new initial combination, and the constraint verification algorithm is used to verify it again until a parameter combination that meets the technical constraints is obtained.

[0069] Optionally, the parameter group can be adapted to the verification environment through macro definition conversion to form a test task, including: parsing the parameter group to generate the parsed parameter group; converting the parsed parameter group into a macro definition format that the verification environment compiler can recognize to generate compiler options; allocating an independent compilation directory for the parameter group and storing the compiler options in the compilation directory; and encapsulating the compilation directory into a test task according to the verification execution requirements.

[0070] S230. Initialize each task execution pool according to the preset number of parallel tasks.

[0071] The preset number of parallel tasks can be configured by the user through the `max_jobs` parameter. During system initialization, a fixed-size task execution pool is created based on `max_jobs`. Each execution pool contains task slots, the number of which is equal to `max_jobs`. Each slot corresponds to one task resource that can be executed in parallel. After initialization, the execution pool is in a state of waiting to receive tasks, waiting for subsequent task distribution.

[0072] S240. Distribute the test tasks to each task execution pool in sequence for compilation and simulation testing, and store the test results in the compilation directory.

[0073] Specifically, the system organizes the queue of test tasks to be executed according to the parameter combination generation order. Then, the execution controller monitors the idle slots in the task execution pool in real time. When a slot becomes idle, a test task is immediately retrieved from the queue and assigned to that slot. After the task enters the slot, the compilation and simulation pipeline process is initiated. First, the Verilog Compiled Simulator (VCS) is invoked to read the macro-defined compiler options in the task's independent compilation directory, compile the UVM verification environment and RTL code, and generate an executable simulation file. After successful compilation, the simulation process is started directly, and test cases are executed. The test results generated during the simulation are written to the task's independent compilation directory in real time as log files.

[0074] S250: Monitor the running status of each test task in real time. When an abnormal situation is detected, mark the test task as an abnormal task, record the reason for failure, and release the task execution pool occupied by the abnormal task.

[0075] Specifically, the system's status monitor collects key metrics for each task in each task slot in real time, including process status, time metrics, and result feedback. Process status refers to whether the compilation or simulation process is running normally or has crashed. Time metrics refer to whether the compilation or simulation has timed out; for example, the preset compilation timeout is 30 minutes, and exceeding this timeout is considered an anomaly. Result feedback refers to the compiler returning compilation error codes or UVM reporting triggered errors during simulation. When any of the above anomalies are detected, the system immediately marks the task as an abnormal task and extracts the cause of failure using log parsing tools. For example, a compilation error might correspond to a macro definition format error, or a simulation timeout might correspond to a logical deadlock under a certain parameter combination. This information is then recorded in the abnormal task list. Simultaneously, the execution controller forcibly terminates the process corresponding to the abnormal task, releasing the task slot it occupies and making the slot idle again, ready to receive new test tasks.

[0076] S260. Extract test results from the compilation directory. The test results include the execution status of the test task, error codes, simulation time, and resource usage.

[0077] It is known that all critical information during the compilation and simulation process of the test task is stored in a dedicated compilation directory as log files, containing the core data needed by the verification personnel. The system will call the log parsing engine to extract the test results from the compilation directory.

[0078] The execution status is determined by the parsing engine identifying keywords in the UVM reporting mechanism, such as UVM_TEST_PASSED indicating a successful test and UVM_TEST_FAILED indicating a failed test or compiler return code. For example, a VCS compiler return 0 for successful compilation and a non-zero value for failure, thus determining the final task status. Error codes refer to the error identifiers and detailed error descriptions extracted from specified fields in the logs when a task fails. Simulation time refers to the time difference calculated by the system from the simulation start and end times recorded in the logs. Resource usage refers to the hardware resource data extracted by the system during the simulation, such as CPU utilization and memory consumption.

[0079] S270. Establish a mapping relationship between parameter combinations and corresponding test results, and statistically analyze test indicators based on the test results of each parameter combination. The test indicators include the total number of tests, the number of passes, the number of failures, and the pass rate.

[0080] Specifically, the system assigns a unique configuration identifier to each parameter combination. This identifier is also included in the naming or metadata files of each compilation directory. By matching these configuration identifiers, a precise mapping relationship can be established between parameter combinations and corresponding test results, ensuring that the parameter configuration corresponding to a specific result can be quickly located during subsequent analysis. Based on this, the system performs statistical calculations on all mapped data to generate test metrics, including total number of tests, number of passes, number of failures, and pass rate. Total number of tests refers to the total number of test tasks corresponding to all parameter combinations; number of passes refers to the number of tasks with a passed execution status; number of failures refers to the number of tasks with a failed execution status; and pass rate is the percentage of passed tasks out of the total number of test tasks.

[0081] S280: Summarize the mapping relationships and test metrics, and generate a test report.

[0082] The test report displays overall test metrics, including total number of tests, number of passes, number of failures, and pass rate, allowing verification personnel to quickly grasp the overall test performance. It also includes the mapping relationship between parameter combinations and test results, listing the specific values ​​of each parameter combination, the corresponding execution status, simulation time, and resource usage, supporting group-by-group traceability. Furthermore, it includes failure configuration analysis, filtering out all parameter combinations and error codes corresponding to failed tasks, categorizing and summarizing them by error type, and providing key performance data comparisons to help verification personnel identify whether certain parameter combinations are more prone to defects. The report is ultimately generated in a structured format, supporting both manual review and import into other analysis tools for further processing.

[0083] S290. Encapsulate the abnormal task, its corresponding parameter combination, and random seed value into a debug package.

[0084] It's important to note that debugging abnormal tasks requires verification personnel to reproduce the failure scenario. Parameter combinations and random seed values ​​are crucial for this reproduction. The parameter combinations determine the chip's hardware configuration environment, while the random seed value determines the test stimulus sequence generated by the UVM verification environment. That is, under the same parameter combination, different random seeds will produce different stimuli; only by using the same seed can the same failure scenario be reproduced. The system collects complete parameter combination information for each abnormal task, the random seed value used to generate the parameter combination, and the corresponding simulation log file. These materials are then packaged into a compressed file, forming a debug package. After obtaining the debug package, verification personnel can directly use the parameter combinations and random seed values ​​to restart the simulation, accurately reproduce the failure scenario, quickly locate defects in the chip design, and significantly improve debugging efficiency.

[0085] Specific application scenarios: Figure 4 This is a structural block diagram of a randomized configuration parameter testing system provided in Embodiment 2 of the present invention. Figure 4 Starting with the JSON parameter configuration file, the process first enters the parameter configuration management module: the parameter definition unit converts the requirements into JSON format, the constraint verification unit performs three-stage verification, the random generation unit samples valid parameter combinations, and these combinations are added to the parameter configuration queue. Next, the verification process execution module starts, the thread controller allocates tasks in parallel according to a preset number, the compiler manager converts the parameters into compiler-recognizable macro definitions, the simulation executor calls tools to compile and execute chip regression tests, and the results are correlated and added back to the queue. Finally, the result statistics and debugging support module works: the result statistician extracts test data, correlates parameters and results, and compiles metrics; the debugging support module generates a report, and simultaneously packages the parameters, seeds, and logs of abnormal tasks into a debug package. Figure 5This is a schematic diagram illustrating the process of adapting a parameter combination to a verification environment, as provided in Embodiment 2 of the present invention. The participants include: parameter configuration, a compiler manager, a UVM verification environment, and RTL code. First, parameter configuration parsing is performed. The compiler manager reads the parameter configuration, parses the parameters, and clarifies the name and value of each parameter. In the macro definition generation stage, the compiler manager converts the parsed parameters into macro definition options that the VCS compiler can recognize. Then, in the compilation execution stage, the compiler manager calls the VCS compiler, passes in the generated macro definition options, and starts the compilation process for the UVM verification environment and RTL code. Finally, in the environment configuration stage, the UVM verification environment and RTL code read the parameter values ​​in the macro definitions through ifdef conditional compilation instructions, complete their own parameterization configuration, and achieve adaptation of the parameter combination to the verification environment. In the compilation feedback stage, when compilation is successful, a success status is output, indicating that the parameter configuration has taken effect and the verification environment can start simulation testing. If compilation fails, an error message is returned, requiring readjustment of parameters or configuration. The entire process uses macro definitions as an intermediate carrier to achieve precise adaptation of parameter configuration to the verification environment, ensuring that different parameter combinations can correspond to different simulation test scenarios.

[0086] The technical solution of this invention provides a standardized resource allocation framework for parallel testing by initializing a task execution pool according to a preset number of parallel tasks, ensuring the orderly scheduling of resources; by distributing tasks sequentially and storing results in the compilation directory, it achieves independent execution of test tasks and dedicated storage of results, avoiding data confusion; by monitoring task status in real time and handling anomalies, it promptly releases resources occupied by abnormal tasks, ensuring a continuous and efficient testing process; by extracting multi-dimensional test results from the compilation directory, it provides comprehensive data support for subsequent analysis; by establishing a mapping relationship between parameter combinations and results and statistically analyzing test indicators, it clearly presents the correlation between parameters and test effects, facilitating quantitative evaluation of verification quality; by summarizing data to generate test reports, it intuitively displays the overall test situation, helping to quickly grasp the verification progress and results; and by encapsulating materials related to abnormal tasks to form a debugging package, it provides complete evidence for reproducing abnormal scenarios, reducing the difficulty of defect localization.

[0087] Example 3

[0088] Figure 6 This is a schematic diagram of a randomized configuration parameter testing device provided in Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a configuration group definition module 310, used to define the name and enable status of each configuration group at the top level of the configuration file, wherein each configuration group contains the number of random samples and the parameter coverage range;

[0089] The test task generation module 320 is used to verify the legality of the configuration group. After the verification is passed, it randomly generates parameter combinations based on the parameter coverage range and transforms the parameter groups to fit the verification environment through macro definition conversion to form a test task.

[0090] The test execution and report generation module 330 is used to execute test tasks in parallel, monitor anomalies, and analyze test results to generate test reports and debug packages.

[0091] Optionally, the test task generation module 320 specifically includes: a legality verification unit, used to: perform syntax checks on the configuration group to verify the correctness of the syntax format and the completeness of the required fields; perform type verification on the configuration group to confirm whether the data type and value range of each parameter value in the configuration group meet the definition requirements; and perform rule verification on the configuration group to verify whether the technical constraint relationship between parameters meets the preset rules.

[0092] Optionally, the test task generation module 320 specifically includes: a parameter combination generation unit, used to: identify the minimum, maximum and critical values ​​corresponding to the parameter coverage range through a boundary value sampling algorithm, and use them as boundary samples; use a uniform random sampling algorithm to uniformly sample within the parameter coverage range using a pseudo-random number generator to obtain uniform samples, and use the boundary samples and uniform samples as initial combinations; and call a constraint verification algorithm to verify the initial combinations to determine the final parameter combinations.

[0093] Optionally, the parameter combination generation unit specifically includes a constraint verification subunit, which is used to: call the constraint verification algorithm to verify the technical constraint relationship between parameters in the initial combination; when the verification passes, use the initial combination as the parameter combination; when the verification fails, resynchronize and start the boundary value sampling and uniform random sampling algorithm to generate a new initial combination, and verify it again through the constraint verification algorithm until a parameter combination that meets the technical constraints is obtained.

[0094] Optionally, the test task generation module 320 specifically includes: a macro definition conversion unit, used for: parsing parameter combinations to generate parsed parameter combinations; converting the parsed parameter combinations into a macro definition format that the verification environment compiler can recognize, generating compiler options; allocating an independent compilation directory for the parameter combinations, storing the compiler options in the compilation directory; and encapsulating the compilation directory into a test task according to the verification execution requirements.

[0095] Optionally, the test execution and report generation module 330 specifically includes: a test task execution unit, used to: initialize each task execution pool according to the preset number of parallel tasks; distribute test tasks to each task execution pool in sequence for compilation and simulation testing, and store the test results in the compilation directory; monitor the running status of each test task in real time, and when an abnormal situation is detected, mark the test task as an abnormal task and record the reason for failure, and release the task execution pool occupied by the abnormal task.

[0096] Optionally, the test execution and report generation module 330 specifically includes: a report generation unit, used for: extracting test results from the compilation directory, wherein the test results include the execution status of the test task, error codes, simulation time, and resource usage; establishing a mapping relationship between parameter combinations and corresponding test results, and statistically analyzing test metrics based on the test results of each parameter combination, wherein the test metrics include the total number of tests, the number of passes, the number of failures, and the pass rate; summarizing the mapping relationship and test metrics to generate a test report; and encapsulating abnormal tasks and their corresponding parameter combinations and random seed values ​​into a debug package.

[0097] The technical solution of this invention achieves hierarchical management of test scenarios and improves parameter configuration flexibility by defining configuration groups. Randomly generating parameter combinations improves verification coverage. Macro definition conversion quickly adapts to the verification environment, shortening configuration time. Parallel execution of test tasks improves resource utilization and test efficiency. Generating test reports provides a clear view of the test results, and generating debug packages provides complete materials for reproducing abnormal scenarios.

[0098] The randomized configuration parameter testing device provided in this embodiment of the invention can execute a randomized configuration parameter testing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0099] Example 4

[0100] Figure 7 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0101] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0102] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a randomized configuration parameter testing method.

[0104] In some embodiments, a randomized configuration parameter testing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the randomized configuration parameter testing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a randomized configuration parameter testing method by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0110] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0111] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for testing randomized configuration parameters, characterized in that, include: The name and enabling status of each configuration group are defined at the top level of the configuration file. Each configuration group contains the number of random samples and the parameter coverage range. The configuration group is validated for legality. After the validation is successful, a parameter combination is randomly generated based on the parameter coverage range. The parameter group is then adapted to the validation environment through macro definition conversion to form a test task. Execute test tasks in parallel and monitor for anomalies, analyze test results to generate test reports and debug packages.

2. The method according to claim 1, characterized in that, The validation of the configuration group includes: Perform a syntax check on the configuration group to verify the correctness of the syntax and the completeness of the required fields; Perform type validation on the configuration group to confirm whether the data type and value range of each parameter value in the configuration group meet the definition requirements; Perform rule validation on the configuration group to verify whether the technical constraints between parameters meet the preset rules.

3. The method according to claim 1, characterized in that, The random generation of parameter combinations based on the parameter coverage range includes: The minimum, maximum, and critical values ​​corresponding to the parameter coverage range are identified using a boundary value sampling algorithm and used as boundary samples. The uniform random sampling algorithm uses a pseudo-random number generator to uniformly sample within the parameter coverage range to obtain uniform samples, and the boundary samples and uniform samples are used as the initial combination. The constraint verification algorithm is invoked to verify the initial combination in order to determine the final parameter combination.

4. The method according to claim 3, characterized in that, The call constraint verification algorithm verifies the initial combination to determine the final parameter combination, including: Call the constraint verification algorithm to verify the technical constraint relationships between parameters in the initial combination; When the verification passes, the initial combination is used as the parameter combination; If the verification fails, the boundary value sampling and uniform random sampling algorithms are restarted to generate a new initial combination, and then the constraint verification algorithm is used to verify it again until a parameter combination that meets the technical constraints is obtained.

5. The method according to claim 1, characterized in that, The step of adapting the parameter group to the verification environment through macro definition transformation to form a test task includes: Parse the parameter combinations to generate parsed parameter combinations; The parsed parameter combinations are converted into a macro definition format that the verification environment compiler can recognize, and compiler options are generated. Allocate a separate compilation directory for the parameter combination and store the compiler options in the compilation directory; According to the verification execution requirements, the compilation directory is packaged into test tasks.

6. The method according to claim 5, characterized in that, The parallel execution of test tasks and monitoring of anomalies include: Initialize each task execution pool according to the preset number of parallel tasks; The test tasks are distributed to each task execution pool in sequence for compilation and simulation testing, and the test results are stored in the compilation directory. Monitor the running status of each test task in real time. When an abnormal situation is detected, mark the test task as an abnormal task, record the reason for failure, and release the task execution pool occupied by the abnormal task.

7. The method according to claim 6, characterized in that, The analysis of test results is used to generate test reports and debug packages, including: Extract test results from the compilation directory, wherein the test results include the execution status of the test task, error codes, simulation time, and resource usage; Establish a mapping relationship between parameter combinations and corresponding test results, and statistically analyze test indicators based on the test results of each parameter combination. The test indicators include the total number of tests, the number of passes, the number of failures, and the pass rate. Summarize the mapping relationships and the test metrics to generate a test report; The abnormal task, its corresponding parameter combination, and random seed value are encapsulated into a debug package.

8. A randomized configuration parameter testing device, characterized in that, include: The configuration group definition module is used to define the name and enabling status of each configuration group at the top level of the configuration file. Each configuration group contains the number of random samples and the parameter coverage range. The test task generation module is used to verify the legality of the configuration group. After the verification is successful, it randomly generates parameter combinations based on the parameter coverage range and uses macro definition conversion to adapt the parameter groups to the verification environment to form a test task. The test execution and report generation module is used to execute test tasks in parallel, monitor anomalies, and analyze test results to generate test reports and debug packages.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions that are used to cause a processor to execute the method of any one of claims 1-7.

Images