Testbed generation method, apparatus, device, storage medium, and program product
Through multi-language model collaborative processing, end-to-end automatic generation from hardware module design specifications to test bench files is achieved, solving the problem of low functional coverage of test bench files in existing technologies, and realizing automated testing and verification and efficient development of hardware modules.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHIWEI CHUANGXIN (NANJING) TECHNOLOGY CO LTD
- Filing Date
- 2024-12-12
- Publication Date
- 2026-06-19
AI Technical Summary
In existing technologies, hardware testing and verification processes rely on manually writing test bench files, resulting in low functional coverage of the generated test bench files and weak correlation between random stimuli and the functions that need to be tested in the hardware modules.
A multi-language model collaborative processing method is adopted. The first language model analyzes the design specifications of the hardware module and generates function points. The second language model generates test points. The third language model generates test bench files. The fourth language model performs correctness detection, realizing end-to-end automatic generation from hardware module design specifications to test bench files.
It significantly improved the functional coverage of testbench files, enabled automated testing and verification of hardware modules, reduced design-verification iteration costs, shortened the verification cycle, and improved development efficiency.
Smart Images
Figure CN122240454A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of hardware testing technology, and in particular to test bench generation methods, apparatus, equipment, storage media, and program products. Background Technology
[0002] With the rapid development of data-intensive applications, the complexity of hardware designs such as chips is increasing, posing a significant challenge to traditional, manual hardware testing and verification processes. In these processes, testbench files are typically written manually to test and verify the design code of hardware modules. The rapid advancement of artificial intelligence technology has provided new opportunities for the automation of hardware testing and verification. The automatic generation of testbench files is a current research hotspot. However, existing technologies mostly focus on the generation of random stimuli, which results in a weak correlation between the generated test stimuli and the functions that need to be tested in the hardware module, leading to low functional coverage of the generated testbench files. Summary of the Invention
[0003] This application provides a test bench generation method, apparatus, device, storage medium, and program product, which can improve the functional coverage of generated test bench files.
[0004] On one hand, embodiments of this application provide a test bench generation method, the method including:
[0005] Obtain the design specifications of the target hardware module to be tested;
[0006] Using the first language model, the functions that need to be tested in the target hardware module are analyzed based on the design specifications of the target hardware module, and at least one functional point is obtained;
[0007] Using a second language model, at least one test point is generated for each function point, and the generated test points are summarized to obtain a test plan. Each test point for a function point is used to describe a test method for the function described by the corresponding function point.
[0008] Using a third language model, test bench generation is performed based on each test point in the test plan to obtain test bench files.
[0009] The testbench files generated by the third language model are processed for correctness detection using the fourth language model.
[0010] If the test bench file generated by the third language model is found to be correct, the test bench file generated by the third language model will be used as the target test bench file for testing the target hardware module.
[0011] Otherwise, based on the detection feedback information obtained from the correctness detection process, the testbench is generated again through the third language model to obtain a new testbench file.
[0012] On one hand, embodiments of this application provide a test bench generation apparatus, the apparatus comprising:
[0013] The acquisition unit is used to acquire the design specifications of the target hardware module to be tested.
[0014] The processing unit is used to analyze the functions that need to be tested in the target hardware module based on the design specifications of the target hardware module using a first language model, and obtain at least one functional point.
[0015] The processing unit is also used to generate at least one test point corresponding to each function point based on the second language model, and to summarize the generated test points to obtain a test plan; wherein, a test point corresponding to a function point is used to describe: a test method for the function described by the corresponding function point;
[0016] The processing unit is also used to generate test bench files based on each test point in the test plan using a third language model.
[0017] The processing unit is also used to perform correctness detection processing on the testbench file generated by the third language model through the fourth language model;
[0018] The processing unit is also used to use the test bench file generated by the third language model as the target test bench file for testing the target hardware module, provided that the test bench file generated by the third language model is correct.
[0019] Otherwise, the processing unit is also used to generate a new testbench file by using the testbench generation process again through the third language model based on the detection feedback information obtained from the correctness detection process.
[0020] On one hand, embodiments of this application provide a computer device, which includes a communication interface and further includes:
[0021] Processor and computer-readable storage medium;
[0022] A computer-readable storage medium for storing computer programs;
[0023] The processor is used to run computer programs to implement the above-described test bench generation method.
[0024] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described test bench generation method.
[0025] On one hand, embodiments of this application provide a computer program product, which includes a computer program adapted to be loaded by a processor and executed by the above-described test bench generation method.
[0026] In this embodiment, a first language model is used to analyze the functions to be tested in the target hardware module based on the design specifications of the target hardware module, obtaining at least one functional point. A second language model is used to generate at least one test point corresponding to each functional point, and the generated test points are summarized to obtain a test plan. A third language model is used to generate a test bench based on each test point in the test plan to obtain a test bench file. A fourth language model is used to perform correctness detection on the test bench file generated by the third language model. If the test bench file generated by the third language model is found to be correct, it is used as the target test bench file for testing the target hardware module. Otherwise, based on the detection feedback information obtained from the correctness detection, the test bench is generated again using the third language model to obtain a new test bench file. This application employs a task divide-and-conquer strategy, dividing the testbench generation task into multiple stages and decomposing complex tasks into multiple manageable atomic tasks. Furthermore, a Mixture of Experts (MoE) architecture is designed and applied in the hardware verification field, assigning each language model to tasks at different stages. This achieves end-to-end automatic generation from hardware module design specifications to testbench files, simplifying and refining complex design requirements. Moreover, the first language model analyzes and extracts the functional points to be tested from the design specifications of the target hardware module, generating testbench files based on these extracted functional points. Functional point mapping ensures that each functional point accurately corresponds to a specific test case, significantly improving the functional coverage of the generated testbench files. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0028] Figure 1 This is a flowchart illustrating a test bench generation method provided in an embodiment of this application;
[0029] Figure 2 This is a flowchart illustrating another test bench generation method provided in an embodiment of this application;
[0030] Figure 3 This is a flowchart illustrating another test bench generation method provided in an embodiment of this application;
[0031] Figure 4 This is a schematic diagram of the structure of a test bench generation device provided in an embodiment of this application;
[0032] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this application or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. Unless otherwise stated, the relative arrangement, numerical expressions, and values of components and steps set forth in these embodiments do not limit the scope of this disclosure. At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar symbols and letters in the following drawings denote similar items, and therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings. Furthermore, in the description of the embodiments of this application, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. Therefore, features defined with "first" or "second" may explicitly or implicitly include one or more features.
[0034] Artificial intelligence (AI) is a field of technology that studies, develops, and applies theories, methods, techniques, and application systems to simulate, extend, and expand human intelligence. It attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI technologies can include computer vision (CV), speech technology, natural language processing (NLP), and machine learning. The major directions of artificial intelligence include machine learning (ML) and deep learning. Machine learning is the core of artificial intelligence, while deep learning is a technique that utilizes deep neural network systems for machine learning. Machine learning and deep learning techniques can include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. Pre-trained models (PTMs, also known as large models or foundational models) developed based on deep learning are trained on massive amounts of unlabeled data. After fine-tuning, pre-trained models can be widely applied to downstream tasks in various areas of artificial intelligence. Pre-trained models can be categorized according to the data modality they process: language models (e.g., GPT, BERT), visual models (e.g., V-MOE), speech models (e.g., VALL-E, ViT), and multimodal models (e.g., ViBERT, Flamingo, Gato). Multimodal models refer to models that establish feature representations for two or more data modalities. Fine-tuning techniques for pre-trained models can include supervised fine-tuning (SFT), instruction fine-tuning, and parameter-efficient fine-tuning (PEFT). Technologies such as […]. Among them, Large Language Models (LLMs) are an important tool in the field of natural language processing. They can generate text relevant to a given input. These models are pre-trained on large-scale corpora and fine-tuned according to human preferences, learning rich linguistic and world knowledge. Through generalization ability, they can be transferred to a wide range of tasks. Currently, large language models can be used for various tasks, such as text generation, question answering systems, coding, and translation.
[0035] This application provides a testbench generation method based on artificial intelligence technology. It can achieve end-to-end automatic generation from hardware module design specifications to testbench files by collaboratively processing multiple stages of the testbench generation task using multiple language models. A first language model can analyze and extract the functional points to be tested from the design specifications of the hardware module under test, and generate testbench files based on these extracted functional points. Function point mapping ensures that each functional point accurately corresponds to a specific test case, thereby improving the functional coverage of the generated testbench files. Optionally, this testbench generation method can be executed by a testbench generation device, which can be a terminal device or a server. Terminal devices can include, but are not limited to, mobile phones, computers, smart wearable devices, smart vehicle devices, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, big data, and artificial intelligence platforms, etc. Optionally, this testbench generation method can also be collaboratively executed by multiple electronic devices with computing power. For ease of explanation, subsequent embodiments will be described as being executed by a testbench generation device.
[0036] Please see Figure 1 , Figure 1 This is a flowchart illustrating a test bench generation method provided in an embodiment of this application. The test bench generation method can be executed by a test bench generation device and may include at least the following steps S101 to S107:
[0037] S101, Obtain the design specifications of the target hardware module to be tested.
[0038] Design specifications are documents that describe the functional and performance requirements of hardware such as chips, helping developers understand these requirements for chip design. Typically, design specifications are described in natural language and may include, but are not limited to, design name, function, interface, and PPA parameter information. PPA parameter information includes performance, power consumption, and area. PPA parameter information is a key indicator for evaluating and optimizing chip design and plays a crucial role in design decisions, especially in high-performance computing, mobile devices, and other power-sensitive applications.
[0039] S102, using the first language model, analyzes the functions that need to be tested in the target hardware module based on the design specifications of the target hardware module, and obtains at least one functional point.
[0040] A function point can be used to describe a function that needs to be tested in the target hardware module, and may include, but is not limited to, one or more of the following information: the name of the function described by the function point (i.e., the function point name), function description, input and output, critical information, and attribute information. In one feasible implementation, the first language model can be a large language model used to perform the task of generating function points from design specifications (referred to as the first task for ease of explanation). The first language model can be selected or trained according to specific needs, and this application embodiment does not impose any restrictions. For example, the first language model can be guided to perform the first task by design prompts. For instance, the first language model can be guided to understand the design specifications of the target hardware module and analyze the functions that need to be tested in order to generate at least one function point. The prompts can be designed according to requirements, and this application embodiment does not impose any restrictions. In one optional implementation, Retrieval-Augmented Generation (RAG) can be performed based on the dataset corresponding to the first task to improve the accuracy of the generated function points. The dataset corresponding to the first task may include the design specifications of the hardware module for reference and the corresponding function points of the hardware module. The test bench generation device can retrieve design specifications similar to the design specifications of the target hardware module from the dataset corresponding to the first task, and input the retrieved design specifications and corresponding function points as examples into the first language model to guide the first language model to generate at least one function point based on the design specifications of the target hardware module by referring to the design specifications and corresponding function points in the examples.
[0041] S103, using the second language model, generate at least one test point corresponding to each function point, and summarize the generated test points to obtain a test plan.
[0042] In this context, a test point corresponding to a function point describes a test method for the function described by that function point. Taking any function point as an example, when the test bench generation device generates at least one test point corresponding to the function point based on the function point using a second language model, it can generate at least one candidate test point according to the function point mapping, and generate descriptive information for each candidate test point to describe the test method of the corresponding function, thus obtaining the corresponding test point. For example, each candidate test point can be described with name, category, excitation signal, value range, etc., to generate descriptive information to constitute the test point. For example, for binary signals, two values, 0 and 1, can be generated; for register signals, values in the range of 0 to 2^width-1 can be generated, and so on. For ease of explanation, the task of generating test points from function points is called the second task. The second language model can be a large language model used to execute the second task. The second language model can be selected or trained according to specific needs, and this application embodiment does not impose any restrictions. For example, a prompt word can be designed to guide the second language model to execute the second task. The prompt word can be designed according to needs, and this application embodiment does not impose any restrictions. In one alternative implementation, Retrieval Enhancement Generation (RAG) can be performed based on the dataset corresponding to the second task to improve the accuracy of the generated test points; the dataset corresponding to the second task may include the function points of the hardware modules used for reference and the test points corresponding to the corresponding function points; the process of performing retrieval enhancement generation based on the dataset corresponding to the second task is similar to the process of performing retrieval enhancement generation based on the dataset corresponding to the first task described above, and will not be described in detail here.
[0043] In one feasible implementation, there can be multiple second language models, each corresponding to a function type. Each second language model is used to handle the second task of generating test points based on function points within the corresponding function type. This fine-grained distinction plays a crucial role in ensuring the model outputs expected data while controlling model illusion, formalization, and content. Based on this, taking any function point as an example, when the test bench generation device generates at least one test point corresponding to that function point using the second language model, it can classify the function point to obtain its function type, and then generate at least one test point corresponding to that function point using the second language model corresponding to that function type. That is, it generates at least one candidate test point based on the function point mapping, and generates descriptive information for each candidate test point to describe the testing method for the corresponding function, thus obtaining the corresponding test point. In one alternative implementation, a second language model corresponding to a function type can be learned from sample data under that function type. The sample data may include at least one function point under that function type and the corresponding test points. For example, the second language model corresponding to that function type may be obtained by fine-tuning the base model from the sample data under that function type, or it may be obtained by pre-inputting the sample data under that function type as an example into the base model to guide the base model to learn, and so on.
[0044] In one alternative implementation, the test plan can be output to support manual correction.
[0045] S104 uses a third language model to generate a test bench based on each test point in the test plan, resulting in a test bench file.
[0046] A testbench file is a file used to test whether the design code (including register transfer level (RTL) code) of a hardware module meets the functional and performance requirements specified in the hardware module's design specifications. It can include a series of test cases, input stimuli, and monitoring mechanisms. A test case is a specific test unit used to verify the functionality and performance of the hardware module's design code. It can contain a specific set of input conditions, expected output results, and execution steps. Each test case aims to check the performance of the design code in a specific scenario, ensuring that it meets the functional and performance requirements specified in the design specifications. In one feasible implementation, the testbench generation device uses a third-language model to generate the testbench file based on each test point in the test plan. When the testbench file is obtained, the third-language model can be used to instantiate test units (i.e., generate test cases), test signals, and test programs based on each test point (and / or the design code of the target hardware module) in the test plan to obtain the testbench file.
[0047] In one feasible implementation, the third language model can be a large language model used to perform the testbench generation task. The third language model can be selected or trained according to specific needs, and this application embodiment does not impose any restrictions. For example, the third language model can be obtained by fine-tuning the base model in specified sample data. The base model can be selected according to specific needs, such as Llama3 70B, GPT4, etc. The specified sample data may include the design code of the hardware module used for fine-tuning, the test plan of the corresponding hardware module, and the testbench file. Alternatively, the third language model can be guided to perform the testbench generation task by designing prompts. The prompts can be designed according to needs, and this application embodiment does not impose any restrictions. In one optional implementation, Retrieval Enhanced Generation (RAG) can be performed based on the dataset corresponding to the test bench generation task to improve the accuracy of the generated test bench file. The dataset corresponding to the test bench generation task may include the design code of the hardware module for reference, as well as the test plan and test bench file of the corresponding hardware module. The test bench generation device can retrieve the test bench file as an example from the dataset corresponding to the test bench generation task as a test bench template, and input the test bench template as an example into the third language model to guide the third language model to generate the test bench based on each test point (and / or the design code of the target hardware module) in the test plan, referring to the test bench template in the example. Optionally, the test bench template can be obtained based on the similarity retrieval between the design code (and / or the test plan of the target hardware module) and the corresponding data in the dataset corresponding to the test bench generation task, which will not be elaborated in this embodiment.
[0048] S105 uses the fourth language model to perform correctness checks on the testbench files generated by the third language model.
[0049] In one feasible implementation, the testbench generation device uses a fourth language model to perform correctness checks on the testbench files generated by the third language model. This can include, but is not limited to, analyzing whether the content is compliant (to prevent model illusion), whether the syntax is correct, whether the code is complete, and whether the test cases conform to the design specification description. The fourth language model can be a large language model used to perform the testbench checking task. The fourth language model can be selected or trained according to specific needs, and this application embodiment does not impose any limitations. For example, prompts can be designed to guide the fourth language model in performing the testbench checking task. These prompts can be designed according to needs, and this application embodiment does not impose any limitations. Further optionally, the testbench generation device can combine a hardware design language specification check dataset for retrieval to complete the correctness checks on the testbench files generated by the third language model in terms of syntax, format, completeness, and usability. The corresponding dataset can include, but is not limited to, the following data: hardware description language standards, correct testbench file (Testbench) examples, and incorrect example templates, for rapid matching and checking by the third language model.
[0050] S106, if the test bench file generated by the third language model is correct, use the test bench file generated by the third language model as the target test bench file for testing the target hardware module.
[0051] S107, otherwise, based on the detection feedback information obtained from the correctness detection process, the test bench is generated again through the third language model to obtain a new test bench file.
[0052] To further improve the coverage of the generated testbench file, this application embodiment also provides a coverage awareness mechanism for analyzing the coverage of the target testbench file and adding test points to generate new test cases to improve coverage if the coverage is insufficient. Based on this, in one feasible implementation, after obtaining the target testbench file for testing the target hardware module, the testbench generation device can further: obtain the design code of the target hardware module; perform simulation testing on the design code of the target hardware module using the target testbench file to obtain a test report; perform coverage analysis processing based on the test report to obtain a coverage analysis report corresponding to the target testbench file; if the coverage analysis report corresponding to the target testbench file indicates that the preset coverage target is met, determine that the testing of the design code of the target hardware module has ended; otherwise, update the test plan based on the coverage analysis report corresponding to the target testbench file to generate a new target testbench file. Optionally, when the testbench generation device updates the test plan based on the coverage analysis report corresponding to the target testbench file, it can be implemented using a fifth language model; please refer to [link to relevant documentation]. Figure 2This is a flowchart illustrating another test bench generation method provided in this application embodiment; the coverage analysis report corresponding to the target test bench file is used to indicate: after testing the design code of the target hardware module based on the target test bench file, the untested code segments and function points of the target hardware module; when the test bench generation device updates the test plan based on the coverage analysis report corresponding to the target test bench file, it can: through the fifth language model, based on the untested code segments and function points of the target hardware module indicated by the coverage analysis report corresponding to the target test bench file, generate new test points, and add the new test points to the test plan to update the test plan.
[0053] In one feasible implementation, the test bench generation device uses the target test bench file to simulate and test the design code of the target hardware module. When a test report is obtained, it can be achieved by calling a test toolchain (an RTL toolchain when the design code is RTL code). The test toolchain may include a compiler and a simulator. The compiler supports compiling the design code to detect the syntax correctness, and the simulator supports simulating the design code to detect the functional correctness and test coverage. Existing tools can be selected for the compiler and simulator as needed, and this application embodiment does not impose any restrictions.
[0054] In one feasible implementation, when the test bench generation device performs coverage analysis processing based on the test report to obtain the coverage analysis report corresponding to the target test bench file, it can first extract the coverage information contained in the test report, and then analyze the extracted coverage information to obtain the coverage analysis report. Optionally, the extraction and processing of coverage information can rely on the coverage indicators required by the preset coverage target. The preset coverage target can be set according to specific needs. For example, in an application scenario, it can be set that when the functional coverage, flip coverage, and line coverage indicators all reach a specified value (e.g., the default value is 95%), the preset coverage target is met. Then, the extraction and analysis of coverage information can include the extraction and analysis of relevant coverage indicators. Optionally, the process of generating a coverage analysis report based on the test report can be implemented by one or more processing modules, such as... Figure 2 As shown, coverage information can be extracted using the coverage extraction module, and the extracted coverage information can be analyzed using the coverage analysis module.
[0055] In one feasible implementation, the test report output by the test toolchain can be input into the coverage extraction module. The coverage extraction module can parse the test report, extract coverage information reflecting the quality of the test bench files, and save the extracted coverage information as a structured data format. This structured coverage information is then input into the coverage analysis module. The coverage analysis module can analyze and process this structured coverage information, including calculating the average, highest, and lowest values of each extracted coverage indicator, analyzing coverage differences between different modules or functional points, identifying key modules or functional points with low coverage, evaluating the quality level of the target test bench files based on preset coverage targets, and generating a coverage analysis report. The generated coverage analysis report may include, but is not limited to, the following: a summary of statistics for each coverage indicator, coverage analysis results at the module / functional point level, and key areas with coverage bottlenecks (including untested code segments and functional points, i.e., uncovered code segments and functional points).
[0056] In one feasible implementation, when the coverage analysis report corresponding to the target test bench file indicates that the preset coverage target is met, the test bench generation device determines that the test of the design code of the target hardware module is completed, and determines that the target test bench file obtained under this condition meets the preset coverage target. That is, the task of generating the test bench file for testing the target hardware module and the verification task of verifying (i.e. testing) the design code of the target hardware module based on the test bench file are both completed and terminated.
[0057] In one feasible implementation, the test bench generation device generates new test points based on the coverage analysis report corresponding to the target test bench file, indicating the untested code segments and functional points of the target hardware module, using a fifth language model. During the process of adding these new test points to the test plan to update the test plan, the fifth language model can be a large language model used to perform this task. The fifth language model can be selected or trained according to specific needs, and this application embodiment does not impose any limitations. For example, prompts can be designed to guide the fifth language model to perform this task.
[0058] In one feasible implementation, augmentation generation (RAG) can be performed based on the augmented dataset to improve the accuracy of the generated test points. Based on this, using the fifth language model, and according to the coverage analysis report corresponding to the target testbench file, new test points are generated for the untested code segments and functional points of the target hardware module. When adding these new test points to the test plan to update the test plan, the following can be done: An augmented dataset can be obtained; wherein the augmented dataset includes at least one augmented data pair, each augmented data pair including initial data and optimized data. The initial data includes an initial test plan, an initial testbench file, and initial coverage; the optimized data includes an optimized test plan, an optimized testbench file, and optimized coverage obtained based on the initial test plan, the initial testbench file, and the initial coverage. The test plan, target testbench file, and corresponding coverage analysis report indicate the coverage rate to construct target data, and determine the feature similarity between the target data and each initial data in the augmented dataset. Based on the feature similarity between the target data and each initial data in the augmented dataset, a target number of augmented data pairs are determined from the augmented dataset. The feature similarity between the target data and the initial data in each determined augmented data pair is greater than or equal to the feature similarity between the target data and the remaining initial data in the augmented dataset. Using the fifth language model, based on the untested code segments and functionalities of the target hardware module indicated in the coverage analysis report corresponding to the target testbench file, and referencing the target number of augmented data pairs, new test points are generated and added to the test plan to update the test plan. The initial coverage and optimized coverage can include the coverage metrics required in the preset coverage target; the target number can be set according to specific needs; the target number of augmented data pairs can be used as examples input to the fifth language model to guide it in generating new test points based on the target number of augmented data pairs. It is known that, in order to facilitate data retrieval, the data contained in the datasets corresponding to various tasks in this application (such as augmented datasets, datasets corresponding to the first task, etc.) can all be stored in vector form.
[0059] As described above, the test bench generation method proposed in this application can achieve end-to-end generation from the design specifications of the target hardware module to be tested to the test bench file. Then, the generated test bench file can be used to test and verify the design code of the target hardware module to automate hardware verification. This allows for the automatic discovery and correction of design defects in the early stages of design, reducing the costly rework costs in the later stages, lowering the design-verification iteration costs, shortening the verification cycle, improving development efficiency, and reducing labor costs.
[0060] This application also provides a coverage extraction and analysis mechanism, and through the fifth language model, adds test points and generates new test cases to improve coverage when coverage is insufficient. The iterative update mechanism based on the test plan ensures the coverage of the test bench files generated based on the design specifications. Furthermore, in various tasks, by introducing RAG and fine-tuning techniques, the corresponding datasets can be updated using data generated during the iteration process implemented in this solution (e.g., test plans with insufficient coverage and test plans with sufficient coverage after optimization during iteration). This helps the language model for the corresponding task learn and utilize specific knowledge and datasets, continuously improving the language model's capabilities and efficiency in test generation.
[0061] In one feasible implementation, the design specifications of the target hardware module are described in natural language. The design specifications described in natural language can be converted into a machine / model-friendly, standardized description to improve the first language model's understanding of the target hardware module's design specifications, thereby improving the accuracy of function point generation and further enhancing the accuracy of the generated Testbench. Based on this, in step S102, when the testbench generation device analyzes the functions to be tested of the target hardware module based on the design specifications of the target hardware module using the first language model and obtains at least one function point, it can: extract and parse information from the design specifications of the target hardware module to obtain the parsed design specifications; perform format conversion processing on the parsed design specifications to obtain the verification specifications described in the verification language; and analyze the functions to be tested of the target hardware module based on the verification specifications described in the verification language using the first language model to obtain at least one function point. In one optional implementation, the test bench generation device can further: acquire the design code of the target hardware module; perform semantic alignment processing between the parsed design specification and the design code of the target hardware module to obtain an intermediate design specification; based on this, when performing format conversion processing on the parsed design specification to obtain a verification specification described in a verification language, this can be done by: performing format conversion processing on the intermediate design specification to obtain a verification specification described in a verification language, so as to supplement the semantic information contained in the design code of the target hardware module and further improve the accuracy of function point generation. Here, the Verification Description Language (VDL) is a Domain Specific Language (DSL) for verification task development, used to parse and map natural language design specifications into structured, machine-friendly descriptions; it is a JSON-like language.
[0062] Optionally, the process of converting the design specifications of the target hardware module into a verification language description can be achieved by extracting a parsed language model; please refer to [link to relevant documentation]. Figure 3This is a flowchart illustrating another testbench generation method provided in this application embodiment. The method includes: an extraction and parsing language model responsible for converting natural language design specifications into VDL format to enhance model readability; a first language model generating function points based on VDL format specifications to ensure comprehensive coverage of design requirements; a second language model generating test points based on function points and summarizing them into a test plan; a third language model generating testbench files based on the test plan; a fourth language model checking the syntax and logic correctness of the generated testbench files; a test toolchain (RTL toolchain, including compiler and simulator) used for testing and verification and outputting test reports indicating the coverage of the testbench files; a coverage extraction module and a coverage analysis module extracted and analyzed the coverage of the testbench files to obtain a coverage analysis report; and a fifth language model optimizing the test plan based on the coverage analysis report to ensure the coverage of the generated testbench files. The extraction and parsing language model can be a large language model used to perform the extraction and parsing task, and can be selected or trained according to specific needs; this application embodiment does not impose any limitations. For example, a design prompt can be used to guide the extraction and parsing language model to perform the corresponding task. Optionally, at least one functional point obtained by analyzing the functions to be tested of the target hardware module through the verification specification described in the verification language, using the first language model, can also be organized based on the verification language. Optionally, functional points organized based on the verification language can be added to the verification specification described in the verification language for the generation of test points. Further optionally, test points and test plans generated based on functional points can also be organized based on the verification language. For example, the design specification described in the verification language can be organized into a specific format, which can define the overall structure of the target hardware module, including the module name, input / output interfaces, parameters, and the definitions of state machines, combinational logic, and sequential logic. Among them, the state definition can represent a state in the state machine, including the state name and the transition conditions to other states; the combinational logic definition can define a combinational logic block, including the logic name, input signals, output signals, and expressions describing the logic; the sequential logic definition can define a sequential logic block, including the logic name, clock signal, reset signal, and a series of assignment statements describing the sequential logic.
[0063] In one feasible implementation, the task of converting the design specifications of a target hardware module into verification specifications described in a verification language can be improved by performing Retrieval Enhanced Generation (RAG) based on the dataset corresponding to the task. This dataset can include the design specifications of the hardware module for reference and the corresponding verification specifications based on the verification language description. Similarly, the task of generating function points from the verification specifications described in the verification language description of the target hardware module can be improved by performing Retrieval Enhanced Generation (RAG) based on the dataset corresponding to the task. This dataset can include the verification specifications described in the verification language description of the hardware module (i.e., the VDL format verification specifications) for reference and the corresponding function points of the hardware module. The RAG process is similar to the RAG process described above and will not be elaborated upon here.
[0064] In this embodiment, a first language model is used to analyze the functions to be tested in the target hardware module based on the design specifications of the target hardware module, obtaining at least one functional point. A second language model is used to generate at least one test point corresponding to each functional point, and the generated test points are summarized to obtain a test plan. A third language model is used to generate a test bench based on each test point in the test plan to obtain a test bench file. A fourth language model is used to perform correctness detection on the test bench file generated by the third language model. If the test bench file generated by the third language model is found to be correct, it is used as the target test bench file for testing the target hardware module. Otherwise, based on the detection feedback information obtained from the correctness detection, the test bench is generated again using the third language model to obtain a new test bench file. This application employs a task divide-and-conquer strategy, dividing the testbench generation task into multiple stages and decomposing complex tasks into multiple manageable atomic tasks. Furthermore, a Mixture of Experts (MoE) architecture is designed and applied in the hardware verification field, assigning each language model to tasks at different stages. This achieves end-to-end automatic generation from hardware module design specifications to testbench files, simplifying and refining complex design requirements. Moreover, the first language model analyzes and extracts the functional points to be tested from the design specifications of the target hardware module, generating testbench files based on these extracted functional points. Functional point mapping ensures that each functional point accurately corresponds to a specific test case, significantly improving the functional coverage of the generated testbench files.
[0065] Based on the above description, this application also discloses a test bench generation apparatus; the test bench generation apparatus may be a computer program running in a computer device, and the computer device may be a device that executes the above-described test bench generation method, such as the above-described test bench generation device. The test bench generation apparatus can execute... Figure 1 , Figure 2 or Figure 3 The steps in the illustrated method flow are shown below. Please refer to [link / reference]. Figure 4 This is a schematic diagram of a test bench generation device provided in an embodiment of this application. The test bench generation device may include an acquisition unit 401 and a processing unit 402, wherein:
[0066] Acquisition unit 401 is used to acquire the design specifications of the target hardware module to be tested;
[0067] Processing unit 402 is used to analyze the functions to be tested of the target hardware module based on the design specifications of the target hardware module using a first language model, and obtain at least one functional point.
[0068] The processing unit 402 is further configured to generate at least one test point corresponding to each function point based on each function point using a second language model, and to summarize the generated test points to obtain a test plan; wherein, a test point corresponding to a function point is used to describe: a test method for the function described by the corresponding function point;
[0069] The processing unit 402 is further configured to perform test bench generation processing based on each test point in the test scheme using a third language model to obtain a test bench file;
[0070] The processing unit 402 is further configured to perform correctness detection processing on the test bench file generated by the third language model through the fourth language model;
[0071] The processing unit 402 is further configured to, if the test bench file generated by the third language model is found to be correct, use the test bench file generated by the third language model as the target test bench file for testing the target hardware module;
[0072] Otherwise, the processing unit 402 is also used to perform test bench generation processing again through the third language model based on the detection feedback information obtained from the correctness detection processing to obtain a new test bench file.
[0073] In one implementation, the design specifications of the target hardware module are described using natural language;
[0074] Processing unit 402 is used to analyze the functions to be tested of the target hardware module based on the design specifications of the target hardware module using a first language model. When at least one functional point is obtained, it can be used for:
[0075] The design specifications of the target hardware module are extracted and parsed to obtain the parsed design specifications.
[0076] The parsed design specifications are converted into a format to obtain a verification specification described in a verification language.
[0077] Using the first language model, the functions to be tested of the target hardware module are analyzed based on the verification specifications described by the verification language, and at least one functional point is obtained.
[0078] In one embodiment, the processing unit 402 is further configured to:
[0079] Obtain the design code of the target hardware module;
[0080] The parsed design specifications are semantically aligned with the design code of the target hardware module to obtain intermediate design specifications.
[0081] The step of converting the parsed design specifications into a format to obtain a verification specification described in a verification language includes:
[0082] The intermediate design specification is converted into a format to obtain a verification specification described in a verification language.
[0083] In one embodiment, the processing unit 402 is further configured to:
[0084] Obtain the design code of the target hardware module;
[0085] The design code of the target hardware module is simulated and tested using the target test bench file, and a test report is obtained.
[0086] Based on the test report, a coverage analysis process is performed to obtain the coverage analysis report corresponding to the target test bench file;
[0087] If the coverage analysis report corresponding to the target test bench file indicates that the preset coverage target is met, the test of the design code of the target hardware module is considered complete.
[0088] Otherwise, the test plan is updated based on the coverage analysis report corresponding to the target testbench file to generate a new target testbench file.
[0089] In one implementation, the coverage analysis report corresponding to the target testbench file is used to indicate: after testing the design code of the target hardware module based on the target testbench file, the target hardware module has corresponding untested code segments and function points;
[0090] When processing unit 402 updates the test plan based on the coverage analysis report corresponding to the target test bench file, it can be used for:
[0091] Using the fifth language model, based on the coverage analysis report corresponding to the target test bench file, new test points are generated for the untested code segments and functional points of the target hardware module, and these new test points are added to the test plan to update the test plan.
[0092] In one implementation, the processing unit 402 is configured to generate new test points based on the coverage analysis report corresponding to the target testbench file, indicating the untested code segments and function points of the target hardware module, using a fifth language model, and add the new test points to the test plan to update the test plan. This can be done by:
[0093] Obtain an augmented dataset; wherein the augmented dataset includes at least one augmented data pair, an augmented data pair includes an initial data and an optimized data, an initial data includes an initial test plan, an initial test bench file and an initial coverage, and an optimized data includes an optimized test plan, an optimized test bench file and an optimized coverage obtained based on the initial test plan, the initial test bench file and the initial coverage;
[0094] Based on the test plan, the target test bench file, and the coverage indicated by the corresponding coverage analysis report, target data is constructed, and the feature similarity between the target data and each initial data in the augmented dataset is determined.
[0095] Based on the feature similarity between the target data and each initial data in the augmented dataset, a target number of augmented data pairs are determined from the augmented dataset; wherein, the feature similarity between the target data and the initial data in each determined augmented data pair is greater than or equal to the feature similarity between the target data and the remaining other initial data in the augmented dataset;
[0096] Using the fifth language model, based on the coverage analysis report corresponding to the target test bench file, the untested code segments and function points of the target hardware module, and the enhanced data pairs with reference to the target number, new test points are generated and added to the test plan to update the test plan.
[0097] In this embodiment, a first language model is used to analyze the functions to be tested in the target hardware module based on the design specifications of the target hardware module, obtaining at least one functional point. A second language model is used to generate at least one test point corresponding to each functional point, and the generated test points are summarized to obtain a test plan. A third language model is used to generate a test bench based on each test point in the test plan to obtain a test bench file. A fourth language model is used to perform correctness detection on the test bench file generated by the third language model. If the test bench file generated by the third language model is found to be correct, it is used as the target test bench file for testing the target hardware module. Otherwise, based on the detection feedback information obtained from the correctness detection, the test bench is generated again using the third language model to obtain a new test bench file. This application employs a task divide-and-conquer strategy, dividing the testbench generation task into multiple stages and decomposing complex tasks into multiple manageable atomic tasks. Furthermore, a Mixture of Experts (MoE) architecture is designed and applied in the hardware verification field, assigning each language model to tasks at different stages. This achieves end-to-end automatic generation from hardware module design specifications to testbench files, simplifying and refining complex design requirements. Moreover, the first language model analyzes and extracts the functional points to be tested from the design specifications of the target hardware module, generating testbench files based on these extracted functional points. Functional point mapping ensures that each functional point accurately corresponds to a specific test case, significantly improving the functional coverage of the generated testbench files.
[0098] Based on the descriptions of the above method and apparatus embodiments, this application also provides a computer device that can be used as the above-described test bench generation device. Please refer to... Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device in this embodiment includes a processor 501, a computer-readable storage medium 502, and a communication interface 503. The processor 501, the computer-readable storage medium 502, and the communication interface 503 can exchange data, and the processor 501 implements the test bench generation method proposed in this embodiment.
[0099] This application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store computer programs and data. The stored computer programs are suitable for being loaded and executed by a processor using the test bench generation method proposed in this application. The computer-readable storage medium 502 may include one or more of volatile memory and non-volatile memory. For example, volatile memory may include random-access memory (RAM), and non-volatile memory may include flash memory, solid-state drive (SSD), etc.
[0100] Processor 501 can be a central processing unit (CPU); processor 501 can also be a combination of a CPU and a graphics processing unit (GPU).
[0101] The communication interface 503 can be used to implement data input and output, and may include, for example, a display screen, microphone or speaker.
[0102] In one embodiment, a processor may load and execute a computer program stored in a computer-readable storage medium to perform the aforementioned tasks. Figure 1 , Figure 2 or Figure 3 The corresponding steps in the method embodiments shown; in specific implementations, the computer program in the computer-readable storage medium can be loaded and executed by a processor as follows:
[0103] Obtain the design specifications of the target hardware module to be tested;
[0104] Using a first language model, the functions that need to be tested in the target hardware module are analyzed based on the design specifications of the target hardware module, and at least one functional point is obtained;
[0105] Using a second language model, at least one test point is generated for each function point, and the generated test points are summarized to obtain a test plan. Each test point for a function point is used to describe a test method for the function described by the corresponding function point.
[0106] Using a third language model, a test bench is generated based on each test point in the test plan to obtain a test bench file.
[0107] The test bench file generated by the third language model is processed for correctness detection using the fourth language model.
[0108] If the test bench file generated by the third language model is found to be correct, the test bench file generated by the third language model shall be used as the target test bench file for testing the target hardware module.
[0109] Otherwise, based on the detection feedback information obtained from the correctness detection process, the testbench is generated again through the third language model to obtain a new testbench file.
[0110] In one implementation, the design specifications of the target hardware module are described using natural language;
[0111] When the processor 501 analyzes the functions to be tested of the target hardware module based on the design specifications of the target hardware module using a first language model and obtains at least one functional point, it can be used for:
[0112] The design specifications of the target hardware module are extracted and parsed to obtain the parsed design specifications.
[0113] The parsed design specifications are converted into a format to obtain a verification specification described in a verification language.
[0114] Using the first language model, the functions to be tested of the target hardware module are analyzed based on the verification specifications described by the verification language, and at least one functional point is obtained.
[0115] In one implementation, the processor 501 is further configured to:
[0116] Obtain the design code of the target hardware module;
[0117] The parsed design specifications are semantically aligned with the design code of the target hardware module to obtain intermediate design specifications.
[0118] The step of converting the parsed design specifications into a format to obtain a verification specification described in a verification language includes:
[0119] The intermediate design specification is converted into a format to obtain a verification specification described in a verification language.
[0120] In one implementation, the processor 501 is further configured to:
[0121] Obtain the design code of the target hardware module;
[0122] The design code of the target hardware module is simulated and tested using the target test bench file, and a test report is obtained.
[0123] Based on the test report, a coverage analysis process is performed to obtain the coverage analysis report corresponding to the target test bench file;
[0124] If the coverage analysis report corresponding to the target test bench file indicates that the preset coverage target is met, the test of the design code of the target hardware module is considered complete.
[0125] Otherwise, the test plan is updated based on the coverage analysis report corresponding to the target testbench file to generate a new target testbench file.
[0126] In one implementation, the coverage analysis report corresponding to the target testbench file is used to indicate: after testing the design code of the target hardware module based on the target testbench file, the target hardware module has corresponding untested code segments and function points;
[0127] When processor 501 updates the test plan based on the coverage analysis report corresponding to the target test bench file, it can be used for:
[0128] Using the fifth language model, based on the coverage analysis report corresponding to the target test bench file, new test points are generated for the untested code segments and functional points of the target hardware module, and these new test points are added to the test plan to update the test plan.
[0129] In one implementation, the processor 501 is configured to generate new test points based on the coverage analysis report corresponding to the target testbench file, indicating the untested code segments and function points of the target hardware module, using a fifth language model, and add the new test points to the test plan to update the test plan. This can be used for:
[0130] Obtain an augmented dataset; wherein the augmented dataset includes at least one augmented data pair, an augmented data pair includes an initial data and an optimized data, an initial data includes an initial test plan, an initial test bench file and an initial coverage, and an optimized data includes an optimized test plan, an optimized test bench file and an optimized coverage obtained based on the initial test plan, the initial test bench file and the initial coverage;
[0131] Based on the test plan, the target test bench file, and the coverage indicated by the corresponding coverage analysis report, target data is constructed, and the feature similarity between the target data and each initial data in the augmented dataset is determined.
[0132] Based on the feature similarity between the target data and each initial data in the augmented dataset, a target number of augmented data pairs are determined from the augmented dataset; wherein, the feature similarity between the target data and the initial data in each determined augmented data pair is greater than or equal to the feature similarity between the target data and the remaining other initial data in the augmented dataset;
[0133] Using the fifth language model, based on the coverage analysis report corresponding to the target test bench file, the untested code segments and function points of the target hardware module, and the enhanced data pairs with reference to the target number, new test points are generated and added to the test plan to update the test plan.
[0134] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned actions. Figure 1 , Figure 2 or Figure 3 The method embodiment shown.
[0135] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the above method embodiments.
[0136] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the processes of the above embodiments and making equivalent changes in accordance with the claims of this application still fall within the scope of this invention.
Claims
1. A test board generation method characterized by, include: Obtain the design specifications of the target hardware module to be tested; Using a first language model, the functions that need to be tested in the target hardware module are analyzed based on the design specifications of the target hardware module, and at least one functional point is obtained; Using a second language model, at least one test point is generated for each function point, and the generated test points are summarized to obtain a test plan. Each test point for a function point is used to describe a test method for the function described by the corresponding function point. Using a third language model, a test bench is generated based on each test point in the test plan to obtain a test bench file. The test bench file generated by the third language model is processed for correctness detection using the fourth language model. If the test bench file generated by the third language model is found to be correct, the test bench file generated by the third language model shall be used as the target test bench file for testing the target hardware module. Otherwise, based on the detection feedback information obtained from the correctness detection process, the testbench is generated again through the third language model to obtain a new testbench file.
2. The method as described in claim 1, characterized in that, The design specifications for the target hardware module are described using natural language. The process involves analyzing the functions that need to be tested in the target hardware module based on the design specifications of the target hardware module using a first language model, resulting in at least one functional point, including: The design specifications of the target hardware module are extracted and parsed to obtain the parsed design specifications. The parsed design specifications are converted into a format to obtain a verification specification described in a verification language. Using the first language model, the functions to be tested of the target hardware module are analyzed based on the verification specifications described by the verification language, and at least one functional point is obtained.
3. The method as described in claim 2, characterized in that, The method further includes: Obtain the design code of the target hardware module; The parsed design specifications are semantically aligned with the design code of the target hardware module to obtain intermediate design specifications. The step of converting the parsed design specifications into a format to obtain a verification specification described in a verification language includes: The intermediate design specification is converted into a format to obtain a verification specification described in a verification language.
4. The method as described in claim 1, characterized in that, The method further includes: Obtain the design code of the target hardware module; The design code of the target hardware module is simulated and tested using the target test bench file, and a test report is obtained. Based on the test report, a coverage analysis process is performed to obtain the coverage analysis report corresponding to the target test bench file; If the coverage analysis report corresponding to the target test bench file indicates that the preset coverage target is met, the test of the design code of the target hardware module is considered complete. Otherwise, the test plan is updated based on the coverage analysis report corresponding to the target testbench file to generate a new target testbench file.
5. The method as described in claim 4, characterized in that, The coverage analysis report corresponding to the target testbench file is used to indicate: after testing the design code of the target hardware module based on the target testbench file, the untested code segments and functional points of the target hardware module are identified. The step of updating the test plan based on the coverage analysis report corresponding to the target testbench file includes: Using the fifth language model, based on the coverage analysis report corresponding to the target test bench file, new test points are generated for the untested code segments and functional points of the target hardware module, and these new test points are added to the test plan to update the test plan.
6. The method as described in claim 5, characterized in that, The step of generating new test points based on the coverage analysis report corresponding to the target testbench file, using the fifth language model, and generating new test points for the untested code segments and functionalities of the target hardware module, and adding these new test points to the test plan to update the test plan, includes: Obtain an augmented dataset; wherein the augmented dataset includes at least one augmented data pair, an augmented data pair includes an initial data and an optimized data, an initial data includes an initial test plan, an initial test bench file and an initial coverage, and an optimized data includes an optimized test plan, an optimized test bench file and an optimized coverage obtained based on the initial test plan, the initial test bench file and the initial coverage; Based on the test plan, the target test bench file, and the coverage indicated by the corresponding coverage analysis report, target data is constructed, and the feature similarity between the target data and each initial data in the augmented dataset is determined. Based on the feature similarity between the target data and each initial data in the augmented dataset, a target number of augmented data pairs are determined from the augmented dataset; wherein, the feature similarity between the target data and the initial data in each determined augmented data pair is greater than or equal to the feature similarity between the target data and the remaining other initial data in the augmented dataset; Using the fifth language model, based on the coverage analysis report corresponding to the target test bench file, the untested code segments and function points of the target hardware module, and the enhanced data pairs with reference to the target number, new test points are generated and added to the test plan to update the test plan.
7. A test bench generation device, characterized in that, include: The acquisition unit is used to acquire the design specifications of the target hardware module to be tested. The processing unit is used to analyze the functions to be tested of the target hardware module based on the design specifications of the target hardware module using a first language model, and obtain at least one functional point. The processing unit is further configured to generate at least one test point corresponding to each function point based on each function point using a second language model, and to summarize the generated test points to obtain a test plan; wherein, a test point corresponding to a function point is used to describe: a test method for the function described by the corresponding function point; The processing unit is also used to generate a test bench based on each test point in the test scheme using a third language model, and obtain a test bench file. The processing unit is also used to perform correctness detection processing on the test bench file generated by the third language model through the fourth language model; The processing unit is further configured to, if the test bench file generated by the third language model is found to be correct, use the test bench file generated by the third language model as the target test bench file for testing the target hardware module; Otherwise, the processing unit is also used to perform testbench generation processing again through the third language model based on the detection feedback information obtained from the correctness detection processing to obtain a new testbench file.
8. A computer device, comprising a communication interface, characterized in that, Also includes: Processor and computer-readable storage medium; The computer-readable storage medium is used to store computer programs; The processor is used to run the computer program to implement the test bench generation method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-6.
10. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-6.