Brain-like processor test case generation method based on large language model

Abstract

The invention discloses a brain-like processor test case generation method based on a large language model (LLM), which comprises the following steps of: S1, extracting key information for processor verification from a chip manual in an LLM-based test case generation part; after the inputs are received, a prompt generator constructs various prompts and generates a series of test cases; s2, performing LLM-assisted test case optimization on the generated series of test cases; s3, entering a test result feedback stage, and compiling and simulating the test case to generate a coverage result; if the coverage rate does not reach the standard, Prompt updating is carried out based on a coverage rate report result, new input is provided for a prompt generator, and a new round of prompt is constructed; s4, a prompt library is constructed, the prompt library comprises test case features, and different test cases are generated through LLM to perform processor verification. According to the method, complexity and limitation of manual rule writing are avoided, the open source prompt library is introduced, the test generation process is standardized, and the efficiency and accuracy of brain-like chip testing are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of chip design verification, and mainly to a method for identifying potential defects before chip tape-out, and specifically to a method for generating test cases for a brain-like processor based on a large language model. Background Art

[0002] With the rapid advancement of technology, the scale of integrated circuit chips has grown from a few hundred transistors to tens of billions or even hundreds of billions. The expansion of chip scale, the increase in design complexity and the increase in integration require more and more manpower and time to be invested in the verification process. According to a 2022 study, processor verification usually accounts for more than 40% of the entire development cycle.

[0003] The simulation-based verification method is the most widely used verification method. Its main steps include generating test stimuli (programs), executing them in the design under test and the reference model, and comparing the execution results. In recent years, many verification methods have been proposed, such as constrained random simulation, coverage-guided mutation fuzz testing, and directed gray-box fuzz testing. These methods have made significant progress in the challenges of processor verification. However, the complexity of writing and maintaining these verification constraints places extremely high demands on engineers. Therefore, developing verification methods that are easier to use has become an important research direction.

[0004] Traditional manual test case writing is not only time-consuming and error-prone, even for experienced verification engineers. Previous work has attempted to reduce the burden through rule-guided random test case generation, but it still requires manual writing of rules for detailed hardware behavior descriptions.

[0005] In order to more efficiently generate test cases for processor verification and reduce manual work, a large language model (LLM) may be an alternative method. LLM has demonstrated excellent capabilities in many fields. The technical problems to be solved by the present invention are: 1. How to effectively extract the core information of the design file in a short text so that LLM can accurately understand and generate high-quality test cases is the key to improving code coverage and verification efficiency, and is also one of the important problems to be solved by the present invention. 2. How to improve the correctness of the test cases generated by LLM during compilation and simulation, and reduce the workload of manual error checking and modification, is the key to solving the problem of LLM generating test cases. 3. How to design and organize prompts so that LLM can efficiently generate high-quality, high-coverage test cases is the main challenge to improving test generation efficiency. This problem is directly related to resource utilization efficiency and the effect of processor verification, and is one of the important problems that the present invention needs to solve. Summary of the invention

[0006] In view of the problems existing in the prior art, the purpose of this invention is to explore the potential of large language models (LLMs) to capture hardware behaviors and build test case generation prompts based on these behaviors with the help of the advancement of LLMs. The focus is on solving how to automatically identify potential design defects in brain-like chips before tape-out, and reduce the errors and time consumption of manual test case creation.

[0007] To achieve the above object, the present invention provides a method for generating test cases for a brain-like processor based on a large language model, comprising the following steps:

[0008] S1. In the LLM-based test case generation part, key information for processor verification is extracted from the chip manual and used as input to the hint generator; after receiving these inputs, the hint generator constructs various hints, which are used as input to the LLM to generate a series of test cases;

[0009] S2. performing LLM-assisted test case optimization on a series of generated test cases to improve these test cases;

[0010] S3. Enter the test result feedback phase. In this phase, the test cases are compiled and simulated to generate coverage results. If the coverage rate does not meet the requirements, the prompt is updated based on the coverage report results to provide new input for the prompt generator and build a new round of prompts.

[0011] S4. Build a hint library, which contains test case features, and generate different test cases through LLM to perform processor verification.

[0012] Further, in step S1, the workflow of test case generation based on LLM includes the following steps:

[0013] S1.1 Rule-based prompting: Establish a set of rule constraints for all test cases generated by LLM to ensure that the generated test cases meet specific standards;

[0014] S1.2 Hint-based test case generation: Based on specific requirements, guide LLM to generate different test cases to meet these requirements.

[0015] Further, in step S1.1, first, the verification rules are extracted from the chip manual, including the compilation tool chain and simulation environment information; based on the above information, the instruction specifications of the test cases in the current compilation and simulation environment are analyzed; at the same time, the architecture information of the processor is obtained to set the task requirements; finally, this information is integrated and sent to the LLM.

[0016] Further, in step S1.2, prompt-based includes instruction-based prompt and module-based prompt.

[0017] Further, based on the prompts of the instructions, the test cases are generated. The specific steps are as follows:

[0018] S1.2.1A Extract the syntax and operation of each RISC-V instruction from the chip manual;

[0019] S1.2.2A organizes this information into prompts and sends them to the LLM;

[0020] S1.2.3A LLM generates inline assembly C language test cases containing relevant instructions based on these prompts.

[0021] Further, generate test cases based on the module prompts. The specific steps are as follows:

[0022] S1.2.1B extract the architecture information of each module from the chip manual;

[0023] S1.2.2B allows the LLM to analyze this information to identify test case features that can verify the module;

[0024] S1.2.3B sends these test case features as hints to LLM so that it generates corresponding test cases.

[0025] Further, in step S2, the process of LLM-assisted test case optimization starts with the initial prompt of generating the test case; then, the generated test case is compiled, and if the compilation fails, the error information in the compilation log is fed back to LLM for necessary modifications; if the error in the same position causes the compilation to fail three times, the instruction format is fed back for correction; after the compilation is successful, the test case is simulated.

[0026] Further, in step S3, the test case generates a set of binary files after compilation and deassembles them into RISC-V assembly instructions; after identifying the RISC-V instructions not included in the feedback file, these instructions are fed back to the LLM to prompt it to generate new test cases containing these instructions; this process is repeated until the coverage exceeds the set threshold.

[0027] Further, in step S4, the construction process of the hint library starts with module-based hints, by providing the LLM with the module's architectural information, allowing the LLM to generate test case features that can verify the module; these features are then provided to the LLM again as hints to generate test cases with these features, and the test cases are then compiled and simulated to obtain coverage reports, which are used to evaluate the effectiveness of these features; if a feature is considered valid, it will be added to the hint library.

[0028] Furthermore, a greedy algorithm is used to evaluate the effectiveness of features.

[0029] Further, initially, when the hint library is empty, the current feature is added to the library and its coverage report is used as a baseline; each new report is merged with the existing report; if the coverage is improved, it means that the newly generated test case has improved the overall coverage report, and the corresponding test case feature will be added to the hint library, and the merged report will become the new coverage baseline; this process is repeated continuously to continuously update the hint library and coverage report.

[0030] Beneficial effects:

[0031] The present invention provides an efficient, accurate and resource-optimized processor verification method, which uses a large language model (LLM) to generate and optimize test cases. Compared with traditional testing methods, this method avoids the tediousness and limitations of manually writing rules, and improves the efficiency and accuracy of brain-like chip testing by introducing an open source prompt library and standardizing the test generation process.

[0032] This method has the following characteristics:

[0033] 1. Efficient test case generation: By developing a dedicated prompt library and prompt framework, LLM is used to generate high-coverage test cases, effectively reducing the workload of verification engineers.

[0034] 2. Accurate test case optimization: After generating test cases, LLM-assisted test case optimization ensures that the generated test cases can be successfully compiled and simulated, improving the reliability of the verification process.

[0035] 3. Automated result feedback: Through the result feedback mechanism, the generated test cases are continuously improved to continuously increase their coverage, thus achieving comprehensive verification of the processor design.

[0036] 4. Optimized hint library construction: A greedy algorithm is used to evaluate and update the hint library to ensure that the test case characteristics in the hint library can maximize the coverage of various functional modules of the processor, thereby improving verification efficiency.

[0037] This innovative verification method not only improves the generation efficiency and coverage of test cases, but also significantly optimizes the overall process of processor verification. Through the technology proposed by this invention, the verification of processor design becomes more efficient, accurate and reliable, providing important technical support for the future field of processor design verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a general architecture diagram of the method for generating test cases for brain-like processors based on a large language model of the present invention;

[0039] Figure 2 It is a test case generation flowchart based on LLM;

[0040] Figure 3 It is a schematic diagram of an example of test case generation based on LLM;

[0041] Figure 4 It is a LLM-assisted test case optimization flow chart;

[0042] Figure 5 It is a test result feedback flow chart;

[0043] Figure 6 It is a flowchart for building a prompt library;

[0044] Figure 7 is a flowchart of the greedy algorithm for building the hint library;

[0045] Figure 8 This is a schematic diagram of generating test cases for brain-like processors. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0048] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] The following combination Figure 1-Figure 8 The specific embodiments of the present invention are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0050] As an advanced processor that simulates brain functions, the complexity of brain-like chips requires more efficient and accurate verification methods. This invention proposes a prompt generation framework based on a large language model (LLM), which uses LLM to deeply understand the key hardware behavior characteristics of neural networks and parallel processing in brain-like chips, automatically generates test cases adapted to the brain-like chip architecture, and ensures comprehensive coverage testing of the processor.

[0051] Figure 1 The overall framework of the test case generation method for brain-inspired processors based on large language models is presented, which usually includes four parts: LLM-based test case generation, LLM-assisted test case optimization, test result feedback, and building a prompt library.

[0052] The workflow is as follows:

[0053] S1. In the LLM-based test case generation part, key information for processor verification is extracted from the chip manual and used as input to the hint generator. After receiving these inputs, the hint generator constructs various hints, which are used as input to the LLM to generate a series of test cases.

[0054] S2. Perform LLM-assisted test case optimization on the generated series of test cases to improve these test cases.

[0055] S3. Enter the test result feedback phase, in which the test cases are compiled and simulated to generate coverage results. If the coverage rate does not meet the standard, the prompt is updated based on the coverage report results to provide new input for the prompt generator and build a new round of prompts.

[0056] S4. Build a hint library. In S1-S3, a large number of test cases will be generated based on the hints and coverage results will be obtained. In this stage, the hints that can improve the overall code coverage in S1-S3 will be collected to build a hint library, which will be used when generating test cases for other processors in the future to improve verification efficiency. The hint library contains a wide range of test case features and can generate different test cases through LLM for processor verification.

[0057] In addition, in order to improve the efficiency of LLM in generating test cases, the present invention compiles representative hints in the hint generator into a hint library, aiming to improve the efficiency of processor verification.

[0058] Specifically, in step S1, in the test case generation stage based on LLM, the main research task of the present invention is to design prompts for generating test cases to achieve high coverage. Figure 2 As shown, the workflow of test case generation based on LLM in step S1 is divided into two steps:

[0059] S1.1 Rule-based prompts: Establish a set of rule constraints for all test cases generated by LLM to ensure that the generated test cases meet specific standards; formulate rule constraints by analyzing the compilation and simulation rules in the verification tool chain to ensure that the generated test cases can pass compilation and simulation more often. Specifically as follows: First, extract the verification rule set from the chip manual. The verification rule set mainly includes relevant information about the compilation tool chain and the simulation environment, and analyze this information to obtain instruction specifications. The purpose of this operation is to improve the compilation and simulation pass rate of the test cases generated by LLM. Propose task requirements for the processor that currently needs to be verified, and combine the task requirements and instruction specifications to form rule-based prompts and send them to LLM, so that the test cases subsequently generated by LLM can meet the rules.

[0060] S1.2 Test generation based on prompts: guide LLM to generate different test cases according to specific requirements to meet these requirements. In the present invention, the specific requirements include two categories, the first category is to generate test cases for verification based on the usage of instructions, and the second category is to generate test cases for verification based on the specific architecture of the module to be verified. The purpose of this step is to generate test cases. There are two strategies for generating test cases. The first is to generate test cases based on instructions, and the second is to generate test cases based on modules. The method of generating test cases based on instructions is as follows: first, the syntax and operation of the instructions are extracted from the chip manual, and combined into a prompt based on the instruction, and the prompt is sent to the LLM to generate a test case containing the instruction. The method of generating test cases based on modules is as follows: first, the architecture information of the module to be verified is extracted from the instruction manual, and combined into a prompt-1 based on the module, which is sent to the LLM to generate features of the test case that can verify the module, and then these features are listed one by one as a prompt-2 based on the module, and sent to the LLM so that it has a test case with these features for verification of the module.

[0061] In order to obtain a higher compilation and simulation success rate, the present invention establishes rules for the test cases generated by the LLM before prompting the LLM. Step S1.1 (i.e. Figure 2 Step 1) in shows the method of constructing rule-based prompts and their contents. First, the verification rules are extracted from the chip manual, including the compilation tool chain and simulation environment information; based on the compilation tool chain and simulation environment information, the instruction specifications of the test cases in the current compilation and simulation environment can be analyzed. At the same time, the architecture information of the processor is obtained to set the task requirements. Finally, this information is integrated and sent to the LLM. The integrated information includes two parts, the first part is the instruction specifications of the test cases in the current compilation and simulation environment, and the second part is to obtain the architecture information of the processor to set the task requirements.

[0062] Step S1.2 shows the steps of using LLM to generate test cases for RISC-V processors. Hint-based includes instruction-based hints and module-based hints.

[0063] First, the instruction-based hint is described in detail. This method aims to cover all RISC-V instructions to ensure that each instruction is verified. The specific steps are as follows:

[0064] S1.2.1A Extract the syntax and operation of each RISC-V instruction from the chip manual.

[0065] S1.2.2A organizes this information into prompts and sends them to the LLM. Prompts are prompts sent to the LLM. Here, prompts refer to Figure 2 Instruction-based prompts in .

[0066] S1.2.3ALLM generates inline assembly C language test cases containing corresponding instructions based on these prompts.

[0067] Next, we will explain module-based hints in detail. Modern processors are composed of various specific functional modules, such as integer units responsible for integer calculations. In order to systematically verify specific modules within the processor, the specific steps of module-based hints to generate test cases are as follows:

[0068] S1.2.1B Extract the architecture information of each module from the chip manual.

[0069] S1.2.2B allows the LLM to analyze this information to identify test case characteristics that can verify the module.

[0070] S1.2.3B sends these test case features as hints to LLM, causing it to generate C language test cases with these features.

[0071] Figure 2 Step 2 in demonstrates the process of generating test cases based on module hints. The characteristics of the test cases generated by LLM can be used as a hint library for verifying general processors. By providing these hints to LLM, it can generate more efficient test cases for processor verification.

[0072] Figure 3 The implementation process of step S1.1 and step S1.2 is shown through a specific embodiment. Figure 3The rule-based prompts in step 1 correspond to step S1.1, the prompt-based test generation in step 2 (instruction-based prompts) corresponds to step S1.2A, and the prompt-based test generation in step 2 (module-based prompts) in the figure corresponds to step S1.2B. First, the rule-based prompts in step 1 are introduced. The rule-based prompts are divided into two parts. One part is the task requirements, which are used to describe the processor architecture that needs to be verified and the type of test cases that need to be generated. The other part is the instruction specification, which is used to prompt LLM for some matters when generating test cases, including prohibited header files, functions, and inline assembly instructions. This part of the content is related to the compilation tool chain and simulation environment, and it is necessary to construct this part of the prompts based on the instructions that the current test environment can support. The main reason why the above-mentioned instructions, functions, and header files cannot be used is that the chip verified in the test stage before tape-out does not contain an operating system, and some instructions require an operating system to execute, so they cannot be supported. Step S1.1 (i.e. Figure 3 Step 1) in the example shows a rule-based prompt example with task requirements and instruction specifications. Taking the RV64GC architecture processor as an example, this task requires writing C language test cases to verify the processor. There are limitations on the instruction specifications in the process, such as not supporting<time.h> header files and the use of instructions such as ebreak. These restrictions are mainly determined by the characteristics of the simulation environment, which usually does not include a complete operating system, so some standard libraries and instructions may not be available in such an environment. This step involves developing rule-based hints tailored to the specific processor being verified. If a processor of a different architecture needs to be verified, the rule-based hints need to be modified without affecting the subsequent steps.

[0073] Figure 3In step 2, the hint-based test generation (instruction-based hints) takes the addi instruction as an example to illustrate the process of generating test cases. In this method, the operation and syntax of the instruction are first extracted from the chip manual, and then combined into an instruction-based hint. For the addi instruction, the extracted syntax is addird,rs1,imm12. Among them, rd is the destination register (Destination Register, rd), rs1 is the source register 1 (Source Register1, rs1), and imm12 is a 12-bit immediate value (Immediate, imm). Its operation is rd←rs1+sign_extend(imm12), which takes out the value of the source register according to the address, adds an immediate value, and then writes the result of the operation back to the result of the destination register. According to the hint, LLM generates an inline assembly C language test case with this instruction. The test case first defines the source register, destination register, and immediate value, and assigns initial values ​​to the source register and immediate value. The destination register is not assigned an initial value because the compiler automatically assigns it a physical register. This is followed by the inline assembly snippet of the test case. The instruction here is the addi instruction mentioned above, and finally the operation result is printed. Through this step, a test case with any RISCV instruction can be generated. Step S1.2 (i.e. Figure 3 Step 2) in the figure demonstrates the process of generating test cases based on prompts. The box in the lower left corner shows prompts based on instructions. Taking the addi instruction as an example, the syntax and operation of the addi instruction are extracted from the chip manual, and an inline assembly C language test case with the correct addi instruction format is generated.

[0074] Figure 3 The right part demonstrates the generation of test cases based on module prompts, taking the integer unit as an example. The architecture information of the integer unit is extracted from the chip manual, and LLM is asked to generate a large number of test case features that can verify the module. These features are then sent to LLM one by one to generate test cases with these features. Among them, the architecture information is information obtained from the chip manual that can clearly describe the various components and functions of the unit. The test case features are the core attributes and characteristics generated by LLM that can define a test case in terms of function, performance, design and implementation.

[0075] Figure 3Step 2 of the prompt-based test generation (module-based prompt) shows the process of generating test cases using the integer module as an example. First, extract the architecture information of the module to be verified from the chip manual. The architecture information here refers to the information obtained from the chip manual that can clearly describe the various components and functions of the unit. Combine it into module-based prompt-1 to prompt LLM to generate the characteristics of the test case that can verify the module. Here, LLM can generate the characteristics of the required test case based on the provided architecture information. For the integer module, the architecture information sent to LLM contains the arithmetic logic unit (ALU) and supports 64-bit integer operations. The test case features generated by LLM include ALU operations and support all 64-bit integer operations, including arithmetic operations such as addition and subtraction and related logical operations of AND or NOT operations. Then extract the characteristics of these test cases generated by LLM one by one to form module-based prompt-2, so that LLM can generate test cases containing these characteristics. Figure 3 In the generated test case, two 64-bit unsigned integers are first defined, and then various arithmetic and logical operations are performed on the two numbers, and finally the operation results are printed out. In this way, a large number of test cases that can verify the specified module can be generated.

[0076] In this way, high-quality test cases can be generated for different instructions and modules, thereby effectively verifying the functionality and performance of the processor.

[0077] Step S2, LLM-assisted test case optimization. After generating test cases based on LLM, a large number of test cases can be generated. However, the test cases generated by LLM cannot guarantee successful compilation and simulation. Therefore, the present invention adopts step S2 to continue LLM-assisted test case optimization. A minimum supervision method is adopted to enable LLM to generate as many correct test cases as possible.

[0078] like Figure 4As shown, the process of LLM-assisted test case optimization starts with the initial prompt of generating test cases. Then, the generated test cases are compiled, and if the compilation fails, the compilation log is fed back to LLM so that necessary test case modifications can be made. If errors in the same position cause compilation failure three times, the instruction format is fed back for correction. The correction of the instruction format is specifically for inline assembly test cases, while C language test cases will automatically skip this step. If the problem still cannot be solved, the test case is deleted. After successful compilation, the test case will be simulated. After rule-based prompts, the simulation error rate can be significantly reduced. However, the present invention will delete those test cases that are successfully compiled but fail in simulation, because these failures are usually related to the simulation environment. After the test case optimization, the remaining test cases can be used to generate coverage reports.

[0079] Figure 4 An example of generating a test case for fmadd.s and performing test case optimization is shown. First, an inline assembly test case is generated for fmadd.s using instruction-based hints. When compiling this test case, if an error occurs, the error message in the compilation log is returned to the LLM for modification. If the error persists after multiple attempts, the syntax and operation of the fmadd.s instruction are returned to the LLM for further modification.

[0080] Figure 4The specific process and examples of LLM-assisted test case optimization are shown. In step S1, the test case generation method based on LLM has been mentioned. In the last step of S1, a test prompt will be sent to LLM to generate a test case. In step S2, corresponding to step S2.1, a test case is generated. The next step is step S2.2 to compile the test case. When this step is reached, there will be two possible results. The first is that the compilation passes, then execute step S2.4 for simulation. The second case is that the compilation fails, then execute step S2.3 to feedback LLM to generate a new test case. In the S2.3 feedback step, it will be divided into three stages. The first stage is the S2.3.1 compilation log feedback step, the second stage is the S2.3.2 instruction format feedback step, and the third stage is the S2.3.3 test case deletion step. Among them, all test cases with compilation errors in the S2.3.1 compilation log feedback step will go through this stage, and only inline assembly test cases with too many errors in the S2.3.2 instruction format feedback step will go through this stage, and test cases that do not contain inline assembly instructions will not go through this stage. The step of deleting test cases in S2.3.3 is a stage that only test cases with more than a threshold number of errors will go through. The step of compiling log feedback in S2.3.1 will combine the test cases with compilation errors and the error information in the compilation log into a prompt and send it to LLM, so that LLM can modify the test cases based on the errors found by the compiler. After executing the feedback in S2.3, a new test case will be generated and the compilation steps in S2.1 and S2.2 will be re-executed. If the error is repeated three times in this step, the step of instruction format feedback in S2.3.2 will be executed. This step will only be executed for inline assembly test cases. This step will prompt LLM the correct usage of the assembly instructions corresponding to this test case, so that LLM can generate a test case with the correct instruction format. If the instruction format feedback in S2.3.2 still cannot generate a correct test case, execute S2.3.3 to directly delete this test case. For test cases that successfully pass the compilation step in S2.2, the simulation step in S2.4 will be executed. After the rule-based prompt in S1.1, the success rate and error rate of simulation can be significantly reduced. However, the present invention will delete those test cases that are successfully compiled but fail in simulation, because these failures are usually related to the simulation environment. After the test case optimization, the remaining test cases can be used to generate coverage reports.

[0081] Figure 4An embodiment of LLM-assisted test case optimization is also shown. Taking the fmadd.s instruction as an example, the specific process of generating test cases based on the instruction prompt method and performing LLM-assisted test case optimization is shown. First, according to S1.2.2A, a prompt is constructed based on the instruction prompt method and sent to LLM, and then the test case is generated by executing step S2.1, and an inline assembly C language test case containing the fmadd.s instruction is generated. The test case is compiled according to the method in step S2, and an error occurs: "test.c:7:Error:illegaloperands`fmadd.sfa5,fa5,fa4,fa3,0'". The reason for this error is that fmadd.s in RISC-V does not require rm (rounding mode) as an explicit parameter, but is implicitly obtained from the fcsr register. However, in the generated test case, it is introduced as an explicit parameter, resulting in a compilation error. After the error occurs, the error information in the compilation log is automatically extracted according to S2.3.1, and the test case and error information are fed back to LLM to generate a correct test case. For inline assembly test cases, if compilation errors still occur after multiple feedback of compilation log information for test case optimization, and the number of errors exceeds the threshold, step S2.3.2 will be started to perform instruction format feedback, and the correct instruction format will be fed back to LLM to generate a test case containing the instruction in the correct format.

[0082] Through this LLM-assisted test case optimization method, the present invention can significantly improve the compilation and simulation success rate of generated test cases, thereby improving the efficiency and effect of processor verification.

[0083] Step S3, after the LLM-assisted test case optimization, all test cases can successfully pass the compilation and simulation. Through this process, a test case result file is generated. Based on these results, the present invention resets a set of prompts to feedback to the LLM and generates new test cases to achieve result feedback. Figure 5 This workflow is illustrated along with an example of result feedback.

[0084] Specifically, the test case generates a set of binary files after compilation and decompiled into RISC-V assembly instructions. By analyzing these files, it is possible to determine which instructions are not covered. After identifying these instructions, this information is fed back to the LLM, prompting it to generate new test cases containing these instructions. This process is repeated until the coverage exceeds the set threshold. Figure 5An example of ISA prompt feedback is shown. Unexecuted instructions are analyzed by disassembling the file. Taking the addi instruction as an example, the figure shows the prompt feedback process based on the instruction, including information such as the syntax and operation of the instruction. After executing step S3, all test cases can be compiled and simulated smoothly. After compiling the test case to obtain a binary file, the RISC-V instructions executed by the test case can be obtained by disassembling, such as Figure 5 As shown, lui and csrs are both RISC-V instructions. Use the search script to search the disassembly file to detect which RISC-V instructions have not appeared in the disassembly file, and the design code of these instructions that have not appeared is covered in the simulation. For the instructions that have not appeared, extract them one by one, and send prompts to LLM according to the method of step S1.2A, so that it generates an inline assembly test case containing the instruction, and confirms that the instruction can be covered during simulation. Figure 5 The instructions not included in the disassembled file shown in are the addi instructions. According to the method in S1.2A, the syntax and operation of the instruction are sent to the LLM to generate a test case containing the instruction.

[0085] During the simulation process, the present invention records the code coverage of each module. In the simulation stage, the test case generated previously is called to execute in the design file, and the code execution of the design file during the simulation process is recorded using an EDA tool. The EDA tool can generate a coverage report for this simulation after the simulation is completed, including the coverage of each file and each module. (The EDA tool used in the present invention to record the code coverage situation can be Incisive software) Some modules can quickly achieve high coverage, usually modules with relatively simple functions. However, for most modules, it is challenging to quickly meet the coverage requirements. For these modules, the present invention sends the previously verified test case features to LLM to generate more diverse test case features. Subsequently, these newly generated test case features are sent back to LLM for another round of test case generation until the coverage meets the requirements. If the coverage of some modules still does not meet the standard after several rounds of testing, the test generation of these modules will be terminated to reduce the testing cost. Figure 5 The feedback process and examples based on module coverage are shown. After simulating the test case to obtain the coverage report, it is determined whether the code coverage of the module to be tested meets the preset indicators. If the code coverage of the module to be tested has met the standard, the code coverage of the module will no longer be collected in the future. If the code coverage does not meet the standard, the previously verified test case features are sent to LLM to generate more diverse test case features. Subsequently, these newly generated test case features are sent back to LLM for another round of test case generation until the coverage meets the requirements. Figure 5Taking the integer unit (IU) as an example, the feedback process based on the module coverage is demonstrated. After obtaining the coverage report, it is determined whether the code coverage meets the standard. (The code coverage recorded here is block coverage (BC) and expression coverage (EC)) The block coverage is only 47%, and the expression coverage reaches 82%, which generally does not meet the 90% pre-tapeout requirement. Therefore, it is regarded as a module with unsatisfactory coverage, and the features of the generated test cases that can verify the module are first sent to the LLM, so that it can generate more features that can verify the test cases of the module, and then generate test cases for verification according to step S1.2B.

[0086] Through this result feedback and optimization cycle, the coverage of test cases can be gradually improved to ensure the comprehensiveness and effectiveness of processor verification.

[0087] Step S4: In order to improve the efficiency of LLM in generating test cases, the present invention provides a special prompt library, which contains a wide range of test case characteristics and can generate different test cases through LLM to perform processor verification. Figure 6 The construction process of the hint library is shown, which starts with module-based hints, by providing the LLM with the architectural information of the module and letting the LLM generate test case features that can verify the module. These features are then provided to the LLM again as hints to generate test cases with these features. These test cases are then compiled and simulated to obtain coverage reports, which are used to evaluate the effectiveness of these features. If a feature is considered valid, it will be added to the hint library.

[0088] The present invention uses a greedy algorithm to evaluate the effectiveness of features. Initially, when the prompt library is empty, the present invention adds the current feature to the library, and its coverage report is used as a benchmark. Each new report is merged with the existing report. If the coverage is improved, the corresponding test case feature will be added to the prompt library, and the merged report will become the new coverage benchmark. This process is repeated continuously, and the prompt library and coverage report are continuously updated.

[0089] Each simulation through the EDA tool can generate a coverage report and a data file containing specific coverage information. The next time the same design file is simulated, the coverage report and data file can also be obtained. The EDA tool can be used to merge two data files. The generated file is a union. If one of the two simulations covers a certain code block, the code block will be considered covered in the merged data file, and a new coverage report will be generated based on this data file.

[0090] BlockCoverage: Block coverage refers to the ratio of code blocks (such as functions, basic blocks, etc.) that have been executed during the verification process to the total code blocks. This coverage helps confirm that each independent code block in the code is executed at least once, thereby discovering areas of untested code. In hardware description languages ​​​​(such as Verilog or VHDL), this usually refers to whether each basic block (a code sequence with an entry and an exit without jump statements in between) is activated.

[0091] ExpressionCoverage: Expression coverage refers to whether all possible expression results (such as true and false in logical expressions) are tested during the verification process. It is used to confirm that each possible value change in the logical expression occurs at least once. This coverage is particularly important because it can reveal errors hidden in conditional expressions that may cause functional failure or abnormal behavior.

[0092] The coverage report is a report that contains the final coverage results of each module. It does not include which specific code is covered, only a final result (percentage data).

[0093] Figure 7 It is a flow chart of the greedy algorithm for constructing the prompt library of the present invention.

[0094] The present invention is achieved by Figure 7 The algorithm shown in the figure is used to build a prompt library to improve the efficiency of generating test cases. It takes the test case feature set as input, and initializes the prompt library and the maximum coverage report as empty sets. Then the test case features are processed one by one: for each test case feature, LLM is called to generate a test case, and then the test case is compiled and simulated to obtain a coverage report. If the maximum coverage report is empty, the current test case feature is directly added to the prompt library, and the coverage report corresponding to the current test case feature is used as the maximum coverage report. Otherwise, the current coverage report is merged with the maximum coverage report to obtain a new report, and the coverage of the new report is compared with that of the maximum coverage report to see if there is an improvement. If the coverage is improved, the current test case feature is added to the prompt library, and the merged new report is used as the coverage report for subsequent comparison. Finally, a complete prompt library will be returned, and the content in the prompt library can be directly extracted when generating test cases in the future, thereby improving the efficiency of test case generation.

[0095] like Figure 6As shown, a test case feature in the hint library - an example of an ALU operation. By extracting this feature and sending it to the LLM, a test case with this feature can be generated for processor verification. The structure of the hint library is hierarchical, and the first level contains the module names to be verified. Under each module name is the test case feature generated by the LLM that can verify the module. In order to verify a specific module, all test case features under the corresponding module can be extracted. If the entire processor needs to be verified, the test case features of all modules need to be extracted.

[0096] Figure 6 What is shown is the process of building a prompt library and the specific content in the prompt library. According to step S1.2B, the architecture of the module is first prompted to LLM so that it can generate features of the test cases that can verify the module, and these features are sent to LLM one by one to generate test cases for verification and generate coverage reports. According to the greedy algorithm mentioned above, if this test case improves the overall coverage, the features of its corresponding test case are collected into the prompt library. The purpose of building a prompt library is mainly to obtain some representative prompts, which can directly generate test cases for verification, improve the efficiency of generating test cases, and do not need more dialogues with LLM.

[0097] After building a complete prompt library, subsequent verification can directly extract the content in the prompt library without having to send the architecture information again according to the steps in S1.2B and return the test case features.

[0098] Through this structured prompt library, the present invention can efficiently generate test cases with wide coverage, thereby significantly improving the efficiency and effect of processor verification.

[0099] In addition, the present invention proposes a method for generating test cases for brain-like processors based on a large language model. Brain-like processors belong to the DSA architecture and have a unique brain-like instruction set. However, LLM does not understand the brain-like instruction set. Therefore, the first step in generating test cases for brain-like processors is to teach LLM how to write test cases based on a custom brain-like instruction set.

[0100] Figure 8 This is a schematic diagram of generating test cases for brain-like processors. The test case generation process is as follows. First, LLM is given detailed instructions on the functions and usage of brain-like instructions. For example, Figure 8The prompts and responses when using LLM to generate instructions including lw.wv are shown. lw.wv is an instruction that loads synaptic weights from memory to the extended weight vector register (Weight Vector Register, WVR), which is a custom brain-like processor instruction. In a brain-like processor, each synaptic weight occupies 4 bits, and WVR is a 128x32-bit weight vector register used to store the weight vector of each synapse. Figure 8 As shown, the specific information of the relevant instructions will be prompted to the LLM, including the specific usage of the instructions, mnemonics, and the actual meaning of each parameter in the instruction in the brain-like processor. After listening to the prompts, the LLM generates a specific test case to traverse and access all WVRs. Figure 8 As can be seen, LLM understands the basic usage of the lw.wv instruction and is therefore able to generate an assembly code snippet containing the lw.wv instruction for loop weight and peak loading.

[0101] Although LLM can generate test case snippets containing lw.wv instructions, generating completely correct test cases is still a challenge for LLM. It usually requires iterative improvement and manual collaboration to ensure that the generated test cases achieve the expected functions. Figure 8 As shown in the figure, the first generated test case has two problems. The first problem is that the target WVR register is not incremented in each loop. The second problem is that the jump condition is wrong and not all 128 WVR registers are operated in the loop. For these errors, LLM is further prompted. LLM shows a strong ability to locate the source of errors and correct them, such as Figure 8 As shown in the figure, only a very vague prompt was provided to indicate that there were 128 WVR registers that needed to be operated. LLM quickly identified the specific location of the error, understood that 128 iterations were required, and accurately corrected the error by changing bnex10 x0,loop to blt x10 128,loop. This shows that LLM has a strong ability to understand the language and can effectively solve problems in fuzzy contexts. After further prompts, the test cases generated by LLM can meet the loop requirements, but the target WVR register still fails to continue to increase with the increase of the loop. Therefore, LLM is prompted again that the target register should also increase. LLM can find that the x12 register corresponding to the WVR does not increase in the loop, and makes modifications to generate correct test cases that meet the requirements.

[0102] Although the test cases initially generated have some errors, it reflects the ability of LLM to retain and effectively apply previously acquired knowledge. Based on this, the present invention further prompts LLM to generate combined test cases containing different brain-like instructions, which are then translated into machine code by the compiler and finally sent to the simulator. This process utilizes the knowledge of LLM to synthesize complex brain-like instruction sequences and generate comprehensive test conditions.

[0103] Any process or method description in the flowchart of the present invention or described in other ways herein can be understood as a module, segment or part of a code including one or more executable instructions for implementing the steps of a specific logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, device or equipment, and the computer-readable medium can be any medium containing storage, communication, propagation or transmission programs for use by execution systems, devices or equipment, including read-only memories, magnetic disks or optical disks, etc.

[0104] In the description of this specification, the description with reference to the terms "embodiment", "example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art can combine or combine different embodiments or examples described in this specification and the features therein without causing any contradiction.

[0105] Although the above content has shown and described the embodiments of the present invention, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in the field can change, modify, replace, modify and other update operations on the above embodiments within the scope of the present invention.

Claims

1. A method for generating test cases for brain-like processors based on a large language model, characterized in that: The following steps are involved: S1. In the LLM-based test case generation part, key information for processor verification is extracted from the chip manual and used as the input of the hint generator; After receiving these inputs, the prompt generator constructs various prompts, which serve as inputs to the LLM to generate a series of test cases; S2. performing LLM-assisted test case optimization on a series of generated test cases to improve these test cases; S3. Enter the test result feedback phase, in which the test cases are compiled and simulated to generate coverage results; If the coverage rate does not meet the requirements, the prompt is updated based on the coverage report results, providing new input to the prompt generator to construct a new round of prompts; S4. Build a prompt library, which contains test case features and generates different test cases for processor verification by prompting LLM.

2. The method for generating test cases for a brain-like processor based on a large language model according to claim 1, characterized in that: In step S1, in the test case generation stage based on LLM, the workflow of test case generation based on LLM includes the following steps: S1.1 Rule-based prompting: Establish a set of rule constraints for all test cases generated by LLM to ensure that the generated test cases meet specific standards; S1.2 Hint-based test case generation: Based on specific requirements, guide LLM to generate different test cases to meet these requirements.

3. The method for generating test cases for a brain-like processor based on a large language model according to claim 2, characterized in that: In step S1.1, first, the verification rules are extracted from the chip manual, including the compilation tool chain and simulation environment information; based on the above information, the instruction specifications of the test cases in the current compilation and simulation environment are analyzed; at the same time, the architecture information of the processor is obtained to set the task requirements; finally, this information is integrated and sent to the LLM.

4. The method for generating test cases for a brain-like processor based on a large language model according to claim 2, characterized in that: In step S1.2, the prompts include instruction-based prompts and module-based prompts.

5. The method for generating test cases for a brain-like processor based on a large language model according to claim 4, characterized in that: Generate test cases based on the prompts of the instructions. The specific steps are as follows: S1.2.1A Extract the syntax and operation of each RISC-V instruction from the chip manual; S1.2.2A organizes this information into prompts and sends them to the LLM; S1.2.3A LLM generates inline assembly C language test cases containing relevant instructions based on these prompts.

6. The method for generating test cases for a brain-like processor based on a large language model according to claim 4, characterized in that: Generate test cases based on module prompts. The specific steps are as follows: S1.2.1B extract the architecture information of each module from the chip manual; S1.2.2B LLM analyzes this information and generates test case features for the verification module; S1.2.3B sends these test case features as hints to LLM so that it generates corresponding test cases.

7. The method for generating test cases for a brain-like processor based on a large language model according to claim 1, characterized in that: In step S2, the process of test case optimization assisted by LLM starts with the initial prompt of generating test cases; then, the generated test cases are compiled, and if the compilation fails, the error information in the compilation log is fed back to LLM so as to make necessary test case modifications; if the error at the same location causes the compilation to fail three times, the instruction format is fed back for correction; After successful compilation, the test case is simulated.

8. The method for generating test cases for a brain-like processor based on a large language model according to claim 1, characterized in that: Step S3: After compiling, the test case generates a set of binary files and disassembles them into RISC-V assembly instructions; identifies RISC-V instructions that do not appear in the disassembled files, and feeds these instructions back to the LLM, prompting it to generate new test cases containing these instructions; this process is repeated until the coverage exceeds the set threshold.

9. The method for generating test cases for a brain-like processor based on a large language model according to claim 1, characterized in that: In step S4, the construction process of the prompt library starts from the module-based prompt, by providing the module's architecture information to the LLM, so that the LLM generates the test case features to verify the module; These features are then again provided as hints to the LLM to generate test cases with these features; The test cases are then compiled and simulated to obtain coverage reports, which are used to evaluate the effectiveness of these features; If a feature is deemed valid, it is added to the hint library.

10. The method for generating test cases for a brain-like processor based on a large language model according to claim 9, characterized in that: When the prompt library is empty, the current feature is added to the library, its coverage report is used as a baseline, and each new report is merged with the existing report; If the coverage rate is improved, it means that the newly generated test case has improved the overall coverage report. The corresponding test case features will be added to the prompt library, and the merged report will become the new coverage benchmark. This process is repeated continuously to continuously update the prompt library and coverage report.

Images