System and method for optimizing coverage of test generating program for CPU random verification using
By using an artificial intelligence system in the CPU constraint random verification, configurable parameters are automatically adjusted, which solves the problems of poor verification coverage and high time-consuming in the existing technology, and achieves more efficient automated verification.
Patent Information
- Application Number
- CN202280097363.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-07
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art in the constrained random verification (CRV) of central processing units (CPUs) relies on manual adjustment of configurable parameters, resulting in poor verification coverage, high time consuming and high cost.
By using an artificial intelligence system, multiple test assemblies are determined based on the configurable parameter set, simulation results are generated, and parameters iteratively updated until predefined criteria are met, improving the efficiency and performance of automated verification.
Improves the CPU's random verification coverage, reduces verification time and cost, and realizes a more efficient automated verification method.
Smart Images
Figure CN119487519A_ABST
Abstract
Description
Technical Field
[0001] The present invention generally relates to verification of chip designs, and in a particular embodiment, to a method and apparatus for constrained random verification (CRV) of a central processing unit (CPU). Background Art
[0002] Functional verification is an important step in the development of integrated circuits or silicon chips. The purpose of functional verification is to determine whether the design of the chip meets the specifications of the chip. Due to the rapid growth in the size and complexity of today's integrated circuits, it has become more difficult and costly to detect and fix design faults in chips. Since modern processors or central processing units (CPUs) may include more than 1 billion transistors, this issue is even more important in the design and development of processors. Most of the time and effort should be spent on functional verification in the early stages. Once the chip is manufactured, the tools and techniques for testing and debugging the hardware are usually limited. Therefore, instead of relying on silicon testing, it is better to capture logical and functional defects in the pre-silicon stage of chip design (such as register transfer level (RTL)).
[0003] Constrained random verification (CRV) is a standard method for industrial design verification of semiconductor chips. During the verification process, sample data and instructions (stimulus) may be sent to a simulator running the Design Under Test (DUT). The output (response) of the simulator is generated and verified to check whether the DUT works properly as required by the specifications. Due to the complexity of chip design, it is difficult to use traditional test benches for exhaustive verification processes to simulate all possible scenarios. Therefore, CRV based on randomized data generation stimuli is becoming more and more popular.
[0004] An important aspect of CRV can be the design of well-designed testbenches that apply pseudo-random stimuli downstream of the DUT. Testbenches are often composed of parameterized tests that are manually created to verify functionality. Some parameters can be used as high-level knobs to control the generation of stimuli. The testbench can then generate a series of related stimuli based on these configurable parameters. Such testbenches are often labor-intensive and may not provide adequate coverage for the necessary test cases. Therefore, there is a need for techniques to improve the efficiency and performance of computer operations in automated verification methods. Summary of the invention
[0005] According to various embodiments, a method for automatically verifying a DUT is provided, the method comprising a computer device determining a plurality of test program sets based on a configurable parameter set. The method also comprises the computer device determining a plurality of simulation results based on the plurality of test program sets. The method also comprises the computing device training an artificial intelligence system using the plurality of simulation results. The method also comprises the computing device updating the configurable parameter set based on the trained artificial intelligence system. The method also comprises the computing device repeating the determining the plurality of test program sets, the determining the plurality of simulation results, the training the artificial intelligence system, and the updating the configurable parameter set until one or more predefined criteria are met.
[0006] According to various embodiments, a device for automatically verifying a DUT is provided, the device comprising a non-transitory memory, the non-transitory memory comprising instructions and one or more processors in communication with the non-transitory memory. The one or more processors execute the instructions to determine a plurality of test program sets according to a configurable parameter set; determine a plurality of simulation results according to the plurality of test program sets; train an artificial intelligence system using the plurality of simulation results; update the configurable parameter set according to the trained artificial intelligence system; repeatedly determine the plurality of test program sets, determine the plurality of simulation results, train the artificial intelligence system, and update the configurable parameter set until one or more predefined criteria are met.
[0007] According to various embodiments, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by a processor of a device, causes the device to perform operations for automatically verifying a DUT and creating a trained artificial intelligence system. The operations include determining a plurality of test program sets according to a configurable parameter set; determining a plurality of simulation results according to the plurality of test program sets; updating the configurable parameter set according to the artificial intelligence system trained using the plurality of simulation results; repeating the determining the plurality of test program sets, the determining the plurality of simulation results, and the updating the configurable parameter set until one or more predefined criteria are met.
[0008] In some embodiments, the artificial intelligence system includes a reinforcement learning algorithm with an internal prediction model, the internal prediction model being used to generate a probability distribution of values. In some embodiments, the one or more predefined criteria include one or more code coverage results that meet predefined requirements, the one or more code coverage results being determined based on the aggregation of the multiple simulation results. In some embodiments, during each iteration of updating the configurable parameter set, the reinforcement learning algorithm uses the one or more code coverage results as a reward to train and update the internal prediction model. In some embodiments, the configurable parameter set is updated based on a sampling of the output of the reinforcement learning algorithm. In some embodiments, the determining of the multiple simulation results includes the computing device applying each of the multiple test program sets as a stimulus to the DUT to generate the multiple simulation results in parallel, the computing device being a computer cluster. In some embodiments, the generating of the multiple simulation results in parallel includes loading the DUT into a CPU simulator and inputting each stimulus into the CPU simulator in parallel as one or more instructions. In some embodiments, the determining of the plurality of simulation results comprises the computing device transmitting the plurality of test program sets to a second device, the second device being a computer cluster, the computer cluster being used to generate the plurality of simulation results in parallel by applying each of the plurality of test program sets as a stimulus to the DUT. The determining of the plurality of simulation results also comprises the computing device receiving the plurality of simulation results from the second device. In some embodiments, the one or more predefined criteria comprise a simulation run time greater than a predetermined threshold. In some embodiments, the DUT comprises a Verilog implementation of a CPU design. In some embodiments, the artificial intelligence system is trained based on a policy gradient algorithm, the policy gradient algorithm comprising a proximal policy optimization algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more fully understand the present invention and its advantages, the following description is made with reference to the accompanying drawings, in which:
[0010] Figure 1 A block diagram illustrating an embodiment of a system for performing the methods described herein;
[0011] Figure 2 A flow chart of an exemplary method for artificial intelligence (AI) based constrained random verification (CRV) is shown;
[0012] Figure 3A and Figure 3BAn exemplary method performed by two separate computing devices is shown;
[0013] Figure 4 An embodiment of a test program generator is shown;
[0014] Figure 5 An embodiment of a simulation environment is shown;
[0015] Figure 6 An embodiment of a reinforcement learning algorithm is shown;
[0016] Figure 7 A block diagram illustrating an embodiment of a computing system for performing the methods described herein;
[0017] Figure 8 A block diagram of another embodiment of a computing system for executing the methods described herein is shown.
[0018] Unless otherwise indicated, corresponding numerals and symbols in the different figures generally refer to corresponding parts. These figures are drawn to clearly illustrate the relevant aspects of the embodiments and are not necessarily drawn to scale. DETAILED DESCRIPTION
[0019] The manufacture and use of embodiments of the present invention will be discussed in detail below. However, it should be understood that the concepts disclosed herein can be embodied in a variety of specific contexts, and the specific embodiments discussed herein are merely illustrative and are not intended to limit the scope of the claims. In addition, it should be understood that various changes, substitutions and modifications may be made herein without departing from the spirit and scope of the present invention as defined by the appended claims.
[0020] In traditional CRV, parameters are usually provided by verification engineers based on years of experience and expertise in related fields. The test sets generated based on these parameters may not be the best choice for test coverage. In addition, manually adjusting these test parameters is cumbersome and time-consuming.
[0021] Various aspects of the present invention provide embodiments of a technique for generating configuration parameters for a test generator, which optimizes the coverage of random verification of a CPU and improves the efficiency and performance of computer operations in an automated verification method. The configuration parameters can be generated by an artificial intelligence (AI) system. Specifically, the proposed technology enables a computing device to determine a test program set based on a configurable parameter set. The computing device can determine simulation results based on the test program set. The computing device can iteratively train the AI system using the simulation results to update the configurable parameter set until one or more predefined criteria are met. In addition, the test program set and the simulation results can be generated in parallel to reduce testing time. The above aspects and other creative aspects will be discussed in more detail below.
[0022] Figure 1 A functional block diagram of an embodiment of a system 100 for performing the methods described herein is shown. The system 100 may be composed of discrete hardware that performs the functions described, or alternatively, as a software module created by one or more processors that execute computer instructions. The system 100 includes an AI system 102, a test program generator 106, and a simulation environment 108. The AI system 102 is used to generate configurable parameters 112 in an iterative manner. The AI system 102 may first analyze the test program and extract the configurable parameters. The AI system 102 may analyze the test program based on some predefined specifications or configuration formats of the test program generator 106. The predefined specifications or configuration formats of the test program generator 106 may be expressed in a format designed to store various data types (e.g., an Extensible Markup Language (XML) file or a JavaScript Object Notation (JSON) file). The AI system 102 may be used to generate an initial set of configurable parameters, and these configurable parameters may be transmitted to the test program generator 106. Whenever the test program generator 106 receives the configurable parameters generated by the AI system 102, the test program generator 106 can generate a test program according to the configurable parameters and transmit the test program to the simulation environment 108. The simulation environment 108 may include a compiler and one or more simulators. The compiler of the simulation environment 108 can be used to compile the hardware description language (HDL) source code (e.g., Verilog) known in the art of the design under test (Design Under Test, DUT) into a target file, or translate the source code into a C source file. One of the multiple simulators of the simulation environment 108 can also be referred to as a DUT simulator, and can be implemented using open source verification tools (e.g., Verilator) or commercial tools (e.g., Synopsys Verilog Compiler and Simulator (Verilog Compiler and Simulator, VCS) known in the art. Each simulator can be used to load and execute the target file and generate simulation results. The simulation environment 108 can also include a user interface for debugging the simulation process and viewing the simulation results. Each time the simulation environment 108 receives a test program generated by the test program generator 106, the simulation environment 108 may generate simulation results according to the test program and transmit the simulation results to the AI system 102. During each iteration, the AI system 102 may update the configurable parameters according to the simulation results and transmit the updated or new configurable parameters to the test program generator 106.
[0023] Configurable parameters 112 may be parameters for controlling the generation of test programs. Configurable parameters 112 may indicate which instructions are selected for each test program set to be sent to the simulation environment 108. Configurable parameters may be a set of options for indicating the selection of instructions for each test program set for the simulation environment 108. Options may include the type of instructions to be included in the test program for the simulation environment 108. Options may also include weights associated with each instruction type. The weight may represent a certain percentage of a plurality of choices and may be a predetermined set of values for indicating a certain aspect of how the test program should be generated. For example, a configurable parameter may indicate whether a branch instruction or a loop instruction is included in the test program. One option may be the ratio of "start jump" using "branch" or "RET". Another option may be the ratio of "reload register" using "move" instructions or "load" instructions. In another example, a configurable parameter may indicate the percentage of a specific type of instruction in the test program. Options may be program specific, which may depend on the designer of the test program generator 106.
[0024] The test program includes instructions to be executed by the design under test (DUT) 110. In order to ensure that the required parts of the DUT 110 are tested, a variety of test programs can be generated to cover as many test scenarios as possible. The simulation environment 108 may include a test bench or simulator for loading the DUT 110. In one embodiment, the DUT 110 includes a design of a chip (e.g., a CPU) implemented using a hardware description language (HDL). For example, the HDL implementation can be an integrated circuit (IC) written in the Verilog or SystemVerilog standard, which is mapped to a physical IC manufactured at a later stage. The test bench in the simulation environment 108 can be used to load and execute the DUT 110. The test program can be compiled into a binary file, and the binary file can be loaded into the test bench as an input to the DUT 110. The test bench can then simulate how the DUT 110 will respond to the instructions in the test program and generate simulation results as outputs of the DUT 110. The output of the DUT 110 may include the output of a final step during execution of the DUT 110 and the output of an intermediate step during execution of the DUT 110 .
[0025] Verification of the DUT 110 can be performed by evaluating the simulation results generated by the simulation environment 108. Verification can include determining whether the DUT 110 meets the specifications, whether the DUT 110 generates the correct response to the input (including the final output and the intermediate output), and whether the specifications are correct. Verification can also include determining whether enough code (e.g., HDL code) for implementing the DUT 110 has been executed during the simulation, i.e., code coverage. In other words, code coverage can be regarded as a quantitative measurement of DUT code execution. Code coverage can use different measurements. For example, code coverage can measure the percentage of DUT code executed in the simulation. In another example, code coverage can measure how many times a specific DUT code line is executed in the simulation. Generally, a greater code coverage means that more DUT code is tested. Untested DUT code can still be executed in the actual application of manufacturing chips. Therefore, the greater the code coverage, the lower the probability that the chip design is unreliable and contains undetected defects.
[0026] Traditional CRV usually relies on verification engineers to manually adjust configurable parameters to improve code coverage. If a set of configurable parameters performs poorly, verification engineers may have to adjust the set of configurable parameters based on their experience and run the simulation again to check the results. This manual adjustment method is very time-consuming and costly. The present invention provides an embodiment technology that enables the AI system 102 to automatically generate configurable parameters by iteratively updating the configurable parameters based on simulation results. In one embodiment, using Figure 1 The AI system 102 is implemented using the reinforcement learning algorithm 104 shown in FIG.
[0027] Figure 2A flow chart of a method 200 for AI-based CRV according to some embodiments is shown. The method 200 may be performed or executed by one or more processing units of a computing device. Examples of processing units include, but are not limited to, a graphics processing unit (GPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an artificial intelligence (AI) accelerator, or a combination thereof. The method 200 may also be performed or executed by a routine, a subroutine, or a software module executed by one or more processing units. The method 200 may also be performed or executed by a combination of hardware and software. In the present invention, the coding of the software for performing or executing the method 200 is within the scope of ordinary technicians in the art. The method 200 may include more or fewer operations than those shown and described, and may be performed or executed in a different order. The computer-readable code or instructions of the software executable by one or more processing units may be stored on a non-transitory computer-readable medium such as a memory of a computing device.
[0028] In one embodiment, method 200 begins at step 202, where a computing device determines a plurality of test program sets based on a configurable parameter set. At step 204, the computing device determines a plurality of simulation results based on the plurality of test program sets. At step 205, the computing device trains an artificial intelligence system using the plurality of simulation results. At step 206, the computing device updates the configurable parameter set based on the trained artificial intelligence system. At step 208, the computing device determines whether one or more predefined criteria are met. If one or more predefined criteria are met, method 200 ends. Otherwise, method 200 proceeds to step 202, where the computing device determines a plurality of test program sets based on the updated configurable parameter set.
[0029] AI systems, test program generators, and simulation environments (e.g. Figure 1The AI system 102, test program generator 106, and simulation environment 108 shown in the figure may or may not be located in the same device. In one embodiment, the AI system, test program generator, and simulation environment are located in the same computing device. In step 202, the computing device determines multiple test program sets according to the configurable parameter set, including the test program generator of the computing device generating multiple test program sets according to the configurable parameter set. The configurable parameter set can be generated by the AI system of the computing device. In step 204, the computing device determines multiple simulation results according to the multiple test program sets, including the simulation environment of the computing device, and the computing device generates multiple simulation results according to the multiple test program sets. In step 205, the computing device uses multiple simulation results to train the AI system, including the computing device using multiple simulation results to train the AI system of the computing device. In step 206, the computing device updates the configurable parameter set according to the trained AI system, including the computing device updating the configurable parameter set according to the trained AI system of the computing device.
[0030] In another embodiment, the AI system is located in a first computing device, and the test program generator and the simulation environment are located in a second computing device. In step 202, the first computing device transmits the configurable parameter set to the second computing device. The test program generator of the second computing device generates multiple test program sets based on the configurable parameter set. In step 204, the simulation environment of the second computing device generates multiple simulation results based on the multiple test program sets. In step 205, the second device transmits the multiple simulation results to the first computing device. The AI system of the first computing device is trained using the multiple simulation results. In step 206, the first computing device updates the configurable parameter set based on the trained AI system of the first computing device. In step 208, the first computing device may determine whether one or more predefined criteria are met based on the multiple simulation results. If one or more predefined criteria are met, method 200 ends. Otherwise, the first computing device may transmit the configurable parameter set to the second computing device and continue to iterate.
[0031] In another embodiment, the AI system is located in a first computing device, the test program generator is located in a second computing device, and the simulation environment is located in a third computing device. In step 202, the first computing device transmits the configurable parameter set to the second computing device. The test program generator of the second computing device generates multiple test program sets based on the configurable parameter set. In step 204, the second computing device transmits the multiple test program sets to the third computing device. The simulation environment of the third computing device generates multiple simulation results based on the multiple test program sets. In step 205, the third device transmits the multiple simulation results to the first computing device. The AI system of the first computing device is trained using the multiple simulation results. In step 206, the first computing device updates the configurable parameter set based on the trained AI system of the first computing device. In step 208, the first computing device may determine whether one or more predefined criteria are met based on the multiple simulation results. If one or more predefined criteria are met, method 200 ends. Otherwise, the first computing device may transmit the configurable parameter set to the second computing device and continue to iterate.
[0032] In yet another embodiment, the AI system and the test program generator are located in a first computing device, and the simulation environment is located in a second computing device. Figure 3A An exemplary method 300 performed by a first computing device is shown. In step 302, a test program generator of the first computing device generates multiple test program sets based on a configurable parameter set. In step 304, the first computing device transmits the multiple test program sets to a second computing device. In step 306, the first computing device receives multiple simulation results from the second computing device. In step 307, the AI system of the first computing device is trained using the multiple simulation results. In step 308, the first computing device updates the configurable parameter set based on the trained AI system of the first computing device. In step 310, the first computing device determines whether one or more predefined criteria are met. If one or more predefined criteria are met, method 300 ends. Otherwise, method 300 proceeds to step 302, where the test program generator of the first computing device generates multiple test program sets based on the updated configurable parameter set.
[0033] Figure 3B An exemplary method 320 performed by a second computing device is shown. At step 322, the second computing device receives a plurality of test program sets from a first computing device. At step 324, a simulation environment of the second computing device generates a plurality of simulation results based on the plurality of test program sets. At step 326, the second computing device transmits the plurality of simulation results to the first computing device.
[0034] In one embodiment, the test program generator can generate multiple test program sets in parallel. Figure 4 An embodiment of a test program generator 402 is shown; Figure 4As shown, the test program generator 402 includes multiple test generators 404, 406, 408 and 410. The test program generator 402 can divide the configurable parameter set into multiple groups. The number of groups can be determined according to the number of multiple test generators 404, 406, 408 and 410. Each group of configurable parameters can be assigned to a test generator in the multiple test generators. The test program generator 402 can run multiple test generators in parallel and generate a set of test program sets according to each set of configurable parameters. The test program generator 402 can be located in a device, which is a computer cluster for running multiple test generators in parallel.
[0035] In one embodiment, the simulation environment can generate multiple simulation results in parallel. Figure 5 An embodiment of a simulation environment 502 is shown. Figure 5 As shown, the simulation environment 502 includes multiple sub-environments 504, 506, 508 and 510. Each sub-environment may include a separate DUT simulator. Each DUT simulator may be an instance of an open source verification tool (e.g., Verilator) or a commercial tool (e.g., Synopsys VCS) in a docker container collection. The simulation environment 502 may divide multiple test program sets into several groups. The number of groups may be determined based on the number of multiple sub-environments 504, 506, 508 and 510. Each group of test program sets may be assigned to one of the multiple sub-environments. The simulation environment 502 may run multiple sub-environments in parallel and generate a set of simulation results based on each group of test program sets. The simulation environment 502 may be located in a device that is a computer cluster for running multiple sub-environments in parallel.
[0036] As described above, the AI system is used to iteratively optimize the configurable parameter set until one or more predefined criteria are met. In various embodiments, the one or more predefined criteria include one or more code coverage results that meet predefined requirements. The one or more code coverage results can be determined based on the aggregation of multiple simulation results. In other words, the AI system is used to determine the configurable parameter set so that when multiple test program sets generated based on the configurable parameter set are loaded into the DUT as input during simulation, one or more code coverage results measured during simulation meet the predefined requirements. To this end, the AI system can use a reinforcement learning algorithm with an internal prediction model that is used to generate a probability distribution of values.
[0037] Reinforcement learning is a machine learning training method based on rewarding desired behaviors and / or punishing undesired behaviors. Reinforcement learning agents are typically able to perceive and interpret their environment, take actions, and learn through trial and error. Reinforcement learning methods can include taking actions (e.g., generating a set of configurable parameters), collecting rewards (i.e., evaluating one or more code coverage results based on the aggregation of multiple simulation results), and using a policy reinforcement algorithm to find the optimal action.
[0038] Figure 6 An embodiment of a reinforcement learning algorithm 600 is shown; the reinforcement learning algorithm 600 includes a policy reinforcement module 602, an internal model 604, and a sampling module 606. The policy reinforcement module 602 is used to train the internal model 604. The internal model 604 can generate a vector. The generated vector can be used as a parameter of a multi-class probability distribution. The sampling module 606 can sample random values generated based on the multi-class probability distribution and generate a predicted action, which determines a configurable parameter set.
[0039] The policy strengthening module 602 can use any policy strengthening algorithm known in the art, such as REINFORCE, actor critic (AC), advantage actor critic (A2C), asynchronous advantage actor critic (A3C), trust region policy optimization (TRPO), and policy gradient-based methods, such as the proximal policy optimization (PPO) algorithm. The policy strengthening module 602 can use one or more code coverage results as rewards and train the internal model 604 to update or predict the configurable parameter set. The updated configurable parameter set can potentially improve one or more code coverage results of the DUT.
[0040] The internal model 604 may include any internal model known in the art, such as a multi-layer perceptron (MLP) model, a convolutional model, an artificial neural network (ANN) model, or any other nonlinear model.
[0041] The multi-class probability distribution may be a Gaussian distribution or a Beta distribution, or any other distribution that can be sampled to generate predicted actions.
[0042] In one embodiment, the reinforcement learning algorithm is state-based. In another embodiment, the reinforcement learning algorithm is stateless.
[0043] In various embodiments, one or more predefined criteria include a simulation run time greater than a predetermined threshold. In other words, the verification program does not run forever. Although the set of configurable parameters that meet the code coverage requirement has not yet been determined, if the entire program (including the simulation) takes more time than the predetermined threshold, the verification program may have to stop.
[0044] Figure 7 A block diagram of an embodiment of a computing system 700 that can be installed in a host device for executing the methods described herein is shown. As shown, the computing system 700 includes a processor 702, a memory 704, and interfaces 706 to 710, which may or may not be configured as Figure 7 . The processor 702 may be any component or a collection of components for performing computing and / or other processing-related tasks, and the memory 704 may be any component or a collection of components for storing programs and / or instructions executed by the processor 702. In one embodiment, the memory 704 includes a non-transitory computer-readable medium. The interface 706, the interface 708, and the interface 710 may be any component or a collection of components that enable the computing system 700 to communicate with other devices / components and / or users. For example, one or more of the interfaces 706, 708, and 710 may be used to transmit data, control, or management messages from the processor 702 to applications installed in a host device and / or a remote device. As another example, one or more of the interfaces 706, 708, and 710 may be used to support a user or a user device (e.g., a personal computer (PC), etc.) to interact / communicate with the computing system 700. The computing system 700 may include Figure 7 Additional components not shown, such as long-term storage (e.g., non-volatile memory, etc.).
[0045] Figure 8A block diagram of another embodiment of a computing system 800 that can be used to implement the devices and methods disclosed herein is shown. For example, the computing system can be any entity in a UE, an access network (AN), a mobility management (MM), a session management (SM), a user plane gateway (UPGW), or an access stratum (AS). A particular device may use all of the components shown or only a subset of the components, and the level of integration may vary from device to device. In addition, a device may include multiple instances of a component, such as multiple processing units, processors, memories, transmitters, receivers, etc. The computing system 800 includes a processing unit 802. The processing unit includes a central processing unit (CPU) 814, a memory 808, and may also include a mass storage device 804, a video adapter 810, and an I / O interface 812 connected to a bus 820.
[0046] Bus 820 can be one or more bus architectures of any type of several bus architectures, including a storage bus or storage controller, a peripheral bus or a video bus. CPU 814 can include any type of electronic data processor. Memory 808 can include any type of non-transient system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM) or a combination thereof. In one embodiment, memory 808 can include a ROM used at startup and a DRAM used to store programs and data when executing a program.
[0047] Mass storage 804 may include any type of non-transitory storage device for storing data, programs, and other information and making them accessible via bus 820. Mass storage 804 may include, for example, one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, or an optical disk drive.
[0048] The video adapter 810 and the I / O interface 812 provide interfaces for coupling external input and output devices to the processing unit 802. As shown, examples of input and output devices include a display 818 coupled to the video adapter 810 and a mouse, keyboard, or printer 816 coupled to the I / O interface 812. Other devices can be coupled to the processing unit 802, and more or fewer interface cards can be utilized. For example, a serial interface such as a Universal Serial Bus (USB) (not shown) can be used to provide an interface for external devices.
[0049] The processing unit 802 also includes one or more network interfaces 806, which may include a wired link (e.g., an Ethernet cable) or a wireless link to an access node or a different network. The network interface 806 enables the processing unit 802 to communicate with a remote unit via a network. For example, the network interface 806 may provide wireless communication via one or more transmitters / transmit antennas and one or more receivers / receive antennas. In one embodiment, the processing unit 802 is coupled to a local area network 822 or a wide area network to perform data processing and communication with a remote device (e.g., other processing units, the Internet, or a remote storage facility).
[0050] In various embodiments, a computing device implemented according to an embodiment of the computing system 800 may include the AI system 102, the test program generator 106, and the simulation environment 108. The computing device may include a memory 808 for storing instructions for executing the method 200. The computing device may also include a CPU 814. The CPU 814 may include one or more processors for executing the method 200. Some steps of the method 200 may be performed in parallel by multiple processors of the CPU 814.
[0051] In various embodiments, a first computing device implemented according to an embodiment of computing system 800 may include AI system 102 and test program generator 106. A second computing device implemented according to an embodiment of computing system 800 may include simulation environment 108. The first computing device and the second computing device may include memory 808, CPU 814, and network interface 806, respectively. The memory 808 of the first computing device may store instructions for executing method 300, which may be executed by one or more processors in CPU 814 of the first computing device. The memory 808 of the second computing device may store instructions for executing method 320, which may be executed by one or more processors in CPU 814 of the second computing device. Both the first computing device and the second computing device may be coupled to network 822 through their respective network interfaces 806. The first computing device and the second computing device may communicate through network 822 and its network interface 806 while executing methods 300 and 320 (e.g., steps 304 and 306 for the first computing device and steps 322 and 326 for the second computing device).
[0052] Although described in detail, it should be understood that various changes, substitutions and modifications may be made without departing from the spirit and scope of the present invention as defined by the appended claims. Furthermore, the scope of the present invention is not limited to the specific embodiments described herein, and as will be readily appreciated by one of ordinary skill in the art from this disclosure, processes, machines, manufactures, compositions of matter, components, methods or steps (currently existing or to be developed later) may perform substantially the same functions or obtain substantially the same results as the corresponding embodiments described herein. Therefore, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of matter, devices, methods or steps.
Claims
1. A method for automatically verifying a design under test (DUT), characterized in that: The method comprises: The computing device determines a plurality of test program sets based on the configurable parameter set; The computing device determines a plurality of simulation results based on the plurality of test program sets; The computing device trains an artificial intelligence system using the plurality of simulation results; The computing device updates the set of configurable parameters according to the trained artificial intelligence system; The computing device repeats determining a plurality of test program sets, determining a plurality of simulation results, training the artificial intelligence system, and updating the set of configurable parameters until one or more predefined criteria are met.
2. The method according to claim 1, characterized in that The artificial intelligence system includes a reinforcement learning algorithm with an internal prediction model, which is used to generate a probability distribution of values.
3. The method according to claim 2, characterized in that The one or more predefined criteria include one or more code coverage results that meet predefined requirements, and the one or more code coverage results are determined based on an aggregation of the plurality of simulation results.
4. The method according to claim 3, characterized in that: During each iteration of updating the set of configurable parameters, the reinforcement learning algorithm trains and updates the internal prediction model using the one or more code coverage results as rewards.
5. The method according to claim 2, characterized in that: The set of configurable parameters is updated based on a sample of an output of the reinforcement learning algorithm.
6. The method according to claim 1, characterized in that Determining a plurality of simulation results comprises: The computing device generates the plurality of simulation results in parallel by applying each of the plurality of test program sets as a stimulus to the DUT, and the computing device is a computer cluster.
7. The method according to claim 6, characterized in that The generating the plurality of simulation results in parallel comprises: Loading the DUT into a CPU simulator; Each stimulus is input into the CPU simulator as one or more instructions in parallel.
8. The method according to claim 1, characterized in that Determining a plurality of simulation results comprises: The computing device transmits the plurality of test program sets to a second device, the second device being a computer cluster, the computer cluster being configured to generate the plurality of simulation results in parallel by applying each of the plurality of test program sets as a stimulus to the DUT; The computing device receives the plurality of simulation results from the second device.
9. The method according to any one of claims 1 to 8, characterized in that The one or more predefined criteria include a simulation run time greater than a predetermined threshold.
10. The method according to any one of claims 1 to 9, characterized in that The DUT includes a Verilog implementation of a central processing unit (CPU) design.
11. The method according to any one of claims 1 to 10, characterized in that The artificial intelligence system is trained based on a policy gradient algorithm, which includes a proximal policy optimization algorithm.
12. A device for automatically verifying a design under test (DUT), characterized in that: The device comprises: A non-transitory memory including instructions; One or more processors in communication with the non-transitory memory, the one or more processors executing the instructions to: Determining a plurality of test program sets based on a configurable parameter set; determining a plurality of simulation results based on the plurality of test program sets; training an artificial intelligence system using the plurality of simulation results; updating the set of configurable parameters according to the trained artificial intelligence system; Determining the plurality of test program sets, determining the plurality of simulation results, training the artificial intelligence system, and updating the set of configurable parameters are repeated until one or more predefined criteria are met.
13. The device according to claim 12, characterized in that The artificial intelligence system includes a reinforcement learning algorithm with an internal prediction model, which is used to generate a probability distribution of values.
14. The device according to claim 13, characterized in that The one or more predefined criteria include one or more code coverage results that meet predefined requirements, and the one or more code coverage results are determined based on an aggregation of the plurality of simulation results.
15. The device according to claim 14, characterized in that During each iteration of updating the set of configurable parameters, the reinforcement learning algorithm trains and updates the internal prediction model using the one or more code coverage results as rewards.
16. The device according to claim 13, characterized in that The set of configurable parameters is updated based on a sample of an output of the reinforcement learning algorithm.
17. The device according to claim 12, characterized in that The instructions for determining the plurality of simulation results include instructions for generating the plurality of simulation results in parallel by applying each of the plurality of test program sets as a stimulus to the DUT, and the device is a computer cluster.
18. The device according to claim 17, characterized in that The instructions for generating the plurality of simulation results in parallel include instructions for loading the DUT into a CPU simulator and instructions for inputting each stimulus into the CPU simulator in parallel.
19. The device according to claim 12, characterized in that The instructions for determining the plurality of simulation results include the following instructions to perform the following operations: transmitting the plurality of test program sets to a second device, the second device being a computer cluster, the computer cluster being configured to generate the plurality of simulation results in parallel by applying each of the plurality of test program sets as a stimulus to the DUT; The plurality of simulation results are received from the second device.
20. The device according to any one of claims 12 to 19, characterized in that The one or more predefined criteria include a simulation run time greater than a predetermined threshold.
21. The device according to any one of claims 12 to 20, characterized in that The artificial intelligence system is trained based on a policy gradient algorithm, which includes a proximal policy optimization algorithm.
22. A non-transitory computer-readable medium having stored thereon instructions, which when executed by an apparatus cause the apparatus to perform operations for automatically verifying a design under test (DUT), characterized in that: The operations include: Determining a plurality of test program sets based on a configurable parameter set; determining a plurality of simulation results based on the plurality of test program sets; training an artificial intelligence system using the plurality of simulation results; updating the set of configurable parameters according to the trained artificial intelligence system; The determining the plurality of test program sets, the determining the plurality of simulation results, the training the artificial intelligence system, and the updating the configurable parameter set are repeated until one or more predefined criteria are met.