AI chip verification system, method, device and storage medium

Abstract

The present disclosure provides an AI chip verification system, method, device and storage medium, wherein the system includes: a use case generator, used to generate a verification case of the object to be verified based on the information of the object to be verified and a preset use case template, wherein the verification case satisfies the format defined by the preset use case template, and the object to be verified includes a target operator or a neural network model in a neural network model; a design to be tested module, used to execute the verification case through the design to be tested to obtain a test result; and a comparison module, used to compare the test result with a reference result to obtain a verification result of the design to be tested on the object to be verified.

Description

Technical Field

[0001] The present disclosure relates to the field of AI chip verification, and specifically to an AI chip verification system, method, device and storage medium. Background Art

[0002] Verification is a very important stage in the chip development process. Verification is required to ensure the correctness of the chip functions before the chip is delivered to the manufacturer for production.

[0003] In the research and development of AI chips, due to the introduction of various types of neural networks, the chips are required to have stronger adaptability and robustness, so the corresponding chip function verification is also required to be able to adapt to various types of neural networks. However, the traditional chip verification method requires the establishment of a special verification environment for the neural network to be verified, and cannot adapt to verify multiple neural networks. Summary of the invention

[0004] The present disclosure provides an AI chip verification system, method, device, storage medium and computer program product.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an AI chip verification system, the system comprising: a use case generator, for generating a verification case of the object to be verified based on the information of the object to be verified and a preset use case template, wherein the verification case satisfies the format defined by the preset use case template, and the object to be verified includes a target operator in a neural network model or a neural network model; a design to be tested module, for executing the verification case through the design to be tested to obtain a test result; and a comparison module, for comparing the test result with a reference result to obtain a verification result of the design to be tested on the object to be verified.

[0006] In the embodiment of the present disclosure, a verification use case is generated according to a preset use case template in a predefined format, so that the AI ​​chip verification system provided in the embodiment of the present disclosure can be adapted to verify various categories of neural network models.

[0007] In some optional embodiments, the preset use case template includes at least one of the following: an instruction flow template file, including the instruction flow template of the verification use case; an initialization data template file, including the initialization data template of the verification use case; a register configuration template file, including the register configuration template of the verification use case.

[0008] In some optional embodiments, the system further includes: an environment driver for driving the verification use case into the design to be tested module based on at least one template file included in the preset use case template, so that the verification use case is executed by the design to be tested.

[0009] In some optional embodiments, the preset use case template includes: a result template file, which contains an output result template of the verification use case; the system also includes: an environment sampler, which is used to sample the test results from the design module to be tested based on the result template file, and transmit the sampled test results to the comparison module.

[0010] In some optional embodiments, the use case generator includes: a directional generator, used to obtain a parameter file of the object to be verified, and generate a verification use case of the object to be verified by parsing object parameters contained in the parameter file of the object to be verified.

[0011] In the disclosed embodiments, the layers of the neural network model can be verified, and the entire neural network model can also be verified, thereby improving the verification depth and flexibility of the neural network of the design under test.

[0012] In some optional embodiments, the parameter file includes: a fixed-point parameter file and a network structure file.

[0013] In some optional embodiments, the preset use case template includes: a reference result file, which contains the reference result of the verification use case; the reference result obtained by the comparison module is obtained from the preset use case template.

[0014] In the disclosed embodiment, the reference result of the verification case can be directly obtained from the verification case, and the verification case does not need to be executed with a reference model, thereby improving the execution efficiency of the comparison device and thus improving the verification efficiency of the design to be tested.

[0015] In some optional embodiments, the use case generator includes: a random generator, used to obtain a configuration file of the object to be verified, generate object parameters of the object to be verified based on the configuration file, and generate a verification case of the object to be verified based on the generated object parameters.

[0016] In the disclosed embodiment, the random generator generates different verification use cases using random technology according to the configuration file and based on specific verification use cases, so that the verification scenario coverage is more comprehensive and the verification completeness is effectively improved.

[0017] In some optional embodiments, the system further includes: a reference model, used to execute the verification use case generated by the use case generator to obtain a corresponding reference result.

[0018] In some optional embodiments, the preset use case template includes: a scheduling core instruction flow template file, which contains the scheduling core instruction flow template of the verification use case; the system also includes: a scheduling instruction compiler, which is used to generate the scheduling core instruction flow of the design to be tested based on the scheduling core instruction flow template file; the use case generator, which is used to generate the verification use case of the object to be verified based on the scheduling core instruction flow generated by the scheduling instruction compiler.

[0019] In the disclosed embodiment, other modules of the AI ​​chip verification system for level 1 DUT are reused, which reduces the chip verification workload, improves the verification efficiency of the design to be tested, and saves verification resources.

[0020] In some optional embodiments, the system further includes a performance statistics device connected to the design under test module, and configured to obtain performance parameters of the design under test executing the verification use case, and obtain performance statistics results based on the performance parameters.

[0021] In the disclosed embodiment, the performance statistics analyzer can count the computing time and bandwidth of the layer-level verification cases of the neural network model executed by the design under test, and can also count the computing time and bandwidth of the network-level verification cases of the entire neural network model executed by the design under test, thereby enriching the performance statistics of the verification of the design under test.

[0022] According to a second aspect of an embodiment of the present disclosure, there is provided an AI chip verification method, comprising: generating a verification case of the object to be verified based on information of the object to be verified and a preset use case template, wherein the verification case satisfies the format defined by the preset use case template, and the object to be verified includes a target operator or a neural network model in a neural network model; executing the verification case through a design to be tested to obtain a test result; comparing the test result with a reference result to obtain a verification result of the design to be tested on the object to be verified.

[0023] In the embodiments of the present disclosure, the verification case generated based on the information of the object to be verified is a verification case generated based on a format predefined in a preset case template. The verification cases generated for the objects to be verified corresponding to different categories of neural networks have a unified format, so the AI ​​chip verification system provided in the embodiments of the present disclosure can be adapted to verify various categories of neural networks.

[0024] In some optional embodiments, the preset use case template includes at least one of the following: an instruction flow template file, including the instruction flow template of the verification use case; an initialization data template file, including the initialization data template of the verification use case; a register configuration template file, including the register configuration template of the verification use case.

[0025] In some optional embodiments, after generating the verification case of the object to be verified based on the information of the object to be verified and the preset use case template, it also includes: based on at least one template file included in the preset use case template, driving the verification case into the design module to be tested, so that the verification case is executed by the design to be tested.

[0026] In some optional embodiments, the preset use case template includes: a result template file, which includes an output result template of the verification case; after executing the verification case through the design to be tested and obtaining the test result, it also includes: based on the result template file, sampling the test result from the design to be tested module, and transmitting the sampled test result to the comparison module.

[0027] In some optional embodiments, generating a verification case for the object to be verified based on the information of the object to be verified and a preset use case template includes: obtaining a parameter file of the object to be verified, and generating a verification case for the object to be verified by parsing the object parameters contained in the parameter file of the object to be verified.

[0028] In some optional embodiments, the parameter file includes: a fixed-point parameter file and a network structure file.

[0029] In some optional embodiments, the preset use case template includes: a reference result file, which contains the reference result of the verification use case.

[0030] In some optional embodiments, generating a verification case of the object to be verified based on the information of the object to be verified and a preset use case template includes: obtaining a configuration file of the object to be verified, generating object parameters of the object to be verified based on the configuration file, and generating a verification case of the object to be verified based on the generated object parameters.

[0031] In some optional embodiments, the method further includes: executing the verification case through a reference model to obtain a corresponding reference result.

[0032] In some optional embodiments, the preset use case template includes: a scheduling core instruction flow template file, which contains the scheduling core instruction flow template of the verification use case; the method also includes: generating the scheduling core instruction flow of the design to be tested based on the scheduling core instruction flow template file; and generating the verification use case of the object to be verified based on the scheduling core instruction flow.

[0033] In some optional embodiments, the method further includes: obtaining performance parameters of the design under test executing the verification use case, and obtaining performance statistical results based on the performance parameters.

[0034] According to a third aspect of an embodiment of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the AI ​​chip verification method described in any one of the second aspects is implemented.

[0035] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the AI ​​chip verification method described in any one of the second aspects is implemented.

[0036] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the program is executed by a processor, the AI ​​chip verification method described in any one of the second aspects is implemented.

[0037] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0039] Figure 1 is a schematic diagram of an AI chip verification system according to an exemplary embodiment;

[0040] Figure 2 is a schematic diagram of a preset use case template according to an exemplary embodiment;

[0041] Figure 3 is a schematic diagram of another AI chip verification system according to an exemplary embodiment;

[0042] Figure 4 is a schematic diagram of another preset use case template according to an exemplary embodiment;

[0043] Figure 5 is a schematic diagram of another AI chip verification system according to an exemplary embodiment;

[0044] Figure 6 is a schematic diagram of another AI chip verification system according to an exemplary embodiment;

[0045] Figure 7 is a schematic diagram of a configuration file according to an exemplary embodiment;

[0046] Figure 8 is a schematic diagram of a secondary design to be tested according to an exemplary embodiment;

[0047] Fig. 9 is a schematic diagram of another AI chip verification system according to an exemplary embodiment;

[0048] Fig.10 The figure is a flow chart of an AI chip verification method according to an exemplary embodiment. DETAILED DESCRIPTION

[0049] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The specific methods described in the following exemplary embodiments do not represent all schemes consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0050] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0051] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] In order to make the AI ​​chip verification solution provided by the present disclosure clearer, the execution process of the solution provided by the present disclosure is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] See also Figure 1 , Figure 1 is an AI chip verification system shown in an embodiment provided by the present disclosure. Figure 1 As shown, the AI ​​chip verification system may include: a use case generator 101, a design to be tested module 102 and a comparison module 103, wherein the use case generator 101 is used to generate a verification case, the verification case is input into the design to be tested module 102 for processing by the design to be tested to obtain a test result, and the test result is input into the comparison module 103 for processing to obtain a verification result of the design to be tested.

[0054] In AI chip verification, it is possible to verify whether the newly designed AI chip can implement certain operations (operators) in the neural network model, such as convolution and pooling in the neural network, and whether these operations can be successfully performed using the AI ​​chip. This verification of certain operators in the neural network model is to use the operators in the neural network model as the object to be verified. Alternatively, it is also possible to verify whether the newly designed AI chip can successfully execute the complete neural network model. This verification of the complete neural network model is to use the entire neural network model as the object to be verified. Among them, the neural network model can be a neural network for image recognition, such as a convolutional neural network, a neural network for speech recognition, or a neural network for text recognition, and the present disclosure does not limit this.

[0055] like Figure 1 In the AI ​​chip verification system shown, the use case generator 101 can parse the information of the object to be verified and obtain the parameters for generating the verification case of the object to be verified. The information of the object to be verified may be the parameter information contained in the fixed-point parameter file and the network structure file, or may be the parameter information in the configuration file corresponding to the object to be verified. For example, taking the convolution operation in the neural network model as the object to be verified, the use case generator 101 can parse the parameter information in the fixed-point parameter file and the network structure file corresponding to the convolution operation to obtain the parameters for generating the verification case of the convolution operation. Alternatively, taking the complete neural network model as the object to be verified, the use case generator 101 can parse the fixed-point parameter file and the network structure file corresponding to the neural network model to obtain the parameters for generating the verification case of the neural network model.

[0056] After the use case generator 101 obtains the parameters for generating the verification case of the object to be verified, it generates the verification case of the object to be verified according to the generated parameters and based on the format defined in the preset use case template. Among them, the preset use case template defines the format of the data in the verification case to be generated. For example, the preset use case template can define the format of the data of a specific part of the verification case to be generated. Specifically, the preset use case template can define the format of the instruction stream data and initialization data in the verification case to be generated, that is, completes the format definition of the specific part (instruction stream data and initialization data). For the data format other than the specific part in the verification case to be generated, the preset use case template can be defined according to the specific situation. Alternatively, the preset use case template can define the format of all data of the verification case to be generated, and the embodiments of the present disclosure are not limited to this.

[0057] In some optional embodiments, the preset use case template includes at least one of the following:

[0058] An instruction flow template file, including an instruction flow template of the verification case;

[0059] An initialization data template file, including an initialization data template for the verification case;

[0060] The register configuration template file includes the register configuration template of the verification case.

[0061] In the above embodiment, the use case generator 101 can generate different data in the verification use case according to each template file in the preset use case template. The use case generator 101 can generate an instruction stream in the format defined by the instruction stream template according to the instruction stream template in the instruction stream template file; generate initialization data in the format defined by the initialization data template according to the initialization data template in the initialization data template file; and generate register configuration data in the format defined by the register configuration template according to the register configuration template in the register configuration template file. The use case generator 101 generates a verification use case for the object to be verified according to the format defined by the preset use case template.

[0062] For example, Figure 2 A preset use case template is shown. The use case generator 101 can generate the instruction flow in the verification use case according to the "instruction flow" template file; generate the initialization data in the verification use case according to the "picture / weight data" template file and the "Initial list" template file: "picture / weight data" and "Initial list" data; generate the register configuration data in the verification use case according to the "Registerconfig" template file.

[0063] After the use case generator 101 generates the verification case, the design under test module 102 can obtain the verification case and execute the verification case by the design under test (DUT) to obtain a test result.

[0064] In some optional embodiments, such as Figure 3 The system also includes: an environment driver 301, which is used to drive the verification use case into the design under test module 102 based on at least one template file included in the preset use case template, so that the verification use case is executed by the design under test.

[0065] In the above embodiment, the environment driver 301 can parse the specific data in the verification case and excite the parsed data. For example, the environment driver 301 can drive the instruction stream, initialization data and register configuration data in the verification case to the design module 102 under test, so that the design under test can execute the verification case.

[0066] by Figure 2The verification case generated by the preset use case template shown is used as an example for explanation. The environment driver 301 can parse the "image / weight data" file, "Initial list" file, "instruction stream" file and Registerconfig file in the verification case, and the Driver component in UVM (Universal Verification Methodology) is used to stimulate and drive the four parts of the design module 102 to be tested.

[0067] Part 1: Use the register configuration data in the verification case to configure the registers of the design under test. The register configuration operation must strictly follow the interface protocols such as AHB (Advanced High performance Bus, system bus) and APB (Advanced Peripheral Bus, peripheral bus).

[0068] Part 2: Initialize the external storage model VIP model connected to the design under test. The initialization of the external storage model VIP model requires the use of VIP backdoor technology to operate the memory area used.

[0069] Part 3: Initialize the memory model of the internal storage model of the design under test. The initialization of the memory model of the internal storage model of the DUT requires encapsulating the DUT memory function and performing precise address writing operations on the internal shared storage space.

[0070] Part 4: Stimulate the instruction stream in the verification case. Using the timing characteristics of the UVM Driver, the instruction stream in the verification case instruction stream file is transmitted to the instruction interface of the DUT in the specified format.

[0071] The environment driver 301 drives the data of the verification case to the design under test module 102, and the design under test executes the verification case. The design under test included in the design under test module 102 can be a complete chip of a new design to be verified, or a specific module of the new design constituting a complete chip to be verified, which is not limited in the present disclosure.

[0072] After the design under test in the design under test module 102 executes the verification case, the test result corresponding to the verification case is obtained. The comparison module 103 can obtain the test result of the design under test after executing the verification case from the design under test module 102.

[0073] In some optional embodiments, the preset use case template includes: a result template file, which includes the output result template of the verification use case. Figure 3The system further includes: an environment sampler 302 for sampling the test result from the design module to be tested 102 based on the result template file, and transmitting the sampled test result to the comparison module 103.

[0074] Reference Figure 5 The preset use case template of the example, wherein the "Dump list" template file can be used as a result template file, including the output result template of the verification use case. The environment sampler 302 can sample the test results obtained after the design under test executes the verification use case from the design under test module 102 according to the output result template in the "Dump list" template file.

[0075] For example, the environment sampler 302 is extended through the Monitor component in UVM, and the sampling task of the result data dump_data is customized. According to the timing requirements in the DUT input / output interface, the result data in the external storage model is sampled into a temporary array. Then, according to the format of the output result template in the result template file, for example, according to the format of the "Dump list" template file, the temporary array is printed into the file. The sampled result data includes the network-level operation results of the AI ​​network, and the sampling condition is the msg signal at the end of the network layer.

[0076] in, Figure 3 In the AI ​​chip verification system shown, the environment driver 301 and the environment sampler 302 are independent of the design module to be tested 102. In other possible implementations, the environment driver 301 and the environment sampler 302 can be integrated in the design module to be tested 102, and the present disclosure does not impose specific restrictions on this.

[0077] The reference result used in the comparison module 103 for comparison with the test result can be the result data directly obtained by the comparison module 103 from the verification case, or can be the result data obtained after the verification case is executed by the reference model. Among them, the reference model is a model for verifying the various functions of the AI ​​chip to be verified or the design to be tested, and is responsible for simulating the logical behavior of the design to be tested DUT. For example, it is implemented using the System C coding model, which is responsible for generating the Golden result for comparison, and when generating the Golden result, it is necessary to ensure consistency with the data collected by the environmental sampler.

[0078] In some achievable ways, the comparison module 103 is responsible for comparing the test results of the environmental sampler 302 with the reference results corresponding to the verification case for correctness, and supports multiple forms of result data comparison. For example, it supports the comparison of the result data stored inside the DUT; it supports the comparison of the network results in the external storage model. In terms of comparison time, data comparison can be performed at the end of each network layer of the network to compare the data results of each layer; the data results of all network layers can be compared at the end of the execution of the verification case. Thus, the R&D personnel can make targeted modifications to the design to be tested that is currently being verified based on the verification results.

[0079] In the embodiment of the present disclosure, verification cases are generated according to a preset use case template in a predefined format, so that the verification cases generated for the objects to be verified corresponding to different categories of neural networks have a unified format, so the AI ​​chip verification system provided in the embodiment of the present disclosure can be adapted to verify various categories of neural networks.

[0080] In some optional embodiments, such as Figure 5 As shown, another AI chip verification system is provided in an embodiment of the present disclosure, in which the use case generator 101 includes a directional generator 401. The directional generator 401 can obtain a parameter file of the object to be verified, parse the object parameters contained in the parameter file of the object to be verified, and generate a verification case of the object to be verified. Among them, the object parameters in the parameter file are the parameters required by the directional generator 401 when generating a verification case of the object to be verified. For example, the parameter file may be one or more of a fixed-point parameter file or a network structure file, or may be other parameter files containing object parameters, which is not limited by the embodiment of the present disclosure.

[0081] Taking the example that the parameter file includes two files, a fixed-point parameter file and a network structure file, the directional generator 401 can obtain the fixed-point parameter file and the network structure file of the object to be verified, parse the object parameters contained in the two files, and generate a verification case of the object to be verified based on the object parameters.

[0082] Specifically, C++ is used as a programming language for explanation. The directional generator 401 can parse the fixed-point parameter file and the network structure file of the neural network model, and call the Printf function to generate an instruction stream according to the object parameters obtained by the analysis. Based on the generated instruction stream, the directional generator 401 generates the instruction stream of the verification case according to the format defined by the instruction stream template in the preset use case template; further, according to other template files in the preset use case template, other data in the verification case is generated to obtain a complete verification case. Through the environment driver 301, the data of the verification case is driven to the design module to be tested 102, and the verification case is executed by the design to be tested DUT. The test results corresponding to the verification case are obtained by sampling through the environment sampler 302. The relevant specific process is similar to the relevant description in the aforementioned embodiment, and will not be repeated here.

[0083] In the embodiment of the present disclosure, the verification case generated by the directional generator 401 may include a reference result of the verification case. Correspondingly, the preset case template includes: a reference result file including the reference result of the verification case. Figure 4 Taking the preset use case template shown as an example, the "Layer-level data" template file can be used as a reference result file, which contains the reference result of the verification case generated by the directional generator 401. In this way, the comparison module 103 can directly obtain the reference result from the verification case, and compare the reference result with the test result obtained from the design module to be tested 102.

[0084] In another achievable manner, when generating a verification case, the directional generator 401 can obtain a reference result by executing the verification case through a reference model, regardless of whether or not a reference result of the verification case is generated, and the reference result is transmitted to the comparison module 103 by the reference model. The reference model is a model for verifying various functions of the chip to be verified or the design to be tested, and is responsible for simulating the logical behavior of the design to be tested DUT.

[0085] In the AI ​​chip verification method of this embodiment, the comparison module can directly obtain the reference result of the verification case from the verification case, and does not need to refer to the model to execute the verification case, thereby improving the execution efficiency of the comparison device and thus improving the verification efficiency of the design to be tested.

[0086] In some optional embodiments, the directional generator 401 may generate a layer or several layers in the neural network model as a verification case of the object to be verified. For example, a convolutional layer in the convolutional neural network model may be generated as a verification case of the object to be verified. A layer or several layers in the neural network model as the object to be verified may be referred to as a hierarchical verification object.

[0087] by Figure 5Taking the AI ​​chip verification system shown as an example, when supporting hierarchical verification, the directional generator 401 can parse the parameter file corresponding to the target network layer in the neural network model, and generate a verification case with the target network layer as the hierarchical verification object according to the format defined by the preset use case template. Among them, the target network layer can be one or more network layers in the neural network model, such as a convolutional layer, a pooling layer, or multiple convolutional layers, multiple fully connected layers, etc.

[0088] Specifically, for example, a certain convolution layer in a convolutional neural network model needs to be verified to test whether the design under test can successfully perform the operation of the convolution layer. In the disclosed embodiment, the directional generator 401 generates a verification case for the convolution layer as a hierarchical verification object by parsing the fixed-point parameter file and network structure file corresponding to the convolution layer in the neural network model, based on the format defined by the preset use case template.

[0089] In the disclosed embodiment, the directional generator 401 can also generate a complete neural network model as a verification case of the object to be verified. For example, a verification case corresponding to a convolutional neural network model can be generated. This method of using the entire neural network model as the object to be verified can be called a network-level verification object.

[0090] Specifically, for example, a convolutional neural network model for image recognition needs to be verified to test whether the design under test can fully execute the operations in the convolutional neural network model. In the disclosed embodiment, the directional generator 401 generates a verification case of the convolutional neural network model as a network-level verification object based on the format defined by a preset use case template by parsing the fixed-point parameter file and network structure file corresponding to the convolutional neural network model.

[0091] The AI ​​chip verification method provided by the embodiment of the present disclosure can verify the hierarchy of the neural network model and can also verify the entire neural network model, thereby improving the verification depth and flexibility of the neural network of the design under test. Thus, the bugs generated in chip development can be more accurately located and corrected, accelerating the convergence of chip verification.

[0092] In some optional embodiments, such as Figure 6 As shown, another AI chip verification system is provided in an embodiment of the present disclosure, in which the use case generator 101 includes a random generator 501, and a reference model 502 is also included between the verification use case and the comparison device 103. Among them, the reference model 502 is a model for verifying the various functions of the AI ​​chip to be verified or the design to be tested, and is responsible for simulating the logical behavior of the design to be tested DUT. For example, it is implemented using a System C coding model, which is responsible for generating a Golden result for comparison, and when generating a Golden result, it is necessary to ensure consistency with the data collected by the environmental sampler.

[0093] In this embodiment, the random generator 501 can obtain a configuration file of the object to be verified, generate object parameters of the object to be verified using randomization technology based on the parameters in the configuration file, and generate a verification case using the randomly generated object parameters. The configuration file is a file configured by the verification personnel for the design to be verified, such as an Excel file in which the verification personnel configures parameters using Excel. Figure 7 This example shows an Excel configuration file.

[0094] The random generator 501 can parse the configuration file and use the parameters configured in the configuration file to randomly generate the object parameters of the object to be verified. For example, the random constraint technology of the SystemVerilog language is used to randomly generate the source operand range, source operand address, target operand range, target operand address and calculation parameters of the operator. Figure 7 Taking the configuration file of the example as an example, the random generator 501 can parse the n parameters in case-1 in the Excel file, and randomly generate some of the n parameters using the random constraint technology. Among them, "some parameters" can be one of the n parameters, or several of the n parameters, which is not limited by the embodiment of the present disclosure. After randomization, multiple groups of parameters different from case-1 are obtained as object parameters of the object to be verified, and the random generator 501 can generate verification cases based on these randomly generated object parameters.

[0095] like Figure 7 Shown is an Excel file configured by the verifier for the neural network model to be verified and the design to be tested. The configuration file includes parameters for generating multiple specific verification use cases. For example, the row where serial number 1 is located includes n parameters of case-1: param-1-1...param-1-n, and the corresponding case-2, case-3...case-n, etc. are all composed of n parameters. The n configuration parameters in each case correspond to a specific verification use case of the case. The random generator 501 can use randomization technology to generate verification use cases with a wider coverage based on specific verification use cases.

[0096] In the process of generating verification cases, the random generator 501 uses random technology to generate the object parameters of the object to be verified, so the reference result of the generated verification case is unknown. Figure 6The AI ​​chip verification system shown includes a reference model 502. The reference model 502 is used to execute the verification case generated by the random generator 501 to obtain the reference result of the verification case. The reference model 502 is a model for verifying the functions of the chip to be verified or the design to be tested, and is responsible for simulating the logical behavior of the design to be tested DUT. For example, it is implemented using a System C coding model, which is responsible for generating Golden results for comparison, and when generating Golden results, it is necessary to ensure consistency with the data collected by the environmental sampler.

[0097] In the above embodiment, the random generator 501 generates different verification use cases using random technology according to the configuration file and based on specific verification use cases, so that the verification scenario coverage is more comprehensive and the verification completeness is effectively improved.

[0098] Figure 6 In the AI ​​chip verification system shown, the use case generator 101 only includes the random generator 501. In other achievable embodiments, the use case generator 101 may include the directional generator 401 and the random generator 501 at the same time. Figure 5 and Figure 6 The AI ​​chip verification system shown is shown separately for the sake of clarity in the description of the embodiment, without any form of limitation.

[0099] In the AI ​​chip verification process, the design under test to be verified can be divided into Level 1 DUT and Level 2 DUT. Figure 8 The Level 2 DUT adds a scheduling core based on the Level 1 DUT. The Level 1 DUT can execute the instruction flow in the verification case. The scheduling core in the Level 2 DUT can execute the scheduling core instruction flow to carry, schedule, control and distribute the instruction flow in the verification case, thereby improving the flexibility of the design under test.

[0100] by Figure 4 Take the verification case generated by the preset use case template as an example. The level 1 DUT can execute the instruction flow generated according to the instruction flow template file in the verification case; the scheduling core in the level 2 DUT can execute the scheduling core instruction flow generated according to the scheduling core instruction flow template file in the verification case, and carry, schedule, control and distribute the instruction flow in the verification case. The use case generator (including the random generator and the directional generator) provided in the above embodiment cannot generate a "scheduling core instruction flow", so it can only generate verification cases corresponding to the level 1 DUT.

[0101] like Fig. 9As shown, another AI chip verification system is provided in an embodiment of the present disclosure. Corresponding to the level 2 DUT, the AI ​​chip verification system also includes a scheduling instruction compiler 601. The scheduling instruction compiler 601 can compile the instruction control flow to obtain a scheduling core instruction flow executable by the scheduling core in the level 2 DUT. Among them, the instruction control flow is an instruction control flow described in a certain programming language for controlling the instruction flow in the verification use case. For example, the scheduling instruction compiler 601 compiles the instruction control flow described in C language code, and outputs a binary assembly file executable by the scheduling core in the level 2 DUT, that is, the scheduling core instruction flow is obtained.

[0102] correspond Fig. 9 The AI ​​chip verification system shown in the figure has a preset use case template including: a scheduling core instruction flow template file, which contains a scheduling core instruction flow template for the verification use case. The scheduling instruction compiler 601 compiles the instruction control flow and generates a scheduling core instruction flow in the verification use case according to the format defined by the scheduling core instruction flow template. Figure 4 Taking the preset use case template as an example, the use case generator 101 can generate other data other than the "scheduling core instruction flow" in the level 2 verification use case based on other template files other than the "scheduling core instruction flow template file", thereby generating a complete verification use case corresponding to the level 2 DUT.

[0103] Compared with the AI ​​chip verification system corresponding to the level 1 DUT, the AI ​​chip verification system in the disclosed embodiment only adds a scheduling instruction compiler 601 and reuses other modules of the AI ​​chip verification system for the level 1 DUT. This reduces the chip verification workload, improves the verification efficiency of the design to be tested, and saves verification resources.

[0104] In some optional embodiments, such as Fig. 9 As shown, the AI ​​chip verification system also includes a performance statistics device 602 connected to the design module to be tested 102. It can be understood that this is only an exemplary specific structure of the AI ​​chip verification system, and it can also be other structural forms. For example, the scheduling instruction compiler 601 may not be included, and it is only used to verify the system of level 1 DUT. In the embodiment of the present disclosure, the performance statistics device 602 can obtain the performance parameters of the design to be tested executing the verification case from the design module to be tested 102, and perform performance statistics based on the obtained performance parameters.

[0105] The performance counter 602 can count the time for the design under test to execute the verification case. Specifically, the performance counter 602 can count the time for each layer operation of the neural network model from the reset signal being pulled high, or count the entire execution time of the neural network model.

[0106] For example, if the verification case executed by the design to be tested is a verification case of a hierarchical verification object, the performance statistics device 602 can count the execution time of the corresponding network layer in the neural network model in the design to be tested, such as counting the operation time of the convolutional layer in the design to be tested. For example, if the verification case executed by the design to be tested is a verification case corresponding to the entire neural network model, the performance statistics device 602 can count the time it takes for the design to test to operate the entire neural network model, such as counting the time it takes for the entire convolutional neural network model to operate in the design to be tested.

[0107] In some achievable methods, the performance statistics device 602 can also count the actual bandwidth of the design under test. Specifically, the performance statistics device 602 can count the amount of read and write data of all interfaces in the design under test, and calculate the actual bandwidth of the design under test in combination with the statistical operation time of the design under test corresponding to the amount of read and write data. For example, if the design under test executes a level verification use case for the hierarchy, the performance statistics device 602 can calculate the actual bandwidth of the network layer corresponding to the neural network model executed by the design under test. For example, the actual bandwidth of the verification use case of the convolutional layer executed by the design under test. For example, if the design under test executes a verification use case for the network level, the performance statistics device 602 can calculate the actual bandwidth of the entire neural network model executed by the design under test. For example, the actual bandwidth of the entire convolutional neural network executed by the design under test.

[0108] In the above embodiment, a performance counter is added to the AI ​​chip verification system, which can count the performance parameters such as the time and bandwidth of the verification case executed by the design under test. In addition, the performance counter can count the operation time and bandwidth of the verification case of the layer of the neural network model executed by the design under test, and can also count the operation time and bandwidth of the verification case of the network level of the entire neural network model executed by the design under test, enriching the performance statistics of the verification of the design under test.

[0109] In some optional embodiments, such as Fig.10 As shown, the embodiment of the present disclosure shows an AI chip verification method. The AI ​​chip verification method or AI chip verification system provided in the embodiment of the present disclosure can be applied to a server, terminal device or other type of electronic device that performs AI chip verification. Fig.10 The AI ​​chip verification flow shown includes the following steps:

[0110] Step 101, based on the information of the object to be verified and a preset use case template, generate a verification case of the object to be verified, wherein the verification case satisfies the format defined by the preset use case template, and the object to be verified includes a target operator in a neural network model or a neural network model.

[0111] In this step, the information of the object to be verified can be parsed to obtain the parameters for generating the verification case of the verification object. The object to be verified can be an operator in a neural network model, or can be the entire neural network model. The information of the object to be verified can be the parameter information contained in the fixed-point parameter file and the network structure file, or can be the parameter information in the configuration file corresponding to the verification object.

[0112] After obtaining the parameters for generating the verification case of the verification object, the verification case of the object to be verified is generated according to the generated parameters and based on the format defined in the preset case template. The preset case template defines the format of the data in the verification case to be generated, and the specific description can refer to the relevant description of some embodiments of the verification system.

[0113] In some achievable methods, a parameter file of the object to be verified can be obtained, the object parameters contained in the parameter file of the object to be verified can be parsed, and a verification case of the object to be verified can be generated. For relevant descriptions, please refer to the description of the relevant part of the verification system embodiment. This implementation method can generate corresponding reference results in the verification case, and does not require the reference model to execute the verification case, thereby improving the efficiency of verification of the design to be tested. In addition, it is possible to verify the hierarchy of the neural network model and the entire neural network model, thereby improving the depth and flexibility of verification of the neural network by the design to be tested.

[0114] In other achievable methods, a configuration file of the object to be verified can be obtained, and object parameters of the object to be verified can be generated using randomization technology based on the parameters in the configuration file, and verification cases can be generated using the randomly generated object parameters. For relevant specific descriptions, please refer to the description of the relevant parts of the embodiment of the verification system disclosed in the present invention. This implementation method uses randomization technology to generate different verification cases based on specific verification cases, so that the verification scenario coverage is more comprehensive, which effectively improves the completeness of the verification.

[0115] Step 102, executing the verification case through the design to be tested to obtain a test result.

[0116] After generating a verification case of the object to be verified, the verification case can be executed by the design to be tested to obtain the test result after execution. The design to be tested can be included in the design to be tested module, and the data of the verification case is driven to the design to be tested module by the environment driver and executed by the design to be tested. After the design to be tested executes the verification case, the test result after the execution of the verification case can be sampled from the design to be tested module by the environment sampler. The relevant description is consistent with the relevant description of the verification system embodiment in the present disclosure, and will not be repeated here.

[0117] Step 103: compare the test result with the reference result to obtain a verification result of the design to be tested on the object to be verified.

[0118] The reference result compared with the test result of the test case in this step can be the result data already existing in the verification case, or can be the result data obtained after executing the verification case through the reference model. Among them, the reference model is a model for verifying the functions of the AI ​​chip to be verified or the design to be tested, and is responsible for simulating the logical behavior of the design to be tested DUT.

[0119] In the disclosed embodiment, the format of the verification case is predefined using a preset case template, and the verification cases generated for the objects to be verified corresponding to different categories of neural networks have a unified format, so the AI ​​chip verification method can be adapted to verify various categories of neural networks.

[0120] In some optional embodiments, the instruction control flow may be compiled to obtain a scheduling core instruction flow executable by the scheduling core in the level 2 DUT. The corresponding preset use case template includes: a scheduling core instruction flow template file, which includes a scheduling core instruction flow template for the verification use case.

[0121] Specifically, the instruction control flow can be compiled, and the scheduling core instruction flow in the verification case can be generated according to the format defined by the scheduling core instruction flow template. Then, other data other than the "scheduling core instruction flow" in the level 2 verification case can be generated according to other template files other than the "scheduling core instruction flow template file", thereby generating a complete verification case corresponding to the level 2 DUT. This method of generating a level 2 DUT verification case can reuse the module for generating a level 1 DUT verification case, reduce the workload of chip verification, and improve verification efficiency.

[0122] In some optional embodiments, the performance parameters of the verification case executed by the design under test can be obtained, and performance statistics can be performed based on the obtained performance parameters. For example, by counting the time it takes for the design under test to execute the verification case, the actual bandwidth of the design under test can be counted. The specific statistical process can refer to the description of the relevant part of the verification system embodiment, which will not be described here. The performance statistics of this embodiment can count the operation time and bandwidth of the verification case of the hierarchy of the neural network model executed by the design under test, and can also count the operation time and bandwidth of the verification case of the network level of the entire neural network model executed by the design under test, thereby enriching the performance statistics of the verification of the design under test.

[0123] The present disclosure also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the AI ​​chip verification method of any embodiment of the present disclosure when executing the program.

[0124] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the AI ​​chip verification method of any embodiment of the present disclosure.

[0125] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present disclosure is not limited thereto.

[0126] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the AI ​​chip verification method of any embodiment of the present disclosure.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the inventions claimed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not claimed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0128] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0129] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An AI chip verification system, characterized in that: The system comprises: A use case generator, configured to generate a verification case of the object to be verified based on the information of the object to be verified and a preset use case template, wherein the verification case satisfies the format defined by the preset use case template, and the object to be verified includes a target operator in a neural network model or a neural network model; The design under test module is used to execute the verification case through the design under test to obtain the test result; A comparison module, used to compare the test result with the reference result to obtain a verification result of the design to be tested on the object to be verified; The use case generator includes: a random generator, which is used to obtain a configuration file of the object to be verified, generate object parameters of the object to be verified based on the configuration file, and generate a verification case of the object to be verified based on the generated object parameters; Wherein, the configuration file includes parameters for generating a plurality of specific verification use cases; The random generator is also used to randomly generate at least one parameter among multiple parameters, and the at least one parameter includes: at least one of a source operand range, a source operand address, a target operand range, a target operand address, and a calculation parameter of an operator.

2. The system according to claim 1, characterized in that The preset use case template includes at least one of the following: An instruction flow template file, including an instruction flow template of the verification case; An initialization data template file, including an initialization data template for the verification case; The register configuration template file includes the register configuration template of the verification case.

3. The system according to any one of claims 1 to 2, characterized in that The system further comprises: An environment driver is used to drive the verification use case into the design to be tested module based on at least one template file included in the preset use case template, so that the verification use case is executed by the design to be tested.

4. The system according to claim 1, characterized in that The preset use case template includes: a result template file, including an output result template of the verification use case; The system further includes: an environment sampler, which is used to sample the test result from the design module to be tested based on the result template file, and transmit the sampled test result to the comparison module.

5. The system according to claim 1, characterized in that The use case generator includes: The directional generator is used to obtain the parameter file of the object to be verified, and generate a verification case of the object to be verified by parsing the object parameters contained in the parameter file of the object to be verified.

6. The system according to claim 5, characterized in that The parameter files include: fixed-point parameter files and network structure files.

7. The system according to any one of claims 5 to 6, characterized in that The preset use case template includes: a reference result file, including the reference result of the verification use case; The reference result obtained by the comparison module is obtained from the preset use case template.

8. The system according to claim 1, characterized in that The system further comprises: The reference model is used to execute the verification use cases generated by the use case generator to obtain corresponding reference results.

9. The system according to claim 1, characterized in that The preset use case template includes: a scheduling core instruction flow template file, including the scheduling core instruction flow template of the verification use case; The system further comprises: A scheduling instruction compiler, used for generating a scheduling core instruction stream of the design to be tested based on the scheduling core instruction stream template file; The use case generator is used to generate a verification use case for the object to be verified based on the scheduling core instruction stream generated by the scheduling instruction compiler.

10. The system according to claim 1, characterized in that The system also includes a performance statistics device connected to the design under test module, which is used to obtain performance parameters of the design under test executing the verification case and obtain performance statistics results based on the performance parameters.

11. An AI chip verification method, characterized in that: The method comprises: Based on the information of the object to be verified and the preset use case template, generating a verification case of the object to be verified, wherein the verification case satisfies the format defined by the preset use case template, and the object to be verified includes a target operator in a neural network model or a neural network model; Execute the verification case through the design to be tested to obtain the test result; Comparing the test result with the reference result to obtain a verification result of the design to be tested on the object to be verified; The step of generating a verification case of the object to be verified based on the information of the object to be verified and a preset case template includes: Acquire a configuration file of the object to be verified, generate object parameters of the object to be verified based on the configuration file, and generate a verification case of the object to be verified based on the generated object parameters; Wherein, the configuration file includes parameters for generating a plurality of specific verification use cases; The generating of the object parameters of the object to be verified based on the configuration file includes: randomly generating at least one parameter among a plurality of parameters based on the configuration file, the at least one parameter including: at least one of a source operand range, a source operand address, a target operand range, a target operand address and a calculation parameter of an operator.

12. The method according to claim 11, characterized in that The preset use case template includes at least one of the following: An instruction flow template file, including an instruction flow template of the verification case; An initialization data template file, including an initialization data template for the verification case; The register configuration template file includes the register configuration template of the verification case.

13. The method according to any one of claims 11 to 12, characterized in that After generating a verification case of the object to be verified based on the information of the object to be verified and the preset case template, the method further includes: Based on at least one template file included in the preset use case template, the verification use case is driven into the design to be tested module, so that the verification use case is executed by the design to be tested.

14. The method according to claim 11, characterized in that The preset use case template includes: a result template file, including an output result template of the verification use case; After the verification case is executed through the design to be tested to obtain the test result, the method further includes: Based on the result template file, the test result is sampled from the design module to be tested, and the sampled test result is transmitted to the comparison module.

15. The method according to claim 11, characterized in that The step of generating a verification case of the object to be verified based on the information of the object to be verified and a preset case template includes: A parameter file of the object to be verified is obtained, and a verification case of the object to be verified is generated by parsing the object parameters contained in the parameter file of the object to be verified.

16. The method according to claim 15, characterized in that The preset use case template includes: a reference result file, which contains the reference results of the verification use case.

17. The method according to claim 11, characterized in that The preset use case template includes: a scheduling core instruction flow template file, including the scheduling core instruction flow template of the verification use case; The method further comprises: Based on the scheduling core instruction stream template file, generating the scheduling core instruction stream of the design to be tested; Based on the scheduling core instruction stream, a verification case of the object to be verified is generated.

18. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 11 to 17 is implemented.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 11 to 17 are implemented.

Images