Test case, rule generation, chip test method, device, equipment and medium

Abstract

Embodiments of the present application disclose a test case, a test case rule generation method, a chip test method, a device, equipment and a medium. The test case generation method comprises: generating a plurality of candidate test cases according to an original case rule library matched with a chip to be tested; calling at least one standard operation operator provided by a golden system to execute each candidate test case and obtaining an execution result of each candidate test case; and dividing each candidate test case into a normal test case set and an abnormal test case set according to the execution result. The technical solution of the embodiments of the present application can quickly, accurately and automatically generate test cases meeting test requirements.

Description

Technical Field

[0001] The embodiments of the present invention relate to testing technology, and more particularly to a test case, test case rule generation, chip testing method, apparatus, equipment and medium. Background Technology

[0002] A test case is a set of test inputs, execution conditions, and expected results designed for a specific objective. It is used to verify whether a system under test (e.g., an application program or a data processing chip) meets a specific data processing requirement. Test cases generally include normal test cases and abnormal test cases. Normal test cases are those that are expected to execute normally, while abnormal test cases are those that are expected to produce abnormal errors.

[0003] Whether it's normal test cases or abnormal test cases, they are generally designed manually by experienced testers. Testers need to thoroughly understand the various functional modules or implementation mechanisms of the system under test in order to write test cases that meet the requirements.

[0004] In the process of realizing this invention, the inventors discovered the following defects in the prior art: test cases can only be manually generated by experienced testers who have a deep understanding of the system under test, which places high demands on testers, has high implementation costs, and requires a lot of time. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for generating test cases and test case rules, as well as chip testing methods, to quickly, accurately, and automatically generate test cases that meet testing requirements.

[0006] In a first aspect, embodiments of the present invention provide a method for generating test cases, the method comprising:

[0007] Based on the original test case rule base that matches the chip under test, multiple alternative test cases are generated;

[0008] Call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution results of each alternative test case;

[0009] Based on the execution results, each candidate test case is divided into a normal test case set and an abnormal test case set.

[0010] Secondly, embodiments of the present invention also provide a method for generating test case rules, the method comprising:

[0011] Based on the original test case rule base that matches the chip under test, multiple alternative test cases are generated;

[0012] Call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution results of each alternative test case;

[0013] Based on the execution results, each candidate test case is divided into a normal test case set and an abnormal test case set;

[0014] Generate a normal test case rule base and an abnormal test case rule base based on the normal test case set and / or the abnormal test case set.

[0015] Thirdly, embodiments of the present invention also provide a chip testing method, which includes:

[0016] Obtain the target test cases;

[0017] Call at least one standard operation operator provided by the Golden system to execute the target test case and obtain the first execution result of the target test case;

[0018] Invoke at least one hardware execution unit included in the chip under test to execute the target test case, and obtain the second execution result of the target test case;

[0019] If the first execution result and the second execution result are inconsistent, the chip under test is determined to have failed the test.

[0020] Fourthly, embodiments of the present invention also provide a test case generation apparatus, which includes:

[0021] The alternative test case generation module is used to generate multiple alternative test cases based on the original test case rule library that matches the chip under test.

[0022] The execution result acquisition module is used to call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution result of each alternative test case;

[0023] The test case partitioning module is used to divide each candidate test case into a normal test case set and an abnormal test case set based on the execution results.

[0024] Fifthly, embodiments of the present invention also provide a test case rule generation apparatus, which includes:

[0025] The alternative test case generation module is used to generate multiple alternative test cases based on the original test case rule library that matches the chip under test.

[0026] The execution result acquisition module is used to call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution result of each alternative test case;

[0027] The test case partitioning module is used to divide each candidate test case into a normal test case set and an abnormal test case set based on the execution results.

[0028] The rule base generation module is used to generate normal test case rule bases and abnormal test case rule bases based on normal test case sets and / or abnormal test case sets.

[0029] Sixthly, embodiments of the present invention also provide a chip testing apparatus, the chip testing apparatus comprising:

[0030] The target test case acquisition module is used to acquire target test cases;

[0031] The first execution result acquisition module is used to call at least one standard operation operator provided by the Golden system to execute the target test case and obtain the first execution result of the target test case;

[0032] The second execution result acquisition module is used to call at least one hardware execution unit included in the chip under test to execute the target test case and obtain the second execution result of the target test case;

[0033] The test verification module is used to determine that the chip under test has failed the test if the first execution result and the second execution result are inconsistent.

[0034] In a seventh aspect, embodiments of the present invention also provide a computer device, the computer device comprising:

[0035] One or more processors;

[0036] Storage device for storing one or more programs.

[0037] When one or more programs are executed by one or more processors, the one or more processors implement a test case generation method as described in any of the embodiments of the present invention, or implement a test case rule generation method as described in any of the embodiments of the present invention, or implement a chip testing method as described in any of the embodiments of the present invention.

[0038] Eighthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a test case generation method as described in any of the embodiments of the present invention, or implements a test case rule generation method as described in any of the embodiments of the present invention, or implements a chip testing method as described in any of the embodiments of the present invention.

[0039] The technical solution of this invention uses the standard arithmetic operators in the Golden system as the correctness verification standard for the chip under test. Based on the execution results of the test cases in the Golden system, and using only a roughly constructed original test case rule base, it can quickly, accurately, and automatically generate normal test cases, abnormal test cases, and test case rules that meet the testing requirements. This allows for effective usability verification of the chip under test, significantly reducing the manpower and time costs required to construct test cases and improving the testing efficiency of the chip under test. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the implementation of a test case generation method according to Embodiment 1 of the present invention.

[0041] Figure 2 This is a flowchart illustrating the implementation of a test case rule generation method according to Embodiment 2 of the present invention.

[0042] Figure 3 This is a flowchart illustrating the implementation of a test case rule generation method in Embodiment 3 of the present invention.

[0043] Figure 4 This is a flowchart illustrating the implementation of a chip testing method according to Embodiment 4 of the present invention;

[0044] Figure 5 This is a structural diagram of a test case generation device according to Embodiment 5 of the present invention;

[0045] Figure 6 This is a structural diagram of a test case rule generation device according to Embodiment Six of the present invention;

[0046] Figure 7 This is a structural diagram of a chip testing device according to Embodiment 7 of the present invention;

[0047] Figure 8 This is a structural diagram of a computer device according to Embodiment 8 of the present invention. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0049] It should also be noted that, for ease of description, the accompanying drawings show only the parts relevant to the invention and not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. The process may correspond to a method, function, procedure, subroutine, subprogram, etc.

[0050] Example 1

[0051] Figure 1 This is a flowchart illustrating the implementation of a test case generation method according to Embodiment 1 of the present invention. This embodiment is applicable to situations where a roughly configured initial test case rule base is used to quickly, accurately, and automatically generate test cases that meet testing requirements. This method can be executed by a test case generation device, which can be implemented in software and / or hardware, and is generally integrated into various computer devices with data processing capabilities (e.g., desktop computers, servers, or laptops). The method of this embodiment specifically includes the following steps:

[0052] S110. Generate multiple alternative test cases based on the original test case rule base that matches the chip under test.

[0053] In this embodiment, the chip under test refers to a hardware chip that needs to be tested for usability using test cases. The chip under test can execute test cases and obtain matching execution results.

[0054] Optionally, the chip under test may include one or more hardware execution units, which are used to perform set data processing and calculation operations in hardware. The data type may include: numerical values, vectors, arrays, matrices, or images, etc., and this embodiment does not impose any limitations on this.

[0055] Optionally, the data processing and calculation operations that the hardware execution unit can perform can be: operations between at least two numerical values, such as addition, subtraction, multiplication, or division; operations between at least two matrices, such as matrix addition, matrix multiplication, or convolution between matrices; operations on the same numerical value or matrix, such as squaring, square rooting, matrix transpose, or finding the inverse of a matrix; or image processing operations on a specified image, such as image flipping, image cropping, or image format conversion.

[0056] To test the aforementioned chip under test, test cases (normal test cases and abnormal test cases) that meet the testing requirements of the chip under test need to be constructed. In this embodiment, testers do not need to deeply study and master the implementation mechanism and principles of the chip under test to design very precise test cases; only a relatively broad and coarse original test case rule base needs to be set up.

[0057] The original use case rule base stores one or more original use case rules. These rules may include operators, the range of data values ​​corresponding to the operators, and the data types (e.g., integer, long integer, or floating-point) that match the operators. The operators included in the original use case rules are matched with the hardware execution units in the chip under test. Optionally, different data types (numerical, matrix, or array, etc.) correspond to different operators.

[0058] Optionally, operators may include: square root operator, the data type corresponding to the square root operator can be floating point or integer, and the range of data values ​​corresponding to the square root operator can be [-∞, +∞].

[0059] Understandably, it's sufficient to broadly set matching data value ranges and data types based solely on the physical meaning of the operators, without guaranteeing that the original test case rules are the optimal rules for the hardware execution unit in the chip under test. Specifically, considering the need to construct multiple normal test cases and multiple abnormal test cases to test the chip under test, it's unnecessary to guarantee that the original test case rules necessarily satisfy the physical meaning of the operators. For example, the square root must be positive, the denominator of the two divided numbers cannot be 0, or the row and column values ​​of the two multiplied matrices must satisfy a set numerical relationship. This ensures the diversity and completeness of the alternative test cases constructed from the original test case rule base.

[0060] In this embodiment, one or more matching original test cases can be obtained from the original test case rule library by setting the original test case rule combination method, and multiple matching alternative test cases can be constructed based on the obtained original test case rules.

[0061] Alternative test cases can be normal or abnormal test cases used to test the chip under test. In a specific example, an alternative test case consists of one or more specific data and one or more operators forming a calculation formula. For example: 9 + 7 - 6, or matrix A * matrix B, etc.; it can also be an input image with a specified size or format, etc.

[0062] Optionally, multiple alternative test cases that match the original test case rule base can be generated using a pre-defined test case generator.

[0063] S120. Call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution results of each alternative test case.

[0064] In this embodiment, because the role of humans in the test case design process is reduced, it is necessary to predetermine the type of test case (normal test case or abnormal test case) or the execution result of the test case (normal execution or abnormal error) through other means. Based on this, the inventors creatively propose using the Golden system as a verification standard for the correctness of the test case execution result.

[0065] Optionally, the alternative test cases include at least one operator, and the chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system, and thus the standard arithmetic operators correspond one-to-one with the hardware execution units of the chip under test.

[0066] Understandably, widely used deep learning models (also known as neural networks) and frameworks offer a variety of standard computation operators, such as numerical operations, vector operations, matrix operations, convolution operations, and various image processing operations. These standard operators can run on a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Therefore, these neural networks or learning frameworks can be used as a Golden system to execute various alternative test cases.

[0067] Depending on the specific application or the operators required by the chip, relevant networks or learning frameworks adopted in the industry can be selected, such as TensorFlow, Caffe, PyTorch, or Keras within the Golden system. As a Golden system, the selected Golden system includes reference implementations of various required operators. Because Golden systems that implement these operators are relatively mature and stable, they can serve as a standard for verifying the correctness of test case execution results.

[0068] Of course, it is understood that the above examples of specific forms of the Golden system are merely illustrative. In fact, those skilled in the art can also use other types of neural networks or learning frameworks adopted in the industry, and this embodiment does not limit them.

[0069] As mentioned earlier, since at least one standard operator provided by the Golden system matches the operator used in the alternative test cases, each alternative test case can be executed by calling the standard operator in the Golden system that matches the operator in the alternative test cases, and the execution result of each alternative test case can be obtained.

[0070] Specifically, if the Golden system can execute a candidate test case normally and obtain the corresponding test case result data, then the execution result of the candidate test case is determined to be normal. If the Golden system cannot execute a candidate test case normally and outputs one or more abnormal errors, then the execution result of the candidate test case is determined to be abnormal.

[0071] S130. Based on the execution results, divide each candidate test case into a normal test case set and an abnormal test case set.

[0072] In this embodiment, if the execution result of a candidate test case is normal, the candidate test case can be assigned to the normal test case set; if the execution result of a candidate test case is abnormal, the candidate test case can be assigned to the abnormal test case set.

[0073] Correspondingly, by using normal test case sets and abnormal test case sets respectively, the chip under test can be effectively tested, and then based on the test results, the various hardware execution modules in the chip under test can be effectively modified or optimized.

[0074] The technical solution of this invention uses the standard arithmetic operators in the Golden system as the correctness verification standard for the chip under test. Based on the execution results of the test cases in the Golden system, and using only a roughly constructed original test case rule base, it can quickly, accurately, and automatically generate normal and abnormal test cases that meet the testing requirements, thereby enabling effective usability verification of the chip under test. This significantly reduces the manpower and time costs required to construct test cases and improves the testing efficiency of the chip under test.

[0075] In one optional embodiment of this example, generating alternative test cases based on the original test case rule base may include:

[0076] Obtain at least one test case template to be populated, the test case template including at least one operator to be populated and at least one data item to be populated;

[0077] Obtain at least one operator from the original use case rule base as the target operator;

[0078] Construct the target data item based on at least one data value range and at least one data type corresponding to the target operator included in the original use case rule base;

[0079] Fill the target operator and target data item into each test case module to generate each alternative test case.

[0080] The rules stored in the original test case rule base can include: operators, data value ranges, and data types. Correspondingly, a mapping relationship can be established between data types (numerical, matrix, or array, etc.) and operators in the original test case rule base, and separate mapping relationships can be established between operators and data value ranges and data types. Based on the above, various alternative test cases can be constructed.

[0081] In a specific example, a test case template includes two numeric operators to be filled and three data items to be filled, each containing a single numeric value. Correspondingly, multiple sets of numeric operators can be obtained from the original test case rule base. Each set of numeric operators includes two data operators, for example: Numeric operator set 1 {"+", "*"}, Numeric operator set 2 {"-", " / "}, etc.

[0082] Simultaneously, for each target operator included in the set of numeric operators, corresponding target data items can be constructed using at least one matching data value range and at least one matching data type from the original use case rule base. Specifically, the corresponding target data items can be determined based on the data value range by random selection or selection according to a set rule order. For example, for set 1 of numeric operators, the generated target data item sets are {5.34, 6.23, 7.12} and {-7.33, 40.00, 5.03}, and for set 2 of numeric operators, the generated target data item sets are {7, -5, 0} and {-9, 1368, -2230}.

[0083] Accordingly, based on the target operator and the target data item, the alternative test cases obtained by the test case filling module can be: alternative test case 1: 5.34+6.23*7.12, alternative test case 2: (-7.33)+40.00*5.03, alternative test case 3: 7-(-5) / 0 and alternative test case 4: (-9)-1368 / (-2230).

[0084] Of course, those skilled in the art will understand that alternative test cases can be constructed in other ways; this is merely an example of one possible implementation.

[0085] The advantage of this setup is that it can automatically generate comprehensive and complete alternative test cases based on the original test case rules included in the original test case rule base, saving a lot of manpower costs and ensuring the diversity of alternative test cases in the alternative test case set.

[0086] Example 2

[0087] Figure 2 This is a flowchart illustrating the implementation of a test case rule generation method according to Embodiment 2 of the present invention. This embodiment is applicable to situations where a roughly configured initial test case rule library is used to quickly, accurately, and automatically generate test case rules that meet testing requirements. This method can be executed by a test case rule generation device, which can be implemented in software and / or hardware and is generally integrated into various computer devices with data processing capabilities (e.g., desktop computers, servers, or laptops). Accordingly, the method of this embodiment specifically includes the following steps:

[0088] S210. Generate multiple alternative test cases based on the original test case rule base that matches the chip under test.

[0089] S220. Call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution results of each alternative test case.

[0090] Optionally, the alternative test cases include at least one operator, and the chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system.

[0091] S230. Based on the execution results, divide each candidate test case into a normal test case set and an abnormal test case set.

[0092] S240. Generate a normal test case rule base and an abnormal test case rule base based on the normal test case set and / or the abnormal test case set.

[0093] In this embodiment, based on the execution results of each candidate test case output by the Golden system, the candidate test cases generated through the original test case rule base can be divided into a set of normal test cases and a set of abnormal test cases, thereby enabling the testing of the chip under test. However, if the chip under test needs to be tested using new normal or abnormal test cases, the operations from S210 to S230 must be executed again to obtain them, which is a rather cumbersome process.

[0094] In this embodiment, the inventors creatively propose to construct a normal test case rule base for generating normal test cases and an abnormal test case rule base for generating abnormal test cases, based on the normal test case set, abnormal test case set, or combination of normal test case set and abnormal test case set already determined using the Golden system. This simplifies the steps for generating normal test cases and abnormal test cases and improves the efficiency of generating normal test cases and abnormal test cases.

[0095] Similar to the original use case rule base, the normal use case rule base can include: operators, the range of data values ​​that match the operators, and data types; the abnormal use case rule base can include: operators, the range of data values ​​that match the operators, and data types.

[0096] Optionally, based on the normal test case set and / or the abnormal test case set, a normal test case rule base and an abnormal test case rule base can be generated, which may include:

[0097] Generate a normal test case rule base based on the normal test case set, and generate an abnormal test case rule base based on the abnormal test case set; or

[0098] Based on the abnormal test case set, an abnormal test case rule base is generated, and based on the original test case rule base and the abnormal test case rule base, a normal test case rule base is generated.

[0099] Specifically, matching normal test case rule bases and abnormal test case rule bases can be generated separately based on normal test case sets and abnormal test case sets. Alternatively, after generating an abnormal test case rule base based on the abnormal test case set, the complement of the abnormal test case rule base in the original test case rule base can be calculated as the normal test case rule base.

[0100] Of course, you can also generate a normal test case rule base based on the normal test case set, and then calculate the complement of the normal test case rule base in the original test case rule base to serve as the abnormal test case rule base.

[0101] However, considering that the proportion of abnormal test case rules in the entire original test case rule base should be very small, that is, the normal test case rule base generally includes far more normal test case rules than the abnormal test case rule base, the abnormal test case rule base can be generated first based on the abnormal test case set.

[0102] Optionally, a rule generator can be pre-constructed, and each abnormal test case included in the abnormal test case set can be input into the rule generator to obtain each abnormal test case rule. Then, an abnormal test case rule library can be constructed based on each abnormal test case rule. Alternatively, the abnormal test case rule library can be constructed manually or semi-manually, etc. This embodiment does not limit this.

[0103] The technical solution of this invention uses the standard arithmetic operators in the Golden system as the correctness verification standard for the chip under test. Based on the execution results of the test cases in the Golden system, and using only a roughly constructed original test case rule base, it can quickly, accurately, and automatically generate a normal test case rule base and an abnormal test case rule base that meet the testing requirements. Furthermore, it can generate a large number of normal and abnormal test cases through these rule bases to effectively verify the usability of the chip under test. This significantly reduces the manpower and time costs required to construct test cases and improves the testing efficiency of the chip under test.

[0104] Based on the above embodiments, after generating a normal test case rule base and an abnormal test case rule base according to the normal test case set and / or the abnormal test case set, it may further include:

[0105] Generate at least one normal test case based on the normal test case rule base, and / or generate at least one abnormal test case based on the abnormal test case rule base.

[0106] The advantage of this setup is that there is no need to redetermine normal and abnormal test cases from the newly generated alternative test cases using the Golden system. Normal and abnormal test cases that meet the requirements can be generated simply by using the generated normal and abnormal test case rule bases. This further reduces the manpower and time costs required to construct test cases and improves the testing efficiency of the chip under test.

[0107] Example 3

[0108] Figure 3This is a flowchart illustrating the implementation of a test case rule generation method according to Embodiment 3 of the present invention. This embodiment further refines the above embodiment. In this embodiment, the operation of generating an abnormal test case rule library based on an abnormal test case set is set as follows: based on the test case parameter information of each abnormal test case, the abnormal test cases are clustered to obtain multiple clusters; based on the test case parameter information of each abnormal test case in each cluster, a system of linear equations corresponding to each cluster is constructed, wherein the linear equations include at least one abnormal test case rule to be solved; the system of linear equations corresponding to each cluster is solved to obtain matching abnormal test case rules; and an abnormal test case rule library is generated based on the solved abnormal test case rules.

[0109] Accordingly, the method of this embodiment of the invention specifically includes the following steps:

[0110] S310. Generate multiple alternative test cases based on the original test case rule base that matches the chip under test.

[0111] S320. Call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution results of each alternative test case.

[0112] Optionally, the alternative test cases include at least one operator, and the chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system.

[0113] S330. Based on the execution results, divide each candidate test case into a normal test case set and an abnormal test case set.

[0114] S340. Based on the test case parameter information of each abnormal test case, cluster the abnormal test cases to obtain multiple clusters.

[0115] The user parameter information can include the operator and the specific values ​​of the data. Accordingly, the abnormal test cases can be clustered according to the type of the operator or the type of the data to ensure that the abnormal test cases included in the same cluster are of the same type.

[0116] Optionally, all abnormal test cases that include a specific operator (e.g., matrix transpose operator) can be grouped into the same cluster, or all abnormal test cases that correspond to the same data type (numerical, matrix, or array, etc.) can be grouped into the same cluster. This embodiment does not impose any restrictions on this.

[0117] S350. Based on the test case parameter information of each abnormal test case in each cluster, construct a system of linear equations corresponding to each cluster.

[0118] The linear relationship equation includes at least one exception use case rule to be solved.

[0119] In this embodiment, multiple abnormal use case rules to be solved can be constructed, and each abnormal use case rule may include one or more unknowns.

[0120] S360. Solve the linear equations corresponding to each cluster to obtain the matching exception use case rules.

[0121] Generally, the number of abnormal test cases far exceeds the number of variables contained in the abnormal test case rules to be solved. Therefore, the linear equation system obtained for each cluster is usually an overdetermined system, and multiple sets of possible abnormal test case rules can be obtained for each linear equation system. After obtaining multiple sets of possible abnormal test case rules, the optimal solution can be found through a certain fitting method to obtain one or more abnormal test case rules that match each cluster.

[0122] S370. Generate an exception use case rule library based on the exception use case rules obtained from the solution.

[0123] S380. Generate a normal test case rule base based on the normal test case set.

[0124] In this embodiment, the method for generating a normal test case rule base based on a normal test case set is similar to the method for generating an abnormal test case rule base based on an abnormal test case set. For example, it could be:

[0125] Based on the test case parameter information of each normal test case, the normal test cases are clustered to obtain multiple clusters. Based on the test case parameter information of each normal test case in each cluster, a system of linear equations corresponding to each cluster is constructed. Each linear equation includes at least one normal test case rule to be solved. The system of linear equations corresponding to each cluster is solved to obtain matching normal test case rules. Based on the solved normal test case rules, a normal test case rule base is generated.

[0126] The technical solution of this invention is based on the normal test case set, abnormal test case set, or combination of normal test case set and abnormal test case set determined by the Golden system. It automatically constructs a linear equation system to solve the abnormal test case rules and normal test case rules, and constructs the corresponding abnormal test case rule library and normal test case rule library. This can simplify the generation steps of normal test cases and abnormal test cases and improve the generation efficiency of normal test cases and abnormal test cases.

[0127] Of course, it is understandable that, in addition to solving abnormal use case rules and normal use case rules by constructing a system of linear equations, other methods can be used to generate abnormal use case rules and normal use case rules. For example, abnormal use case rules and normal use case rules can be learned by using a pre-constructed abnormal rule generation model or normal rule generation model. This embodiment does not limit this.

[0128] Example 4

[0129] Figure 4 This is a flowchart illustrating the implementation of a chip testing method according to Embodiment 4 of the present invention. This embodiment is applicable to situations where the Golden system is used as the correctness verification standard to verify the chip under test. The method can be executed by a chip testing device, which can be implemented in software and / or hardware, and is generally integrated into various computer devices with data processing capabilities (e.g., desktop computers, servers, or laptops). Accordingly, the method of this embodiment specifically includes the following steps:

[0130] S410. Obtain the target test cases.

[0131] The target test case can be a normal test case, an abnormal test case, or a test case with an unknown execution result. The target test case can be manually constructed, obtained from the normal test case set or abnormal test case set obtained through any embodiment of the present invention, or automatically generated from the normal test case rule base or abnormal test case rule base obtained through any embodiment of the present invention; no restrictions are imposed here.

[0132] Optionally, the target test case includes at least one operator, and the chip under test includes a hardware execution unit that matches the operator.

[0133] S420: Call at least one standard operation operator provided by the Golden system to execute the target test case and obtain the first execution result of the target test case.

[0134] Optionally, standard operators are matched with other operators.

[0135] As mentioned above, in the various embodiments of the present invention, standard operators in the Golden system are used as the correctness verification standard for the execution results of test cases. Furthermore, each candidate test case can be executed by calling the standard operators in the Golden system that match the operators in the candidate test cases, and the execution results of each candidate test case can be obtained.

[0136] The first execution result can be normal or abnormal. When the first execution result of the target test case is normal, it means that the target test case is executed normally and the corresponding test case result data is output. When the first execution result of the target test case is abnormal, it means that the target test case has generated an abnormal error during execution and cannot output the corresponding test case result data.

[0137] S430: Call at least one hardware execution unit included in the chip under test to execute the target test case and obtain the second execution result of the target test case.

[0138] The second execution result can be normal or abnormal. When the second execution result of the target test case is normal, it means that the target test case was executed normally and the corresponding test case result data was output. When the second execution result of the target test case is abnormal, it means that the target test case encountered an abnormal error during execution and could not output the corresponding test case result data.

[0139] S440. If the first execution result and the second execution result are inconsistent, it is determined that the chip under test has failed the test.

[0140] In this embodiment, if the first execution result and the second execution result are inconsistent—that is, the first execution result is normal while the second execution result is abnormal, or the first execution result is abnormal while the second execution result is normal—it indicates that the chip under test cannot achieve the expected effect (it does not conform to the correctness verification standard of the standard arithmetic operator in the Golden system), and thus it is determined that the chip under test has failed the test.

[0141] Furthermore, if the first and second execution results are consistent across a large number of target test cases, it can be concluded that each hardware execution unit of the chip under test can achieve the same effect as the standard arithmetic operators implemented by various software in the Golden system, and the chip under test passes the test.

[0142] The technical solution of this invention uses standard arithmetic operators in the Golden system as the correctness verification standard for test case execution results. It does not require pre-determining whether the test case type is normal or abnormal. As long as the execution results of the chip under test and the Golden system for the same target test case are consistent, it can quickly determine whether the chip under test has passed the verification. This provides a new way to test the chip under test and improves the testing efficiency of the chip under test.

[0143] Example 5

[0144] Figure 5 This is a structural diagram of a test case generation device provided in Embodiment 5 of the present invention, as shown below. Figure 5As shown, the device includes: a candidate test case generation module 510, an execution result acquisition module 520, and a test case partitioning module 530. Wherein:

[0145] The alternative test case generation module 510 is used to generate multiple alternative test cases based on the original test case rule library that matches the chip under test.

[0146] The execution result acquisition module 520 is used to call at least one standard operation operator provided by the golden system to execute each alternative test case and obtain the execution result of each alternative test case;

[0147] The test case partitioning module 530 is used to divide each candidate test case into a normal test case set and an abnormal test case set based on the execution results.

[0148] The technical solution of this invention uses the standard arithmetic operators in the Golden system as the correctness verification standard for the chip under test. Based on the execution results of the test cases in the Golden system, and using only a roughly constructed original test case rule base, it can quickly, accurately, and automatically generate normal and abnormal test cases that meet the testing requirements, thereby enabling effective usability verification of the chip under test. This significantly reduces the manpower and time costs required to construct test cases and improves the testing efficiency of the chip under test.

[0149] Based on the above embodiments, the alternative test cases include at least one operator, and the chip under test includes a hardware execution unit matching the operator; the operator matches the standard arithmetic operators provided by the Golden system. Based on the above embodiments, the alternative test case generation module 510 can specifically be used for:

[0150] Obtain at least one test case template to be populated, the test case template including at least one operator to be populated and at least one data item to be populated;

[0151] Obtain at least one operator from the original use case rule base as the target operator;

[0152] Construct the target data item based on at least one data value range and at least one data type corresponding to the target operator included in the original use case rule base;

[0153] Fill the target operator and target data item into each test case module to generate each alternative test case.

[0154] Based on the above embodiments, the Golden system may include: TensorFlow, Caffe, PyTorch, or Keras.

[0155] The test case generation device provided in this embodiment of the invention can execute the test case generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0156] Example 6

[0157] Figure 6 This is a structural diagram of a test case rule generation device provided in Embodiment Six of the present invention, as shown in the figure. Figure 6 As shown, the device includes: a candidate test case generation module 610, an execution result acquisition module 620, a test case partitioning module 630, and a rule base generation module 640. Wherein:

[0158] The alternative test case generation module 610 is used to generate multiple alternative test cases based on the original test case rule library that matches the chip under test.

[0159] The execution result acquisition module 620 is used to call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution result of each alternative test case;

[0160] The test case partitioning module 630 is used to divide each candidate test case into a normal test case set and an abnormal test case set based on the execution results.

[0161] The rule base generation module 640 is used to generate normal test case rule bases and abnormal test case rule bases based on normal test case sets and / or abnormal test case sets.

[0162] The technical solution of this invention uses the standard arithmetic operators in the Golden system as the correctness verification standard for the chip under test. Based on the execution results of the test cases in the Golden system, and using only a roughly constructed original test case rule base, it can quickly, accurately, and automatically generate a normal test case rule base and an abnormal test case rule base that meet the testing requirements. Furthermore, it can generate a large number of normal and abnormal test cases through these rule bases to effectively verify the usability of the chip under test. This significantly reduces the manpower and time costs required to construct test cases and improves the testing efficiency of the chip under test.

[0163] Based on the above embodiments, the rule base generation module 640 may specifically include:

[0164] The first rule base generation unit is used to generate a normal test case rule base based on the normal test case set, and to generate an abnormal test case rule base based on the abnormal test case set; or the second rule base generation unit is used to generate an abnormal test case rule base based on the abnormal test case set, and to generate a normal test case rule base based on the original test case rule base and the abnormal test case rule base.

[0165] Based on the above embodiments, the first rule base generation unit or the second rule base generation unit can be specifically used for:

[0166] Based on the test case parameter information of each abnormal test case, the abnormal test cases are clustered to obtain multiple clusters. Based on the test case parameter information of each abnormal test case in each cluster, a system of linear equations corresponding to each cluster is constructed. The linear equations include at least one abnormal test case rule to be solved. The system of linear equations corresponding to each cluster is solved to obtain the matching abnormal test case rules. Based on the solved abnormal test case rules, an abnormal test case rule base is generated.

[0167] Based on the above embodiments, it may further include: a test case generation module, used for:

[0168] After generating the normal test case rule base and the abnormal test case rule base based on the normal test case set and / or the abnormal test case set, the method further includes: generating at least one normal test case based on the normal test case rule base, and / or generating at least one abnormal test case based on the abnormal test case rule base.

[0169] The test case rule generation device provided in this embodiment of the invention can execute the test case rule generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0170] Example 7

[0171] Figure 7 This is a structural diagram of a chip testing device provided in Embodiment 7 of the present invention, as shown below. Figure 7 As shown, the device includes: a target test case acquisition module 710, a first execution result acquisition module 720, a second execution result acquisition module 730, and a test verification module 740. Wherein:

[0172] The target test case acquisition module 710 is used to acquire target test cases;

[0173] The first execution result acquisition module 720 is used to call at least one standard operation operator provided by the Golden system to execute the target test case and obtain the first execution result of the target test case;

[0174] The second execution result acquisition module 730 is used to call at least one hardware execution unit included in the chip under test to execute the target test case and acquire the second execution result of the target test case;

[0175] The test verification module 740 is used to determine that the chip under test has failed the test if the first execution result and the second execution result are inconsistent.

[0176] The technical solution of this invention uses standard arithmetic operators in the Golden system as the correctness verification standard for test case execution results. It does not require pre-determining whether the test case type is normal or abnormal. As long as the execution results of the chip under test and the Golden system for the same target test case are consistent, it can quickly determine whether the chip under test has passed the verification. This provides a new way to test the chip under test and improves the testing efficiency of the chip under test.

[0177] The chip testing apparatus provided in the embodiments of the present invention can execute the chip testing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0178] Example 8

[0179] Figure 8 This is a schematic diagram of the structure of a computer device provided in Embodiment 8 of the present invention, as shown below. Figure 8 As shown, the computer device includes a processor 80, a memory 81, an input device 82, and an output device 83; the number of processors 80 in the computer device can be one or more. Figure 8 Taking a processor 80 as an example; the processor 80, memory 81, input device 82, and output device 83 in a computer device can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0180] The memory 81, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the module corresponding to the test case generation method in the embodiments of the present invention, or the module corresponding to the test case rule generation method in the embodiments of the present invention, or the module corresponding to the chip testing method in the embodiments of the present invention. The processor 80 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 81, thereby implementing the test case generation method as described in any embodiment of the present invention. This method includes:

[0181] Based on the original test case rule base that matches the chip under test, generate multiple alternative test cases; call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution result of each alternative test case; based on the execution result, divide each alternative test case into a normal test case set and an abnormal test case set.

[0182] Alternatively, a method for generating test case rules as described in any embodiment of the present invention may be implemented. This method includes:

[0183] Based on the original test case rule base that matches the chip under test, generate multiple alternative test cases; call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution result of each alternative test case; based on the execution result, divide each alternative test case into a normal test case set and an abnormal test case set; based on the normal test case set and / or the abnormal test case set, generate a normal test case rule base and an abnormal test case rule base.

[0184] Alternatively, a chip testing method as described in any embodiment of the present invention may be implemented. This method includes:

[0185] Obtain the target test case; call at least one standard operation operator provided by the Golden system to execute the target test case and obtain the first execution result of the target test case; call at least one hardware execution unit included in the chip under test to execute the target test case and obtain the second execution result of the target test case; if the first execution result and the second execution result are inconsistent, it is determined that the chip under test has failed the test.

[0186] The memory 81 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 81 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 81 may further include memory remotely located relative to the processor 80, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] Input device 82 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the computer device. Output device 83 may include display devices such as a display screen.

[0188] Example 9

[0189] Embodiment 9 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a method for generating test cases as described in any embodiment of the present invention. The method includes:

[0190] Based on the original test case rule base that matches the chip under test, generate multiple alternative test cases; call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution result of each alternative test case; based on the execution result, divide each alternative test case into a normal test case set and an abnormal test case set.

[0191] Alternatively, a method for generating test case rules as described in any embodiment of the present invention may be performed, the method comprising:

[0192] Based on the original test case rule base that matches the chip under test, generate multiple alternative test cases; call at least one standard operation operator provided by the Golden system to execute each alternative test case and obtain the execution result of each alternative test case; based on the execution result, divide each alternative test case into a normal test case set and an abnormal test case set; based on the normal test case set and / or the abnormal test case set, generate a normal test case rule base and an abnormal test case rule base.

[0193] Alternatively, a chip testing method as described in any embodiment of the present invention may be performed, the method comprising:

[0194] Obtain the target test case; call at least one standard operation operator provided by the Golden system to execute the target test case and obtain the first execution result of the target test case; call at least one hardware execution unit included in the chip under test to execute the target test case and obtain the second execution result of the target test case; if the first execution result and the second execution result are inconsistent, it is determined that the chip under test has failed the test.

[0195] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the above-described method operations, but can also perform related operations in the methods provided in any embodiment of the present invention.

[0196] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0197] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for generating test cases, characterized in that, include: Based on the original test case rule base that matches the chip under test, multiple alternative test cases are generated. The original test case rule base stores one or more original test case rules. The original test case rules include operators, the data value range corresponding to the operators, and the data type that matches the operators. Call at least one standard operation operator provided by the Golden system to execute each of the candidate test cases and obtain the execution results of each of the candidate test cases. The standard operation operator includes at least one of numerical operation, vector operation, matrix operation, and convolution operation. Based on the execution results, each of the candidate test cases is divided into a normal test case set and an abnormal test case set; The alternative test cases include at least one operator, and the chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system.

2. The method according to claim 1, characterized in that, Based on the original test case rule base that matches the chip under test, alternative test cases are generated, including: Obtain at least one test case template to be populated, wherein the test case template includes at least one operator to be populated and at least one data item to be populated; Obtain at least one operator as the target operator from the original use case rule base; Construct a target data item based on at least one data value range and at least one data type included in the original use case rule base that correspond to the target operator; The target operator and the target data item are filled into each of the test case modules to generate each of the alternative test cases.

3. The method according to claim 1, characterized in that, The golden system includes: TensorFlow, Caffe, PyTorch, or Keras.

4. A method for generating test case rules, characterized in that, include: Based on the original test case rule base that matches the chip under test, multiple alternative test cases are generated. The original test case rule base stores one or more original test case rules. The original test case rules include operators, the data value range corresponding to the operators, and the data type that matches the operators. Call at least one standard operation operator provided by the Golden system to execute each of the candidate test cases and obtain the execution results of each of the candidate test cases. The standard operation operator includes at least one of numerical operation, vector operation, matrix operation, and convolution operation. Based on the execution results, each of the candidate test cases is divided into a normal test case set and an abnormal test case set; Based on the normal test case set and / or the abnormal test case set, generate a normal test case rule base and an abnormal test case rule base; The alternative test cases include at least one operator, and the chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system.

5. The method according to claim 4, characterized in that, Based on the normal test case set and / or the abnormal test case set, generate a normal test case rule base and an abnormal test case rule base, including: Based on the normal test case set, generate a normal test case rule base, and based on the abnormal test case set, generate an abnormal test case rule base; or Based on the abnormal test case set, an abnormal test case rule base is generated, and based on the original test case rule base and the abnormal test case rule base, the normal test case rule base is generated.

6. The method according to claim 5, characterized in that, Based on the aforementioned set of abnormal test cases, an abnormal test case rule base is generated, including: Based on the test case parameter information of each abnormal test case, the abnormal test cases are clustered to obtain multiple clusters; Based on the test case parameter information of each abnormal test case in each cluster, construct a system of linear equations corresponding to each cluster. The linear equations include at least one abnormal test case rule to be solved. Solve the linear equations corresponding to each cluster to obtain matching exception use case rules; An exception use case rule library is generated based on the obtained exception use case rules.

7. The method according to any one of claims 4-6, characterized in that, After generating the normal test case rule base and the abnormal test case rule base based on the normal test case set and / or the abnormal test case set, the method further includes: Based on the normal test case rule base, generate at least one normal test case, and / or, based on the abnormal test case rule base, generate at least one abnormal test case.

8. A chip testing method, characterized in that, include: Obtain target test cases, wherein the target test cases include at least one operator, and the target test cases are obtained from a normal test case set or an abnormal test case set, wherein the normal test case set or the abnormal test case set is obtained by the method described in any one of claims 1-3; or, the target test cases are automatically generated based on a normal test case rule base or an abnormal test case rule base, wherein the normal test case rule base or the abnormal test case rule base is obtained by the method described in any one of claims 4-7. The system invokes at least one standard operation operator provided by the Golden system to execute the target test case and obtains the first execution result of the target test case. The standard operation operator includes at least one of numerical operation, vector operation, matrix operation, and convolution operation. The target test case is executed by calling at least one hardware execution unit included in the chip under test, and a second execution result of the target test case is obtained; If the first execution result and the second execution result are inconsistent, the chip under test is determined to have failed the test. The chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system.

9. A test case generation device, characterized in that, include: The alternative test case generation module is used to generate multiple alternative test cases based on the original test case rule library that matches the chip under test. Each alternative test case includes at least one operator. The original test case rule library stores one or more original test case rules. The original test case rules include operators, the data value range corresponding to the operators, and the data type that matches the operators. The execution result acquisition module is used to call at least one standard operation operator provided by the Golden system to execute each of the candidate test cases and obtain the execution result of each of the candidate test cases. The standard operation operator includes at least one of numerical operation, vector operation, matrix operation, and convolution operation. The test case partitioning module is used to partition each of the candidate test cases into a normal test case set and an abnormal test case set based on the execution results. The alternative test cases include at least one operator, and the chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system.

10. A test case rule generation device, characterized in that, include: The alternative test case generation module is used to generate multiple alternative test cases based on the original test case rule library that matches the chip under test. The original test case rule library stores one or more original test case rules, and the original test case rules include operators, the data value range corresponding to the operators, and the data type that matches the operators. The execution result acquisition module is used to call at least one standard operation operator provided by the Golden system to execute each of the candidate test cases and obtain the execution result of each of the candidate test cases. The standard operation operator includes at least one of numerical operation, vector operation, matrix operation, and convolution operation. The test case partitioning module is used to partition each of the candidate test cases into a normal test case set and an abnormal test case set based on the execution results. The rule base generation module is used to generate a normal test case rule base and an abnormal test case rule base based on the normal test case set and / or the abnormal test case set. The alternative test cases include at least one operator, and the chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system.

11. A chip testing device, characterized in that, include: A target test case acquisition module is used to acquire target test cases, wherein the target test cases include at least one operator, and the target test cases are acquired from a normal test case set or an abnormal test case set, wherein the normal test case set or the abnormal test case set is obtained by the method described in any one of claims 1-3; or, the target test cases are automatically generated based on a normal test case rule base or an abnormal test case rule base, wherein the normal test case rule base or the abnormal test case rule base is obtained by the method described in any one of claims 4-7. The first execution result acquisition module is used to call at least one standard operation operator provided by the Golden system to execute the target test case and obtain the first execution result of the target test case. The standard operation operator includes at least one of numerical operation, vector operation, matrix operation, and convolution operation. The second execution result acquisition module is used to call at least one hardware execution unit included in the chip under test to execute the target test case and acquire the second execution result of the target test case; The test verification module is used to determine that the chip under test has failed the test if the first execution result and the second execution result are inconsistent. The chip under test includes a hardware execution unit that matches the operator; the operator matches the standard arithmetic operators provided by the Golden system.

12. A computer device, characterized in that, The computer device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the test case generation method as described in any one of claims 1-3, or implement the test case rule generation method as described in any one of claims 4-7, or implement the chip testing method as described in claim 8.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the test case generation method as described in any one of claims 1-3, or the test case rule generation method as described in any one of claims 4-7, or the chip testing method as described in claim 8.

Images