Verification methods, apparatus, electronic devices, storage media, program products and chips

Abstract

The present disclosure provides a verification method and device, electronic equipment, storage medium, program product and chip, and relates to the technical field of chip verification. The method comprises the following steps: obtaining an initial test case set and a first check item set corresponding to a measured object; determining a second check item set corresponding to the initial test case set; determining a check item belonging to the first check item set and not belonging to the second check item set as a target check item; updating the initial test case set based on the target check item to obtain a target test case set; and verifying the measured object based on the target test case set. Therefore, before the measured object is verified by using the test case, the target check item not covered by the current initial test case set is determined first, and then the initial test case set is updated based on the target check item, so as to improve the coverage rate of the target test case set, and then the measured object is verified by using the target test case set, so that the verification coverage rate is ensured and the verification efficiency of the measured object is improved.

Description

Technical Field

[0001] This disclosure relates to the field of chip verification technology, and in particular to a verification method, apparatus, electronic device, storage medium, program product, and chip. Background Technology

[0002] Chip verification refers to the process of verifying a chip design to ensure that it meets requirements and specifications, mitigate risks, and identify and correct any defects.

[0003] In related technologies, after generating test cases corresponding to test points, these test cases are directly used to drive the chip and reference model, obtaining the chip's output results and the reference model's output results. Functional coverage is then calculated based on these identical results. If the functional coverage does not meet the requirements, the test cases are adjusted, and the adjusted test cases are used to drive the chip and reference model again until the functional coverage meets the requirements, thus completing the chip verification. This method has low verification efficiency.

[0004] Therefore, improving chip verification efficiency while ensuring verification coverage has become a key research direction. Summary of the Invention

[0005] This disclosure aims to at least partially address one of the technical problems in the related art.

[0006] The first aspect of this disclosure provides a verification method, including:

[0007] Obtain the initial set of test cases and the first set of inspection items corresponding to the object under test;

[0008] Determine the second set of check items corresponding to the initial set of use cases;

[0009] The inspection items that belong to the first set of inspection items but not to the second set of inspection items are identified as target inspection items;

[0010] Based on the target inspection items, the initial test case set is updated to obtain the target test case set;

[0011] The tested object is verified based on the target use case set.

[0012] A second aspect of this disclosure provides a verification apparatus, comprising:

[0013] The acquisition module is used to acquire the initial test case set and the first check item set corresponding to the object under test.

[0014] The first determining module is used to determine the second set of check items corresponding to the initial set of test cases;

[0015] The second determining module is used to determine the inspection items that belong to the first set of inspection items but do not belong to the second set of inspection items as target inspection items;

[0016] The update module is used to update the initial test case set based on the target inspection items to obtain the target test case set;

[0017] The verification module is used to verify the object under test based on the target use case set.

[0018] A third aspect of this disclosure provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the verification method as proposed in the first aspect of this disclosure.

[0019] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the verification method as described in the first aspect of this disclosure.

[0020] A fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the verification method as proposed in the first aspect of this disclosure.

[0021] A sixth aspect of this disclosure provides a chip that has been verified by the verification method described in the first aspect of this disclosure.

[0022] The verification methods, apparatus, electronic devices, storage media, program products, and chips provided in this disclosure have the following beneficial effects:

[0023] In this embodiment, an initial set of test cases and a first set of inspection items corresponding to the object under test are first obtained, and a second set of inspection items corresponding to the initial set of test cases is determined. Then, inspection items belonging to the first set of inspection items but not to the second set of inspection items are identified as target inspection items. Based on these target inspection items, the initial set of test cases is updated to obtain the target set of test cases. Finally, the object under test is verified based on the target set of test cases. Therefore, before verifying the object under test using test cases, target inspection items not covered by the current initial set of test cases are identified, and the initial set of test cases is updated based on these target inspection items to improve the coverage of the target set of test cases. Using the target set of test cases to verify the object under test can improve the verification efficiency while ensuring verification coverage.

[0024] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0025] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0026] Figure 1 This is a flowchart illustrating a verification method.

[0027] Figure 2 This is a schematic flowchart of a verification method provided in an embodiment of the present disclosure;

[0028] Figure 3 A schematic flowchart illustrating a verification method provided in another embodiment of this disclosure;

[0029] Figure 4 A flowchart illustrating a verification method provided in another embodiment of this disclosure.

[0030] Figure 5 This is a schematic diagram of the structure of a verification device provided in another embodiment of the present disclosure;

[0031] Figure 6 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0032] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0033] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0034] In related technologies, the basic process of chip verification includes: (1) Extracting test points: analyzing the chip specifications and requirements, and clarifying the specific points that need to be verified; (2) Formulating verification plan: determining the verification methods and strategies, including functional coverage, code coverage and test case formulation; (3) Test case execution: discovering potential defects and problems by executing test cases; (4) Completeness analysis: ensuring that all test points have been covered, including the completeness confirmation of functions, code and interfaces.

[0035] By following the steps described above, chip verification can minimize errors and problems after chip production, thereby improving product quality and reliability.

[0036] Figure 1 This is a flowchart illustrating a verification method. Figure 1 As shown, it includes the following steps:

[0037] (1) Extract test points: According to the test requirements, extract all test points that need to be tested based on the function and interface definition of the chip under test, including: module function, interface data bit width, timing logic relationship, etc.

[0038] (2) Test case construction: Based on the test points, test cases are constructed. The test cases are used to drive the reference model and the chip under test.

[0039] (3) Coverage Model Construction: Based on the test points and test requirements, generate SV files for the check items that need to be tested and covered, which will be used for subsequent coverage result statistics. The test points here include single-function test points and combined-function test points. Combined-function check points are composed of several single-function test point conditions.

[0040] (4) Reference model driven: Input test cases and stimulate the reference model to output correct test vectors, including: prototype data, print parameters, structure data, key information of intermediate variables, etc.

[0041] (5) Test chip driver: Input test cases, run the designed chip code on simulation platforms such as VCS, and generate output results.

[0042] (6) Output result matching: The correct test vectors output by the reference model are matched with the output results of the chip code under test to determine whether the design logic of the chip code under test is consistent with the reference model and is correct.

[0043] (7) Coverage Result Statistics: Extract the check items described in the SV file from the correctly matched output results in step (6), and perform coverage statistics using tools such as Verdi. If all functional tests are completed, the test ends. Otherwise, test cases need to be added and modified, and the test needs to be retested.

[0044] In some embodiments, if it is found that there are checks that cannot be covered after traversing the test cases (these checks are often unreasonable and difficult to find during the build process), the coverage model needs to be modified and the tests need to be re-run.

[0045] In the testing process of related technologies, after generating test cases corresponding to test points, the test cases are directly used to drive the chip and reference model to obtain the chip's output results and the reference model's output results. Functional coverage is then calculated based on these identical results. If the functional coverage does not meet the requirements, the test cases are adjusted, and the adjusted test cases are used to drive the chip and reference model again until the functional coverage meets the requirements, completing the chip verification. This process often requires multiple iterations and is time-consuming. In particular, the driving of the reference model, the driving of the IC under test, and the result comparison are high-time-consuming items that heavily rely on server resources, making the verification process take several weeks.

[0046] This invention primarily addresses the issue of lengthy verification processes in related technologies. Before driving the chip under test and reference model using test cases, the test cases are adjusted to ensure that the finalized test cases cover all inspection items. Then, the determined test cases are used to drive the chip under test and reference model. This approach separates the server resource-dependent steps of driving the reference model and chip under test, as well as the result comparison steps, from the iterative process, reducing verification convergence time and improving verification efficiency.

[0047] The verification methods, apparatus, electronic devices, and storage media of embodiments of this disclosure are described below with reference to the accompanying drawings.

[0048] Figure 2 This is a schematic flowchart of a verification method provided in an embodiment of the present disclosure.

[0049] This disclosure illustrates the example of the verification method being configured in a verification device, which can be applied to any electronic device or chip to enable the electronic device or chip to perform verification functions.

[0050] like Figure 2 As shown, the verification method may include the following steps:

[0051] Step 201: Obtain the initial set of test cases and the first set of inspection items corresponding to the object under test.

[0052] The object being tested can be hardware, such as chips, processors, graphics cards, and memory, or software, such as applications. This disclosure does not limit the scope of the test.

[0053] The initial test case set contains one or more initial test cases. Initial test cases can be test cases generated in advance based on the test points corresponding to the test being tested.

[0054] In some embodiments, the initial test cases may be generated manually based on test points or automatically based on test points. This disclosure does not limit this.

[0055] In some embodiments, test points include single test points and combined test points. Test points are used to construct test cases; a single test point is an input variable, output variable, or intermediate variable corresponding to the object under test. A combined test point can be a combination of single test points.

[0056] The first set of check items contains check items corresponding to one or more test points, and each check item is a value or combination of values ​​corresponding to one or more test points.

[0057] For example, if the test points that the test subject is concerned with are single test point A, single test point B, and combined test points A and B, the value of single test point A is {1,2,3}, the value of single test point B is {4,5,6}, and the combined value of combined test points A and B is {[1,5],[3,4]}, then the first set of check items contains 8 check items, namely A(1); A(2); A(3); B(4); B(5); B(6); A(1)B(5); A(3)B(4).

[0058] In some embodiments, all check items in the first check item set can be stored using data file formats such as JSON or XLS. This facilitates comparison with the second check item set of the initial test case set.

[0059] Step 202: Determine the second set of check items corresponding to the initial set of use cases.

[0060] In some embodiments, a subset of target inspection items corresponding to each initial use case in the initial use case set can be determined first, and then a second set of inspection items can be determined based on the subset of target inspection items corresponding to each initial use case.

[0061] In some embodiments, a subset of target inspection items may contain one or more inspection items. This disclosure does not limit the number of inspection items.

[0062] In some embodiments, the control parameter values ​​corresponding to the input parameter values ​​in the initial use case are determined, wherein the initial use case is the use case in the initial use case set. Then, based on the input parameter values ​​and control parameter values ​​corresponding to the initial use case, the initial check item subset corresponding to the initial use case is determined. The intersection between the initial check item subset corresponding to the initial use case and the first check item set is determined as the target check item subset corresponding to the initial use case. Finally, the union of all target check item subsets corresponding to the initial use cases is determined as the second check item set.

[0063] The initial check item subset includes at least one of the following: a check item consisting of input parameters and their corresponding input parameter values; a check item consisting of control parameters and their corresponding control parameter values; and a check item consisting of input parameters, their corresponding input parameter values, control parameters, and their corresponding control parameter values.

[0064] The control parameters include intermediate parameters and output parameters.

[0065] In some embodiments, the control parameter values ​​corresponding to the input parameter values ​​in the initial test case can be determined based on the mapping relationship between the input parameters and control parameters of the object under test.

[0066] In some embodiments, the input parameter values ​​corresponding to each initial use case can also be input into the pseudo-reference model to obtain the control parameter values ​​corresponding to each initial use case.

[0067] The pseudo-reference model is generated based on the reference model and contains the mapping relationship between input parameters and control parameters.

[0068] In some embodiments, the reference model may include a parameter mapping module and a calculation module. The parameter mapping module includes the mapping relationship between input parameters and control parameters, and the calculation module performs calculations based on the control parameter values. The pseudo-reference model only contains the parameter mapping module of the reference model. This allows for the rapid determination of the target inspection item subset corresponding to each use case.

[0069] The reference model is used to determine the expected output result of the tested object after processing the input parameter values. The reference result can be understood as the expected result.

[0070] Step 203: The inspection items that belong to the first set of inspection items but not to the second set of inspection items are identified as target inspection items.

[0071] It should be noted that check items that belong to the first set of check items but not to the second set of check items are check items not covered by the test cases.

[0072] In some embodiments, there may be one or more target inspection items. This disclosure does not limit this.

[0073] Step 204: Update the initial set of test cases based on the target inspection items to obtain the target set of test cases.

[0074] In some embodiments, new use cases that can cover the target check item can be generated for the target check item, and then the new use cases can be added to the initial use case set to obtain the target use case set.

[0075] Step 205: Verify the object under test based on the target use case set.

[0076] In this embodiment of the disclosure, after obtaining the target use case set, the target use case set can be used to verify the object under test and obtain the verification result.

[0077] In some embodiments, the input parameter values ​​corresponding to each target use case in the target use case set are input into the object under test and the reference model corresponding to the object under test, respectively, to obtain the running results output by the object under test and the reference results output by the reference model. Then, based on the matching results between the running results and the reference results, the verification results corresponding to the object under test are determined.

[0078] The reference model is used to determine the reference result output by the tested object after processing the input parameter values.

[0079] In some embodiments, if the execution result is exactly the same as the reference result, the object under test is considered to have passed verification. If the execution result for any target use case is different from the reference result, the object under test is considered to have passed verification. Furthermore, the object under test can be modified based on the difference between the execution result and the reference result.

[0080] In this embodiment, an initial set of test cases and a first set of inspection items corresponding to the object under test are first obtained, and a second set of inspection items corresponding to the initial set of test cases is determined. Then, inspection items belonging to the first set of inspection items but not to the second set of inspection items are identified as target inspection items. Based on these target inspection items, the initial set of test cases is updated to obtain the target set of test cases. Finally, the object under test is verified based on the target set of test cases. Therefore, before verifying the object under test using test cases, target inspection items not covered by the current initial set of test cases are identified, and the initial set of test cases is updated based on these target inspection items to improve the coverage of the target set of test cases. Using the target set of test cases to verify the object under test can improve the verification efficiency while ensuring verification coverage.

[0081] Figure 3 This is a flowchart illustrating a verification method provided in an embodiment of the present disclosure, as shown below. Figure 3 As shown, the verification method may include the following steps:

[0082] Step 301: Obtain the initial test case set and the first check item set corresponding to the object under test.

[0083] Step 302: Determine the second set of check items corresponding to the initial set of use cases.

[0084] Step 303: The inspection items that belong to the first set of inspection items but not to the second set of inspection items are identified as target inspection items.

[0085] The specific implementation of steps 301 to 303 can be found in the detailed descriptions of other embodiments of this disclosure, and will not be repeated here.

[0086] Step 304: Determine the target test points based on the target inspection items.

[0087] In some embodiments, a first test point corresponding to the target inspection item can be determined first. If the first test point is a combined test point, each second test point that makes up the first test point can be determined, and then the first test point and each second test point can be determined as the target test point.

[0088] For example, if the target inspection item A(a3)B(b2) has a first test point that is a combination of test points A and B, then the second test point is a single test point A and a single test point B. The target test points include single test point A, single test point B and combination of test points A and B.

[0089] It should be noted that a test case may cover not only the checks corresponding to the combined test point, but also the checks corresponding to the individual test points that make up the combined test point. For example, a test case that covers the check item A(a3)B(b2) may also cover the checks A(a3) and B(b2). Therefore, if the checks corresponding to the combined test point are not covered, it may be due to poorly generated test cases for the combined test point or the individual test points that make up the combined test point.

[0090] Therefore, in this embodiment of the disclosure, when the first test point corresponding to the target inspection item is a combined test point, the first test point and each of the second test points that make up the first test point can be determined as the target test point, and test cases can be regenerated.

[0091] In some embodiments, when the first test point is a single test point, the first test point is directly determined as the target test point.

[0092] Step 305: Determine the first set of test cases corresponding to the target test point.

[0093] In some embodiments, test cases can be regenerated for the target test point to obtain a first set of test cases.

[0094] In some embodiments, to ensure that the first set of test cases generated for the target test point can cover all the checks corresponding to the target test point as much as possible, a set of candidate test cases corresponding to the target test point can be generated first. Then, a first number of all different checks corresponding to the candidate test case set and a second number of all checks corresponding to the target test point in the first check item set can be determined. The ratio of the first number to the second number is determined as the first coverage rate corresponding to the candidate test case set. If the first coverage rate is less than a first threshold, the operation of generating a candidate test case set is returned until the first coverage rate of the generated candidate test case set is greater than or equal to the first threshold. Finally, the set of candidate test cases with a first coverage rate greater than or equal to the first threshold is determined as the first test case set.

[0095] The first threshold can be 100%, 99%, etc. This disclosure does not limit it.

[0096] In some embodiments, a set of candidate test cases corresponding to the target test point can be generated by combining the target inspection items.

[0097] In some embodiments, after determining the candidate use case set, the check items corresponding to each candidate use case in the candidate use case set can be determined first, and then all different check items corresponding to the candidate use case set can be determined. For a detailed description of the specific implementation, please refer to the description of determining the second check item set corresponding to the initial use case set, which will not be repeated here.

[0098] Step 306: Using the first test case set, update the second test case set in the initial test case set that corresponds to the target test point to obtain the target test case set.

[0099] In some embodiments, the second use case set in the initial use case set can be replaced with the first use case set to obtain the target use case set.

[0100] Step 307: Verify the object under test based on the target use case set.

[0101] In this embodiment, after identifying the target inspection items not covered by the initial test case set, target test points can be determined based on the target inspection items. This leads to the determination of a first test case set corresponding to the target test point. The first test case set is then used to update the second test case set corresponding to the target test point in the initial test case set, resulting in a target test case set. Finally, the object under test is verified based on the target test case set. Therefore, target test points can be determined based on the target inspection items not covered by the initial test case set. Test cases can then be regenerated for these target test points, ensuring that the regenerated test cases cover the target inspection items. This improves the coverage of the target test points, thereby increasing the coverage of the target test case set. Furthermore, this improves the verification efficiency of the object under test while maintaining verification coverage.

[0102] Figure 4 This is a flowchart illustrating a verification method provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, the verification method may include the following steps:

[0103] Step 401: Obtain the initial test case set and the first check item set corresponding to the object under test.

[0104] Step 402: Determine the second set of check items corresponding to the initial set of use cases.

[0105] Step 403: The inspection items that belong to the first set of inspection items but not to the second set of inspection items are identified as target inspection items.

[0106] The specific implementation of steps 401 to 403 can be found in the detailed descriptions of other embodiments of this disclosure, and will not be repeated here.

[0107] Step 404: Determine the third quantity corresponding to all inspection items in the second inspection item set, and the fourth quantity corresponding to all inspection items in the first inspection item set.

[0108] Step 405: The ratio of the third quantity to the fourth quantity is determined as the second coverage rate corresponding to the initial use case set.

[0109] Step 406: If the second coverage rate is less than the second threshold, the check items that belong to the first check item set but not to the second check item set are identified as target check items.

[0110] It should be noted that when the second coverage rate is less than the second threshold, the coverage of the initial test case set is low. If the initial test case set is used directly to verify the object under test, the verification effect is poor. If the second coverage rate is greater than or equal to the second threshold, direct verification of the object under test can also achieve better results. Therefore, in this embodiment of the present disclosure, when the second coverage rate corresponding to the initial test case set is less than the second threshold, the target inspection items not covered by the test cases can be identified, and the initial test case set can be updated to improve the coverage of the obtained target test case set.

[0111] In some embodiments, the second threshold may be the same as or different from the first threshold. This disclosure does not limit this.

[0112] Step 407: Update the initial test case set based on the target inspection items to obtain the target test case set.

[0113] The specific implementation of step 407 can be found in the detailed descriptions of other embodiments of this disclosure, and will not be repeated here.

[0114] Step 408: Determine the fifth number of all different inspection items corresponding to the target use case set.

[0115] In some embodiments, after determining the target use case set, the check items corresponding to each target use case in the target use case set can be determined first, and then all different check items corresponding to the target use case set can be determined. For a detailed description of the specific implementation, please refer to the description of determining the second check item set corresponding to the initial use case set, which will not be elaborated here.

[0116] Step 409: The ratio of the fifth quantity to the fourth quantity is determined as the third coverage rate corresponding to the target use case set.

[0117] Step 410: If the third coverage rate is less than the second threshold, use the target use case set as the initial use case set, and return to execute the operation to obtain the target use case set until the third coverage rate corresponding to the target use case set is greater than or equal to the second threshold.

[0118] It should be noted that if the third coverage rate corresponding to the target use case set is less than the second threshold, it means that the coverage rate of the target use case set still does not meet the requirements. If the target object is directly verified using a target use case set whose coverage rate does not meet the requirements, the verification results may be inaccurate. Therefore, in order to accurately verify the target object, if the third coverage rate corresponding to the target use case set is less than the second threshold, the target use case set can be further updated so that the third coverage rate of the updated target use case set is greater than or equal to the second threshold.

[0119] In some embodiments, when the third coverage is less than the second threshold, the target use case set is used as the initial use case set, and then the second check item set corresponding to the initial use case set is determined. Check items that belong to the first check item set but not to the second check item set are determined as target check items. Based on the target check items, the initial use case set is updated to obtain the target use case set, until the third coverage corresponding to the target use case set is greater than or equal to the second threshold.

[0120] Step 411: Verify the object under test based on the target use case set.

[0121] In this embodiment, if the second coverage rate corresponding to the initial test case set is less than the second threshold, target inspection items not covered by the initial test case set can be identified. Then, based on these target inspection items, the initial test case set is updated to obtain a target test case set. If the third coverage rate corresponding to the target test case set is less than the second threshold, the target test case set is used as the initial test case set. Further uncovered target inspection items are identified, and the initial test case set is updated again until the third coverage rate corresponding to the target test case set is greater than or equal to the second threshold. Then, the object under test is verified. Therefore, by updating the initial test case set when the second coverage rate is less than the second threshold, and ensuring that the third coverage rate of the updated target test case set is greater than or equal to the second threshold, the verification efficiency of the object under test is improved while maintaining verification coverage.

[0122] To implement the above embodiments, this disclosure also proposes a verification device.

[0123] Figure 5 This is a schematic diagram of the structure of the verification device provided in the embodiments of this disclosure.

[0124] like Figure 5 As shown, the verification device 500 may include:

[0125] The acquisition module 501 is used to acquire the initial test case set and the first check item set corresponding to the object under test;

[0126] The first determining module 502 is used to determine the second set of check items corresponding to the initial set of test cases;

[0127] The second determining module 503 is used to determine the inspection items that belong to the first set of inspection items but do not belong to the second set of inspection items as target inspection items;

[0128] Update module 504 is used to update the initial test case set based on the target inspection items to obtain the target test case set;

[0129] Verification module 505 is used to verify the object under test based on the target test case set.

[0130] In some embodiments, the update module 504 is used to:

[0131] Based on the target inspection items, determine the target test points;

[0132] Determine the first set of test cases corresponding to the target test point;

[0133] Using the first set of test cases, update the second set of test cases in the initial set of test cases that corresponds to the target test point to obtain the target set of test cases.

[0134] In some embodiments, the update module 504 is used to:

[0135] Determine the first test point corresponding to the target inspection item;

[0136] If the first test point is a combination of test points, determine the second test points that make up the first test point;

[0137] The first test point and each of the second test points are determined as target test points.

[0138] In some embodiments, the update module 504 is used to:

[0139] Generate a set of candidate test cases corresponding to the target test point;

[0140] Determine the first number of all different check items corresponding to the candidate test case set, and the second number of all check items corresponding to the target test point in the first check item set;

[0141] The ratio of the first quantity to the second quantity is determined as the first coverage rate corresponding to the candidate use case set;

[0142] If the first coverage rate is less than the first threshold, return to the operation of generating a set of candidate test cases until the first coverage rate corresponding to the generated set of candidate test cases is greater than or equal to the first threshold.

[0143] The set of candidate use cases with a first coverage rate greater than or equal to a first threshold is determined as the first use case set.

[0144] In some embodiments, the second determining module 503 is configured to:

[0145] Determine the third quantity corresponding to all inspection items in the second inspection item set, and the fourth quantity corresponding to all inspection items in the first inspection item set;

[0146] The ratio of the third and fourth quantities is determined as the second coverage rate corresponding to the initial set of test cases;

[0147] If the second coverage rate is less than the second threshold, the check items that belong to the first check item set but not to the second check item set are identified as target check items.

[0148] In some embodiments, a processing module is further included for:

[0149] Determine the fifth number of all distinct inspection items corresponding to the target use case set;

[0150] The ratio of the fifth quantity to the fourth quantity is determined as the third coverage rate corresponding to the target use case set;

[0151] If the third coverage rate is less than the second threshold, the target use case set is used as the initial use case set, and the operation to obtain the target use case set is returned until the third coverage rate corresponding to the target use case set is greater than or equal to the second threshold.

[0152] In some embodiments, the verification module 505 is configured to:

[0153] The input parameter values ​​corresponding to each target test case in the target test case set are input into the test object and the reference model corresponding to the test object, respectively, to obtain the running results output by the test object and the reference results output by the reference model. The reference model is used to determine the reference results output by the test object after processing the input parameter values.

[0154] Based on the matching results between the running results and the reference results, the verification results corresponding to the tested object are determined.

[0155] In some embodiments, the first determining module 502 is configured to:

[0156] Determine the control parameter values ​​corresponding to the input parameter values ​​in the initial test cases, where the initial test cases are the test cases in the initial test case set;

[0157] Based on the input parameter values ​​and control parameter values ​​corresponding to the initial use case, determine the initial check item subset corresponding to the initial use case. The initial check item subset includes at least one of the following: a check item consisting of input parameters and corresponding input parameter values; a check item consisting of control parameters and corresponding control parameter values; and a check item consisting of input parameters, corresponding input parameter values, control parameters, and corresponding control parameter values.

[0158] The intersection of the initial check item subset corresponding to the initial test case and the first check item set is determined as the target check item subset corresponding to the initial test case;

[0159] The union of the target check item subsets corresponding to all initial use cases is used to determine the second check item set.

[0160] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0161] The verification apparatus of this embodiment first acquires an initial set of test cases and a first set of inspection items corresponding to the object under test, and determines a second set of inspection items corresponding to the initial set of test cases. Then, inspection items belonging to the first set of inspection items but not to the second set of inspection items are identified as target inspection items. Based on these target inspection items, the initial set of test cases is updated to obtain a target set of test cases. Finally, the object under test is verified based on the target set of test cases. Therefore, before verifying the object under test using test cases, target inspection items not covered by the current initial set of test cases are identified, and the initial set of test cases is updated based on these target inspection items to improve the coverage of the target set of test cases. By using the target set of test cases to verify the object under test, the verification efficiency of the object under test can be improved while ensuring verification coverage.

[0162] To implement the above embodiments, this disclosure also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the verification method proposed in the foregoing embodiments of this disclosure.

[0163] Figure 6 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 6 The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0164] like Figure 6 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0165] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0166] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0167] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 6 Not shown; usually referred to as a "hard drive"). Although Figure 6 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0168] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0169] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0170] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the foregoing embodiments.

[0171] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the verification method proposed in the foregoing embodiments of this disclosure.

[0172] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program that, when executed by a processor, implements the verification method as proposed in the foregoing embodiments of this disclosure.

[0173] To achieve the above embodiments, this disclosure also proposes a chip, which is obtained by the verification method proposed in the foregoing embodiments of this disclosure.

[0174] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0175] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0176] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0177] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0178] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0179] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0180] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0181] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0182] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method of verification, characterized by, The method comprises: acquiring an initial test case set and a first inspection item set corresponding to a to-be-tested object; determining a second inspection item set corresponding to the initial test case set; determining, as a target inspection item, an inspection item belonging to the first inspection item set and not belonging to the second inspection item set, wherein an inspection item set comprises one or more test points corresponding inspection items, and the inspection item comprises one or more test points corresponding values or value combinations; determining a first test point corresponding to the target inspection item; in a case where the first test point is a combination test point, determining each second test point constituting the first test point; determining the first test point and the each second test point as a target test point; determining a first test case set corresponding to the target test point; updating, by using the first test case set, a second test case set corresponding to the target test point in the initial test case set to obtain a target test case set; verifying the to-be-tested object based on the target test case set.

2. The method of claim 1, wherein, The determination of the first test case set corresponding to the target test point comprises: generating a candidate test case set corresponding to the target test point; determining a first number of all different inspection items corresponding to the candidate test case set and a second number of all inspection items corresponding to the target test point in the first inspection item set; determining a first coverage rate of the candidate test case set as a ratio of the first number to the second number; in a case where the first coverage rate is less than a first threshold, returning to perform the operation of generating a candidate test case set until the first coverage rate of the generated candidate test case set is greater than or equal to the first threshold; determining, as the first test case set, the candidate test case set whose first coverage rate is greater than or equal to the first threshold.

3. The method of claim 1, wherein, The determination of the target inspection item comprises: determining a third number of all inspection items corresponding to the second inspection item set and a fourth number of all inspection items corresponding to the first inspection item set; determining a second coverage rate of the initial test case set as a ratio of the third number to the fourth number; in a case where the second coverage rate is less than a second threshold, determining, as the target inspection item, an inspection item belonging to the first inspection item set and not belonging to the second inspection item set.

4. The method of claim 3, wherein, After the initial test case set is updated based on the target inspection item to obtain a target test case set, the method further comprises: determining a fifth number of all different inspection items corresponding to the target test case set; determining a third coverage rate of the target test case set as a ratio of the fifth number to the fourth number; in a case where the third coverage rate is less than the second threshold, taking the target test case set as the initial test case set and returning to perform the operation of obtaining the target test case set until the third coverage rate of the target test case set is greater than or equal to the second threshold.

5. The method according to any one of claims 1 to 4, characterized in that, The verification of the to-be-tested object based on the target test case set comprises: input parameter values corresponding to each target use case in the target use case set into the to-be-tested object and a reference model corresponding to the to-be-tested object, to obtain a running result output by the to-be-tested object and a reference result output by the reference model, wherein the reference model is used to determine the reference result output by the to-be-tested object after processing the input parameter values; determine a verification result corresponding to the to-be-tested object according to a matching result between the running result and the reference result.

6. The method according to any one of claims 1 to 4, characterized in that, The determining the second check item set corresponding to the initial use case set comprises: determining control parameter values corresponding to input parameter values in an initial use case, wherein the initial use case is a use case in the initial use case set; determining an initial check item sub-set corresponding to the initial use case according to the input parameter values and the control parameter values corresponding to the initial use case, wherein the initial check item sub-set comprises at least one of the following: a check item composed of an input parameter and a corresponding input parameter value, a check item composed of a control parameter and a corresponding control parameter value, and a check item composed of an input parameter, a corresponding input parameter value, a control parameter and a corresponding control parameter value; determining a target check item sub-set corresponding to the initial use case as an intersection between the initial check item sub-set corresponding to the initial use case and the first check item set; determining a union of the target check item sub-sets corresponding to all the initial use cases as the second check item set.

7. A verification device, characterized in that The apparatus comprises: an acquisition module configured to acquire an initial use case set and a first check item set corresponding to a to-be-tested object; a first determination module configured to determine a second check item set corresponding to the initial use case set; a second determination module configured to determine a target check item as a check item belonging to the first check item set and not belonging to the second check item set, wherein a check item set comprises check items corresponding to one or more test points, and the check items comprise values or value combinations corresponding to the one or more test points; a third determination module configured to determine a first test point corresponding to the target check item; a fourth determination module configured to determine each second test point constituting a first test point in a case where the first test point is a combination test point; a fifth determination module configured to determine a target test point as the first test point and the each second test point; a sixth determination module configured to determine a first use case set corresponding to the target test point; an update module configured to update a second use case set corresponding to the target test point in the initial use case set by using the first use case set, to obtain a target use case set; a verification module configured to verify the to-be-tested object based on the target use case set.

8. An electronic device, comprising: The computer program is executed by the processor to implement the verification method in any one of claims 1-6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the verification method in any one of claims 1-6.

10. A computer program product, characterised in that, The computer program comprises a computer program which, when executed by a processor, implements the authentication method according to any one of claims 1-6.

11. A chip, characterized by The chip is obtained by the authentication method according to any one of claims 1-6.

Images