Methods, apparatus, media, and products for verifying checkers in test cases

Abstract

The present disclosure relates to the technical field of chip verification, and particularly provides a method, device, medium and product for verifying a checker in a test case. The method comprises: obtaining a code point capable of being used for error injection in a design under test; performing error injection on the code point to obtain an error-injected design code; running a test case corresponding to the design under test on the error-injected design code to obtain a checking result; and in a case where the checking result indicates that no error is reported in the test, confirming that the checker in the test case has an incompleteness risk. It can be seen that the completeness of the test case is verified by using the embodiments of the present disclosure.

Description

Technical Field

[0001] This disclosure relates to the field of chip verification technology, and in particular to a method, apparatus, medium, and product for verifying an inspector in a test case. Background Technology

[0002] Currently, chip IP verification is achieved through the combination of testbench and testcase. Testbench is a complete, automated simulation test environment used to "host" and "test" the design under test (DUT). Testcase is a test scenario or stimulus sequence manually written by verification engineers, defining under what conditions, what data is input to the DUT, and what results or states are expected.

[0003] In related technologies, the checker in the test case can only be guaranteed by the verification engineer's own verification experience. If the checker fails or is incomplete, there is a risk of missing functional verifications. Ensuring the validity and completeness of the checker in the test case is a challenge for IP verification. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned problems. This disclosure provides a method, apparatus, medium, and product for verifying inspectors in test cases.

[0005] According to a first aspect of this disclosure, a method for verifying an inspector in a test case is provided, comprising:

[0006] Identify code points in the design under test that can be used for bug fixing; Error injection is performed on the aforementioned code points to obtain the error-injection design code; Run the test cases corresponding to the design under test on the punctured design code to obtain the inspection results; If the inspection results indicate that the test did not report any errors, it is confirmed that the checker in the test case has a risk of incompleteness.

[0007] Furthermore, according to the method of the checker in the verification test cases of the first aspect of this disclosure, before running the test cases corresponding to the design under test on the punctured design code and obtaining the check results, the method further includes: Confirm that the coverage of the test cases reaches the preset coverage threshold.

[0008] Furthermore, according to the method of the checker in the verification test cases of the first aspect of this disclosure, code points in the design under test that can be used for bug fixing are obtained, including: From the design under test, find code statements that meet the set conditions, which are used to indicate that the semantics of the code statements can be modified; The code statement is used as the code point that can be used for error annotation.

[0009] Furthermore, according to the method of the checker in the verification test cases of the first aspect of this disclosure, error injection is performed on the code points to obtain the error-injected design code, including: If the code point includes an operator, modify the operator or modify the operator's operation parameters to obtain the error annotation design code.

[0010] Furthermore, according to the method of the checker in the verification test cases of the first aspect of this disclosure, the test cases corresponding to the design under test are run on the punctured design code to obtain the check results, including: Obtain the design specifications of the design under test; The inspector is invoked to detect the error-injection design code according to the design specifications, and the inspection results are obtained.

[0011] Furthermore, according to the method of the checker in the verification test cases of the first aspect of this disclosure, when the number of code points is N, error injection is performed on the code points to obtain the error-injected design code, including: Perform error injection on each i-th code point to obtain the error injection design code corresponding to the i-th code point, where i≤N; The method further includes: If the check result of the error injection design code corresponding to the i-th code point indicates that there is an error in the test, and i < N, update i = i + 1, return to the step of performing error injection on the i-th code point one by one to obtain the error injection design code corresponding to the i-th code point, until i = N.

[0012] According to a second aspect of this disclosure, an apparatus is provided for verifying an inspector in a test case, comprising: The acquisition module is used to acquire code points in the design under test that can be used for error annotation. The injection module is used to inject errors into the code points to obtain the error-injection design code; The execution module is used to run test cases corresponding to the design under test on the error-injected design code and obtain the inspection results. The determination module is used to confirm that the checker in the test case has a risk of incompleteness if the check result indicates that the test has no errors.

[0013] According to a third aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method described in the first aspect.

[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program / instructions stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0016] As will be described in detail below, the method for verifying the checker in test cases according to embodiments of this disclosure involves injecting errors into code points of the design under test to obtain error-injected design code, running test cases corresponding to the design under test on the error-injected design code, and obtaining the checker's check result in the test case. This check result can be used to determine whether the checker in the test case has any incompleteness risk. Therefore, embodiments of this disclosure achieve the verification of the completeness of test cases.

[0017] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0018] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 This is a flowchart illustrating a method for applying an inspector in a verification test case according to an embodiment of this disclosure.

[0020] Figure 2 This is another flowchart illustrating a method for using an inspector in a verification test case according to an embodiment of this disclosure.

[0021] Figure 3 This is a structural diagram of an apparatus for an inspector used in a verification test case according to an embodiment of the present disclosure.

[0022] Figure 4 This is a hardware block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0023] Figure 5 This is a schematic diagram illustrating a computer program product according to an embodiment of the present disclosure. Detailed Implementation

[0024] The technical methods of the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings.

[0025] The checker in a test case can check whether the test case runs according to the design specifications of the design under test. For example, for a module that implements login functionality, the code implementing this module contains statements that check whether the username and password match.

[0026] When testing the module's functionality using test cases, the checker in the test cases checks whether the module's code contains statements that compare the username and password according to the module's design specifications. If so, it further outputs the login result of the module under the given username and password example in the test case. The login result indicates whether the login was successful or failed.

[0027] If the checker detects that a module's code statement checks whether the username and password do not match, since this statement is inconsistent with the username and password matching specifications, the checker will issue an error and will not output the subsequent login result.

[0028] In related technologies, the checkers in test cases can only be guaranteed by the verification engineer's own verification experience. If the checkers are invalid or incomplete, there is a risk of missing functional verifications. Ensuring the validity and completeness of checkers in test cases is a challenge for IP verification.

[0029] To facilitate understanding of this embodiment, a method for verifying a checker in a test case, as disclosed in this disclosure, will first be described in detail. The execution subject of this method for verifying a checker in a test case is generally an electronic device with a certain computing capability, such as a terminal device, a server, or other processing device. In some possible implementations, the method for verifying a checker in a test case can be implemented by a processor calling computer-readable instructions stored in memory.

[0030] See Figure 1 The diagram shows a flowchart of a method for verifying an inspector in a test case according to an embodiment of this disclosure. The method includes the following steps: Step 101: Obtain the code points in the design under test that can be used for error annotation.

[0031] In this embodiment, the design under test (DUT) is typically referred to as DUT. In IP verification, the DUT is the hardware design module to be verified. This module can be a complete processor (such as a CPU), a communication interface controller (such as USB, PCIe), an encryption engine, an image processing unit, or any digital circuit module with specific functions. The sole purpose of building the entire verification platform (Testbench) and writing all test cases is to prove whether the behavior of this DUT conforms to its design specifications.

[0032] In this embodiment, the code points in the design under test that can be used for bug injection refer to code locations intentionally designed during software development to test the system's fault tolerance, error handling capabilities, or fault recovery mechanisms, and which can trigger simulated errors. These points are typically used for fault injection testing to help verify the system's behavior under abnormal conditions.

[0033] In this embodiment, the code points that can be used for error correction can include the following categories: Injection points at the code logic level; Injection points that depend on component interactions; Injection points at the data and state levels; Concurrency and timing-related injection points; Injection points at the system and environment levels.

[0034] Injection points at the code logic level can be further divided into injection points related to function / method return values ​​and injection points related to conditional branch error triggering. The former usually appears in scenarios simulating function call failure or returning an abnormal value, while the latter usually appears in scenarios that trigger error paths under specific conditions (such as insufficient resources or timeout).

[0035] Injection points that depend on component interaction can include injection points related to external service / API calls and injection points related to database / middleware operations. The former usually appears in scenarios such as simulating network timeouts, service unavailability, and protocol errors.

[0036] Injection points at the data and state levels can be further divided into injection points related to data validation boundaries and injection points related to resource state simulation. The former occurs in scenarios involving abnormal paths of input validation and data integrity checks, while the latter occurs in scenarios simulating disk fullness, memory exhaustion, file corruption, etc.

[0037] Concurrency and timing-related injection points include those related to race condition triggering and those related to timeout and retry logic. The former appears in scenarios simulating race conditions in multithreaded / distributed environments, while the latter appears in scenarios testing the robustness of timeout handling and retry mechanisms.

[0038] System and environment-level injection points can be further divided into injection points related to configuration and parameter errors and injection points related to hardware / low-level simulation. The former occurs in scenarios simulating incorrect configurations (such as incorrect formats or invalid paths), while the latter occurs in scenarios such as CPU failures, I / O errors, and network partitions.

[0039] In some alternative embodiments, the verification engineer may identify, based on experience, code points in the design under test that can be used for bug fixing.

[0040] In some alternative embodiments, code statements that meet certain conditions can also be found in the design under test. These conditions are used to indicate that the semantics of the code statements can be modified; the code statements are then used as code points that can be used for error correction.

[0041] In this embodiment, the set conditions include, but are not limited to, operators and / or belonging to a specific type. The operators here include, but are not limited to, addition, subtraction, multiplication, division, inversion, and XOR. The specific type of code statements includes, but is not limited to, conditional statements, statements expressing function / method return values, and statements instructing external service / API calls.

[0042] In this embodiment, the semantics of a code statement refers to what a piece of code should do when the program runs, that is, its actual meaning, function, and effect. When the semantics of a code statement are changed, its actual meaning, function, and / or effect change.

[0043] For example, if the original code statement is to update parameter a = a + 1, and the modified code statement is to update parameter a = a + 2, it means that the effect of the code statement has changed. Because the update magnitude of parameter a has changed, some parameters a cannot be traversed, and the part of parameters that is a = a + 1 is missing. This will affect the effect of the solution and reduce its effectiveness.

[0044] For example, if the original code statement was to update parameter a = a + 1, and the modified code statement is to update parameter a = a - 1, then the actual meaning of the code statement has changed. The original solution aimed to analyze the output of the functional module as parameter a gradually increases. After the semantics of the code statement are modified, the modified solution focuses on the output of the functional module as parameter a gradually decreases. The modified code statement causes the module's functionality to change completely in the opposite direction to the original solution, making the solution's purpose contradictory.

[0045] Step 102: Inject errors into the code points to obtain the error-injection design code.

[0046] In this embodiment, error injection refers to introducing errors into the code to observe the reaction of the checker in the test case.

[0047] In some embodiments, where the code point includes an operator, the operator or the operator's operation parameters are modified to obtain the error annotation design code.

[0048] In this embodiment, the operation parameter of the operator is the operand of the operator. Taking a+b as an example, the symbol "+" is the operator, and the two parameters a and b are the operation parameters or operands of the operator "+".

[0049] In this embodiment, modifying an operator refers to changing the operator in the code point to another operator. The modification of the operator can be random, as long as the operator before and after the modification is not the same operator. For example, changing "+" to "-", "AND" to "OR", or ">" to "<", etc.

[0050] In this embodiment, modifying the operator's parameters can be done by randomly changing one parameter to another, with the parameters before and after the modification being different. Alternatively, the parameters can be modified according to certain preset rules, such as increasing or decreasing the parameters, or doubling or decreasing the parameters.

[0051] In some embodiments, code point error injection can be achieved by manually modifying the code, using debugging tools, writing test cases, using specialized error injection tools, or proxy-based injection.

[0052] Among the methods of manually modifying code, errors are manually inserted into the code, such as manually modifying conditional statements, changing variable values, or throwing exceptions.

[0053] In the use of debugging tools, the debugger is used to modify variables and change program flow at runtime.

[0054] In writing test cases, simulate error conditions in unit tests or integration tests, such as using Mock to simulate the failure of dependent components.

[0055] When using specialized error injection tools, for Java applications, tools such as Pumba and ChaosMonkey can be used; for C++, tools such as LLVM's LibFuzzer can be used.

[0056] In proxy-based injection, errors are injected before and after method calls via proxies or AOP (Aspect-Oriented Programming).

[0057] In some embodiments, in order to cover the design under test to a greater extent, the test case coverage is required to reach a preset coverage threshold before running test cases on the punctured design code.

[0058] The coverage threshold here can be set manually based on experience or according to actual needs, for example, setting the coverage threshold to 100%.

[0059] It should be understood that test coverage is a quantitative metric for measuring the completeness and effectiveness of testing; it indicates the extent to which test cases cover the design under test. It should also be understood that in practice, tools can be used to obtain test case coverage.

[0060] Step 103: Run the test cases corresponding to the design under test on the code that was injected with errors, and obtain the inspection results.

[0061] It should be understood that when the design under test encompasses multiple functions, multiple test cases are required to test the design under test. In this case, when running test cases on the bug-annotated design code, it is necessary to regress all test cases on the bug-annotated design code. Here, regression test cases refer to re-executing test cases after modifying the code to confirm whether the modification introduces new errors or causes errors in other code.

[0062] In one alternative embodiment, when running test cases on the error-injected design code, the design specifications of the design under test are obtained; the inspector is invoked to detect the error-injected design code according to the design specifications and obtain the inspection results.

[0063] It should be understood that the design specifications for the design under test refer to the design ideas and principles used to indicate the design of the design under test. These design specifications typically include the design methods of the design under test (such as equivalence class partitioning, boundary value analysis, cause-effect graphing, etc.), coding specifications (such as naming conventions, structural specifications, commenting conventions, etc.), data management specifications, environment management specifications, and code specifications (such as using the Page Object pattern and following the DRY principle, etc.).

[0064] In this embodiment, the risk of incompleteness of the test case checker refers to the risk that the checker fails to detect erroneous code in the test case that does not conform to the design specifications of the test case, resulting in undiscovered defects in the test case.

[0065] In one example, the design specification of the design under test defines the parameter a > b. If, after code point annotation, the parameter a > b in the design under test is changed to a < b, when checking the annotated design code according to the design specification, since a b in the design specification, the checker in the test case will output the test error check result. This indicates that the checker in the test case corresponding to the design under test has no integrity risk, that is, the checker can detect the error in the test case.

[0066] It should be understood that when the design under test corresponds to multiple test cases, that is, when the design under test requires multiple test cases for verification testing, since all test cases are run on the code that is injected with errors, it is not possible to know which test case has a problem simply by looking at the check results of the test that indicates no error. Therefore, if the error detection results of the test that indicates no error are obtained, it is necessary to analyze all test cases to find out why no error was reported.

[0067] Additionally, if the checker doesn't report an error, it could be due to a problem with the test case itself, or the design under test (DUT) might have a problem. For example, the DUT might have a coding error; the original design specification might define parameter a > b, but the code might have written parameter a < b. When annotating the code, the parameter a b. When the checker examines the annotated code according to the design specification, it won't report an error because both the annotated code and the design specification use parameter a > b. However, the reason for this situation is not a problem with the checker but an error in the DUT itself, so the DUT needs to be investigated.

[0068] Step 104: If the inspection results indicate that the test did not report any errors, confirm that the checker in the test case has a risk of incompleteness.

[0069] In this embodiment, if the inspection result indicates that the test has errors, it means that the checker in the test cases of the design under test is working properly, there is no risk of incompleteness, and the design under test is written according to the design specifications. In this case, if there are multiple code points in the design under test that can be injected with errors, the error injection is performed on the next code point in the design under test until all code points in the design under test have been injected with errors.

[0070] As an example, if the design under test has N code points that can be injected with errors, after injecting errors into the i-th code point to obtain the i-th injected design code, and the inspection result of the i-th injected design code indicates that there is an error in the test, determine whether i < N. If yes, update i = i + 1, and return to execute the step of injecting errors into the design under test at the i-th code point to obtain the i-th injected design code, until i = N, and complete the error injection for all code points of the design under test.

[0071] In the solution provided in this embodiment, error injection is performed at code points in the design under test to obtain the injected design code. Then, test cases corresponding to the design under test are run on the injected design code to obtain the checker results in the test cases. These check results can determine whether the checker in the test cases has any incompleteness risk. Therefore, this embodiment of the disclosure achieves the verification of the completeness of test cases.

[0072] This disclosure also provides a method for verifying the checker in a test case, such as... Figure 2 As shown, the method may include the following steps: Step 201: The verification personnel complete the construction of the testbench and test cases required for IP verification, and ensure that the test case coverage reaches 100%.

[0073] Step 202: Analyze the design under test and count the code points in the design under test that can be used for error injection.

[0074] Step 203: Select one code point at a time to inject errors into the design under test.

[0075] It should be understood that when there are multiple code points in the design under test that can be used for error injection, these code points can be numbered so that error injection can be performed one by one.

[0076] Step 204: Regress all test cases corresponding to the design under test after the error injection to obtain the inspection results.

[0077] Step 205: Determine if there are any errors in the check results. If so, return to step 203; otherwise, proceed to step 206.

[0078] It should be understood that when the inspection results show an error, it means that the inspector has no risk of incompleteness. When returning to step 203, the next code point is selected to inject errors into the design under test.

[0079] Once it is confirmed that all code points have completed the error injection into the design under test, the completeness analysis of the test case checker is terminated.

[0080] Step 206: If the checker is found to be incomplete, analyze the design under test and the test case to confirm the reason for the absence of errors.

[0081] This disclosure also provides an apparatus for verifying a checker in a test case, which is used to execute the method for verifying a checker in a test case provided in any of the above embodiments. Figure 3 As shown, the device includes: Module 31 is used to acquire code points in the design under test that can be used for error annotation. Injection module 32 is used to inject errors into the code points to obtain the error-injection design code; The execution module 33 is used to run test cases corresponding to the design under test on the error-annotated design code and obtain the inspection results; The determination module 34 is used to confirm that the checker in the test case has a risk of incompleteness if the check result indicates that the test has no errors.

[0082] In some embodiments, the device is further used for: Before obtaining the inspection results, the test cases corresponding to the design under test are run on the punctured design code to confirm that the coverage of the test cases reaches a preset coverage threshold.

[0083] In some embodiments, the acquisition module 31 is used for: From the design under test, find code statements that meet the set conditions, which are used to indicate that the semantics of the code statements can be modified; The code statement is used as the code point that can be used for error annotation.

[0084] In some embodiments, the injection module 32 is used for: If the code point includes an operator, modify the operator or modify the operator's operation parameters to obtain the error annotation design code.

[0085] In some embodiments, the running module 33 is used to: Obtain the design specifications of the design under test; The inspector is invoked to detect the error-injection design code according to the design specifications, and the inspection results are obtained.

[0086] In some embodiments, when the number of code points is N, error injection is performed on the code points to obtain the error-injected design code, including: Perform error injection on each i-th code point to obtain the error injection design code corresponding to the i-th code point, where i≤N; This device is also used for: If the check result of the error injection design code corresponding to the i-th code point indicates that there is an error in the test, and i < N, update i = i + 1, return to the step of performing error injection on the i-th code point one by one to obtain the error injection design code corresponding to the i-th code point, until i = N.

[0087] The apparatus for the checker in the verification test cases provided in this disclosure and the method for the checker in the verification test cases provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0088] This disclosure also provides an electronic device for executing the method of the checker in the above-described verification test cases. Please refer to... Figure 4 It illustrates a schematic diagram of an electronic device provided by some embodiments of this disclosure. For example... Figure 4 As shown, the electronic device 4 includes: a processor 401, a memory 402, and a bus 403, wherein the processor 401 and the memory 402 are connected via the bus 403; the memory 402 stores a computer program that can run on the processor 401, and when the processor 401 runs the computer program, it executes the method of the checker in the verification test case provided in any of the foregoing embodiments of this disclosure.

[0089] The memory 402 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0090] Bus 403 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 402 is used to store programs. After receiving an execution instruction, processor 401 executes the program. The checker method in the verification test case disclosed in any of the foregoing embodiments of this disclosure can be applied to processor 401, or implemented by processor 401.

[0091] Processor 401 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 401 or by instructions in software form. The processor 401 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), an Off-the-shelf Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 402. Processor 401 reads the information in memory 402 and, in conjunction with its hardware, completes the steps of the above method.

[0092] The electronic device provided in this disclosure and the checker method in the verification test cases provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0093] This disclosure also provides a computer-readable storage medium corresponding to the checker method in the verification test cases provided in the foregoing embodiments. The computer-readable storage medium is an optical disc, on which a computer program (i.e., a computer program product) is stored. When the computer program is run by a processor, it executes the checker method in the verification test cases provided in any of the foregoing embodiments.

[0094] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0095] The computer-readable storage medium provided in the above embodiments of this disclosure and the checker method in the verification test cases provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0096] This disclosure also provides a computer program product; please refer to [reference needed]. Figure 5 The computer program product 500 carries program code, namely computer program 501. The instructions included in the computer program 501 can be used to execute the steps of the method of the checker in the verification test case described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0097] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0098] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0099] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0100] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0101] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0102] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0103] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0104] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for verifying the checker in a test case, characterized in that, include: Identify code points in the design under test that can be used for bug fixing; Error injection is performed on the aforementioned code points to obtain the error-injection design code; Run the test cases corresponding to the design under test on the punctured design code to obtain the inspection results; If the inspection results indicate that the test did not report any errors, it is confirmed that the checker in the test case has a risk of incompleteness.

2. The method according to claim 1, characterized in that, Before running the test cases corresponding to the design under test on the punctured design code to obtain the inspection results, the process also includes: Confirm that the coverage of the test cases reaches the preset coverage threshold.

3. The method according to claim 1, characterized in that, Obtain code points in the design under test that can be used for bug fixing, including: From the design under test, find code statements that meet the set conditions, which are used to indicate that the semantics of the code statements can be modified; The code statement is used as the code point that can be used for error annotation.

4. The method according to claim 3, characterized in that, Error injection is performed on the aforementioned code points to obtain the error injection design code, including: If the code point includes an operator, modify the operator or modify the operator's operation parameters to obtain the error annotation design code.

5. The method according to any one of claims 1-4, characterized in that, Run the test cases corresponding to the design under test on the punctured design code to obtain the inspection results, including: Obtain the design specifications of the design under test; The inspector is invoked to detect the error-injection design code according to the design specifications, and the inspection results are obtained.

6. The method according to claim 1, characterized in that, When there are N code points, error injection is performed on these code points to obtain the error-injection design code, including: Perform error injection on each i-th code point to obtain the error injection design code corresponding to the i-th code point, where i≤N; The method further includes: If the check result of the error injection design code corresponding to the i-th code point indicates that there is an error in the test, and i < N, update i = i + 1, return to the step of performing error injection on the i-th code point one by one to obtain the error injection design code corresponding to the i-th code point, until i = N.

7. An apparatus for verifying an inspector in a test case, characterized in that, include: The acquisition module is used to acquire code points in the design under test that can be used for error annotation. The injection module is used to inject errors into the code points to obtain the error-injection design code; The execution module is used to run test cases corresponding to the design under test on the error-injected design code and obtain the inspection results. The determination module is used to confirm that the checker in the test case has a risk of incompleteness if the check result indicates that the test has no errors.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.

Images