Chip verification automation regression test method and computer readable storage medium
By utilizing a seed set that contributes to the coverage of older versions and random seeds for automated regression testing in chip verification, the problems of low simulation efficiency and slow coverage improvement in chip verification are solved, achieving rapid coverage improvement and automated testing, thus improving testing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SPL ELECTRONICS TECH CO LTD
- Filing Date
- 2022-08-30
- Publication Date
- 2026-05-15
AI Technical Summary
Existing chip verification regression testing methods suffer from low simulation efficiency, slow coverage improvement, and reliance on human resources, leading to uncertainty.
By acquiring a set of prior seeds that contribute to the coverage of older code versions, performing regression tests on these seeds, and then combining them with random seeds for automated regression testing, seeds with no coverage contribution are removed, and seeds with coverage contribution are collected to form a new seed list, thereby achieving automated and rapid coverage improvement without human intervention.
To quickly achieve coverage requirements in the early stages of regression testing, and to ensure seed quality by continuously updating the seed list, efficient and automated regression testing can be achieved, shortening simulation time and avoiding the occupation of human resources.
Smart Images
Figure CN115328798B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of chip verification technology, specifically relating to an automated regression testing method for chip verification and a computer-readable storage medium. Background Technology
[0002] Chip verification involves using appropriate verification languages, tools, and methods to verify whether the chip design meets the chip's defined requirements and specifications before production, whether risks have been fully mitigated, and to identify and correct all defects.
[0003] In chip verification (IC verification), especially module and subsystem verification, versions are frequently updated and design code is modified. Because some test case stimuli have random constraints, extensive simulations are required to ensure quality. Therefore, a large number of regression test cases are often performed in the later stages, and each test case needs to be run many times with a random seed, as different seeds provide different coverage. For this scenario, there are generally three regression testing methods: the first is to run simulations endlessly for each test case based on a random seed, which is time-consuming and inefficient; the second is to add a large number of targeted test cases to cover code that is difficult to cover, which is manpower-intensive and inefficient; the third is a compromise between the first two methods, adding some targeted test cases and letting random seeds run the simulations, thus achieving a balance in simulation time and manpower. All three conventional regression testing methods suffer from low simulation efficiency during regression testing, meaning that the improvement in code coverage is slow.
[0004] Typically, the third method is often used for simulation in IC verification. However, this method still requires human resources, and the efficiency of simulation is uncertain due to human factors, resulting in unsatisfactory simulation efficiency. In other words, the rate of coverage improvement is still slow when using targeted test cases. Summary of the Invention
[0005] The purpose of this invention is to provide an automated regression testing method and a computer-readable storage medium for chip verification, in order to solve the problem that the coverage cannot be quickly improved in the early stage of regression testing in the prior art.
[0006] To address the aforementioned technical problems, this invention provides an automated regression testing method for chip verification, comprising the following steps:
[0007] 1) Obtain the prior seed set; the prior seed set includes simulation seeds that contribute to the coverage of simulation of older version code; the coverage includes at least one of code coverage and functional coverage;
[0008] 2) Perform regression testing on the updated code to be regressed using simulation seeds in the prior seed set; the updated code is obtained by updating the old version code in step 1).
[0009] 3) Use random simulation seeds to perform regression testing on the updated code again until the set coverage is achieved.
[0010] Its beneficial effects are as follows: By obtaining a prior seed set that contributes to the coverage of the old version code simulation, after the code version is updated, the simulation seeds and random seeds in the seed set are used to perform automated regression testing. This enables rapid regression coverage in the early stage of regression testing, that is, the coverage rate quickly approaches the coverage requirement. After using the simulation seeds in the prior seed set for regression testing, the random seeds are used to perform regression testing again, further improving the coverage rate. Moreover, the regression testing of this invention is automated regression testing, which does not require human intervention. Therefore, the automated regression testing method for chip verification of this invention can meet the coverage requirements while improving the efficiency of regression testing.
[0011] Furthermore, in step 2), when performing regression testing using simulation seeds in the prior seed set, simulation seeds that do not contribute to coverage are removed from the prior seed set based on the regression test results; the remaining simulation seeds in the prior seed set are included in the seed set of the updated code.
[0012] Considering that even simulation seeds that contributed to coverage in the previous version (old code version) may not contribute to coverage in the updated code version after a code version update, this invention aims to ensure that the seed list can be used for fast regression testing not only for the current code update but also for subsequent code updates. Therefore, during regression testing using simulation seeds in the seed list, this invention continues to assess whether each simulation seed contributes to coverage and removes those that do not. By removing simulation seeds that do not contribute to coverage during regression testing, the seed list after regression testing still consists of simulation seeds that contribute to coverage of the "current code version" (the code version being tested), allowing for verification in subsequent code versions. This ensures the continuous use and quality of the seed list, thereby improving the efficiency of regression testing for subsequent code versions.
[0013] Furthermore, in step 3), when performing regression testing on the updated code again using random simulation seeds, simulation seeds that contribute to coverage are also included in the seed set of the updated code.
[0014] By collecting simulation seeds that contribute to coverage from random simulation seeds and including the collected simulation seeds in the seed set, it is ensured that the simulation seeds in the seed set still meet the coverage requirements of the "current code version" (the code version for regression testing), which can be used for verification in subsequent code versions. This enables the continuous use of the seed list and ensures the quality of the seed list, thereby enabling rapid coverage of functionalities and design code during regression testing of subsequent code versions.
[0015] Furthermore, in step 3), when the code version to be re-tested is subjected to regression testing using random simulation seeds, simulation seeds that have been found to have bugs will also be included in the seed set of the updated code.
[0016] To cover functionalities where design code is prone to errors, we also collect seeds of bugs that have been found and subsequently fixed, and include these seeds in the seed list. By including these seeds in the seed list, we can avoid situations where bugged designs are reverted due to typos during the design process.
[0017] Furthermore, the coverage contribution is defined as follows: when performing regression testing, the simulation seed of the current regression test satisfies the following condition: if the sum of the coverage values obtained by all the simulation seeds that have been regressed plus the simulation seed of the current regression test is greater than the sum of the coverage values obtained by all the simulation seeds that have been regressed, then the simulation seed of the current regression test is a simulation seed that contributes to the coverage.
[0018] By comparing the coverage set generated by the current regression test seed with the coverage sets generated by all previously tested simulation seeds during the regression testing process, we can determine whether the current regression test seed contributes to the coverage of the "current code version" (the code being tested). If it does, the current test seed is added to the seed set; otherwise, it is either not added or removed (removed if it already exists in the seed set). Through this process, after regression testing with random seeds, the new seed set formed contains the minimum set of simulation seeds that meet the coverage requirements. This further improves the efficiency of continuing regression testing with simulation seeds from the new seed set after the code version is updated again, and enables rapid regression coverage in the early stages of regression testing.
[0019] Furthermore, simulation seeds that contribute to coverage are obtained as follows:
[0020] Determine whether cov(a+b)>cov(a) is true, where a is all the simulation seeds that have been tested in regression, b is the simulation seed of the current regression test, cov(a) represents the sum of the coverage results from the simulation of all the simulation seeds that have been tested in regression, and cov(a+b) represents the sum of the coverage results from the simulation of all the simulation seeds that have been tested in regression plus the simulation seed of the current regression test. If true, then the current regression test seed is a simulation seed that contributes to the coverage.
[0021] Further, in step 1), the prior seed set is the simulation seed set of the old version code verified on the chip.
[0022] Before updating the code version, chip verification must have been performed on the previous code version. Therefore, we only need to obtain the simulation seed set of the old version code in the chip verification from the chip verification to obtain the previous seed set.
[0023] Furthermore, in step 1), the prior seed set also includes simulation seeds that have discovered bugs in older versions of the code.
[0024] To cover functionalities where design code is prone to errors, simulation seeds for which bugs have been found in older versions of the code are also included in the prior seed set, which can prevent the situation of restoring a bugged design due to typos during the design process.
[0025] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing program instructions executable by a processor to implement the automated regression testing method for chip verification described above, and to achieve the same beneficial effects as the method. Attached Figure Description
[0026] Figure 1 This is a flowchart of the automated regression testing method for chip verification according to the present invention. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0028] Example of an automated regression testing method for chip verification:
[0029] In this embodiment, a pre-stored seed list is obtained (the existing seed set is recorded in the seed list). After the code version is updated, regression testing is performed using the simulation seeds in the seed list. After the regression test is completed, regression testing is performed again on the updated code using random seeds to obtain seeds that contribute to the coverage from the random seeds. The newly obtained seeds are added to the seed list to form a new seed list. After the code version is updated, the new seed list can be used for efficient automated regression testing, that is, to achieve rapid regression coverage.
[0030] By obtaining a list of seeds that contribute to coverage, after a code version update, automated regression testing is performed using both simulated seeds and random seeds from the seed list. This enables rapid regression coverage. Furthermore, after performing regression testing using simulated seeds from the seed list, another regression test is performed using random seeds to further improve coverage. Moreover, the regression testing in this embodiment is automated and requires no human intervention. Therefore, the automated regression testing method for chip verification of this invention can meet coverage requirements while improving regression testing efficiency.
[0031] like Figure 1 The specific method for implementing automated regression testing for chip verification is as follows:
[0032] 1) Collect simulation seeds that contribute to the coverage of the current code version and add the collected simulation seeds to the seed set.
[0033] The current code version is the previous code version (i.e., the old code version). Before the code version was updated, chip verification must have been performed on the old code version. Therefore, we only need to obtain the simulation seed set of the old code version in the chip verification from the chip verification to obtain the previous seed set.
[0034] In order to cover the functional points where design code is prone to errors, this embodiment also collects simulation seeds that have discovered bugs in the current code version, and includes simulation seeds that have discovered bugs in the unupdated code version in the seed set, which can avoid the situation of restoring bug designs due to typos when designing code.
[0035] By forming a seed set (recorded in a seed list) of simulation seeds that contribute to the coverage of the current code version, the simulation seeds in the seed list meet the coverage requirements of the current code version. Therefore, when the code is updated, the simulation seeds in the seed list can be used to simulate and cover the design code as quickly as possible.
[0036] 2) After the code version is updated, use the simulation seeds in the seed list to perform regression testing.
[0037] After the code version is updated, all the recorded seeds (i.e., the simulation seeds in the seed list) are run. Since the simulation seeds in the seed list have met the coverage requirements before the code version update, when using the simulation seeds in the seed list for automated regression testing after the code version update, the design code can be simulated and covered as quickly as possible, bugs can be found in time, simulation efficiency can be improved and the verification progress can be accelerated.
[0038] Considering that even simulation seeds that contributed to coverage in the previous version may not contribute to coverage in the updated code version after a code version update, this embodiment, in order to ensure that the seed list can not only enable rapid regression testing for the current code update but also enable rapid regression testing when the code is updated again after this update, still judges whether the simulation seeds in the seed list contribute to coverage during regression testing, and removes simulation seeds that do not contribute to coverage. By removing simulation seeds that do not contribute to coverage during regression testing, the seed list after the regression test still consists of simulation seeds that contribute to coverage of the "current code version" (the code version being retested), which can be used for verification in subsequent code versions. This achieves continuous use of the seed list and ensures the quality of the seed list.
[0039] In this embodiment, the criterion for determining whether the simulation seed contributes to the coverage is: whether cov(a+b)>cov(a) holds true, where a is the set of seeds that have been regressed and whose seed list has been recorded, b is the simulation seed of the current regression test, cov(a) represents the sum of coverage results from the simulation of the set of seeds that have been regressed and whose seed list has been recorded, and cov(a+b) represents the sum of coverage results from the simulation of the set of seeds that have been regressed and whose seed list has been recorded plus the simulation seed of the current regression test. If true, the simulation seed of the current regression test contributes to the coverage; if false, the simulation seed of the current regression test does not contribute to the coverage. That is, whether it contributes depends on whether the result of the current simulation seed improves the coverage of the sum of results from the set of seeds that have been regressed and whose seed list has been recorded. The improvement in coverage includes at least one improvement in functional coverage and code coverage.
[0040] 3) After the regression test is completed, continue to perform regression tests using random seeds.
[0041] Although the simulation seeds in the seed list meet the coverage requirements of the code version before the update, and these seeds can quickly increase coverage to near saturation after the code version update and promptly detect bugs in the new code version, the new code version may have added new features or bugs. Therefore, to meet the coverage requirements of the updated code version, in addition to using the existing seed list for rapid regression simulation, it is also necessary to continue simulation using random seeds. If bugs are found during this process, they will be located and fixed. By continuing to use random seeds for regression testing, the coverage requirements of the updated code version are guaranteed.
[0042] 4) Collect the seeds that contribute to the coverage from the random seeds and add the collected seeds to the seed list.
[0043] To ensure the seed list can perform rapid regression testing not only on the currently updated code but also on subsequent code updates, this embodiment, during automated regression testing using random seeds, also determines whether the random seeds contribute to coverage (as per the "criteria for judging whether simulated seeds contribute to coverage" in step 2). This allows for the collection of new seeds useful for coverage, which are then added to the seed list to ensure its completeness. Seeds found to contain bugs during this process are also included in the seed list.
[0044] By collecting seeds that contribute to coverage from random seeds and adding the collected seeds to the seed list, the simulation seeds in the seed list are guaranteed to still meet the coverage requirements of the "current code version" (the code version of the regression test), which can be used for verification in subsequent code versions. This enables the continuous use of the seed list and ensures the quality of the seed list.
[0045] Furthermore, when performing regression testing using random seeds, simulation seeds containing previously discovered bugs are also acquired. To cover functionalities where design code is prone to errors, seeds where bugs were discovered and subsequently fixed are also collected and included in the seed list. By including previously discovered and fixed seeds in the seed list, the possibility of reverting to a buggy design due to typos during code design can be avoided.
[0046] When performing regression testing on updated code, a seed list that contributes to the coverage of the current code is obtained. After the code version is updated, the seed list is first used for rapid regression verification, followed by automated regression testing using random seeds. Rapid regression testing can simulate and cover the design code as quickly as possible, promptly identify bugs, improve simulation efficiency, and accelerate the verification process. Furthermore, when performing regression testing on updated code, simulation seeds that do not contribute to the updated code version are removed from the seed list, and random seeds that contribute to the updated code version are added to the seed list. This method updates the simulation seeds in the seed list, ensuring that the simulation seeds in the seed list consistently contribute to the coverage of the "current code version" (the code version being regressed) while also meeting the coverage requirements of the "current code version" (the code version being regressed). Therefore, the regression testing method of this invention ensures that the design code is updated to the latest version, avoids the restoration of bugged designs due to typos, and enables the sustainable use of the seed list. Furthermore, this method performs highly efficient automated regression testing for chip verification. This includes using old seeds (i.e., simulation seeds in the seed list) to quickly regress for efficient coverage and collecting and recording seeds that contribute to coverage. Both of these are fully automated simulation processes. Therefore, the automated regression testing method for chip verification in this embodiment not only meets the coverage requirements of regression testing but also shortens the simulation time and accelerates the verification progress.
[0047] Examples of computer-readable storage media:
[0048] The computer-readable storage medium in this embodiment has program instructions that can be executed by a processor to implement the method steps of automated regression testing for chip verification. The specific method of automated regression testing for chip verification has been described clearly enough in the embodiment of the method of automated regression testing for chip verification, and will not be repeated here.
[0049] Furthermore, the above-mentioned automated regression testing method for chip verification can be automated by computer scripts, including collecting and recording seeds, determining whether a seed contributes, performing regression simulation, and updating the seed list. Therefore, no updates are required, making it a one-time solution that is universal across various verification platforms. This eliminates the need for manpower, automating the verification process and improving efficiency.
[0050] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. The scope of patent protection of the present invention shall be determined by the claims. Similarly, any equivalent structural changes made based on the description and drawings of the present invention shall also be included within the scope of protection of the present invention.
Claims
1. An automated regression testing method for chip verification, characterized in that, Includes the following steps: 1) Obtain the prior seed set; the prior seed set includes simulation seeds that contribute to the coverage of simulation of older code versions; the coverage includes at least one of code coverage and functional coverage; 2) Perform regression testing on the updated code to be regressed using simulation seeds in the prior seed set. During the regression test, determine whether the simulation seeds in the prior seed set contribute to the coverage. Remove simulation seeds that do not contribute to the coverage and obtain a list of seeds that contribute to the coverage of the updated code. The updated code is obtained by updating the old version code in step 1). 3) Use random simulation seeds to perform regression testing on the updated code again until the set coverage is achieved. During this regression test, determine whether the random simulation seeds contribute to the coverage. Add random simulation seeds that contribute to the coverage to the seed list. The seed list is used for subsequent code version verification. The criterion for determining whether the simulated seed in steps 2) and 3) contributes to the coverage rate is: if the sum of the coverage rates of all simulated seeds that have been regressed plus the current simulated seed that is being regressed is greater than the sum of the coverage rates of all simulated seeds that have been regressed, then the current simulated seed that is being regressed is considered to contribute to the coverage rate.
2. The automated regression testing method for chip verification according to claim 1, characterized in that, In step 2), when performing regression testing using simulation seeds in the prior seed set, simulation seeds that do not contribute to coverage are removed from the prior seed set based on the regression test results; the remaining simulation seeds in the prior seed set are then included in the seed set of the updated code.
3. The automated regression testing method for chip verification according to claim 1, characterized in that, In step 3), when the updated code is re-tested using random simulation seeds, simulation seeds that contribute to coverage are also included in the seed set of the updated code.
4. The automated regression testing method for chip verification according to claim 3, characterized in that, In step 3), when the code version to be retested is subjected to regression testing again using random simulation seeds, simulation seeds that have been found to have bugs will also be included in the seed set of the updated code.
5. The automated regression testing method for chip verification according to claim 2 or 3, characterized in that, In step 3), when the code version to be retested is subjected to regression testing again using random simulation seeds, simulation seeds that have been found to have bugs and then corrected are also included in the seed set of the updated code.
6. The automated regression testing method for chip verification according to claim 1, characterized in that, In step 1), the prior seed set is the simulation seed set for old version code verification on the chip.
7. The automated regression testing method for chip verification according to claim 1, characterized in that, In step 1), the prior seed set also includes simulation seeds that have discovered bugs in older versions of the code.
8. A computer-readable storage medium, characterized in that, It stores program instructions that can be executed by a processor to implement the steps of any of the automated regression testing methods for chip verification as described in claims 1 to 7.