Techniques for test case prioritization

Abstract

A, preferably computer-implemented, method of test case prioritization, the method comprising: obtaining a test case coverage for each one of a plurality of the test cases, wherein the test cases serve for testing an integrated circuit, e.g., in a hardware description language, preferably on the register transfer level, determining a priority of each test case based on the respective test case coverage, wherein the priority indicates an execution sequence of the test cases.

Description

[0001] Techniques for Test Case Prioritization

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to the verification of integrated circuits. Moreover, the present disclosure relates to improvement in test benches for, and in regression testing of integrated circuits.

[0004] BACKGROUND

[0005] Verification planning and management involves identifying the features of a device under test, such as an integrated circuit, that need to be verified, prioritizing those features, measuring progress, and adjusting the allocation of verification resources so that verification closure can be reached on the required timescale. The verification of an integrated circuit can be accomplished using formal verification (also known as property checking) , simulation, emulation, or FPGA prototyping .

[0006] Regression testing is a method pursued in all levels of system testing. The prime aspiration of running a Regression Test is to assure that modified or amended component of the system does not give way for bugs in the unmodified portion of the system. The re-execution of test cases is performed to verify that the previous functionality and the present modifications is desirably functioning.

[0007] SUMMARY

[0008] The various regression testing techniques, and test techniques in general, comprise test case minimization, test case selection and / or test case prioritization. The aim of test case minimization is to eliminate the redundant test cases, while test case selection is performed to reduce the size of a test suite, i.e., the number of test cases. Test case prioritization is concerned with ordering of test cases for de- tection of faults at the earliest. Hence, test case prioritization is the process of arranging test cases in an order according to some criteria. For example, one or more test cases may be available for the existing version of an integrated circuit. The required test cases, or test case group, comprise the test cases that are required to be executed for the modified version of the software. In addition, some out-dated test cases may exist that are not required by both the original integrated circuit and the modified integrated circuit.

[0009] Thus, test case prioritization may help in ordering test cases based on certain criteria such as maximization of fault coverage in order to reduce the testing cost and speed up the delivery of the, e.g., modified, integrated circuit. For example, the test cases may be obtained from a test case selection and are supplied as input to an algorithm for test case prioritization.

[0010] Test case prioritization techniques try to (re-) order test cases based on decreasing priority. The priority of test cases is established using some predefined testing criterion. Test cases having higher priority are preferred over lower priority ones in the process of testing, e.g., regression testing. Prioritization can be achieved on the basis of one or more objectives such as faults, code coverage, cost of execution, etc . .

[0011] The improved performance of the prioritized test cases in terms of reduced run time is a forever goal in this evertime-constrained world. Furthermore, intelligent prioritization would mean that more and more faults are exposed as early as possible. Thus, the multi-ob ective goal is to deliver an ordering of test cases, e.g., in a test suite or test bench, with the capacity to reveal maximum faults in the least possible time. It is an obj ect to make use of the test coverage and its failure detection probability when it comes to improving veri fication techniques .

[0012] According to a first aspect , a, preferably computer-implemented, method of test case prioriti zation is proposed . The method comprising obtaining a test case coverage for each one of a plurality of the test cases , wherein the test cases serve for testing an integrated circuit , e . g . , in a hardware description language , preferably on the register trans fer level . The method further comprising determining a priority of each test case based on the respective test case coverage , wherein the priority indicates an execution sequence of the test cases .

[0013] According to a second aspect , using reinforcement learning for determining a priority indicating an execution sequence of plurality of test cases is proposed .

[0014] According to a third aspect , a system, preferably comprising a processor and a memory, for test case prioriti zation, operative to perform the methods steps according to the first and / or second aspect is proposed .

[0015] According to a fourth aspect , a computer program, preferably stored in a non-transitory memory, comprising program code which when executed performs the method steps according to any one of the first and / or second aspect is proposed .

[0016] BRIEF DESCRITPION OF THE DRAWINGS

[0017] FIG 1 shows a veri fication process of an integrated circuit .

[0018] FIG 2 shows an integrated circuit .

[0019] FIG 3 shows a plurality of test cases .

[0020] FIG 4 shows a test case coverage .

[0021] FIG 5 shows a test case prioriti zation based on test case coverage . FIG 6 shows the use of reinforcement learning for test case prioritization.

[0022] FIG 7 shows the use of multiple agents for test case prioritization .

[0023] FIG 8 to 14 show exemplary embodiments of test case prioritization techniques.

[0024] FIG 15 shows a system for test case prioritization.

[0025] DETAILED DESCRIPTION

[0026] A verification process of an integrated circuit is shown in FIG 1. The verification may be an iterative process and is typically repeated until the desired functionality of the integrated circuit is achieved.

[0027] The integrated circuit may be given in the form of a description, e.g., an RTL code, in a hardware description language. The RTL code describes how data is transformed as it is passed between registers of the integrated circuit. The transforming of the data is performed by the combinational logic of the integrated circuit that exists between the registers. The RTL code may be subject to an RTL-level simulation. The RTL-level simulation simulates the behavior of the integrated circuit and allows for a verification of the integrated circuit prior to any translation of the RTL code by synthesis or implementation tools.

[0028] The RTL-level simulation is typically performed to verify RTL code syntax, and to confirm that the RTL code is functioning as intended. In this step, the integrated circuit is primarily described in RTL and consequently, no timing information is required.

[0029] After successful completion of the verification the RTL code may be synthesized. That is, the abstract specification of integrated circuit, given in the form of the RTL code, is turned into an implementation of the integrated circuit in terms of logic gates. The RTL code may be in a hardware description language, such as VHDL or Verilog. Some synthesis tools generate bitstreams for programmable logic devices such as PALs or FPGAs, while others target the creation of ASICs. Logic synthesis is one step in design of integrated circuits in the electronic design automation. Other steps comprise place and route and verification and validation.

[0030] Now, a verification plan may comprise a list of features to be tested. Features listed in the verification plan, i.e., to be tested, may be cross-referenced to the original integrated circuit design specification or to the RTL code or RTL implementation. Features may be identified as being mission- critical, primary, secondary, optional, stretch goals, and so forth, and may be prioritized.

[0031] During the verification process the same verification environment is executed repeatedly using different tests, also denoted as test cases herein, e.g., until the verification goals are met. One test can differ from another by configuring and / or constraining the verification environment in a different way or merely by starting a simulation with a different random seed. The one or more tests executed, e.g., by and / or in the verification environment, can be graded according to a variety of criteria including functional coverage, bugs detected, and simulation run time. The best-performing tests, according to some combination of these criteria, can then be used as the basis for a regression test suite. Here, a set of tests may be selected that achieve the maximum coverage in the minimum run time. Furthermore, tests that do not add to the functional coverage achieved by the best set of tests may be discarded. Thus, a highly optimized regression test suite, i.e., comprising a plurality of tests, may be obtained .

[0032] A verification environment may comprise a test bench. The test bench may also be given in the form of a hardware description language, HDL, code written for the simulator that: The test bench may instantiate and / or initialize the integrated circuit, e.g., said RTL code of the integrated cir- cult , may generate and / or apply stimulus to the integrated circuit , and / or may monitor the output result and / or checks for functional correctness of the integrated circuit .

[0033] Thus , a test bench may be used to display the simulation output to a file , a waveform, or to a display screen . A test bench may be used to apply stimulus to inputs of the integrated circuit . Thus , the test bench may be used to store the test conditions , inter alia the test cases , test case parameters and / or the test case results .

[0034] Coverage-driven veri fication uses a random veri fication technique that starts with a random stimulus . This approach relies on the brute power of randomi zation and compute server farms to cover the state space of the integrated circuit .

[0035] Code coverage measures the execution of the actual RTL code . Functional coverage , on the other hand, attempts to measure whether the features described in the veri fication plan have actually been executed by the integrated circuit . It is for example possible to achieve 100% code coverage but only 50% functional coverage .

[0036] An exemplary integrated circuit is shown in FIG 2 . The integrated circuit comprises registers and a multiplexer that routes the register' s outputs to an adder from where the results are routed again to further registers .

[0037] The current mainstream technology employed to capture the design of a complex integrated circuit , IC, is based upon the use of hardware description languages , HDLs , such as VHDL or Verilog . These languages support the description of circuits at a range of abstraction levels varying from gate level netlists up to purely algorithmic behaviors .

[0038] An integrated circuit may be (part of ) an application speci fic integrated circuit , AS IC, or a system-on-chip, SoC . A SoC may contains one or more programmable processors , on-chip memory, peripheral devices, and specifically designed hardware modules.

[0039] With the help of input stimulus, an error (bug) of the integrated circuit is produced (red color) but it requires support from the SEL line, e.g., provided by a test bench, to select this path so that this path is visible on the external ports. In the shown structure, currently SEL line is HIGH hence it is selecting the green channel, hence unless the SEL line is LOW the design error will keep hidden inside the design & will not be caught.

[0040] The integrated circuit may be described in RTL code. For example, the multiplexer may be given as: if sei = ' 1 ' then f <= a; else f <= b; end if;

[0041] Therein, a is routed through to the output if sel=l and b is routed to the output if sel=0.

[0042] Thus, the input stimulus may be decisive for a test case coverage. And since the input stimulus is dependent on the (random) seed is derived from, a different random seed may result in a different input stimulus, or simply stimulus. As described herein, it is inter alia proposed to make use of the (random) seed for the purpose of test case prioritization, e.g., by an artificial neural network. The ANN may serve as the agent in reinforcement learning, RL . The seed may be input into the ANN, or the seed may effect the test case coverage. That is, the seed may determine the stimulus data that is generated. In turn, the stimulus data is responsible for and / or may determine the test case coverage. Hence, the test case coverage may be determined based on the stimulus data and / or on the (random) seed of the stimulus data. An RTL description or RTL code may comprise a description of one or more hardware modules by means of a datapath and a control unit. The datapath is composed structurally, using a set of memory elements (registers) , processing elements (ALUs, multipliers, etc) and interconnection between these. The control unit is a behavior description of how data flows and is transformed in the datapath. RTL modeling assumes clock cycle level accuracy in the datapath and control unit descriptions. The RTL code may be input in a synthesis tools producing an IC design on a logic level.

[0043] In any case there may be more abstraction levels beyond the RTL which are known as system level, e.g., the transaction level, which may be provided in an HDL . For example, SystemC is a language that enables the use of several abstraction levels, including RTL and upper levels such as the transaction level.

[0044] Now turning to FIG 3 a plurality of tests, also referred to as test cases are shown. The test cases are identified by respective identifiers, e.g., the names of the test cases. In addition to the identifiers of the test cases the random seeds based on which the test cases have been executed are shown. These seeds may be stored together with the cases and / or the identifiers. Still further, the result or status of the respective test case may be stored and / or displayed as shown in FIG 3. The status or result of a test case may be pass or fail. In addition, other results or status such as warning are possible. Such an indication of the result of the test case may be associated with the respective test case. In addition, further information relating to the test case may be stored and / or associated with each respective test case. For example, the processing time, i.e., the duration of an execution of a respective test case, may stored. Furthermore, a point in time, e.g., given by a date, of an execution of the test case may be stored andVor associated to the respective test case. It should be understood that as already described the test cases serve for testing the desired features and / or functionalities of the integrated circuit. The test cases may be executed based on stimulus generated from the (random) seed.

[0045] Now turning to FIG 4, a coverage summary report shows how much of the integrated circuit has been covered by the one or more test cases. As shown in FIG 4 the test cases have covered 61.01% of the integrated circuit, while 38.99% has not been covered. This coverage summary report helps to evaluate the effectiveness of the testing strategy and identify the areas of the design that may require further testing.

[0046] Hence, tests may be run with one or more random seeds in order to achieve a desired coverage. The test(s) may be re-run with the same or modified stimulus in order to take account for a modified integrated circuit and / or to achieve a desired test coverage of the integrated circuit.

[0047] Test case coverage or coverage is used to measure tested and untested portions of the integrated circuit. Coverage may be defined as the percentage of verification objectives that have been met. For example, code coverage measures how much of the integrated circuit or its "design Code" is exercised. This includes the execution of design blocks, Number of Lines, Conditions, FSM, Toggle and Path. Code coverage is a metric that measures the degree of coverage of the integrated circuit by one or more test case. A code coverage tool, e.g., a simulator tool, may be used to automatically extract the code coverage from the integrated circuit or the design code. The quality of verification or test may be measured with help of code coverage tool. The set of features provided by code coverage tools usually includes line / block coverage, arc coverage for state machines, expression coverage, event coverage and toggle coverage.

[0048] Functional coverage is a user defined metric that measures the extent to which the design specification, as enumerated by features in the test plan, was exercised. It can be used to measure whether interesting scenarios, corner cases, specification invariants, or other applicable design conditions— captured as features of the test plan— have been observed, validated, and tested.

[0049] Functional coverage may be defined as a user-defined metric that measures how much of the design specification has been exercised in verification.

[0050] During the verification process the same verification environment, comprising the test bench and / or the test groups, may be executed repeatedly using different tests until the coverage goals are met. One test can differ from another by configuring or constraining the verification environment in a different way or merely by starting simulation with a different random seed.

[0051] Tests can be graded according to a variety of criteria including functional coverage, bugs detected, and simulation run time. The best-performing tests, according to some combination of these criteria, can then be used as the basis for the regression test suite: Typically a set of tests are selected that achieve the maximum coverage in the minimum run time, and discard any tests that do not add to the functional coverage achieved by the best set of tests. Towards the end of the verification process it is possible to arrive at a highly optimized regression test suite.

[0052] Sufficient information should be captured from each test execution to allow the simulation run to be reproduced. This information will include the random seed, the revision identifiers of the integrated circuit, e.g., its RTL code, and the verification code, and all relevant software version numbers.

[0053] For proper testing the test bench needs to generate proper input stimulus to activate an error in the integrated circuit. The test bench also needs to generate input stimulus to propagate all effects resulting from the design error to an output port. The test bench may also contain a monitor that can detect the design error that was first activated then propagated to a point for detection.

[0054] Even if the tests do not take a long time, the cost of testing is still expensive due to the required human effort to set up, execute and monitor the tests. For these reasons, test case prioritization is needed to provide earlier feedback for the developers and improve the cost-effectiveness, e.g., of regression testing.

[0055] There is a growing interest in test case prioritization techniques in the last years. There are many strategies that can be used to prioritize the test cases. For example, prioritizing the test cases that execute the most recent changes of the integrated circuit, e.g., changes in the RTL code of the integrated circuit. Another strategy is to use the test coverage data, where the tests that have more coverage are given higher priority. One more alternative is to use the faults history data, based on the hypothesis that the tests that recently failed are more likely to fail again. Other test case prioritization techniques may be based on the requirements, bug tracking reports, natural language processing of tests descriptions, flow model, fault severity estimates, and / or estimates of fault propagation probability. In any case, prioritization of test cases can hardly be solved with the use of non-adaptive methods only. The traditional greedy methods are inefficient since test case prioritization is an NP-hard problem because of the dynamic and stochastic nature of regression testing. For example, for an integrated circuit with a large code base with frequent edits, frequent changes in tests, or changing customer priorities over development stages .

[0056] As shown in FIG 5, a plurality of test cases may be obtained, e.g., as a result of test case selection. It should be understood that the test cases may be updated and / or that new test cases, e.g., for testing new feature of the integrated circuit may be added. Hence, the number and / or function of the test cases may change. This may even depend on the (random) seed that is used to generate stimulus (data) for the test case. The update, change and / or modification of the one or more test cases is illustrated as a bend arrow in FIG 5. For the one or more test cases a test case coverage may be determined, e.g., by a coverage tool, e.g., Questa, as proffered by Siemens. The coverage may be determined for one or more, a plurality of, if not all of the test cases. Based on the coverage a test case prioritization may be determined. Again, the prioritization may be determined for one or more, a subset of or for each one of the plurality of test cases. Therein, there is no need to provide the test case itself to the test case prioritization but an identifier is sufficient based on which the priority may be assigned or associated to the test case again. In addition to the test case (identifier) the coverage and additional information as described herein may be provided and / or processed in order to arrive at a priority for the respective test case. The priority is output by the test case prioritization, which may be a software program, for example given as program code in a programming language and which is for example executed by a processor and / or which is stored in a memory. The output of the test case prioritization may be stored in a memory and / or transferred to a test case execution, which also may be a software program, for example given as program code in a programming language and which is for example executed by a processor and / or which is stored in a memory. The test case execution may comprise a test bench which perform the test cases in an order according to the priority of the test case. That is, the test cases may be ordered in a sequence according to their priority. To that end the test cases are executed and the integrated circuit is thus subject to the respective tests. As the case may be, and as described herein, the integrated circuit may be modified making an adaptive testing necessary. Therefore, an adaptation of the priority of a test case (or the one or more test cases) and / or the sequence of test may become necessary. The result (s) of the test case ( s ) may be obtained and may be re-used to adapt the prioritiza- tion of the test cases as described herein, for example by providing a reward for a fail test case.

[0057] Through the use of machine learning algorithms, and in particular artificial neural networks, the knowledge from previous tests can be used to adapt to the changes. Reinforcement learning, RL, is one of the machine learning algorithms that may be used. RL requires no prior knowledge or any model of the environment. RL has the ability to learn from progressive interactions. RL may be applied in test case prioritization using the failure history, for example as proposed in "Reinforcement learning for automatic test case prioritization and selection in continuous integration, " arXiv preprint arXiv : 1811.04122 , 2018 for the development of software. However, test coverage is not yet used with the RL technique. Moreover, the RL techniques are not used in the context of verification tests for integrated circuits, e.g. given as an RTL code, where a test can be executed using different seeds. It is proposed to use the failure history and / or test coverage to enhance the prioritization techniques known. Moreover, it is proposed to use multiple agents for reinforcement learning. It is thus proposed to build on the hypothesis that tests that have more coverage can detect more failures and that coverage can be effective in test case prioritization. It is the aim to make use of the test coverage and its failure detection probability.

[0058] Reinforcement Learning, RL, is used to give each test case a priority. The priority may be given as a priority score, e.g., as a real-valued variable, or priority rank, e.g., a integer-valued variable. The priority may be an output of the artificial neural network, i.e., the agent of the RL algorithm. The artificial neural network may receive the test cases or a corresponding identifier and other information such as for example the test case coverage and / or history as an input. After the one or more test cases are prioritized, a schedule is made up of the prioritized test cases according to their corresponding given priority in ascending or descending order. RL may be applied as a model-free and / or online learning method for the prioritization problem. A mod- el-free method ensures that the solution does not have an initial understanding of the nature of the world and how the taken actions impact it. This is feasible for the prioritization of test cases, as there is no rigid pattern behind the presence and identification of failures within the integrated circuit .

[0059] Online learning describes a method that is learning constantly during its runtime. This is often particularly useful for testing of an integrated circuit, where metrics may shift over time based on the implementation objectives or changes in the test suite. Therefore, the process of prioritization for test cases should be constantly modified, e.g., in order to adapt the order of execution of the test cases.

[0060] FIG 6 depicts how an agent in RL learns the appropriate possible actions based on the current state in order to achieve the goal. The state corresponds to the test case (identifier) and may include additional information about the test case, such as the test case coverage and / or history. The agent receives a reward for each action it takes, reflecting whether the action brought the agent closer to the goal or not. Here, the action corresponds to a test case priority. The goal of the RL agent is to collect as many rewards as possible. The reward may depend on the result of the execution or run of the one or more test cases.

[0061] However, after a while, the agent is torn between exploiting known rewards and exploring new actions that will still contribute to the target. Therefore, the RL agent should balance between exploration and exploitation until converging to the best suite of actions, i.e., the optimal test case priority. The test case can be represented by a test case state variable. The test case state variable may comprise a test case identifier, e.g., name of the test case. The state variable may comprise the environment's response to the (previous) test case(s) , e.g., the result of the test case(s) , such as fail, pass, and / or warning. In any case the environment's response, i.e., the result of the (previous) test case(s) , such as fail, pass, and / or warning, may be included or comprised in the reward, which reward may serve as an input for the ANN. The state variable may represent the metadata of a test case, or one or more test cases. The state variable may comprise one or more of the following inf ormation / features : a. Last execution time: This feature represents the awareness of the information of the last time a test case was executed. This feature ensures that additional priority is given to test cases that have been introduced recently or to test cases that have not been executed for a long time. b. Duration of test case: This feature reflects the knowledge of the approximate test case duration. It ensures that the run time of test cases is considered. For example, a preference may be given to one or more test cases that have a short run time. This is based on the hypothesis that the more test cases that are scheduled as early as possible, the more failures are detected in less time. However, this must not be correct and test cases with a short runtime may be assigned a lower priority by the ANN. c. Previous execution verdicts: This feature represents the results as a vector of the previous test runs where a value of 1 means that a test failed, and 0 otherwise. The number of previous execution verdicts to be considered is determined by a tuned parameter: history length. The history length parameter is tuned based on an experiment conducted to find the best fit for the problem.

[0062] The test case coverage in the last run reflects the knowledge of how much code the test case covers. Based on the hypothesis that test cases with higher coverage, detect more failures. Therefore, test cases with higher coverage should be given higher priorities. The agent may implement a policy or policy function . The agent itsel f may be implemented by an arti ficial neural network . The policy function may serve for selecting the appropriate action based on an input state , the agent has to follow a policy function that maps the states to their corresponding appropriate actions . In the context of test case prioriti zation, a policy function maps a given test case to a priority on which the test case is scheduled . An optimal policy is the policy function that gives the test cases the optimal priorities to detect all the failures as early as possible . In general , a policy function is an approximation to the optimal policy . It is a loose approximation at first, but it adapts to the optimal policy over time and by gathering experience . Initially, the agent has no knowledge about the ef fects of the actions that can be taken because the policy function does not hold any state-action pairs yet . Accordingly, the agent starts to take random actions and learns the ef fects of the taken actions by receiving the corresponding rewards .

[0063] After one or more , preferably all , test cases are prioriti zed, scheduled, and / or executed, the reward function returns the verdict of a test case as an individual reward for each test case . The priority of a test case for the current cycle may be a decimal number that is submitted by the RL agent as an action . The scheduling of failed test cases is intended and therefore reinforced . I f a test case passed, no speci fic reward is given as including it neither improved nor reduced the schedule ' s quality according to the information available . No reward is provided for test cases that are not scheduled . The agent then learns to adapt its actions according to the received rewards .

[0064] As mentioned, the agent may be implemented in the form of an arti ficial neural network . Thus , one or more arti ficial neural networks , ANNs , may be used to approximate the policy function . The one or more ANNs map a given state to an ex- pected action i . e . , the priority of the test case . Multilayer Perceptron (MLP ) is a class of ANN . An MLP network is composed of multiple layers of nodes of perception . Any MLP network consists of at least three layers . The first layer is the input layer . The input layer takes the test case state as an input feature . Then, one or more hidden dense layers . Dense layers are layers in which all the nodes are interconnected together . At last , the output layer that makes a prediction . The number and si ze of the hidden layers may be decided based on an experience and / or computational considerations .

[0065] After having some experience , the agent has to choose between exploiting the learned actions and exploring the environment , corresponding to the execution and / or result of the test cases , by taking new actions . The agent has to balance between exploration and exploitation to take the appropriate actions . To implement this behavior, a random value may be chosen from a Gaussian distribution . The random value is added to the output ( action) of ANN to achieve exploration . The variance of the Gaussian distribution is determined by the exploration rate . The exploration rate is the agent ' s proclivity to explore the environment rather than exploit the learned policy . A higher exploration rate increases the variance of the Gaussian distribution, increasing the deviation to the output ( action) and allowing the agent to explore the environment more . Initially, the exploration rate has a high value so that the agent starts by exploring the environment . As the agent learns more about the environment , the exploration rate may be reduced starts after each cycle . Accordingly, the tendency of the agent to explore the environment is less than the earlier CI cycles and depends more on the gained experience .

[0066] Experience replay is a technique for improving the ef ficiency and stability of the arti ficial neural network by having a buf fer that keeps a certain number of the most recently gathered experience for training . Experience replay enables reus- ing the data multiple times for training, instead of ignoring them after collection. Hence, experience replay helps in improving the stability of the artificial neural network. The experience replay is implemented as a buffer queue, where the oldest gathered experience is removed when a new experience is collected. Experience batches are then randomly sampled from the buffer for the training process as specified. Prioritized sampling is used instead of uniform sampling, where recent experiences are given higher probabilities to be randomly chosen. The batches may be collected at fixed intervals for the purpose of training.

[0067] Turning to FIG 7, in multi-agent reinforcement learning, several agents sharing the same environment have the objective of finding the optimal policy function. Again, an agent may be given by an artificial neural network, that is multiple artificial neural networks are provided. Multi-agent RL provides higher stability since the actions taken are not based on one agent's experience but based on the experience learned from multiple agents which makes the actions taken more robust. Thus, test case prioritization based on multi-agent RL, i.e., MultiAgent Reinforced Test Case Prioritization is proposed. The multiple agents are used to obtain a final action, i.e., test case priority. Each agent learns, i.e. is trained, using different experience replay intervals, activation function, and / or solver for weight optimization. Afterwards, each agent takes an action based on the provided state variable. Then, for example the arithmetic mean of, the agents' actions are calculated and may be used as the final priority of the test case as shown in FIG 7. The agents may obtain the same reward, for example for determining the test case priority. The agents may work in parallel threads. Hence, no additional time for running the multiple agents is added.

[0068] The test case prioritization, e.g., comprising the machine learning and / or reinforcement learning algorithm, may be integrated in a software (product) , such as Verification IQ, for executing and / or visualizing, a verification of an inte- grated circuit. The test case prioritization may be triggered upon launching a new regression run and may assign each testcase a priority which is used to order the testcases. After the regression run finishes execution, the test case prioritization is re-run again, for example to be trained on the results of the testcases.

[0069] It is thus proposed to use an adaptive algorithm for test case prioritization that does not require previous knowledge about the regressions and / or does not depend on the language in which the test cases are written. Furthermore, the test case prioritization may be applied to the testing of an integrated circuit, e.g., in the RTL domain. Code coverage may be used in addition to test case history for the test case prioritization. Still further, multiple agents may be used for reinforcement learning in testcase prioritization. Moreover, test cases which run with different seeds can be prioritized.

[0070] Further exemplary steps are provided in the FIG 8 to 14.

[0071] For the purpose of test case prioritization a test case coverage for each one of a plurality of the test cases may be obtained in a step SI as shown in FIG 8. Therein, the test cases serve for testing an integrated circuit, e.g., in a hardware description language, preferably on the register transfer level. In a step S2 a priority of one or more test cases, or preferably of each test case, may be determined based on the respective test case coverage. Therein, the priority may indicate an execution sequence of the test cases. In a step 3, the priority may be assigned to the respective test case and / or to an identifier of the test case. Thus, a priority may be determined for one test case at a time. As a result, a priority of one or more test cases of the plurality of test cases may be determined. Preferably, a priority of each one of the plurality of test cases is obtained.

[0072] Turning to FIG 9, a plurality of test cases and / or identifiers for each one of the plurality of test cases may be ob- tained. The test cases and / or identifiers may be provided in a (ordered) list.

[0073] In a step S5, a test case coverage for each one of a plurality of the test cases may be determined, e.g., by a coverage tool. The test case coverage may be assigned to the respective test case and / or test case identifier. As the case may be, the test case coverage may only be determined for a subset of the plurality of test cases.

[0074] In a step S6, similarly to step S2, the priority of each test case based on the respective test case coverage may be determined. Again, a pair of test case (identifier) and (associated) test case coverage may be processed at a time. Hence, after determining the priority of a first test case (identifier) and (associated) first test case coverage, the priority of a second test case (identifier) and (associated) second test case coverage may be determined, until the priority of all, at least a subset of, or at least (a single) one of the test cases has been determined.

[0075] The priority of each test case may be determined based on at least one of the following: a point in time, e.g., in the past, of an execution of the test case, a seed for generating random test stimulus for the test case, a duration of an execution of the test case, and / or a pass or fail indication of the test case, e.g., during the last n excutions of the test case or during a past time period. Therein, n preferably is a positive integer. As mentioned, the seed may be used to determine the test case coverage, e.g., by inputting the seed into the ANN. The seed may be considered indirectly by its effect on the test case coverage, since the seed determines the stimulus data that is used for executing or running a test case. Therefore, the test case coverage of a test case may be used for determining the priority of a test case.

[0076] In a step S7, a sequence of the plurality of test cases may be determined, wherein the sequence is ordered according to the priority assigned to the test cases. That is, the sequence may be given as an (ordered) list of test cases or test case identifiers, e.g., in ascending or descending order, for example stored in a memory. The sequence determines the order in which the plurality of test cases are (to be) executed - or run, e.g. in a test bench or other verification software .

[0077] In a step S8, as shown in FIG 10, a reward may be obtained for each test case, in particular a reward for a failed test case, and particularly no reward for a passed test case may be obtained, for example for each one of the plurality of test cases. In a step S9, the priority of a test case may be determined based on the reward, e.g., in addition to the state variable of the test case. Accordingly, the sequence of test cases may be determined based on the priority, wherein the priority is determined based on the reward and / or the test case coverage. Consequently, in a step S10, as before, the sequence of test cases is determined based on the priority.

[0078] Turning to FIG 11 and step Sil, the steps as described herein may be repeated after one or more executions of the test case(s) and / or after one or more executions of the sequence of test cases. Thereby, the test case prioritization and / or the sequence of test cases may be adapted in order to take account of the changed integrated circuit and / or new test cases, e.g., determined based on test case selection.

[0079] As shown in step S12 of FIG 12, the state variable of a test case may be obtained as a result of one or more executions of said test case or the sequence of test cases. In step S13, a reward and / or a state variable representing a test case may be input into an artificial neural network, ANN. The ANN may be, e.g., a multi-layer perceptron. The state variable comprises one or more of the following, e.g., for each one of the plurality of test cases: a test case coverage, a point in time, e.g., in the past, of an execution of the test case, a seed for generating random test stimulus for the test case, a duration of an execution of the test case, and / or a pass or fail indication of the test case, e.g., during the last n excutions of the test case or during a past time period. Thus, in a step S14, the ANN may output a priority of the test case, preferably for each one of the plurality of test cases, e.g., after processing the coverage and / or the state variable of a test case.

[0080] As shown in step S15 of FIG 13, the output of the ANN may be adapted based on a random variable. The adapted output may then be used as priority of the test case, e.g., for exploring the search space of possible execution sequences of the test cases. In a step S16, the value range of the random variable may be decreased with an increasing number of test case executions .

[0081] Now turning to FIG 14, the artificial neural network may be trained before inferring the priority of the one or more test cases. The ANN may be trained based on historic state variables of the plurality of test cases, and / or the corresponding reward and / or priority, preferably wherein the training data is, e.g., randomly, sampled from historic state variables, e.g., for experience replay, in particular wherein the training data is updated based on newly received state variables of newly executed test cases, for example wherein the historic state variables in the training data are replaced by the newly received state variables based on a First In - First Out principle.

[0082] Preferably, in a step S17, a plurality of different artificial neural networks, ANN, is trained. For example, the training of the different ANN may be based on different training data and / or hyperparameters, e.g., based on different samples from a training data set. In a step S18, for example in the productive phase, the priority of the test case, preferably for each one of the plurality of test cases, may be determined based on the plurality of different ANN, e.g., the plurality of outputs of the different ANN, e.g. by adding and / or weighing the outputs.

[0083] Thus, using reinforcement learning for determining a priority indicating an execution sequence of plurality of test cases, for example according to the steps as described herein is proposed .

[0084] Turning to FIG 15, a system for test case prioritization is shown. The test case prioritization may be performed in and / or by software. Hitherto, the software may be provided in the form of program code that is executed by a processor of the test case prioritization system. The test case prioritization may be stored and / or run in a cloud environment, e.g., a server farm. The test case prioritization may be stored and / or run on a processing device, such as a computer. The test case prioritization may be implemented using program code. The test case prioritization may be communicatively coupled to a test bench for test case execution. The test bench and / or the test cases may be stored and / or executed on a processing device, such as a computer or cloud environment as the case may be. The test case execution may be given as a software (module) . Hitherto, the test case execution may be provided in the form of program code that is executed by a processor of the test case execution system. The test case execution may receive a priority and / or sequence of test cases for executing the test cases, e.g., from the test case prioritization. In case one or more test case priorities are received, the test case execution (engine) may sort or order the test cases according to their respective priority in order to arrive at the sequence of test cases. The test case execution and / or the test case prioritization may also comprise or be communicatively coupled to a coverage tool. The coverage tool may serve for determining the coverage of one or more of the plurality of test cases.

[0085] After the integrated circuit, IC, e.g., given in the form of a hardware description language, for example as RTL code, has been tested, the integrated circuit may be adapted based on a result of the test. That is, the RTL code of the IC design may be adapted. If all requirements are satisfied and / or if all tests are passed and thus presumable the IC design fulfills the desired features, the IC design may be synthesized into a hardware implementation. The IC design may thus be signed-off and / or taped-out. The IC design may then be stored in a file, e.g., on a non-transitory memory. Subsequently, a semiconducting material may be processed according to the hardware implementation. Thus, an integrated circuit may be fabricated and / or the fabrication of the IC may be controlled. For example, the Graphic Design System, GDSII, or the Open Artwork System Interchange Standard, OASIS, file format may be used to store the IC design. The IC design may thus be stored in a binary database file format. The IC design may represent and / or may be used for a (photo-)mask layout for produced during the integrated circuit.

[0086] Thus, the IC design, e.g., in a file format, may be received at a fabrication process to fabricate the integrated circuit. For example, a device manufacturing process may include providing the IC design file to a mask manufacturer to create one or more masks, such as masks to be used with photolithography processing. The mask may be used during the fabrication process to generate one or more wafers, which may be tested and separated into dies. A die may include a circuit including a device that includes the integrated circuit (according to the IC design) .

Claims

PATENT CLAIMS1. A, preferably computer-implemented, method of test case prioritization, the method comprising: obtaining a test case coverage for each one of a plurality of the test cases, wherein the test cases serve for testing an integrated circuit, e.g., in a hardware description language, preferably on the register transfer level, determining a priority of each test case based on the respective test case coverage, wherein the priority indicates an execution sequence of the test cases.

2. The method according to the preceding claim, further comprising assigning the priority to the respective test case and / or to an identifier of the test case.

3. The method according to any one of the preceding claims, further comprising obtaining a plurality of test cases and / or identifiers for each one of the plurality of test cases, e.g., in a list, for testing an integrated circuit.

4. The method according to any one of the preceding claims, further comprising determining a sequence of the plurality of test cases, wherein the sequence is ordered according to the priority assigned to the test cases.

5. The method according to any one of the preceding claims, further comprising the step of determining the priority of each test case based on at least one of the following: a point in time, e.g., in the past, of an execution of the test case, a seed for generating random test stimulus for the test case, a duration of an execution of the test case, and / ora pass or fail indication of the test case, e.g., during the last n excutions of the test case or during a past time peri- od .

6. The method according to any one the preceding claims, further comprising the step of obtaining a reward for each test case, in particular a reward for a failed test case, and particularly no reward for a passed test case.

7. The method according to any one of the preceding claims, further comprising the step of determining the priority of a test case based on the reward and / or determine the sequence of test cases based on the priority.

8. The method according to any one of the preceding claims, further comprising the step of repeating the steps of any one of the preceding claims after one or more executions of the test case(s) and / or after one or more executions of the sequence of test cases.

9. The method according to any one of the preceding claims, further comprising inputting a reward and / or a state variable representing a test case into an artificial neural network, ANN, e.g., a multi-layer perceptron, wherein the state variable comprises one or more of the following: a test case coverage, a point in time, e.g., in the past, of an execution of the test case, a seed for generating random test stimulus for the test case, a duration of an execution of the test case, and / or a pass or fail indication of the test case, e.g., during the last n excutions of the test case or during a past time period .

10. The method according to any one of the preceding claims, further comprising the step of obtaining the state variable of a test case as a result of one or more executions of said test case or the sequence of test cases.

11. The method according to any one of the preceding claims, further comprising the step of outputting by the ANN a priority of the test case, preferably for each one of the plurality of test cases.

12. The method according to any one of the preceding claims, further comprising the step of adapting the output of the ANN based on a random variable and using the adapted output as priority of the test case, e.g., for exploring the search space of possible execution sequences of the test cases.

13. The method according to the preceding claim, further comprising decreasing the value range of the random variable with increasing number of test case executions.

14. The method according to any one of the preceding claims, further comprising wherein the ANN is trained based on historic state variables of the plurality of test cases, and / or the corresponding reward and / or priority, preferably wherein the training data is, e.g., randomly, sampled from historic state variables, e.g., for experience replay, in particular wherein the training data is updated based on newly received state variables of newly executed test cases, for example wherein the historic state variables in the training data are replaced by the newly received state variables based on a First In - First Out principle.

15. The method according to any one of the preceding claims, further comprisingtraining a plurality of different artificial neural networks, ANN, for example trained based on different training data and / or hyperparameters, e.g., based on different samples from a training data set, determining the priority of the test case, preferably for each one of the plurality of test cases, based on the plurality of different ANN, e.g., the plurality of outputs of the different ANN, e.g. by adding and / or weighing the outputs.

16. Using reinforcement learning for determining a priority indicating an execution sequence of a plurality of test cases, e.g., according to any one of the preceding claims.

17. A system, preferably comprising a processor and a memory, for test case prioritization, operative to perform the methods steps of any one of the preceding claims.

18. A computer program, preferably stored in a non- transitory memory, comprising program code which when executed performs the method steps according to any one of the preceding claims.

Images