Automated test equipment, process, and computer program for testing one or more devices under test, wherein different test activities utilize a subset of the resources of the device under test

Abstract

An embodiment according to the present invention includes an automated test equipment for testing one or more devices under test. The automated test equipment is configured to automatically and dynamically generate one or more, preferably multiple, test scenarios, and the test scenarios include multiple test activities. Different test activities utilize subsets of the resources of the device under test, and these subsets may overlap. The automated test equipment is configured to generate multiple test scenarios such that the resources of the device under test associated with the multiple test activities of the test scenarios do not conflict with each other.

Description

Technical Field

[0001] Embodiments in accordance with the present invention relate to automated test equipment. Other embodiments in accordance with the present invention relate to system-level testing. Other embodiments in accordance with the present invention relate to system-on-chip testing. Embodiments in accordance with the present invention relate to coverage testing. Background Art

[0002] Although certain devices pass structural and / or parametric tests, they fail system-level testing (SLT), which is an empirical observation or fact. This is typically attributed to unrealistic test conditions during structural testing.

[0003] For example, compared to system-level testing, the activity patterns across die regions are very different during structural testing. Structural testing spends most of its time on shift operations with very low frequencies but very high pattern activities. This results in an unnatural power load, and thus the relationship between the voltage curve and the position and time on the die is unrealistic.

[0004] Another example is that due to the difficulty of automatic test pattern generation, power domain crossing and / or clock domain crossing are usually not included in structural testing. Faults in this area are neither sensitized nor detected.

[0005] Structural tests are affected by unrealistic test conditions and are optimized for excellent fault detection capabilities. In contrast, SLT consists entirely of legal user scenarios. Without knowing any fault models, it is generally agreed or commonly believed that failing SLT will identify truly bad devices.

[0006] System-level testing has certain disadvantages. For example, SLT is by no means exhaustive because the device under test (DUT) runs only a small fraction of the possible uses and scenarios in only one selected environment, such as power supply voltage, frequency, and temperature.

[0007] Another disadvantage is that it is difficult to debug when SLT fails because the time between fault sensitization and detection may be very long and / or the exact order of all activities is unknown and / or the simulation time of SLT will be too long.

[0008] In addition, the running time of SLT is very long, and ten minutes or more is not an exception.

[0009] In addition, it is difficult to deploy SLT in a test chamber because a large number of system boards must be maintained for each DUT.

[0010] There is a need to improve system-level testing to provide a better trade-off among possible user scenarios, running duration, and practicality. Summary of the Invention

[0011] Embodiments in accordance with the present invention include an automated test equipment for testing one or more devices under test (DUTs), such as a system-on-chip (SoC). The automated test equipment (ATE) is configured to automatically and dynamically generate one or more (preferably multiple) test scenarios, and the test scenarios include multiple test activities. Different test activities (such as A1, A2, A3) utilize subsets of DUT resources, and these subsets can overlap, such as A1: CPU2, Mem3, A2: Mem3, A3: MPEG. The automated test equipment is configured to generate multiple test scenarios such that the resources of the DUT associated with the multiple test activities of the test scenarios do not conflict with each other.

[0012] It can be expected that on-chip system testing (OCST) will find more relevant problems than SLT because it exposes the DUT to more legitimate test conditions compared to SLT. When a problem occurs during the combination of test activities, the problem can be considered relevant, which can simulate a real use case or be part of a real use case.

[0013] OCST may change external test conditions, such as power supply voltage and frequency, to sensitize faults and / or internal test conditions (such as the intensity of test activities, such as the speed of data movement), so as to sensitize workload-related faults. Once sensitized to faults, OCST can provide good observability to detect failed behavior.

[0014] OCST can prove its effectiveness by showing which subset of the unpassed cases of SLT it can find. In other words, OCST may be able to find the unpassed subset of SLT.

[0015] Ideally, OCST tests can be reproduced in a simulation environment, such as for debugging. OCST should also utilize existing on-chip resources, such as design for test (DFT), design for debug (DFD), scan chains, logic built-in self-test (LBIST), and / or memory built-in self-test (MBIST). In addition, multithreading can be supported at the interface between multi-core DUTs and multiple ATE resources.

[0016] In a preferred embodiment, the multiple test activities of a given test scenario are configured to be executed simultaneously.

[0017] Since the ATE is configured to generate test scenarios in which the test activities use non-overlapping DUT resources, the test activities of the test scenarios do not conflict with each other. That is, the test can be accelerated by executing the test activities of a given test scenario simultaneously.

[0018] The general idea of the present invention is to use the DUT resources to simultaneously run a large number of simple and realistic self - test activities on the DUT, such as data block movement or built - in self - test in a combination of test activity intensity and concurrency changes. Each test activity can check the selected or involved IP blocks, but can also affect the test conditions of other blocks by loading power, clock tree, and thermal coupling.

[0019] According to an embodiment, one or more of the test activities are associated with one or more test parameters (such as voltage, speed, size of data transfer, time between data transfers, block size into which the total data transfer is divided, etc.). The test parameters control the behavior of the test activities. The test parameters are characterized by corresponding test parameter values, such as on / off, 5V, 50GB / s, etc., and / or are associated with one or more constraints or limits (e.g., for dynamically creating test parameter values).

[0020] The same test activity with different parameter values can be executed in a test sequence including one or more test scenarios. A test activity can be associated with more than one test parameter and / or constraint. For example, the test parameter value, which is the value of a test parameter (such as voltage), can vary with the test activity. The test activity may also have some constraints or limitations, for example, to protect the DUT or DUT resources, or to avoid scenarios that do not occur in actual use. Such constraints may be limited write speed, temperature limit, limited use of memory, etc.

[0021] According to an embodiment, one or more of the test activities are characterized in that one or more test parameter values exceed a predetermined limitation or exceed the chip specification, such as too low a voltage value and / or too fast a frequency.

[0022] The DUT test results that exceed or slightly exceed the predetermined limitation or chip specification may be related to the DUT performance within the predefined limitation or chip specification. That is, a DUT with good test results outside the chip specification may imply good performance of the DUT within the chip specification. On the contrary, a failed test outside the chip specification is not considered a hard hint or proof of a failed test within the chip specification, but indicates the presence of a defect.

[0023] According to an embodiment, under the constraint that the resources of the device under test associated with multiple test activities of a given test scenario do not conflict with each other, multiple test activities of the given test scenario are randomly selected to avoid conflicting requirements for the use or test parameter values from different test activities.

[0024] To test for unknown interactions between activities occurring in a real - world working environment or between test activities, test activities need to be randomly mixed in a test scenario. The randomization of parameterizable test activities covers many combinations of local workloads, thus simulating a large number of actual workload patterns that are considered to cause many system - level failures.

[0025] In a preferred embodiment, the automated test device includes a constraint solver for generating a test scenario to prevent conflicts between resources of the device under test.

[0026] Each test activity can have test parameters and optional constraints. For example, a data block write and read - back test activity can have test parameters such as an initial time delay, block size, start address, sub - block size, and time delay before read - back. Constraints can also be at the test scenario level and involve test parameters from multiple test activities.

[0027] A workstation or a controller (such as an on - chip OCST controller or an OCST card) can use the constraint solver to automatically generate a test scenario from test activities. The constraint solver matches test activities that can co - exist simultaneously without violating resource constraints.

[0028] In a preferred embodiment, one or more of the test activities include activating a stress generator arranged on the device under test, such as an on - chip stress generator, such as a traffic generator to internal and / or external buses.

[0029] For example, existing or new design - for - test (DFT) structures can generate controlled stress and increase observability to increase the likelihood of detection (or for more likely fault detection). The stress generator can include a traffic generator to the internal bus and / or a traffic generator to the external input to simulate traffic from missing external devices. The stress generator allows built - in self - test (BIST), such as memory or logic built - in self - test (MBIST, LBIST), to run in selected packaged IP blocks. Additionally, the stress generator can run input / output (I / O) loop - back tests. Further, the stress generator can use a wrapper (such as an IEEE1500 wrapper) to isolate the IP block and switch its scan chain to a linear feedback shift register (LFSR).

[0030] According to an embodiment, the automated test device is configured to generate a test sequence, such as a series of test steps of a test scenario.

[0031] Generate a test sequence of the test scenario to test different combinations of concurrent test activities to simulate possible real - world workload combinations.

[0032] According to an embodiment, the test sequence includes two or more test scenarios having the same test activity, where the test scenarios differ by at least one test parameter value.

[0033] The test sequence may include test scenarios of the same test activity with different test parameter values in order to test the impact of the test parameters on the test results. Changing only one test parameter value of the IP blocks of the DUT may also affect another IP block of the DUT.

[0034] Furthermore, testing scenarios with the same test activity but different test parameter values can further help to locate faulty IP blocks during the test evaluation process.

[0035] In a preferred embodiment, multiple test scenarios of the test sequence are randomly selected and / or sorted in order to simulate realistic workload patterns, for example, by an automatic test equipment or by a controller.

[0036] The randomization of the test scenarios of the test sequence further improves the test process by creating or generating realistic workloads. Ideally, the test scenarios are independent of each other, that is, a change in the execution order of the test scenarios is not expected to change the behavior of the DUT. The randomization of the scenario execution order can, for example, test whether the residual heat or residual data from a previous test scenario will affect the current test scenario.

[0037] In a preferred embodiment, the automatic test equipment is configured to generate a test sequence such that the test sequence is configured to be executed by a controller to collect test data.

[0038] The generated test sequence is executed by a controller or an OCST controller. The task of the controller is to trigger the execution of multiple test activities and read out the test results, such as pass / fail results and / or measurement results. The controller knows which test scenario to execute with which test parameter value. Debugging is required when an error occurs.

[0039] According to an embodiment, the controller is an on-chip processor, for example, together with an operating system, or a controller card, for example, as part of an automatic test equipment, and is configured to communicate with the automatic test equipment and the device under test.

[0040] The controller executes the test sequence on the DUT and provides measurement data or measurement results to the ATE. The on-chip processor or controller card with an operating system in the ATE transfers the measured values or test results or the data collected from the DUT to the ATE.

[0041] In a preferred embodiment, the controller includes one or more interfaces that are configured to read out DUT data, such as memory data and / or device under test sensor data, such as on-chip sensor data. Therefore, the ATE is configured to set the controller to read out DUT data and / or DUT sensor data.

[0042] Read out DUT test data and / or DUT sensor data through the interface to further improve the test by allowing comparison of the result data and / or result sensor data with expected result data or expected sensor data. The controller, preferably an on-chip processor and an optional test operating system, interfaces with the on-chip IJTAG interface to read out on-chip sensor data.

[0043] According to an embodiment, the controller interfaces to one or more sensors distributed over the area of the device under test, such that the controller is configured to read out sensor test data of the one or more sensors. Thus, the automatic test equipment is configured to set the controller to read out sensor test data of one or more sensors distributed over the area of the device under test included by the controller.

[0044] Regardless of whether the controller is an on-chip processor or an OCST control card communicating with the DUT or ATE workstation, the controller may include additional sensors distributed over the DUT die area to detect local anomalies. The sensors applied may include, for example, temperature, current, and / or voltage sensors.

[0045] In a preferred embodiment, the controller is configured to react when the collected test data meets a predetermined condition (e.g., exceeds one or more predetermined thresholds), such as collecting additional data, such as stored system data, program counter, or memory register information. Thus, the automatic test equipment is configured to set the controller to react when the collected test data meets a predetermined condition.

[0046] For example, in a razor circuit, the controller may interface to one or more sensors that have optional threshold alarm capabilities for temperature values (e.g., current / max / min peak temperature), voltage values (e.g., current / max / min peak voltage), and / or timing violations. This alarm capability may trigger, for example, the storage of important system data (e.g., program counter, memory controller register) to simplify debugging.

[0047] According to an embodiment, the controller is configured to transfer the collected test data to the automated test equipment.

[0048] The collected test data, whether test result data or measured data, or any data collected triggered by the measured data or test result data, is provided by the controller to the ATE.

[0049] According to an embodiment, the controller or the automated test equipment is configured to dynamically create test parameter values based on test activity constraints and / or based on the collected test data.

[0050] The collected test data can be analyzed by the ATE or the controller during the test process to modify existing test parameter values or add one or more test scenarios to the test sequence with new test parameter values. The creation of new test parameter values can take into account the constraints of a given test activity.

[0051] In a preferred embodiment, the controller or the automated test equipment is configured to analyze the collected test data to optimize the test sequence, such as solving the minimum coverage set problem, reducing redundancy, and / or never failing the test activity.

[0052] The collected test data is used not only to dynamically create test parameter values but also to optimize the test sequence. The general concept of testing is to identify those test parameters that affect the occurrence of test failures. Redundant test activities can help evaluate the impact of, for example, a single or a small group of test parameters. After identifying the most influential test activities and / or test scenarios, the number of test activities in the test scenario or the test scenario can be reduced to a minimum set of test activities to reduce the running time of the test sequence.

[0053] According to an embodiment, the controller of the automated test equipment is configured to compare or calculate the correlation between the results of system-level tests within a predetermined limit and the collected test data (such as device under test data, device under test sensor data, sensor test data) within and / or outside the predetermined limit to optimize the test time.

[0054] The test is performed outside the predetermined limit or specification of the DUT and compared with the system-level test within the predetermined limit. A test activity with a positive test result beyond the predetermined limit may indicate passing the system-level test within the predetermined limit. Comparing the test results within the predetermined limit with the test results outside the predetermined limit can help reduce the number of test scenarios and / or test activities within the test scenario. It can also help identify or determine the extent to which the test parameter values can be outside the predetermined limit to provide a good correlation between the tests performed within and outside the predetermined limit.

[0055] In a preferred embodiment, the automated test equipment includes an artificial intelligence or machine learning unit, where the controller or the automated test equipment is configured to train the artificial intelligence or machine learning unit using the results of system-level tests and / or using the collected test data and / or using the comparison between system-level tests and the collected test data to optimize the test sequence.

[0056] Furthermore, in a preferred embodiment, the trained artificial intelligence or machine learning unit is configured to predict the results of system-level tests based on the collected test data.

[0057] An artificial intelligence or machine learning unit can be trained with the results of system-level tests combined with the collected test data and / or the collected sensor data to compare the system-level test results with the collected test data in order to predict the results of system-level tests based on the collected test data or measurement data.

[0058] Using the prediction of system-level test results based on the collected test data can further reduce the number of test scenarios and / or the number of test activities in the test scenarios with test sequences.

[0059] In a preferred embodiment, the controller or the automated test equipment is configured to analyze the collected test data to obtain test results, such as to identify the faulty device under test and / or the faulty device under test resources, and / or to classify the device under test.

[0060] Analyzing the collected test data, for example, by using statistical methods may have to define the faulty IP blocks of the DUT or classify the IP blocks of the DUT.

[0061] In a preferred embodiment, the trained artificial intelligence or machine learning unit is configured to analyze the collected test data in order to optimize the test sequence and / or the obtained test results.

[0062] The trained machine learning unit can further improve the test sequence and / or improve the statistical methods to classify the IP blocks of the DUT and / or define the faulty IP blocks of the DUT.

[0063] Corresponding methods and corresponding computer programs are created according to other embodiments of the present invention.

[0064] However, it should be noted that these methods are based on the same considerations as the corresponding automated test equipment. In addition, these methods can be supplemented by any features, functions, and details described herein regarding the automated test equipment, either individually or in combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Embodiments according to the present application will be described hereinafter with reference to the drawings, wherein:

[0066] Figure 1 A schematic diagram of a test arrangement including an automated test equipment for testing one or more devices under test according to an embodiment is shown;

[0067] Figure 2 A block diagram describing the test process performed by the test arrangement according to an embodiment is shown;

[0068] Figure 3 An exemplary test activity table, the input of the constraint solver according to an embodiment is shown;

[0069] Figure 4 Shows an exemplary resource conflict table used by a constraint solver according to an embodiment;

[0070] Figure 5 Shows an exemplary test scenario table used by a constraint solver according to an embodiment;

[0071] Figure 6 Shows an exemplary test step table used by a constraint solver according to an embodiment;

[0072] Figure 7 Shows an empty test result table with an exemplary test sequence created by a constraint solver according to an embodiment;

[0073] Figure 8 Shows an exemplary per - test - step failure table for optimizing the number of test steps according to an embodiment;

[0074] Figure 9 Shows a comparison table between SLT tests and OCST tests according to an embodiment;

[0075] Figure 10 Shows a table for combining the collected test data with SLT results used as a training data set for a machine learning module according to an embodiment;

[0076] Figure 11 Shows a table for combining the collected test data with OCST results used as a training data set for a machine learning module according to an embodiment. Detailed Description

[0077] Hereinafter, different inventive embodiments and aspects will be described. Further embodiments will be defined by the appended claims.

[0078] It should be noted that any embodiment as defined by the claims can be supplemented by any details, features, and functions described herein. In addition, the embodiments described herein can be used alone or, optionally, supplemented by any details, features, and functions included in the claims. Moreover, it should be noted that the various aspects described herein can be used alone or in combination. Thus, details can be added to each of the described aspects without adding details to another aspect. It should also be noted that the present disclosure explicitly or implicitly describes features that can be used in an automated test device. Therefore, any feature described herein can be used in the context of an automated test device.

[0079] In addition, the method-related features and functions disclosed herein can also be used in a device configured to perform such functions. Further, any features and functions regarding the device disclosed herein can also be used in the corresponding method. In other words, the methods disclosed herein can be supplemented by any features and functions described regarding the device.

[0080] The present invention will be more fully understood from the detailed description given below and the accompanying drawings of embodiments of the invention. However, this should not be construed as limiting the invention to the specific embodiments described, but is for explanation and understanding only.

[0081] According to Figure 1 of the embodiment

[0082] Figure 1 A schematic diagram of a test arrangement 100 is shown, the test arrangement 100 including an automated test equipment (ATE) 110 for testing one or more devices under test (DUTs) 120. Figure 1 An exemplary DUT 120 is also included, the exemplary DUT 120 including a plurality of DUT resources 130a-e. The DUT resources 130a-e may include different IP blocks, such as a CPU, a memory, an MPEG, etc.

[0083] The ATE 110 is configured to generate a plurality of test scenarios 140a-c, each test scenario including a plurality of test activities. For example, the test scenario 140c includes test activities 150a-c.

[0084] Each test activity is configured to utilize one or more of the DUT resources 130a-e. For example, the test activity 150a is configured to use the DUT resource 130e. Or for example, the test activity 150b is configured to use the DUT resources 130c and 130d. Another example may be the test activity 150c, which is configured to use the DUT resources 130a and 130b.

[0085] The automated test equipment 110 is configured to generate a plurality of test scenarios, such as test scenarios 140a-c, such that the DUT resources (such as DUT resources 130a-e) associated with the plurality of test activities (such as test activities 150a-c) do not conflict with each other. For example, the test activities 150a-c are grouped into the test scenario 140c such that the DUT resources 130a-e of the test activities 150a-c are non-conflicting, thereby allowing concurrent execution of the test activities 150a-c of a given test scenario 140c.

[0086] The general idea of on-chip system testing (OCST) is to use the DUT resources 130a-e to simultaneously run a large number of simple and real test activities on the DUT 120, where the combination and / or intensity of the concurrent test activities vary. Each test activity can check the involved IP modules, but can also affect the test conditions of other IP blocks by loading power, clock tree, and thermal coupling. Examples of test activities can include moving data blocks or performing built-in self-tests, such as memory built-in self-test (MBIST) or logic built-in self-test (LBIST).

[0087] For example, the OCST controller locally runs structural tests in certain IP blocks, such as LBIST, MBIST, etc. These test activities are clearly self-checking, but can also be used as stress generators to control the test conditions of other concurrently running test activities. In a test scenario, some IP cores run structural tests while some other cores participate in code-based test activities, for example.

[0088] The involved IP blocks can apply design for test (DFT) structures or techniques, such as generating stress to make faults sensitive and / or increasing observability, in order to possibly detect sensitive faults and / or provide access to the existing structure for debugging or for in-system testing.

[0089] The randomization of test activities or parameterizable test activities cover many combinations of local workloads, thus simulating a large number of actual workload patterns that are considered to cause many system-level faults. Existing or new design for test (DFT) structures can generate controlled stress and increase observability to increase the likelihood of fault detection (or for more likely fault detection).

[0090] The DFT structure can improve observability by interfacing, for example, the on-chip OCST controller to the on-chip IJTAG to read out on-chip sensor data.

[0091] For example, in a razor circuit, for current / maximum peak / minimum peak temperature and / or current / maximum peak / minimum peak voltage and / or timing violations, adding sensors with optional threshold alarm capabilities can further improve observability. The alarm can trigger the storage or saving of important system states, such as program counters, memory controller registers, to simplify debugging.

[0092] The observability can be further improved by additional sensors distributed over the die area to detect local anomalies. Or by connecting a processor to track memory and compare its content with the expected. In addition, assertion checkers, such as protocol checkers or CRC or bus traffic recorders, can be added to measure coverage and assist in debugging.

[0093] The test activities are orchestrated in a controlled manner by, for example, an on-chip system test (OCST) controller so that the most effective test conditions can be determined and a failure can be attributed to a specific test activity and its test parameter values.

[0094] According to Figure 2 embodiment

[0095] Figure 2 A block diagram 200 showing the test process performed by a test arrangement similar to Figure 1 the test arrangement 100 is shown.

[0096] The block diagram 200 starts with a test activity table 210. The test activity table 210 includes a column of test activities 212, where each test activity 212 can utilize one or more DUT resources 214. The corresponding DUT resources 214 are provided in the DUT resource 214 column. Additionally, the test activities 212 can have corresponding test parameters 216 and / or constraints 218 provided in separate columns. The constraints can be at the test scenario level and reference test parameters from multiple test activities.

[0097] The test activity table 210 is fed into a constraint solver 250. The constraint solver 250 can be included by the ATE 220, the controller 270, or it can be a separate entity, as Figure 2 shown. Figure 2 The constraint solver 250 of has one input and one output. The input of the constraint solver 250 is fed by the test activity table 210. The output of the constraint solver is a test sequence table 260.

[0098] The test sequence table 260 includes test scenarios 262, similar to Figure 1 the test scenarios 140a - c above. The test scenarios can include one or more test activities 212a - e, where each test activity 212a - e can include test parameter values 216a - e. The test sequence table 260 is provided to the ATE 220 or the controller 270.

[0099] For example, the first scenario can include a first test activity 212a with test parameters P1 216a, P2 216b and a test activity 212b with test parameters P3 216c and P4 216d. The second scenario can include a second test activity 212b with test parameters P3 216c and test parameter P4 216d. The third scenario can include a third test activity 212c with test parameters P3 216c and P5 216e. The fourth scenario can include a fourth test activity 212d with test parameters P2 216b and test parameter P4 266d.

[0100] The controller 270 takes the test sequence list 260 as input and outputs a test result table 280. The test result table 280 provided by the test block 270 includes test results 288 of one or more test activities 212 of the test scenario 262 executed on the DUT 282. The test result table 280 is fed to the ATE 220 and / or returned to the controller 270.

[0101] The ATE 220 or the controller 270 has accepted the test result table 280 as input and provides an improved test sequence list 260 and / or a result table 292 as output.

[0102] The improved test sequence list 260 can also be used as input to the controller 270 to provide a new test result table 280, which can be fed back into the ATE 220 or the controller 270.

[0103] The result table 292 provided by the ATE 220 or the controller 270 includes pass / fail test results 298 of the DUT resources 296. Additionally and / or alternatively, the result table 292 can include a classification of the DUT resources 296.

[0104] A test activity table 210 having constraints 218, test parameters 216, and resources 214 required for the test activities 212 is provided to the constraint solver 250. The test activity table 210 or the test activity library can also include a code pool for the controller 270 or the OCST controller 270 to activate or execute the test activities 212. The library can also know which DUTs or on-chip and ATE resources are used by a given test activity 212.

[0105] The constraint solver 250 is configured to create a test sequence list 260 from the test activity table 210. The test sequence list 260 includes a test scenario 262, where the test scenario 262 includes one or more test activities 212a - e, which can coexist or can be executed simultaneously without violating the resource constraints 218. The test scenario 262 is automatically generated by the constraint solver 250 running on a workstation and / or the controller 270 (such as an OCST card or an on-chip OCST controller). The constraints can be modeled in the PSS.

[0106] The test sequence list 260 provided by the constraint solver 250 includes a scenario 262, where one or more test activities 212a - d associated with one or more test parameters 216a - e can be executed simultaneously. The test parameter values characterizing the respective test parameters 216a - e are randomly selected or have an emphasis on extreme values. For example, the order of the test scenario 262 is randomly generated to simulate real - life workloads.

[0107] The generated test sequence list 260 is provided to a controller 270, such as an on-chip OCST controller, which is configured to execute a test scenario 262 of the test sequence list 260 in order to collect test data 280. The controller 270 may include an interface for communicating with the ATE and / or an interface for communicating with the DUT.

[0108] The controller 270 may read out DUT test data and / or DUT sensor data and / or the controller may include sensors on the DUT area or die area to detect local anomalies. If the measured values and / or sensor values meet certain predetermined conditions, the measured values or the collected data 280 may trigger the controller 270 to collect further information, such as memory information or status information of the DUT.

[0109] The controller 270 is configured to transfer the test results 288 of the test activities 212 of the scenario 262 or the collected test data 280 to the ATE 220. The ATE 220 or the controller 270 is configured to further improve the test process and / or debug or diagnose the DUT.

[0110] The controller 270 (such as an on-chip OCST controller or an OCST card) or the ATE 220 is configured to dynamically create or modify a set of test parameters for a test scenario, which may satisfy constraints and may allow knowledge of the current test parameters for debugging. Methods for creating a set of test parameter values for the test activity 212 include randomization, which follows a desired distribution, which optionally emphasizes extreme values, using, for example, a constraint solver to maximize coverage, or using, for example, nested loops for exhaustive coverage of some test parameters.

[0111] To further improve the test environment, a test learning environment may be required. Extensive characterization tests are the basis of test learning, where many DUTs are exposed to many test steps or many test scenarios. Preferably, not all DUTs 282 are exposed to the same test steps or test scenarios in order to cover a large number of combinations of test parameters 216. To avoid test scenarios being biased towards certain DUTs 282, the test steps or test scenarios are executed in a random permutation order.

[0112] The ATE 220 or the controller 270 may also use a machine learning or AI module trained with the collected test result data 280 and system-level test results. The machine learning module may analyze the collected test result data 280 and system-level test results and may predict system-level test results based on a new set of collected test data 280.

[0113] The AI module can be further trained with measured result data without specification limits (such as on-chip sensor data) or with test parameters beyond specification limits (such as too low voltage or too fast frequency) to predict system-level test failures from test results. A failed test step or test activity beyond specification limits is not evidence of a defective DUT.

[0114] However, a machine learning module or model can be built to predict system-level test results based on test step or test activity results, including measured results without or beyond specification limits, and including test steps or scenarios with test parameters beyond specification and test step or scenario-related attributes such as the test activities and test resources involved.

[0115] Such a model may find some previously missed SLT failures, but may also fail some DUTs that pass SLT and pass all other legitimate OCST tests. These cases may be considered yield losses due to passing the test and can be carefully swapped with other found SLT failures, preferably based on a cost model. Only those additional test steps or scenarios can be included in the production steps required for such a model. Other additional test steps can be removed again.

[0116] ATE 220 or controller 270 can also be configured to debug and / or diagnose the DUT. Since the test result table 280 includes the results 288 of the simultaneously executed test activities 212 of the test scenario 262, the collected test data 280 needs to be further analyzed to identify and / or classify the faulty DUT or DUT resources.

[0117] The general idea of debugging or diagnosis is to identify those test parameters 216 that most affect the occurrence of OCST failures. The test parameters 216 associated with the test activities 212 involving certain actions in a specific IP block provide information cues for debugging.

[0118] A machine learning module trained with a table that combines test steps or scenarios with test activities, DUT resources, test parameters, test results, and overall OCST results (optionally for multiple DUTs) can be used to classify or identify faulty DUT resources. The machine learning module or machine learning feature selection algorithm can identify which test activities, tests, or DUT resources and test results are important for explaining the OCST results that lead to the occurrence of OCST failures.

[0119] In other words, the controller 270 controls the test process. The controller or OCST controller is preferably an on-chip processor with an optional test operating system, but can also be an OCST card communicating with the DUT or ATE workstation.

[0120] For example, the task of the OCST controller can be to trigger the execution of multiple test activities and read out their pass / fail results and / or measurement results. The test activities can include a combination of optional stress generation and / or optional fault detection.

[0121] More examples of test activities are listed below:

[0122] · The ATE sets the external test conditions of the DUT, such as the DUT power supply voltage or frequency, i.e., the OCST controller can control the ATE resources.

[0123] · The ATE performs measurements, such as power supply current measurement.

[0124] · Move data blocks between IP cores and check the content before and / or after.

[0125] · Run a memory test on the on-chip CPU.

[0126] · Run a memory self-test.

[0127] · Compress and decompress images and check the differences between the original image and the decompressed image (stress and check)

[0128] · Run an I / O loopback test.

[0129] · Apply stress generation, using DFT techniques.

[0130] · Use DFT techniques to activate and read out any observability structures.

[0131] The controller always knows which test scenario to execute with which test (activity) parameters, which is necessary for debugging when errors occur.

[0132] In addition, the controller or the OCST controller can dynamically create or modify test activity parameters based on constraints, and / or can manipulate them or import them from a pre-computed list. In addition, the controller or the processor running the OCST controller code can also generate test activities.

[0133] According to Figure 3 Exemplary test activity table

[0134] Figure 3 An exemplary test activity table 300 according to an embodiment is shown. The test activity table 300 is similar to Figure 2 the test activity table 210 of Figure 2 or is an example of the test activity table 210 of

[0135] The test activity table 300 includes columns for test activities, resources, test parameters, and possible results. The test activity table is configured to list all the tests that can be performed on the DUT, as well as the required resources, adjustable parameters, and possible effects or results.

[0136] The test activity table 300 can be used as input by a constraint solver (such as Figure 2 the constraint solver 250) to create a test sequence. The test sequence can define which tests with which test parameters are to be performed on which DUT and in what order.

[0137] The test activity table 300 can include common test activities or DUT-specific test activities. An exemplary test activity table 300 contains some examples of test activities.

[0138] For example, the first test activity A1 can include a processing unit 2 (CPU2) that writes data to a memory 3 (MEM3) and checks the content of MEM3. Activity A1 requires resources R1: CPU2, R2: MEM3, R3: ATE DPS for core supply. The adjustable test parameters for test activity A1 are P1: bandwidth, P2: DPS voltage. The results can include two values r1 and r2, where r1 is a pass / fail value and r2 is a current value.

[0139] For example, test activity A2 is a memory built-in self-test (MBIST) of MEM3, which requires resource R2, i.e., MEM3, has no adjustable parameters, and has a pass / fail value as the result.

[0140] For example, test activity A3 is an MPEG self-test that requires an MPEG resource with an adjustable block size as test parameter P3, and the result is a pass / fail value.

[0141] According to Figure 4 resource conflict table

[0142] Figure 4 Shows a resource conflict table 400 that can be used and / or created by Figure 2 the constraint solver 250. The resource conflict table 400 has a column for test activities and a column for each resource of the DUT. Each test activity in table 400 uses one or more DUT test resources, which are indicated by an "X" in the corresponding resource column.

[0143] The constraint solver (such as Figure 2 the constraint solver 250) can take a test activity table (such as Figure 3The table 300) as input to create a test sequence. The steps to create a test sequence are to create a resource conflict table, such as the resource conflict table 400. The resource conflict table 400 shows which test activities are using the same DUT resources and thus which test activities cannot be run simultaneously.

[0144] For example, the resource conflict table 400 shows the conflicting resources of the test activities in the test activity table 300. For example, test activity A1 uses resources R1, R2, R3 and does not use R4. For example, test activity A2 only uses R2 as a resource. For example, test activity A3 uses the DUT resource R4.

[0145] As shown in the resource conflict table 400, test activity A1 and test activity A2 are conflicting test activities because they both require the test resource R2. Both test activity A1 and A2 require the test resource R2 and thus cannot be run simultaneously. That is, test activity A1 and test activity A2 cannot be run simultaneously, for example, cannot be placed in the same test scenario. Test activities without resource conflicts can be combined into test scenarios.

[0146] According to Figure 5 test scenario table

[0147] Figure 5 shows a test scenario table 500 that can be used or created by Figure 2 the constraint solver 250. The test scenario table 500 includes a test scenario column and a column for each provided test activity. The test scenario table 500 includes all possible test scenarios. One or more test activities of a test scenario run simultaneously.

[0148] The test scenario table 500 shows all possible test scenarios created from test activities (such as Figure 3 test activities A1, A2, and A3 in the test activity table 300). The test activities of a test scenario need to be non - conflicting. Conflicting test activities are shown in the resource conflict table, similar to Figure 4 the resource conflict table 400. Test scenarios with conflicting test activities taken from the resource conflict table are excluded from all possible test scenarios with non - conflicting test activities in the test scenario table (such as the test scenario table 500).

[0149] For example, the test scenario table 500 includes a test scenario column and a column for each test activity (such as A1, A2, A3). According to Figure 4 the resource conflict table 400, A1 and A2 use the same resource R2, so test activity A1 and test activity A2 cannot be run simultaneously and cannot be in the same test scenario.

[0150] For example, Figure 3The test scenarios of exemplary test activities are test scenario S1 with test activity A1, test scenario S2 with test activity A2, test scenario S3 with test activity A3, test scenario S4 with concurrent test activities A1 and A3, and test scenario S5 with concurrent test activities A2 and A3. Due to resource constraints, there is no test scenario that runs test activities A1 and A2 simultaneously.

[0151] Whether the test activities actually run simultaneously can depend on test parameter settings and other unknown factors. A test sequence or test suite consists of multiple test steps that execute a given test scenario, where specified test parameter values are utilized for its test activities.

[0152] According to Figure 6 test step table

[0153] Figure 6 Shows a test step table 600 that can be used or created by Figure 2 the constraint solver 250. The test step table includes columns for test steps, test scenarios, test activities, and test parameters corresponding to the test activities.

[0154] The test step column of the test step table 600 includes serial numbers to identify different test steps. The test scenario column can include all possible test scenarios at least once. A test scenario can be tested multiple times with several different test parameters. Similar to Figure 5 the test scenario table 500 of , the test activities of a test scenario are indicated by "X" in the corresponding test activity column.

[0155] If a test scenario includes a test activity, the corresponding test parameter column includes test parameter values. The test parameter values are preferably generated randomly, optionally following a predetermined distribution and / or optionally concentrated at extreme values that tend to cause more problems than randomly generated test parameter values by a specific percentage.

[0156] A test sequence is generated from the test step table 600 by randomly mapping the test steps of the test step table 600 to the DUT or mapping them to the DUT following a predetermined distribution (so as to cause more problems than randomly mapped test sequences), for example Figure 2 the test sequence table 260 of .

[0157] The test scenario column includes all possible test scenarios, such as test scenarios S1 to S5 from the test scenario table 500, where a test scenario can be tested with several different test parameters.

[0158] For example, the first test step may be a first test scenario S1 with a test activity A1, corresponding to test parameter P1 with a bandwidth of 10 GB / S, and test parameter P2 with a DPS voltage of 0.9 V. For example, test step 2 includes the same test scenario S1 with different test parameters P1 with a bandwidth of 20 GB / S, and P2 with a DPS voltage of 0.86 V.

[0159] For example, test step 3 may include test scenario S2, which includes test activity A2 without any adjustable test parameters.

[0160] For example, test step 4 includes test scenario S3 with a test activity A3, where test parameter P3 is a block size of 128 kB. For example, test step 5 includes the same scenario S3 with a test parameter P3 of 1 MB block size.

[0161] For example, test step 6 includes test scenario S4 with test activities A1 and A3, with test parameter P1 with a bandwidth of 50 GB / S, test parameter P2 with a DPS voltage of 1.04 V, and test parameter P3 with a block size of 6 MB. For example, step 7 includes the same scenario S4 with test parameter P3 with a bandwidth of 3 GB / S, test parameter P2 with a DPS voltage of 0.97 V, test parameter P3 with a block size of 500 KB. For example, step 8 includes the same scenario S4 with test parameter P1 with a bandwidth of 27 GB / S, test parameter P2 with a DPS voltage of 0.88 V, test parameter P3 with a block size of 21 MB.

[0162] For example, test step 9 includes test scenario S5 with test activity A2 without test parameters, and test activity A3 with test parameter P3 with a block size of 7 MB. For example, test step 10 includes the same test scenario as S5 with test parameter P3 with a block size of 780 KB. For example, step 11 includes the same test scenario as S5 with test parameter P3 with a block size of 13 MB.

[0163] According to Figure 7 test result table

[0164] Figure 7 An empty test result table 700 is shown. After the test activities of the test sequence table 260 are performed, the controller can fill in this table. The test result table includes columns for the DUT, test steps, test results of the test activities, and an overall test result column. Figure 2 The test result table 700 includes a test sequence, such as

[0165] The test result table 700 includes a test sequence, for example Figure 2The test sequence list 260. The first two columns of the test result table 700, the DUT and test step columns, define on which DUT which test step is to be executed. The test sequence is generated by randomly mapping the test steps of the test step table 600 to the DUTs or mapping them to the DUTs according to a predetermined distribution and / or satisfying a predetermined condition (so as to trigger more problems than a randomly mapped test sequence) from Figure 6 the test step table 600 of

[0166] An exemplary test sequence may include, for example, DUT1 tested by test steps 6, 11, 7, 1, and 8, and DUT2 tested by test steps 5, 2, 10, 4, and 9.

[0167] A controller (such as an on-chip controller or a controller card) executes tests according to the test sequence. The test activity results represented by the columns r1 (# failed), r2 (current), r3 (# failed), r4 (p / f) are collected by the controller. The overall OCST result may be based on the results of all test activities or a subset of the test activity results. Additionally or alternatively, the OCST result may include a DUT classification made based on the test activity results or some of the test activity results.

[0168] In this example, the overall OCST result is calculated based on the results r1, r3, and r4, that is, the measured result r2 does not contribute to the overtest result because it has no specification limit in this example.

[0169] According to Figure 8 per test step failed table

[0170] Figure 8 The per-test-step fail table 800 is shown, including the DUT column and columns for each test step. The DUTs being tested are listed in the DUT column. The rows of the DUTs being tested contain the letter "P" (if the DUT being tested has passed the given test step) and the letter "F" with a highlighted background (if the DUT has failed the given test step) in the column for the given test step.

[0171] The controller or ATE is configured to use the per-test-step fail table 800 to optimize (preferably reduce) the number of test steps. For example, the number of tests that never fail and / or redundant test steps can be reduced. The task of the controller or ATE may include selecting the smallest set of test steps, which is known as the minimum cover set problem and has a known solution.

[0172] For example, in a case where the per-test-step fail table 800 can include the test results of four DUTs (DUT 1 - 4). For example, DUT 1 may pass all test steps except test step 6. For example, DUT 2 may pass all test steps except test steps 2 and 4. For example, DUT 3 may pass all test steps except test steps 1 and 6. For example, DUT 4 may pass all test steps except test step 8.

[0173] For example, when the controller or ATE analyzes the per-test-step fail table 800, the controller can conclude that test steps 3, 5, and 7 will never fail and can thus be removed from the production test. Additionally, test step 4 is redundant with respect to test step 2 and can also be removed.

[0174] According to Figure 9 Compare OCST and SLT

[0175] Figure 9 An exemplary comparison table 900 showing the OCST test results and the SLT test results is presented. The exemplary comparison table 900 shows the differences that may occur when the same set of DUTs are tested using the OCST test method and the SLT test method.

[0176] In this example, 9900 devices passed both the OCST and SLT test methods, and 70 devices failed. 25 devices or DUTs failed the OCST test but passed the SLT test, while only 5 DUTs passed the OCST test and failed the SLT test.

[0177] In other words, in the above example, OCST missed 5 devices that failed SLT, but found problems in 25 DUTs that SLT did not detect. Assuming all test steps describe real scenarios with real test parameter values, this would be a good balance.

[0178] According to Figure 10 Training table based on the binding test results and SLT results

[0179] Figure 10 A training table 1000 that combines the collected test data with the SLT results is presented. The training table 1000 is configured to be used as a training table or training data set for a machine learning or AI module. The training table 1000 can include all test activities, test resources, and test results of the test steps performed on the DUTs, as well as the corresponding SLT results. The training table 1000 can include multiple test steps performed on multiple DUTs.

[0180] The training table 1000 may include test data collected outside or slightly outside the DUT specification in combination with SLT results, where the SLT tests are conducted within the DUT specification. For example, in such a case, a failed OCST result is not considered a strong indication of a non - compliant DUT, but a passed OCST test beyond the specification limits may be considered a strong indication of a good DUT and may lead to classifying the DUT as a high - quality DUT.

[0181] The ATE or the controller may include a machine - learning unit or an AI module, which can be trained by the training table 1000 and can be configured to predict SLT results based on newly conducted test steps on the DUT. The machine - learning unit can also be used to improve the test process.

[0182] According to Figure 11 training table with test results

[0183] Figure 11 A training table 1100 showing the collected test data and the overall OCST results is shown. The training table 1100 is configured to be used as a training table or a training data set for a machine - learning or an AI module. The training table 1100 may include test activities, test resources, and test results of test steps conducted on the DUT, as well as the corresponding overall OCST results.

[0184] Table 1100 is used for debugging and diagnostic purposes. The general idea of debugging is to identify those test parameters that most affect the occurrence of a failed OCST. The ATE or the controller may include an AI or a machine - learning unit to identify and / or classify DUT resources or DUTs that often fail. For efficiency reasons, the table may be limited to failed devices. Analyses can be performed on many DUTs to identify frequently occurring failure mechanisms.

[0185] The machine - learning unit or module can be trained by the test result table 1100 to predict failed DUTs or DUT resources.

[0186] Alternative implementation

[0187] Although some aspects have been described in the context of an apparatus, it is evident that these aspects also represent a description of a corresponding method, where a block or a device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of method steps also represent a description of corresponding blocks or items or features of a corresponding apparatus.

[0188] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation can be carried out using a digital storage medium on which electronic-readable control signals are stored, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM, or flash memory, which cooperates (or is capable of cooperating) with a programmable computer system to perform the corresponding method.

[0189] Some embodiments according to the present invention include a data carrier having electronic-readable control signals, which is capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0190] Generally, embodiments of the present invention may be implemented as a computer program product having program code that, when the computer program product runs on a computer, is operable to perform one of the methods. The program code may be stored, for example, on a machine-readable carrier.

[0191] Other embodiments include a computer program stored on a machine-readable carrier for performing one of the methods described herein.

[0192] In other words, embodiments of the method of the present invention are thus computer programs having program code for performing one of the methods described herein when the computer program runs on a computer.

[0193] Therefore, another embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) on which a computer program for performing one of the methods described herein is recorded. The data carrier, digital storage medium, or recording medium is generally tangible and / or non-transitory.

[0194] Therefore, another embodiment of the method of the present invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, via a data communication connection (such as via the Internet).

[0195] Another embodiment includes a processing device, such as a computer or a programmable logic device, which is configured or adapted to perform one of the methods described herein.

[0196] Another embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0197] Another embodiment according to the present invention includes a device or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a storage device, etc. For example, the device or system may include a file server for transmitting the computer program to the receiver.

[0198] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0199] The devices described herein can be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0200] The devices described herein or any component of the devices described herein can be implemented at least in part in hardware and / or software.

[0201] The methods described herein can be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

Claims

1. An automated test device (110, 220) for testing one or more devices under test (120, 282). The automated test device is configured to generate one or more test scenarios (140a - c, 262), the test scenarios including a plurality of test activities (150a - c, 212, 212a - d). Different test activities utilize a subset of the resources of the device under test (130a - e, 214). The automated test device is configured to generate the plurality of test scenarios such that the resources of the device under test associated with the plurality of test activities of the test scenarios do not conflict with each other.

2. The automated test device according to claim 1, wherein the plurality of test activities of a given test scenario are configured to be executed simultaneously.

3. The automated test device according to claim 1 or 2, wherein one or more of the test activities are associated with one or more test parameters (216, 216a - e) and / or with one or more constraints (218), the one or more test parameters (216, 216a - e) being characterized by corresponding test parameter values.

4. The automated test device according to claim 1, wherein one or more of the test activities are characterized by one or more test parameter values outside a predetermined limit.

5. The automated test device according to claim 1 or 2, wherein the plurality of test activities of the test scenario are randomly selected under the constraint that the resources of the device under test associated with the plurality of test activities of the test scenario do not conflict with each other.

6. The automated test device according to claim 1 or 2, wherein the automated test device includes a constraint solver (250) for generating test scenarios to prevent conflicts between the resources of the device under test.

7. The automated test device according to claim 1 or 2, wherein one or more of the test activities include activating a pressure generator arranged on the device under test.

8. The automated test device according to claim 4, wherein the automated test device is configured to generate a test sequence of test scenarios.

9. The automated test device according to claim 8, wherein the test sequence includes two or more test scenarios having the same test activities, the test scenarios differing in at least one test parameter value.

10. The automated test device according to claim 8 or 9, wherein the plurality of test scenarios of the test sequence are randomly selected and / or sorted.

11. The automated test device according to claim 8, wherein the automated test device is configured to generate the test sequence such that the test sequence is configured to be executed by a controller (270) to collect test data.

12. The automated test device according to claim 11, wherein the controller is an on - chip processor or a controller card configured to communicate with the automated test device and with the device under test.

13. The automated test equipment according to claim 11, wherein the controller includes one or more interfaces configured to read out device - under - test data and / or device - under - test sensor data.

14. The automated test equipment according to any one of claims 11 to 13, wherein the controller includes one or more sensors distributed over the device - under - test area such that the controller is configured to read out sensor test data of the one or more sensors.

15. The automated test equipment according to any one of claims 11 to 13, wherein the controller is configured to react when the collected test data (280) meets a predetermined condition.

16. The automated test equipment according to any one of claims 11 to 13, wherein the controller is configured to transmit the collected test data to the automated test equipment.

17. The automated test equipment according to any one of claims 11 to 13, wherein the controller or the automated test equipment is configured to dynamically create test parameter values based on test activity constraints and / or based on the collected test data.

18. The automated test equipment according to any one of claims 11 to 13, wherein the controller or the automated test equipment is configured to analyze the collected test data to optimize the test sequence.

19. The automated test equipment according to any one of claims 11 to 13, wherein the controller or the automated test equipment is configured to compare the results of system - level tests within the predetermined limits with the collected test data within and / or outside the predetermined limits to optimize the test sequence.

20. The automated test equipment according to any one of claims 11 to 13, wherein the automated test equipment includes an artificial intelligence or machine learning unit, wherein the controller or the automated test equipment is configured to train the artificial intelligence or the machine learning unit using the results of system - level tests and / or using the collected test data and / or using the comparison between the system - level tests and the collected test data to optimize the test sequence.

21. The automated test equipment according to claim 20, wherein the trained artificial intelligence or machine learning unit is configured to predict the results of system - level tests based on the collected test data.

22. The automated test equipment according to any one of claims 11 to 13, wherein the controller or the automated test equipment is configured to analyze the collected test data to obtain test results.

23. The automated test equipment according to claim 21, wherein the trained artificial intelligence or machine learning unit is configured to analyze the collected test data to optimize the test sequence and / or obtain test results.

24. A processing method for testing one or more devices under test by an automated test equipment, wherein the automated test equipment generates one or more test scenarios, and the test scenarios include a plurality of test activities; wherein different test activities utilize a subset of the device - under - test resources; The automated test equipment generates the multiple test scenarios such that resources of the device under test associated with multiple test activities of the test scenarios do not conflict with each other.

25. A computer-readable storage medium storing a computer program which, when executed on a computer or a signal processor, implements the method of claim 24.

Images