Automated test equipment, processes, and computer programs that test one or more devices under test, wherein different test activities utilize a subset of the resources of the device under test.

Abstract

An embodiment according to the present invention includes an automated test device for testing one or more devices under test. The automated test device is configured to automatically and dynamically generate one or more, preferably multiple, test scenarios comprising multiple test activities. Different test activities utilize a subset of resources of the devices under test that may overlap. The automated test device is configured to generate multiple test scenarios so that the resources of the devices under test associated with the multiple test activities of the test scenarios do not conflict with one another.

Description

Technology Field

[0001] Embodiments according to the present invention relate to automatic test equipment. Further embodiments according to the present invention relate to system-level testing. Further embodiments according to the present invention relate to on-chip system testing. Embodiments according to the present invention relate to covering testing. Background Technology

[0002] It is an empirical observation or truth that some devices fail System Level Testing (SLT) even though they have passed structural and / or parameter tests. This is generally attributed to unrealistic test conditions during structural testing.

[0003] For example, the activity patterns across the die region during structural testing differ significantly from those of system-level testing. While structural testing utilizes most of its time for very low-frequency shift operations, pattern activity is very high. This results in unnatural loads on the power supply, making the voltage profiles for die location and time unrealistic.

[0004] An additional example is that power domain crossings and / or time domain crossings are often not included in structural testing due to the difficulty of generating automated test patterns. Errors in this area will not be detected and will not be identified.

[0005] Structural testing is optimized for superior defect detection capabilities but faces challenges due to unrealistic test conditions. In contrast, SLT consists solely of genuine user scenarios. It is agreed, or generally believed, that even without knowing any defective models, a failure in SLT identifies a truly bad device.

[0006] System-level testing has some drawbacks, such as the fact that SLT is incomplete because the Device Under Test (DUT) runs only a small subset of usage and scenarios, which are possible only in a single selected environment, such as supply voltage, frequency, and temperature.

[0007] Another disadvantage is that SLT failures are difficult to debug because the time between failure detection and detection can be significantly long, and / or the exact sequence of all activities is unknown, and / or the simulation time of SLT is too long.

[0008] Also, SLT has a very long execution time, and 10 minutes or more is no exception.

[0009] In addition, SLT is difficult to deploy in a test house because a large number of system boards must be maintained for each DUT.

[0010] Improvements to system-level testing are needed to provide a better trade-off between possible user scenarios, execution time length, and practicality. means of solving the problem

[0011] An embodiment according to the present invention provides an automated test equipment for testing one or more devices under test, such as a system-on-chip (SoC). The automated test equipment (ATE) is configured to automatically and dynamically generate multiple test scenarios, which include one or more, preferably multiple, test activities. Different test activities, such as A1, A2, and A3, use overlapping subsets of DUT resources (e.g., A1: CPU2, Mem3, A2: Mem3, A3: MPEG). The automated test equipment is configured to use multiple test scenarios so that the DUT resources related to the multiple test activities of the test scenarios do not conflict with one another.

[0012] On-Chip System Testing (OCST) is expected to identify more relevant issues than SLT because it exposes the DUT to a wider variety of legitimate test conditions. If an issue occurs while simulating a real-world use case or in a combination of test activities that may be part of a real-world use case, the issue can be considered relevant.

[0013] OCST can change external test conditions, such as supply voltage and frequency, to detect faults, and / or internal test conditions, such as the intensity of test activities, such as the rate of data movement, to detect workload-related faults. When a fault is detected, OCST provides excellent observability for detecting fault behavior.

[0014] OCST can demonstrate its effectiveness by showing that it can identify which subsets of SLT have failed. In other words, OCST can find the failed subsets of SLT.

[0015] Ideally, OCST can be reproduced in a simulation environment, for example, for debugging. OCST must also utilize existing on-chip resources such as Design for Test (DFT), Design for Debug (DFD), scan chains, Logit Built-In Self-Tests (LBIST), and / or Memory Built-In Self-Test (MBIST). Additionally, multi-threading can be supported at the interface between the multi-core DUT and multiple ATE resources.

[0016] In a preferred embodiment, a plurality of test activities of a given test scenario are configured to be executed simultaneously.

[0017] ATE is configured to generate test scenarios that use DUT resources where test activities do not overlap, so the test activities of the test scenarios do not conflict with each other. In other words, the speed of testing can be increased by executing the test activities of a given test scenario simultaneously.

[0018] The general idea of ​​the present invention is to execute multiple simple and realistic self-test activities simultaneously on a DUT using DUT resources, such as data block movement or built-in self-tests, with various combinations of test activity intensity and concurrency. Each test activity can not only check selected or related IP blocks but also contribute to test conditions of other blocks through supply load, clock tree, and thermal coupling.

[0019] According to one embodiment, one or more test activities are associated with one or more test parameters, such as voltage, speed, size of data transmission, time between data transmissions, and block size in which the total data transmission is divided. The test parameters control the operating state of the test activities. The test parameters are characterized by respective test parameter values, such as on / off, 5V, 50GB / s, etc., and / or are associated with one or more constraints and limitations, such as dynamic generation of test parameter values.

[0020] The same test activity with different parameter values ​​may be executed in a test sequence containing one or more test scenarios. A test activity may be associated with one or more test parameters and / or constraints. For example, test parameter values, such as voltage, may vary from test activity to test activity. A test activity may also have certain constraints or limitations, for instance, to protect the DUT or DUT resources, or to avoid scenarios that would not occur in actual use. These constraints may include limited write rates, temperature limits, or limited memory usage.

[0021] According to one embodiment, one or more test activities are characterized by one or more test parameter values ​​that deviate from predefined limits or chip specifications, such as a voltage value that is too low and / or a frequency that is too fast.

[0022] Test results of a DUT that fall outside or slightly outside of predefined limits or chip specifications can correlate with the performance of the DUT within those limits or specifications. In other words, a DUT with good test results outside the chip specifications can serve as a hint of good performance for the DUT within the chip specifications. Conversely, a test failure outside the chip specifications may be considered as an indication of weakness, rather than a definitive hint or evidence of a test failure within the chip specifications.

[0023] According to one embodiment, a plurality of test activities of a given test scenario are randomly selected under the constraint that the resources of the test target device associated with the plurality of test activities of the given test scenario do not conflict with each other, in order to avoid conflict requirements of use or test parameter values ​​of different test activities.

[0024] Random mixing of test activities within a test scenario is necessary to test activities occurring in a real-world operating environment or unknown interactions between test activities. Randomization of parameterizable test activities covers multiple mixes of local workloads and thus mimics many real-world workload patterns considered responsible for numerous system-level faults.

[0025] In a preferred embodiment, the automated test equipment includes a constraint solver for generating test scenarios to prevent conflicts between the resources of the device under test.

[0026] Each test activity may have test parameters and optional constraints. For example, data block write and reread activities may have test parameters such as initial time delay, block size, starting address, subblock size, and time delay before reread. Constraints may exist at the test scenario level, including test parameters for multiple test activities.

[0027] Workstations or controllers, such as on-chip OCST controllers or OSCT cards, can use a constraint solver to automatically generate test scenarios from test activities. The constraint solver matches test activities that can coexist simultaneously without violating resource constraints.

[0028] In a preferred embodiment, one or more test activities include a stress generator placed on the device under test, such as an on-chip stress generator, such as a traffic generator for an internal and / or external bus.

[0029] For example, existing or new Design for Test (DFT) structures can increase observability and generate controlled stress to increase detectability (or for more fault detection). The stress generator may include a traffic generator for the internal bus and / or a traffic generator for external inputs to simulate traffic from the absence of external inputs. The stress generator can execute embedded self-tests (BIST), such as memory or logic embedded self-tests (MBIST, LBIST), on selected wrapped IP blocks. Additionally, the stress generator can execute input / output (I / O) loopback tests. Furthermore, the stress generator can use a wrapper, such as an IEEE1500 wrapper, to isolate IP blocks and switch scan chains using linear-feedback shift registers (LFSRs).

[0030] According to one embodiment, the automatic test equipment is configured to generate a test sequence, such as a sequence of test steps of a test scenario.

[0031] The test sequences of the test scenarios are generated to test various mixes of concurrent test activities to mimic possible real-world workload combinations.

[0032] According to one embodiment, the test sequence includes two or more test scenarios of the same test activity, and the test scenarios differ by at least one test parameter value.

[0033] The test sequence may include test scenarios with the same test activity and different test parameters to test the influence of test parameters on the test results. Changing only one test parameter value of an IP block of a DUT may also affect or have an effect on IP blocks of other DUTs.

[0034] In addition, performing test scenarios with the same test activity but different test parameter values ​​can be more helpful in finding defective IP blocks during the test evaluation process.

[0035] In a preferred embodiment, a plurality of test scenarios of the test sequence are randomly arranged and / or selected, for example, by an automated test equipment or controller, to mimic actual workload patterns.

[0036] Randomizing test scenarios within a test sequence can further improve the test process by creating or generating actual workloads. Ideally, test scenarios are independent of one another; that is, changes in the execution order of test scenarios are not expected to alter the behavior of the DUT. Randomizing the execution order of scenarios can, for example, test whether residual columns or data from a previous test scenario might affect the current test scenario.

[0037] In a preferred embodiment, the automatic test equipment is configured to generate a test sequence, and the test sequence is configured to be executed by a controller to collect test data.

[0038] The generated test sequence is executed by a controller or an OCST controller. The controller's task is to trigger the execution of multiple test activities and read test results, such as success / failure outcomes and / or measurement results. The controller knows which test scenarios are executed with which test parameter values. This is necessary for debugging when errors occur.

[0039] According to one embodiment, the controller is, for example, an on-chip processor having an operating system, or a controller card that is part of an automated test equipment configured to communicate with, for example, automated test equipment and a device under test.

[0040] The controller executes a test sequence on the DUT and provides measurement data or measurement results to the ATE. An on-chip processor with an operating system or a controller card of the ATE communicates data or test results or measurement results collected from the DUT to the ATE.

[0041] In a preferred embodiment, the controller includes one or more interfaces configured to read DUT data, such as memory data and / or DUT data, such as on-chip sensor data. Accordingly, the ATE is configured to set up the controller to read DUT data and / or DUT sensor data.

[0042] Reading DUT test data and / or DUT sensor data through the interface also improves the test by allowing the predicted result data or predicted sensor data to be compared with the result data and / or result sensor data. The controller, the OCST controller, is preferably an on-chip processor having an optional test operating system, which interfaces with the on-chip IJTAG to read the on-chip sensor data.

[0043] According to one embodiment, a controller interfaces with one or more sensors distributed across a device area under test, and the controller is configured to read sensor test data of one or more sensors. Accordingly, an automated test equipment is configured to set up the controller to read sensor test data of one or more sensors distributed across a device area under test that includes the controller.

[0044] Regardless of whether the controller is an OCST control card or an on-chip processor communicating with the DUT or ATE workstation, the controller may include additional sensors distributed across the die area of ​​the DUT to detect local abnormalities. The applied sensors may include, for example, temperature, current, and / or voltage sensors.

[0045] In a preferred embodiment, the controller may be configured to respond, for example, by collecting additional data such as stored system data, program counters, or memory register information, when a predetermined condition is met, such as when the collected test data exceeds one or more predefined thresholds. Accordingly, the automated test equipment may be configured to set up the controller to respond when the collected test data meets a predetermined condition.

[0046] For example, the controller may interface with one or more sensors having temperature values, such as current / max / minimum peak temperatures; voltage values, such as current / max / minimum peak voltages; and / or an optional threshold alarm capability for, for example, timing violations in a shaver circuit. The alarm capability may trigger the saving of critical system data, such as a program counter or memory controller registers, to simplify debugging, for example.

[0047] According to one embodiment, the controller is configured to communicate collected test data to automatic test equipment.

[0048] Regardless of test result data or measurement data or any collected data triggered by measurement data or test result data, the collected test data is provided to the ATE by the controller.

[0049] According to one embodiment, the controller or automated test equipment is configured to dynamically generate test parameter values ​​based on test activity constraints and / or collected test data.

[0050] The ATE or controller may analyze collected test data to modify existing test parameter values ​​during the test process or to add one or more test scenarios to a test sequence with new test parameter values. The generation of new test parameter values ​​may take into account the constraints of a given test activity.

[0051] In a preferred embodiment, the controller or automated test equipment is configured to analyze collected test data to, for example, solve the minimum covering set problem, reduce redundancy, and / or optimize the test sequence so that the test activity never fails.

[0052] The collected test data is used not only to dynamically generate test parameter values ​​but also to optimize the test sequence. The general idea behind the test is to identify test parameters that influence the occurrence of test failures. Redundant test activities can help evaluate the impact of, for example, a single or small number of test parameters. After identifying the most influential test activities and / or test scenarios, the number of test scenarios or the test activities within them can be reduced to the minimum set of test activities to decrease the execution time of the test sequence.

[0053] According to one embodiment, a controller of an automatic test equipment is configured to compare or calculate the correlation between the results of a system-level test within a predefined limit and collected test data, such as test target device data, test target device sensor data, and sensor test data, within and / or outside the predefined limit, for example, to optimize test time.

[0054] Tests are performed outside of predefined limits or the specifications of the DUT and are compared to system-level tests within the predefined limits. Test activities with positive results outside the predefined limit range may indicate that the system-level tests within the predefined limits have passed. Comparing test results within the predefined limits with those outside can help reduce the number of test scenarios and / or test activities within those scenarios. Additionally, by identifying or determining the range within which test parameter values ​​can fall outside the predefined limits, it can help provide a good correlation between tests performed inside and outside the predefined limits.

[0055] In a preferred embodiment, the automated test equipment includes an artificial intelligence or machine learning unit, and the controller or the automated test equipment is configured to optimize the test sequence by training the artificial intelligence or machine learning unit with the results of a system-level test and / or collected test data and / or a comparison of the system-level test and the collected test data.

[0056] In addition, in a preferred embodiment, a trained artificial intelligence or machine learning unit is configured to predict the results of a system-level test based on collected test data.

[0057] An artificial intelligence or machine learning unit is trained on the results of a system-level test combined with collected test data and / or collected sensor data to compare system-level test results with collected test data, and can predict system-level test results based on the collected test data or measurement data.

[0058] Using the prediction of system-level test results based on collected test data can also reduce the number of test activities and / or test scenarios within the test sequence.

[0059] In a preferred embodiment, the controller or automated test equipment may be configured to obtain test results by analyzing collected test data, such as identifying defects in a defective device under test and / or a resource of the device under test, and / or classifying the device under test.

[0060] For example, by analyzing collected test data using statistical methods, it may be necessary to define defective IP blocks of the DUT or classify the IP blocks of the DUT.

[0061] In a preferred embodiment, a trained artificial intelligence or machine learning unit is configured to analyze collected test data for the optimization of a test sequence and / or for the acquisition of test results.

[0062] The already trained machine learning unit can also improve statistical methods and / or test sequences by classifying IP blocks of the DUT and / or defining defective IP blocks of the DUT.

[0063] Additional embodiments according to the present invention generate each method and each computer program.

[0064] However, it should be noted that the method is based on the same considerations as the corresponding automated test equipment. Additionally, the method may be supplemented, individually or in combination, by any of the features, functions, and details described herein with respect to the automated test equipment. Brief explanation of the drawing

[0065] Embodiments according to the present invention will be described later with reference to the attached drawings. FIG. 1 illustrates a schematic diagram of a test device including an automatic test device for testing one or more test target devices according to one embodiment. FIG. 2 illustrates a block diagram illustrating a test process performed by a test device according to one embodiment. FIG. 3 illustrates an exemplary test activity table and the input of a constraint solver according to one embodiment. FIG. 4 illustrates an exemplary resource conflict table used by a constraint solver according to one embodiment. FIG. 5 illustrates an exemplary test scenario table used by a constraint solver according to one embodiment. FIG. 6 illustrates an exemplary test step table used by a constraint solver according to one embodiment. FIG. 7 illustrates an empty test result table having an exemplary test sequence generated by a constraint solver according to one embodiment. FIG. 8 illustrates an example of a failure table per test step used to optimize the number of test steps according to one embodiment. FIG. 9 illustrates a comparison table between the SLT test and the OCST test according to one embodiment. FIG. 10 illustrates a table that combines test data with SLT results, which is used as a training data set for a machine learning module according to one embodiment. FIG. 11 illustrates a table that combines collected test data with OCST results, which is used as a training data set for a machine learning module according to one embodiment. Specific details for implementing the invention

[0066] Hereinafter, various embodiments and aspects of the present invention will be described. Additionally, further embodiments will be defined by the appended claims.

[0067] It should be noted that any embodiment defined by the claims may be supplemented by any details, features, and functions described herein. Additionally, the embodiments described herein may be used individually and may also be optionally supplemented by any details, features, and functions included in the claims. Furthermore, it should be noted that individual aspects described herein may be used individually or in combination. Thus, details may be added to each described aspect while details may not be added to another described aspect. Additionally, it should be noted that the present invention explicitly or implicitly describes functions available in automated test equipment. Therefore, all functions described herein may be used in relation to automated test equipment.

[0068] Furthermore, the features and functions disclosed herein regarding the method may also be used in a device configured to perform such functions. In addition, any features and functions disclosed herein regarding the device may also be used in the method. In other words, the method disclosed herein may be complemented by any features and functions described regarding the device.

[0069] The present invention will be more fully understood from the detailed description given below and the accompanying drawings of embodiments of the invention, but the invention should not be limited to the specific embodiments described, and this is for illustrative and illustrative purposes only.

[0070] Example of FIG. 1

[0071] FIG. 1 illustrates a schematic diagram of a test device (100) including an automated test equipment (ATE) for testing one or more device under test (DUT) (120). FIG. 1 further includes an exemplary DUT (120) including a plurality of DUT resources (130a-e). The DUT resources (130a-e) may include different IP blocks such as CPU, memory, MPEG, etc.

[0072] The ATE (110) is configured to generate multiple test scenarios (140a-c), each of which includes multiple test activities. For example, a test scenario (140c) includes test activities (150a-c).

[0073] Each test activity is configured to use one or more DUT resources (130a-e). For example, test activity (150a) is configured to use DUT resource (130e). Or, for example, test activity (150b) is configured to use DUT resources (130c, 130d). Another example may be test activity (150c) configured to use DUT resources (130a, 130b).

[0074] The automatic test equipment (110) is configured to generate multiple test scenarios, such as test scenarios (140a-c), so that DUT resources, such as DUT resources (130a-e), associated with multiple test activities, such as test activities (150a-c), do not conflict with each other. For example, test activities (150a-c) are grouped into test scenarios (140c), so that the DUT resources (130a-e) of test activities (150a-c) do not conflict, and simultaneous execution of test activities (150a-c) of a given test scenario (140c) is possible.

[0075] The general idea of ​​on-chip system testing (OCST) is to use DUT resources (130a-e) to run a large number of simple and realistic test activities simultaneously on the DUT (120), with varying combinations and / or intensities of the concurrent test activities. Each test activity not only verifies the associated IP block but also contributes to the test conditions of other IP blocks through supply load, clock tree, and thermal coupling. Examples of test activities may include moving data blocks or running embedded self-tests such as memory-embedded self-tests (MBIST) or logic-embedded self-tests (LBIST).

[0076] For example, the OCST controller executes structural tests, such as LBIST and MBIST, locally within some IP blocks. While these test activities are clearly self-checking, they can also act as stress generators to control test conditions for other test activities running simultaneously. In a test scenario, some IP cores execute structural tests, while others are involved, for instance, in code-based test activities.

[0077] The associated IP block is applied to design for test (DFT) structures or techniques, such as generating stress to make them sensitive to defects and / or increase observability, to enable sensitive couplings to be detected and / or to provide access to existing structures for debugging or in-system testing.

[0078] Randomization of test activities or parameterizable test activities covers many combinations of local workloads and thus mimics a large number of real-world workload patterns considered responsible for many system-level defects. Existing or new Design for Test (DFT) structures can generate controlled stress and increase observability to enhance the likelihood of defect detection (or for more probable defect detection).

[0079] The DFT structure can improve observability, for example, by an on-chip OSCT controller interfacing with an on-chip IJTAG to read on-chip sensor data.

[0080] For example, additional sensors with temperature values ​​such as current / maximum / minimum peak temperatures, voltage values ​​such as current / maximum / minimum peak voltages, and / or optional threshold alarm capabilities for timing violations, such as in a shaver circuit, can further enhance observability. This alarm can trigger the retention or storage of critical system data, such as program counters or memory controller registers, to simplify debugging.

[0081] Observability can be further improved by detecting local anomalies with additional sensors distributed across the die. Alternatively, a processor can be connected to track memory by comparing the contents with expectations. Additionally, assertion checkers, such as protocol checkers, CRCs, or bus traffic loggers, can be added to measure coverage and support debugging.

[0082] Test activities are coordinated in a controlled manner, for example, by an on-chip system test (OCST) controller, so the most effective test conditions can be determined, and failure can be attributed to specific test activities and corresponding test parameter values.

[0083] Example according to Fig. 2

[0084] FIG. 2 illustrates a block diagram (200) describing a test process performed by a test device similar to the test device (100) of FIG. 1.

[0085] The block diagram (200) starts with a test activity table (210). The test activity table (210) includes a test activity column (212), and each test activity (212) may use one or more DUT resources (214). The corresponding DUT resources (214) are provided in the DUT resource (214) column. Additionally, the test activity (212) may have corresponding test parameters (216) and / or constraints (218) provided in separate columns. Constraints may exist at the test scenario level and may refer to test parameters of multiple test activities.

[0086] The test activity table (210) is input into the constraint solver (250). The constraint solver (250) may be included in the ATE (220) and controller (270), or it may be an independent entity as shown in FIG. 2. The constraint solver (250) in FIG. 2 has one input and one output. The input of the constraint solver (250) is input by the test activity table (210). The output of the constraint solver is the test sequence table (260).

[0087] The test sequence table (260) includes a test scenario (262) similar to the test scenario (140a-c) of FIG. 1. The test scenario may include one or more test activities (212a-e), and each test activity (212a-e) may include test parameter values ​​(216a-e). The test sequence table (260) is provided to the ATE (220) or controller (270).

[0088] For example, the first scenario may include a first test activity (212a) having test parameters P1 (216a) and P2 (216b), and a second test activity (212b) having test parameters P3 (216c) and P4 (216d). The second scenario may include a second test activity (212b) having test parameters P3 (216c) and P4 (216d). The third scenario may include a third test activity (212b) having test parameters P3 (216c) and P5 (216e). The fourth scenario may include a fourth test activity (212d) having test parameters P2 (216b) and P4 (216d).

[0089] The controller (270) takes a test sequence table (260) as input and outputs a test result table (280). The test result table (280) provided by the test block (270) contains the test results (288) of one or more test activities (212) of a test scenario (262) executed on the DUT (282). The test result table (280) is input to the ATE (220) and / or fed back to the controller (270).

[0090] The ATE (220) or controller (270) receives a test result table (280) as input and provides an improved test sequence table (260) and / or a result table (292) as output.

[0091] The improved test sequence table (260) also provides a new test result table (280) that can be used as an input to the controller (270) and input back into the ATE (220) or the controller (270).

[0092] A result table (292) provided by the ATE (220) or controller (270) includes the success / failure test results (298) of the DUT resource (296). Additionally and / or alternatively, the result table (292) includes the classification of the DUT resource (296).

[0093] A test activity table (210) having constraints (218), test parameters (216), and resources (214) required for a test activity (212) is provided to a constraint solver (250). The test activity table (210) or a library of test activities may also include a code pool for a controller (270) or an OCST controller (270) for enabling or executing the test activity (212). The library may also know whether DUT or on-chip and ATE resources are used by a given test activity (212).

[0094] The constraint solver (250) is configured to generate a test sequence table (260) from a test activity table (210). The test sequence table (260) includes test scenarios (262), and the test scenarios (262) include one or more test activities (212a-e) that can be executed simultaneously or coexist without violating resource constraints (218). The test scenarios (262) can be automatically generated by the constraint solver (250) running on a workstation and / or a controller (270), such as an OSCT card or an on-chip OSCT controller. Constraints can be modeled in the PSS.

[0095] The test sequence table (260) provided by the constraint solver (250) includes scenarios (262), and one or more test activities (212a-d) associated with one or more test parameters (216a-e) can be executed simultaneously. The test parameter values ​​characterizing each test parameter (216a-e) are randomly selected or focus on extreme values. The order of the test scenarios (262) is, for example, randomly generated to simulate an actual workload.

[0096] The generated test sequence table (260) is provided to a controller (270), such as an on-chip OCST controller, configured to execute the test scenario (262) of the test sequence table (260) to collect test data (280). The controller (270) may include an interface for communicating with the ATE and / or an interface for communicating with the DUT.

[0097] The controller (270) can read DUT test data and / or DUT sensor data and / or the controller (270) can detect local anomalies by including sensors across the DUT area or die area. The measurement or collected data (280) may trigger the controller (270) to collect additional information, such as status information or memory information of the DUT, if the measurement and / or sensor values ​​satisfy specific predefined conditions.

[0098] The controller (270) is configured to communicate the test results (288) or collected test data (280) of the test activity (212) of the scenario (262) to the ATE (220). The ATE (220) or the controller (270) is also configured to improve the test process and / or debug or diagnose the DUT.

[0099] A controller (270), such as an on-chip OCST controller or an OCST card, or an ATE (220) is configured to dynamically generate or modify a set of test parameters for a test scenario that can satisfy constraints and know the current test parameters for debugging. The method of generating test parameter values ​​for a test activity (212) involves randomization, which follows a desired distribution that selectively highlights extreme values, for example, by using a constraint solver to maximize coverage, or for example, by using nested loops for thorough coverage of some test parameters.

[0100] To further improve the test environment, a test learning environment may be required. Extensive characterization testing is the basis of test learning, and many DUTs are exposed to many test steps or many test scenarios. To cover many combinations of test parameters (216), it is desirable that not all DUTs are exposed to the same test step or test scenario. To prevent bias in test scenarios for a specific DUT (282), the test steps or test scenarios are executed in a random permutation order.

[0101] The ATE (220) or controller (270) may also use a machine learning or AI module trained on collected test result data (280) and system-level test results. The machine learning module can analyze the collected test result data (280) and system-level test results and predict system-level test results based on a new set of collected test data (280).

[0102] The AI ​​module can also be trained to predict system-level test failures from test results using measured result data without specification limitations, such as on-chip sensor data, or test parameters outside specification limits, such as significantly low voltage or significantly high frequency. Failure of test phases or test activities outside specification limits is not evidence of a bad DUT.

[0103] However, the machine learning module or model may be configured to predict system-level test results from test phase or test activity results that include out-of-specification test parameters and test phase or scenario-related attributes, such as measurement results that are free from or outside of specification limits, and related test activities and test resources.

[0104] Such a model can identify some previously undetected SLT failures, but failures may also occur in some DUTs that have passed SLT and all other true OCST tests. These cases can be considered as yield losses due to testing and, preferably, can be carefully traded for additionally discovered SLT failures based on a cost model. Only these additional testing steps or scenarios may be included in the production steps required by the model. Other additional testing steps can be eliminated.

[0105] The ATE (220) or controller (270) may also be configured to debug and / or diagnose the DUT. Since the test result table (280) contains the results of the test activities (212) of the test scenarios (262) executed concurrently, further analysis of the collected test data (280) is required to identify and / or classify the defective DUT or DUT resources.

[0106] The general idea of ​​debugging or diagnosis is to identify the test parameters (216) that have the greatest influence on the occurrence of OCST failures. Test parameters (216) associated with test activities (212) involving specific behaviors of specific IP blocks provide useful hints for debugging.

[0107] A machine learning model trained by a table combining test activities, DUT resources, test parameters, test results, and overall OCST results, and optionally test phases or scenarios with multiple DUTs, can be used to classify and identify defective DUT resources. A machine learning module or machine learning feature selection algorithm can identify test activities, tests or DUT resources, and test results that are important in explaining the OCST results contributing to the occurrence of OCST failure.

[0108] In other words, the controller (270) controls the test process. The controller, or OCST controller, is preferably an on-chip processor having an optional test operating system, but may also be an OCST card that communicates with the DUT or ATE workstation.

[0109] The operation of the OCST controller may, for example, trigger the execution of multiple test activities and read their pass / fail results and / or measurement results. Test activities may include a combination of optional stress generation and / or optional fault detection.

[0110] Additional examples of test activities are listed below.

[0111] ● The ATE sets external test conditions of the DUT, such as the DUT supply voltage or frequency, that is, the OCST controller can control the ATE resources.

[0112] ● ATE performs measurements such as supply current measurement.

[0113] ● Move data blocks between IP cores and check previous and / or subsequent contents.

[0114] ● Run memory test on the on-chip CPU.

[0115] ● Run memory self-test.

[0116] ● Image compression and decompression, and checking the difference between the original and decompressed images. (Stress & Check)

[0117] ● Run I / O loopback test.

[0118] ● Application of stress generation using DFT technology.

[0119] ● Activating and reading arbitrary observable structures using DFT technology

[0120] The controller always knows which test scenario is being executed with the test (activity) parameters, which is necessary for debugging in case of errors.

[0121] Additionally, the controller or OCST controller can dynamically generate or modify test activity parameters based on constraints and / or operate on or retrieve them from a pre-calculated list. Additionally, the processor executing the controller or OCST controller code can also generate test activities.

[0122] Exemplary test activity table according to Fig. 3

[0123] FIG. 3 illustrates an exemplary test activity table (300) according to one embodiment. The test activity table (300) is an example of the test activity table (210) of FIG. 2 or is similar thereto.

[0124] The test activity table (300) includes columns of test activities, resources, test parameters, or possible results. The test activity table is configured to list all possible tests that can be executed on the DUT, along with the required resources, adjustable parameters, and possible results or outcomes.

[0125] The test activity table (300) can be used as an input for generating a test sequence by a constraint solver such as the constraint solver (250) of FIG. 2. The test sequence can define test activities including which test parameters are executed in which order for which DUT.

[0126] The test activity table (300) may include general test activities or DUT-specific test activities. Some examples of test activities are included in the exemplary test activity table (300).

[0127] For example, the first test activity (A1) may include a processing unit 2 (CPU2) that writes data to memory 3 (MEM3) and checks the contents of MEM3. The activity (A1) requires resources R1: CPU2, R2: MEM3, R3: ATE DPS for core supply. The adjustable test parameters of the test activity (A1) are P1: bandwidth and P2: DPS voltage. The result may include two values: a fail / pass value r1 and a current value r2.

[0128] For example, test activity (A2) is the memory-embedded self-test (MBIST) of MEM3, which requires resource R2, i.e., MEM3, which has no adjustable parameters and consequently has a pass / fail result.

[0129] For example, test activity (A3) is an MPEG self-test that requires an MPEG resource with an adjustable block size as test parameter P3, along with a result of a pass / fail value.

[0130] Resource conflict table according to Fig. 4

[0131] FIG. 4 illustrates a resource conflict table (400) that can be used and / or generated by the constraint solver (250) of FIG. 2. The resource conflict table (400) has a column for test activities and a column for all resources of the DUT. Each test activity in the table (400) uses one or more DUT test resources, which are indicated by an 'X' in each resource column.

[0132] A constraint solver, such as the constraint solver (250) of FIG. 2, may take a test activity table, such as the table (300) of FIG. 3, as input for generating a test sequence. The step of generating a test sequence is to generate a resource conflict table, such as the resource conflict table (400). The resource conflict table (400) shows which test activities use the same DUT resource and therefore which test activities cannot be executed simultaneously.

[0133] For example, the resource conflict table (400) shows the conflicting resources of the test activities in the test activity table (300). For example, test activity A1 uses resources R1, R2, and R3 and does not use R4. For example, test activity A2 uses only R2 as a resource. For example, test activity A3 uses DUT resource R4.

[0134] As shown in the resource conflict table (400), test activity A1 and test activity A2 are conflicting test activities because both activities require test resource R2. Since both test activities A1 and A2 require test resource R2, they cannot be executed simultaneously. That is, test activities A1 and A2 cannot be executed simultaneously, just as they cannot enter the same test scenario. Test activities without resource conflicts can be combined into a test scenario.

[0135] Test scenario table according to Fig. 5

[0136] FIG. 5 illustrates a test scenario table (500) that can be used or generated by the constraint solver (250) of FIG. 2. The test scenario table (500) includes a test scenario column and a column for each provided test activity. The test scenario table (500) includes all possible test scenarios. One or more test activities of a test scenario are executed simultaneously.

[0137] The test scenario table (500) illustrates all test scenarios that can be generated from test activities such as test activities A1, A2, and A3 of the test activity table (300) of FIG. 3. The test activities of the test scenarios must not conflict. Conflict test activities are illustrated in a resource conflict table similar to the resource conflict table (400) of FIG. 4. From all possible test scenarios, by excluding test scenarios that have conflict test activities taken from the resource conflict table, a test scenario table having non-conflict test activities, such as the test scenario table (500), is obtained.

[0138] For example, the test scenario table (500) includes a test scenario column and columns for all test activities, such as A1, A2, and A3. According to the resource conflict table (400) of FIG. 4, since A1 and A2 use the same resource R2, test activity A1 and test activity A2 cannot be executed at the same time and cannot be in the same test scenario.

[0139] For example, the test scenarios of the exemplary test activities in FIG. 3 are test scenario S1 with test activity A1, test scenario S2 with test activity A2, test scenario S3 with test activity A3, test scenario S4 with concurrent test activities A1 and A3, and test scenario S5 with concurrent test activities A2 and A3. Due to resource constraints, there is no test scenario in which test activities A1 and A2 are executed simultaneously.

[0140] Whether test activities are actually executed simultaneously may depend on test parameter settings and other unknown factors. A test sequence or test suite consists of multiple test phases that execute a given test scenario with specified test parameter values ​​for the test activities.

[0141] Test step table according to Fig. 6

[0142] FIG. 6 illustrates a test phase table (600) that can be used or generated by the constraint solver (250) of FIG. 2. The test phase table includes columns for test phases, test scenarios, and test activities, and columns for test parameters corresponding to the test activities.

[0143] The test phase column of the test phase table (600) contains execution numbers to identify different test phases. The test scenario column may include all possible test scenarios at least once. A test scenario may be tested more than once using several different test parameters. Similar to the test scenario table (500) of FIG. 5, the test activity of a test scenario is indicated by an 'X' in the corresponding test activity column.

[0144] If the test scenario includes a test activity, the corresponding test parameter column contains test parameter values. The test parameter values ​​are preferably selected randomly, optionally follow a predefined distribution, and / or optionally have a specific percentage of concentration in extreme values ​​that tend to cause more problems than the randomly generated test parameter values.

[0145] A test sequence such as the test sequence table (260) of FIG. 2 is generated from the test step table (600) by randomly mapping the test steps of the test step table (600) to the DUT or following a predefined distribution to cause more problems than the randomly mapped test sequence.

[0146] The test scenario column includes all possible test scenarios, such as test scenarios S1 to S5 of the test scenario table (500), and the test scenarios can be tested with various different test parameters.

[0147] For example, the first test step may be a first test scenario S1 having a test activity A1 corresponding to a bandwidth of 10 GB / S, which is test parameter P1, and a DPS voltage of 0.9 V, which is test parameter P2. For example, the second test step may be the same test scenario S1 having a test activity A1 corresponding to a bandwidth of 20 GB / S, which is another test parameter P1, and a DPS voltage of 0.86 V, which is test parameter P2.

[0148] For example, test phase 3 may include test scenario S2, which includes test activity A2, which has no adjustable parameters.

[0149] For example, test step 4 includes test scenario S3 with test activity A3, and test parameter P3 has a block size of 128kB. For example, test step 5 may include the same scenario S3 with test parameter P3 having a block size of 1MB.

[0150] For example, test step 6 includes test scenario S4 having test activities A1 and A3 with a test parameter P1 of 50 GB / S bandwidth, P2 of a DPS voltage of 1.04 V, and a test parameter P3 of a block size of 6 MB. For example, step 7 includes the same test scenario S4 with a test parameter P1 of 3 GB / S bandwidth, P2 of a DPS voltage of 0.97 V, and a test parameter P3 of a block size of 500 KB. For example, step 8 includes the same test scenario S4 with a test parameter P1 of 27 GB / S bandwidth, P2 of a DPS voltage of 0.88 V, and a test parameter P3 of a block size of 21 MB.

[0151] For example, test step 9 includes test activity A2 with no test parameters and test scenario S5 with a block size of 7 MB with test parameter P3. For example, test step 10 includes the same test scenario S5 with a block size of 780 KB with test parameter P3. For example, test step 11 includes the same test scenario S5 with a block size of 13 MB with test parameter P3.

[0152] Test result table according to Fig. 7

[0153] FIG. 7 illustrates an empty test result table (700) that can be filled by a controller after performing the test activity of the test sequence table (260) of FIG. 2. The test result table includes columns for the test results of the DUT, test phase, and test activity, and a column for the overall test result.

[0154] The test result table (700) contains a test sequence such as the test sequence table (260) of FIG. 2. The first two columns of the test result table (700), the DUT and test step columns, define which test step was performed on which DUT. The test sequence is generated from the test step table (600) of FIG. 6 by randomly mapping the test steps of the test step table (600) to the DUT, following a predefined distribution to cause more problems than the randomly mapped test sequence, and / or satisfying predefined conditions.

[0155] An exemplary test sequence may include, for example, DUT1 tested by test steps 6, 11, 7, 1 and 8, and DUT2 tested by test steps 5, 2, 10, 4 and 9.

[0156] A controller, such as an on-chip controller or controller card, performs tests according to a test sequence. The results of the test activities represented by columns r1 (#fails), r2 (current), r3 (#fails), and r4 (p / f) are collected by the controller. The overall OCST results may be based on the results of all test activities or a subset of the test activity results. Additionally, or alternatively, the OCST results may include a classification of the DUT based on the test activity results or a portion of the test activity results.

[0157] In this example, the total OCST result is calculated from results r1, r3, and r4, meaning that in this example, measurement result r2 does not contribute to the test result because there are no specification limitations.

[0158] Failure table per test step according to Fig. 8

[0159] FIG. 8 illustrates a failure table (800) per test step, including a DUT column and a column for all test steps. Tested DUTs are listed in the DUT column. If a DUT passes a given test step, the row of the tested DUT contains the letter 'P' in the given test step column, and if a DUT fails a given test step, it contains the letter 'F' with a highlighted background.

[0160] The controller or ATE is configured to use a failure table (800) per test step to optimize, preferably reduce, the number of test steps. For example, the number of tests that never fail and / or redundant test steps can be reduced. The work of the controller or ATE may include selecting a minimum set of test steps known as the minimum covering set problem, for which there is a known solution.

[0161] For example, the failure table (800) per test step may include the test results of four DUTs, DUTs 1-4. For example, DUT 1 may pass all test steps except test step 6. For example, DUT 2 may pass all test steps except test steps 2 and 4. For example, DUT 3 may pass all test steps except test steps 1 and 6. For example, DUT 4 may pass all test steps except test step 8.

[0162] For example, when the controller or ATE analyzes the failure table (800) per test step, the controller may reach the conclusion that test steps 2, 4, and 6 never fail and therefore can be removed from production testing. Additionally, test step 4 overlaps with test step 2 and can also be removed.

[0163] Comparison of OCST and SLT according to Fig. 9

[0164] FIG. 9 illustrates an exemplary comparison table (900) between OCST test results and SLT test results. The exemplary comparison table (900) illustrates the differences that may occur when testing the same set of DUTs with both OCST and SLT test methods.

[0165] In this example, 9,900 devices passed and 70 failed in both the OCST and SLT test methods. There were 25 devices or DUTs that passed the SLT test but failed the OCST test, but only 5 DUTs that passed the OCST test and failed the SLT test.

[0166] In other words, in the above example, OCST missed 5 devices that failed SLT but found problems in 25 DUTs that SLT could not find, which is a good balance assuming that it describes a realistic scenario where all test steps have realistic test values.

[0167] Training table combining test results and SLT results according to Fig. 10

[0168] FIG. 10 illustrates a training table (1000) that combines collected test data and SLT results. The training table (1000) is configured to be used as a training table or training data set for a machine learning or AI module. The training table (1000) may include all test activities, test resources, and test results of test steps performed on a DUT having corresponding SLT results. The training table (1000) may include multiple test steps executed on multiple DUTs.

[0169] The training table (1000) may include test data collected outside or slightly outside the specifications of the DUT, combined with SLT results, and the SLT test was performed within the specifications of the DUT. For example, in this case, a failed OCST result is not considered a strong signal of DUT failure, but a passing OCST test outside the specification limits can be considered a strong signal of a good DUT and the DUT can be classified as a high-quality DUT.

[0170] The ATE or controller may include a machine learning unit or AI module that can be trained by a training table (1000) and may be configured to predict SLT results from a newly performed test step on the DUT. The machine learning unit may also be used to improve the test process.

[0171] Training table having test results according to Fig. 11

[0172] FIG. 11 illustrates a training table (1100) containing collected test data and the entire OCST result. The training table (1100) is configured to be used as a training table or training data set for a machine learning or AI module. The training table (1100) may include all test activities, test resources, and test results of the test phase performed on the DUT having the corresponding OCST result.

[0173] The table (1100) is used for debugging and diagnostic purposes. The general idea of ​​debugging is to identify test parameters that affect the cases where the OCST fails most frequently. The ATE or controller may include an AI or machine learning unit to identify and / or classify frequently failing DUT resources or DUTs. For efficiency, the table may be limited to failed devices. The analyzer can be performed across multiple DUTs to identify frequently occurring failure mechanisms.

[0174] A machine learning unit or module can be trained by a test result table (1100) to predict a failed DUT or DUT resource.

[0175] Alternative implementation

[0176] Although some aspects have been described in the context of a device, these aspects also represent descriptions of the corresponding method, and the blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent descriptions of the corresponding features, items, and blocks of the device.

[0177] Depending on specific implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be implemented using a digital storage medium storing electrically readable control signals, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM, or flash memory, which works in cooperation with (or may work in cooperation with) a programmable computer system to perform each method.

[0178] Some embodiments according to the present invention include a data carrier having an electrically readable control signal, which can cooperate with a programmable computer system to perform one of the methods described herein.

[0179] Generally, embodiments of the present invention may be implemented as a computer program product having program code, and the program code is operated to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, in a machine-readable carrier.

[0180] Other embodiments include a computer program stored in a machine-readable carrier that performs one of the methods described herein.

[0181] Further embodiments of the method of the present invention are, therefore, data carriers (or digital storage media or computer-readable media) on which a computer program for performing one of the methods described herein is recorded. Data carriers, digital storage media or recording media are generally tangible and / or non-tangible.

[0182] Further embodiments of the method of the present invention are, therefore, a sequence of data streams or signals representing a computer program that performs one of the methods described herein. The sequence of data streams or signals may be configured to be transmitted, for example, through a data communication connection, such as the Internet.

[0183] Additional embodiments include processing means, such as a computer, or a programmable logic device configured or adjusted to perform one of the methods described herein.

[0184] Additional embodiments include a computer installed with a computer program for performing one of the methods described herein.

[0185] Further embodiments according to the present invention include a device or system configured to transmit a computer program for performing one of the methods described herein to a receiver (e.g., electrically or optically). The receiver may be, for example, a computer, a mobile device, a memory device, etc. The device or system may include, for example, a file server for transmitting the computer program to the receiver.

[0186] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field-programmable gate array may work in cooperation with a microprocessor to perform one or more of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0187] The device described in this specification may be implemented using a hardware device, or a computer, or a combination of a hardware device and a computer.

[0188] Any device described in this specification, or any component of a device described in this specification, may be implemented at least partially in hardware and / or software.

[0189] The method described in this specification may be performed using a hardware device, or a computer, or a combination of a hardware device and a computer.

Claims

Claim 1 An automatic test device (110, 220) for testing one or more test target devices (120, 282), wherein the automatic test device is configured to generate one or more test scenarios (140a-c, 262) including a plurality of test activities (150a-c, 212, 212a-d), each test activity utilizes a subset of resources of the test target device, wherein the automatic test device is configured to generate a plurality of test scenarios such that resources of the test target device associated with the plurality of test activities of the test scenarios do not conflict with each other, and wherein the plurality of test activities of the test scenarios are randomly selected under the constraint that resources of the test target device associated with the plurality of test activities of the test scenarios do not conflict with each other. Claim 2 An automatic test device according to claim 1, wherein the plurality of test activities of a given test scenario are configured to be executed simultaneously. Claim 3 An automated test device according to claim 1 or 2, wherein the one or more test activities are associated with one or more test parameters (216, 216a-e) characterized by each test parameter value and / or one or more constraints (218). Claim 4 An automated test device according to claim 1 or 2, wherein one or more of the above test activities are characterized by one or more test parameters that exceed predefined limits. Claim 5 delete Claim 6 In claim 1 or 2, the automatic test equipment comprises a constraint solver (250) that generates a test scenario to prevent the occurrence of conflicts between the resources of the device to be tested. Claim 7 An automatic test device according to paragraph 4, wherein one or more of the above test activities include activating a stress generator placed in the device under test. Claim 8 In paragraph 1 or 2, the automatic test equipment is configured to generate a test sequence of a test scenario. Claim 9 In paragraph 8, the test sequence comprises two or more test scenarios having the same test activity, and the test activity is an automatic test device having at least one different test parameter value. Claim 10 In paragraph 8, the plurality of test scenarios of the test sequence are randomly arranged and / or selected, automatic test equipment. Claim 11 In paragraph 8, the automatic test equipment is configured to generate the test sequence, the test sequence is configured to be executed by a controller (270) to collect test data. Claim 12 In paragraph 11, the automatic test equipment is an on-chip processor or controller card configured to communicate with the automatic test equipment and the device to be tested. Claim 13 In paragraph 11, the automatic test equipment, wherein the controller comprises one or more interfaces configured to read device data under test and / or device sensor data under test. Claim 14 In claim 11, the controller comprises one or more sensors distributed across the area of ​​the device to be tested, and the automatic test equipment is configured such that the controller reads sensor test data of the one or more sensors. Claim 15 In paragraph 11, the controller is configured to respond when the collected test data (280) meets predetermined conditions, an automatic test device. Claim 16 In paragraph 11, the automatic test equipment is configured such that the controller communicates the collected test data to the automatic test equipment. Claim 17 In paragraph 11, the controller or the automatic test equipment is configured to dynamically generate test parameter values ​​based on test activity constraints and / or collected test data. Claim 18 In paragraph 11, the controller or the automatic test equipment is configured to analyze the collected test data to optimize the test sequence, the automatic test equipment. Claim 19 In paragraph 11, the controller or the automatic test equipment compares the results of a system-level test within the predefined limits with the collected test data within the predefined limits and / or outside the predefined limits in order to optimize the test sequence. Claim 20 In claim 11, the automatic test equipment comprises an artificial intelligence or machine learning unit, and the controller or the automatic test equipment is configured to train the artificial intelligence or machine learning unit by the results of a system-level test and / or the collected test data and / or the comparison of the system-level test and the collected test data in order to optimize a test sequence. Claim 21 In paragraph 20, the automated test equipment configured such that the trained artificial intelligence or machine learning unit predicts the results of the system-level test based on the collected test data. Claim 22 In paragraph 11, the controller or the automatic test equipment is configured to obtain test results by analyzing the collected test data. Claim 23 In paragraph 20, the automated test equipment configured such that the trained artificial intelligence or machine learning unit analyzes the collected test data to optimize the test sequence and / or obtain test results. Claim 24 A process for testing one or more test target devices by an automated test device, wherein the automated test device generates one or more test scenarios including a plurality of test activities, each test activity utilizes a subset of resources of the test target device, and the automated test device is configured to generate the plurality of test scenarios such that resources of the test target device associated with the plurality of test activities of the test scenarios do not conflict with each other, and the plurality of test activities of the test scenarios are randomly selected under the constraint that resources of the test target device associated with the plurality of test activities of the test scenarios do not conflict with each other. Claim 25 A computer program stored on a computer-readable storage medium that implements the process of claim 24 when executed by a computer or signal processor.

Images