System and method for multi-dimensional dynamic component average testing

Abstract

Embodiments of the present invention provide systems and methods for multi-dimensional part average testing to test devices and analyze test results to detect outliers, according to embodiments of the present invention. Testing may include, for example, calculating multivariate (e.g., bivariate) statistics using delta measurements of similar devices, ratios of measurements, or principal component analysis to identify eigenvectors and eigenvalues ​​to define meta-parameters. Raw test result data can be transformed into residual space and robust regressions performed to prevent outlier results from influencing the regression, thereby advantageously reducing overkill.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of device testing. More particularly, embodiments of the present invention relate to methods and systems for component average testing.

Background Art

[0002] A device or equipment under test (e.g., Device Under Test: DUT) is usually tested to determine the performance and uniformity of the device before it is sold. The device can be tested using various test cases, and the results of the test cases are compared with the expected output results. If the results of the test cases do not match the expected output values, the device can be considered a failed device or an outlier, and the device can be binned based on performance, etc.

[0003] The DUT is usually tested by an automatic test device or automated test equipment (Automatic or Automated Test Equipment: ATE), and the ATE can be used to perform complex tests using software and automation to improve test efficiency. The DUT can be any type of semiconductor device, wafer, or component intended to be integrated into a final product such as a computer or other electronic device. By using ATE to remove defective or unsatisfactory chips during manufacturing, the yield quality can be significantly improved.

[0004] In the field of automotive device testing, additional safety is considered, and in some cases, automotive devices are tested using zero tolerance methods. Part Average Testing (PAT) is a type of device testing based on statistical analysis that identifies parts (e.g., wafers or dies) with characteristics significantly different from other parts within the same lot, wafer, or vicinity of a sub-wafer. These differences in parts can indicate which parts are expected to fail (e.g., go out of tolerance). Generally, in part average testing, data is collected from previously tested parts, and the average value (or median) of the previous measurements is compared to the current part. If the measurement value of the current part is outside a specific range, that part is indicated as an outlier. This range can be based, for example, on the number of standard deviations from the mean or on the calculation of quartile range values.

[0005] Unfortunately, these conventional approaches to part average testing are limited to use with individual test parameters using univariate statistical techniques. Furthermore, current techniques for setting the limit values for part average testing are somewhat simplistic and can result in more overkill than desired. SUMMARY OF THE INVENTION

[0006] What is needed is a multivariate approach to device testing that is superior to univariate techniques in detecting outlier units and limiting overkill and escape. Accordingly, embodiments of the present invention provide a system and method for multi-dimensional part average testing for testing a device and analyzing test results to detect outliers according to embodiments of the present invention. The testing can include, for example, the calculation of multivariate (e.g., bivariate) statistics using principal component analysis to identify eigenvectors and eigenvalues to define delta measurements, ratios of measurements, or meta-parameters of similar devices.

[0007] According to one embodiment, a method for dynamic component average testing is disclosed. The method includes determining a test limit value based on past test result data, testing a plurality of devices according to the test limit value and obtaining test results, calculating multivariate statistics using the test results, calculating an average value or a median of the multivariate statistics, calculating a difference between the multivariate statistics and the average value or the median, updating the test limit value based on the average value or the median and generating an updated test limit value, and identifying outliers of the plurality of devices based on the average value or the median and the updated test limit value.

[0008] According to some embodiments, the method includes determining important measurement values for testing by performing at least one of principal component analysis (PCA), independent component analysis (ICA), autoencoding, machine learning, or other similar analysis methods for identifying important factors.

[0009] According to some embodiments, the step of calculating multivariate statistics using the test results includes at least one of forming bivariate pairs, forming ratios, and forming deltas.

[0010] According to some embodiments, the step of calculating multivariate statistics using the test results includes clustering the test results according to result types.

[0011] According to some embodiments, the method includes transforming the average value or the median into a residual space using a non-linear, non-monotonic transformation to amplify the outlier results.

[0012] According to some embodiments, the method includes excluding outlier results before updating the test limit value based on the average value or the median.

[0013] According to some embodiments, the method includes performing weighted least squares regression before updating the test limit based on the mean or median value.

[0014] According to another embodiment, an apparatus for performing a dynamic component average test is disclosed. The apparatus includes a processor and a memory that communicates with the processor for storing test data and instructions. The processor executes instructions for performing a method of a multivariate (or multi-dimensional) component average test. The method includes determining a test limit value based on past test result data, testing a plurality of devices according to the test limit value and obtaining test results, calculating multivariate statistics using the test results, calculating an average value or a median value of the multivariate statistics, calculating a difference between the multivariate statistics and the average value or the median value of the multivariate statistics, updating the test limit value based on the average value or the median value and generating an updated test limit value, and identifying outliers of the plurality of devices based on the average value or the median value and the updated test limit value.

[0015] According to some embodiments, the method includes determining important measurement values for testing by performing at least one of principal component analysis (PCA), independent component analysis (ICA), autoencoding, machine learning, or other similar analysis methods for identifying important factors.

[0016] According to some embodiments, the step of calculating multivariate statistics using the test results includes at least one of forming bivariate pairs, forming bivariate ratios, and forming bivariate deltas.

[0017] According to some embodiments, the step of calculating multivariate statistics using the test results includes clustering the test results according to the result type.

[0018] According to some embodiments, the method includes transforming the average value or the median value into the residual space using a non-linear, non-monotonic transformation to amplify the outlier results.

[0019] According to some embodiments, the method includes excluding the outlier results before updating the test limit value based on the average or median value.

[0020] According to some embodiments, the method includes performing weighted least squares regression before updating the test limit value based on the average or median value.

[0021] According to another embodiment, one or more non-transitory computer-readable media storing program instructions executable by one or more processors implement a method that includes determining a test limit value based on past test result data, testing a plurality of devices according to the test limit value and obtaining test results, calculating multivariate statistics using the test results, calculating an average or median value of the multivariate statistics, updating the test limit value based on the average or median value of the multivariate statistics and generating an updated test limit value, and identifying outliers of the plurality of devices based on the average or median value and the updated test limit value.

[0022] According to some embodiments, the method includes determining important measurement values for the test by performing at least one of principal component analysis (PCA), independent component analysis (ICA), autoencoding, machine learning, or other similar analysis methods for identifying important factors.

[0023] According to some embodiments, the step of calculating multivariate statistics using the test results includes at least one of forming bivariate pairs, forming ratios, and forming deltas.

[0024] According to some embodiments, the step of calculating multivariate statistics using the test results includes clustering the test results according to the result type.

[0025] According to some embodiments, the method includes transforming an average or median value into a residual space using a non-linear, non-monotonic transformation to amplify the result of outliers.

[0026] According to some embodiments, the method includes accessing e-test data or data generated during another manufacturing stage.

Brief Description of the Drawings

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0028]

Figure 1

[0029]

Figure 2

[0030]

Figure 3

[0031]

Figure 4

[0032]

Figure 5

[0033]

Figure 6

[0034]

Figure 7

[0035] Here, several embodiments will be referred to in detail. Although the subject matter is described in relation to alternative embodiments, it will be understood that they are not intended to limit the subject matter claimed to these embodiments. On the contrary, the subject matter claimed is intended to cover alternatives, modifications, and equivalents that may be included within the spirit and scope of the subject matter claimed as defined by the appended claims.

[0036] Furthermore, in the following detailed description, many specific details are set forth in order to provide a thorough understanding of the subject matter claimed. However, one of ordinary skill in the art will recognize that embodiments may be practiced without these specific details or their equivalents. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects and features of the subject matter.

[0037] Part of the following detailed description is presented and discussed from a method perspective. The steps and their ordering are disclosed in the figures of this specification (e.g., FIG. 6) that illustrate the operation of the method, but such steps and ordering are exemplary. Embodiments are well-suited to the execution of other various steps or variations of the steps described in the flowcharts of the figures of this specification, and to execution in orders other than those shown and described herein.

[0038] Part of the detailed description is presented with respect to procedures, steps, logical blocks, processes, and other symbolic representations of operations on data bits that can be executed on a computer memory. These descriptions and representations are the means used by those skilled in the data processing art to most effectively communicate the content of their work to other skilled persons. Here, procedures, computer-executed steps, logical blocks, processes, etc. are generally considered to be a series of self-consistent steps or instructions leading to a desired result. A step is a step that requires physically manipulating a physical quantity. Although not necessarily, usually these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise operated on by a computer system. Mainly, it has been found convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, parameters, etc. because they are commonly used.

[0039] However, it should be noted that all of these terms and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to these quantities. As will be apparent from the following discussion, unless otherwise specified, throughout, discussions using terms such as "accessing," "writing," "including," "storing," "transmitting," "associating," "identifying," "encoding," "labeling," etc. refer to the operation of data represented as physical (electronic) quantities in the registers and memories of a computer system and, similarly, to the operation and processes of a computer system, or a similar electronic computing device, that converts the data into other data represented as physical quantities in the memory or registers of the computer system, or other such information storage devices, transmission devices, or display devices.

[0040] Some embodiments may be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers or other devices. Generally, program modules include routines, algorithms, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. Systems and methods for multi-dimensional dynamic component average testing

[0041] Embodiments of the present invention provide a system and method for multi-dimensional component average testing for testing a device and analyzing test results to detect outliers according to embodiments of the present invention. The testing may include, for example, calculating multivariate (e.g., bivariate) statistics using principal component analysis to identify eigenvectors and eigenvalues to define delta measurements, ratios of measurements, or meta-parameters of similar devices. In component average testing (PAT), conventionally, fixed outlier limit values for specific test parameters set prior to processing a set of materials are used. For example, the test parameters may be based on the calculation of the interquartile range value of past test data. In this way, components (e.g., wafers, dies, materials, devices) having characteristics significantly different from the normal distribution of other components within the same lot can be identified. The differences in these components may indicate which components are likely to fail. Using dynamic PAT (DPAT), non-fixed limit values that can be automatically updated over time as new test results become available can be provided.

[0042] FIG. 1 is a flowchart showing an exemplary computer-implemented process 100 for determining static PAT limit values and performing a parts average test (PAT) based on past test data 105. As shown in FIG. 1, the past test data 105 is used to establish the static PAT limit values 110. Then, static PAT screening 120 is used to test devices (e.g., device 115) to determine whether they pass or fail the static PAT limit values. Devices that fail can be binned based on their performance.

[0043] FIG. 2 shows an exemplary DPAT test process 200 for a packaged product that dynamically calculates and adjusts PAT limit values when a unit is tested, according to an embodiment of the present invention. As shown in FIG. 2, when a sufficient number of tests are performed, the PAT limit values are dynamically established based on the accumulated test data. The lot start stage 205 is executed using a predefined PAT limit value, which is adjusted during the test analysis stage 215 based on the test data accumulated during the test phase 210. Dynamic PAT is an efficient approach to parts average testing, but only univariate statistical techniques defined for use with individual test parameters, typically in accordance with Automotive Electronics Council (AEC) standards, are usually used in the analysis of test results. Furthermore, existing approaches to PAT often result in more overkill than desired when executed in accordance with AEC standards.

[0044] Accordingly, embodiments of the present invention can screen devices using multiple parameters in one statistical measurement criterion, such as bivariate parameters. In some embodiments, two leakage measurements are combined to screen devices outside a predetermined range of threshold values of the screening process. Some embodiments use multivariate statistics of any dimension of two or more dimensions to perform the screening process. Embodiments of the present invention can perform parallel execution to screen multiple devices and can implement robust statistics based on past test data. The user of the test equipment (e.g., creator or administrator) performing the screening can indicate, using n screening parameters, which test measurements are intended to be utilized for screening purposes. The number of test results is used to calculate the number y of outlier results.

[0045] FIG. 3 shows an exemplary multi-dimensional part average test process 300 for testing a device and analyzing test results according to an embodiment of the present invention. The test process 300 includes a lot start phase 305, a test execution phase 310, and a post-test execution phase 315. The DUT can be screened using multiple parameters of one statistical measurement criterion. For example, screening can include the calculation of bivariate statistics using principal component analysis to identify eigenvectors and eigenvalues to define delta measurements, ratios of measurements, or meta-parameters of similar devices.

[0046] The lot start phase 305 is executed before the test to determine the initial limit values of the test program. The lot start phase 305 may include prefetching stored past data and other relevant information to seed the limit value calculation for each screening measurement value. The past data may be stored locally in the test system or fetched from a local or remote database or host. The past data may include data from other production lots of the same devices that have already been tested, test insertion data for the same lot that was previously executed, or a combination of the two. Data from the probe phase of the wafer test may be used to establish the initial test limit values for the package test after the wafer is cut. The past data may also include test data from e-tests or parametric tests collected during wafer manufacturing, or other data obtained during device manufacturing, particularly in the case of materials serialized during manufacturing for tracking purposes. Fixed test limit values can also be provided manually by the user during the lot start phase 305.

[0047] The test execution phase 310 includes the collection of PAT parameters and the execution of the test program for the device under test. In one exemplary process for determining the outlier status during the test, bivariate pairs, ratios, deltas, etc. are formed for all measurement values, or clusters of measurement values classified by type (e.g., current, voltage, frequency, etc.), and the median and residuals are calculated for each bivariate pair. The new measurement values accumulated during the test can be used to update the test limit values during the test, and the outlier status can be determined using the aggregated bivariate results. Robust regression (e.g., reweighted least squares regression) can be optionally executed during the test execution phase 310 to exclude the expected outliers from each bivariate pair before using the test results to update the test limit values.

[0048] According to some embodiments, the multi-dimensional component average test process 300 includes comparing the DUT to the signatures of a set of previously tested devices. The DUT is tested using parameters established according to the lot (e.g., the most similar wafer or lot) having the signature that most closely matches the signature of the DUT.

[0049] According to some embodiments, the baseline test limit values are determined according to the number of previous tests (e.g., 50 or 100 tests) performed on the same lot. One approach is to determine the median of the previous measurements to estimate the parameter values of the next DUT (e.g., die). The difference ("residual") between the result of the DUT and the median is calculated. If the value of the residual is significant compared to the baseline, the DUT may be determined to be an outlier (failed).

[0050] Using data fed forward to update the test limit values, individual limit values can be established for different physical parts of the material being tested. For example, e-test data, which is typically collected at a limited number of sites on a wafer, can be used to adjust the limit values applied to new materials in the same part of the wafer. Different limit values can be applied based on which zone of the wafer the material is from. The zones can be defined, for example, radially or as quartiles. Reticle shots on the wafer can be tracked and used to determine which reticle shot was used for each unit of the baseline. For example, the limit values can be set according to which reticle shot was used for each unit. In another example, the units are compared according to their position within the reticle, such as the upper left position of a 2×2 reticle shot.

[0051] After screening measurements are performed with a test program, the measurement values are used in the execution of screening calculations. For example, the measurement data can be passed to the screening computer system in one call (as opposed to being called individually for each test at the time of test execution), thereby saving communication overhead. Bivariate statistics are calculated for all (n)(n - 1) combinations of any two parameters. For example, bivariate statistics can include a ratio (n i / n j ) or a delta (n i -n j ), or other combinations of two parameters suitable for outlier screening. For each combination of parameters, median statistics are calculated using the previous y bivariate measurement criteria. The median of the statistics then becomes the expected value for the current device / die of interest.

[0052] The post - test execution phase 315 is executed after the test execution 310 and analyzes the results from one or more of the most recently tested lots. This phase can include identifying measurement trends and identifying materials that can become uncontrollable, in which case the test system can generate a notification and / or automatically stop testing devices in the material flow. According to some embodiments, an additional monitoring layer functionality is provided during testing for quality improvement. For example, all measurement data and test result data can be maintained and aggregated across all devices being tested, and the aggregated data can be analyzed to identify trends. If a trend indicating a potential quality deviation is identified, the test system can intervene.

[0053] The periods of aggregation and intervention can be various. In one example, the period is limited to the period of testing a single lot of materials, and the intervention is carried out immediately. In this case, the test is aborted and the diagnostic process is initiated. In another example, the period can be longer than the test of a single lot of materials, and the intervention can be a notification (e.g., a message, an email, or other indication) to the responsible person of the trend detected over multiple material lots, and the manufacturing can be stopped or dynamically changed. Thus, the function of DPAT statistics is extended beyond the results of a single device by using DPAT statistics to monitor the aggregated results and trends.

[0054] As shown in FIG. 4, according to an embodiment of the present invention, the difference between the median and the measurement result (the "residual") is calculated. For example, the residual of the latest die can be compared with the residuals of the previous y measurements, and accordingly, the distribution of the dies can be reordered. In the example of FIG. 4, a non-linear, non-monotonic transformation is applied to the raw test data, and the raw data is transformed into the residual space. By this method, the behavior of outliers can be amplified, and as a result, non-conforming devices that have not been detected so far are recognized by outlier screening. The residual transformation can be executed in two passes. According to some embodiments, outliers from the population of y measurement criteria are excluded before the calculation of the median and the residual, enabling robust statistics.

[0055] As shown in FIG. 5, when using robust regression, emphasis can be placed on modeling the good behavior of the device. Robust regression prevents outliers within the population from affecting the regression, thereby advantageously reducing overkill. For example, outliers expected before recalculating the test limit values can be excluded, and the current device under test can be compared with a new baseline. In the example shown in FIG. 5, reweighted least squares regression is performed, and then the total outliers to be excluded from the final regression calculation are identified. Thus, when the limit values are adjusted using robust regression, overkill of suspicious outliers is prevented.

[0056] According to some embodiments, multiple outlier screening techniques and / or multiple threshold setting strategies can be applied to bivariate statistics such that a set of results exists for each item. This approach can be referred to as an ensemble method. The outlier status can be determined based on the number of bivariate statistics that exceed the residual threshold. The threshold value can vary from 1 to (n)(n - 1). Applying multiple outlier techniques to each bivariate statistic allows the threshold to be adjusted beyond (n)(n - 1).

[0057] When using the ensemble method, the final determination of outliers can be limited to only those methods that are shown to be executed accurately within the ensemble (e.g., having a stronger signal and / or better prediction results). For example, a post - analysis can be performed to determine which specific outlier measurement criteria were predicting a result of interest (e.g., device failure during burn - in, device failure in the field, etc.). In this approach, it may be necessary to track the device and feed the results back into the test process for subsequent materials. In another approach, it is determined how far an outlier device is from the center of the statistical distribution of other devices, and more weight is applied to outlier analysis results that are very close to the upper or lower threshold values of the outliers.

[0058] In some cases, various problems generally known as the "curse of dimensionality" can occur by using the leading digits of relatively large measurement values n or by using statistics of dimensions higher than two variables. Therefore, according to some embodiments, the measurement values are grouped by similarity (e.g., current measurement values, voltage measurement values, frequency measurement values, etc.), and only multivariate combinations are generated within the group to avoid overly complex calculations. Alternatively, the measurement values can be clustered according to the functions of the device. For example, all measurement values related to the performance of a specific part of the circuit (e.g., the block implementing the analog-to-digital conversion) can be grouped into one cluster, and the performance measurement values of the power management unit can be grouped into a second cluster. Preliminary analysis can be performed before any test, statically at the start of the test, or periodically throughout the test process to generate specific combinations of parameters such as principal component analysis (PCA), autoencoding, or machine learning.

[0059] Any number of different test parameters can be used for testing and analysis, but some of the most important parameters according to the embodiments include the following. · Resistance · Capacitance · Inductance · Memory · Pressure · Timing / Frequency · Fuse / Trim / Non-Volatile Memory · I / O and DC · Transmit / Receive · Voltage · Analog · Current Monitoring (e.g., I DDQ Leakage)

[0060] Note that the embodiments of the present invention do not require visual inspection data and can be implemented using only the measurement data of conventional test programs.

[0061] According to some embodiments, each device to be tested can be classified using hard bins and / or soft bins, for example, to classify the material being tested, and as a result, different downstream test flows can be assigned to different bins.

[0062] According to some embodiments, in contrast to replacing conventional DPAT with only multivariate statistics, conventional DPAT statistical metrics that use univariate statistics can also be calculated together with multivariate statistics for use in screening decisions.

[0063] According to some embodiments, failed parts are classified (e.g., binned). All outlier calculations can be binned, for example, according to which tests are failed, the value of the residuals, etc.

[0064] FIG. 6 is a flowchart showing an exemplary automated computer-implemented multidimensional component average test process 600 for testing a device and analyzing test results to detect outliers according to an embodiment of the present invention. The test can include, for example, calculations of multivariate (e.g., bivariate) statistics using principal component analysis to identify eigenvectors and eigenvalues for defining delta measurements, ratios of measurements, or meta-parameters of similar devices.

[0065] In step 605, test limits are determined. The test limits define the acceptable range for the device being tested to identify outliers. The test limits can be determined, for example, based on past test data accessed from a memory, local or remote storage system. The past data can be based on, for example, the previous 50 or 100 units tested from the same lot or from the entire lot. According to some embodiments, the test limits are determined according to stored e-test data or data generated during previous manufacturing stages and can be increased, for example, with statistical data or phase correlation data.

[0066] In step 605, it may also include determining important measurement values for testing by performing well-known techniques such as, for example, principal component analysis, autoencoding, or machine learning.

[0067] In step 610, a plurality of devices are tested according to defined test limit values, and test results are obtained. According to some embodiments, raw test results are stored and aggregated to identify trends during or after testing.

[0068] In step 615, multivariate statistics are calculated using the test results. This step may include, for example, forming bivariate pairs, forming bivariate ratios, and forming bivariate deltas, but any suitable multivariate statistical calculation can be used.

[0069] In step 620, the mean or median of the multivariate statistics is calculated. Optionally, the mean or median of the multivariate statistics can be transformed into the residual space to amplify the behavior of outliers, so that non-conforming devices that have not been detected so far are recognized by outlier screening. The residual transformation can be performed in two passes. According to some embodiments, outliers from the population of y measurement criteria are excluded before calculating the median and residuals, enabling robust statistics. Robust regression is optionally performed to prevent outliers within the population from affecting the regression, thereby advantageously reducing overkill. Step 620 may include, for example, performing reweighted least squares regression.

[0070] In step 625, the test limit values are updated based on the mean or median and / or residuals. According to some embodiments, outliers are excluded before updating the test limit values.

[0071] In step 630, outliers of the plurality of devices are identified based on the mean value or median value, and / or the residuals and the updated test limit values. The outliers can be regarded as failed devices and / or binned according to the failed performance, test parameters, etc. Exemplary test system

[0072] Embodiments of the present invention relate to an electronic system for multi-dimensional component averaging testing for testing a device and analyzing test results to detect outliers, according to embodiments of the present invention. The testing can include, for example, the calculation of multivariate (e.g., bivariate) statistics using principal component analysis to identify eigenvectors and eigenvalues for defining delta measurements, ratios of measurements, or meta-parameters of similar devices. In the following discussion, one such exemplary electronic or computer system that can be used as a platform for implementing embodiments of the present invention will be described. For example, computer system 712 can be configured to interface with a test system or apparatus that includes a socket for receiving a device, a thermal management system, a power supply, and an electronic signal system for testing the device.

[0073] In the example of FIG. 7, an exemplary computer system 712 includes a Central Processing Unit (CPU) 701 for executing software applications and an operating system. A Random Access Memory 702 and a Read Only Memory 703 store applications and data for use by the CPU 701. A data storage device 704 provides non-volatile storage for applications and data and can include a fixed disk drive, a removable disk drive, a flash memory device, and a CD-ROM, DVD-ROM, or other optical storage device. The data storage device 704 or the memory 702 / 703 can store past and real-time test data (e.g., test results, limit values, calculations, etc.). Optional user inputs 706 and 707 include devices (e.g., a mouse, a joystick, a camera, a touch screen, a keyboard, and / or a microphone) that communicate input from one or more users to the computer system 712. A communication or network interface 708 enables the computer system 712 to communicate with other computer systems, networks, or devices via an electronic communication network including wired and / or wireless communication and including an intranet or the Internet.

[0074] An optional display device 710 can be any device that can display visual information, such as a final scan report, in response to a signal from the computer system 712 and can include, for example, a flat panel touch sensitive display. The components of the computer system 712 including the CPU 701, the memory 702 / 703, the data storage 704, the user input device 706, and the graphics subsystem 705 can be coupled via one or more data buses 700.

[0075] The embodiments of the present invention have been described above. Although the present invention has been described in specific embodiments, it should be understood that the present invention should not be construed as being limited by such embodiments and should be construed in accordance with the following claims.

Claims

1. A method for dynamic component average testing, the method comprising: Determining a test limit value based on past test result data; Testing a plurality of devices according to the test limit value and obtaining test results; Calculating multivariate statistics using the test results; Calculating an average value or a median value of the multivariate statistics; Calculating a difference between the multivariate statistics and the average value or the median value; Updating the test limit value based on the average value or the median value and generating an updated test limit value; and Identifying outliers of the plurality of devices based on the average value or the median value and the updated test limit value.

2. The method according to claim 1, further comprising determining important measurement values for testing by performing at least one of principal component analysis (PCA); independent component analysis (ICA); autoencoding; or machine learning.

3. The method according to claim 1, wherein the step of calculating the multivariate statistics using the test results includes at least one of forming bivariate pairs; forming ratios; and forming deltas.

4. The method according to claim 1, wherein the step of calculating the multivariate statistics using the test results includes clustering the test results according to result types.

5. The method according to claim 1, further comprising transforming the average value or the median value into a residual space using a non-linear, non-monotonic transformation to amplify the outlier results.

6. The method according to claim 1, further comprising excluding outlier results before updating the test limit value based on the average value or the median value.

7. The method according to claim 1, further comprising performing reweighted least squares regression before updating the test limit value based on the average value or the median value.

8. An apparatus for performing dynamic component average testing, the apparatus comprising: A processor; and A memory communicating with the processor for storing test data and instructions, wherein the processor executes instructions for performing a method of multivariate component average testing, the method comprising: Determining a test limit value based on past test result data; Testing a plurality of devices according to the test limit value and obtaining test results; A step of calculating multivariate statistics using the test results; A step of calculating the mean or median of the multivariate statistics; A step of calculating the difference between the multivariate statistics and the mean or median; A step of updating the test limit value based on the mean or median and generating an updated test limit value; and An apparatus comprising a step of identifying outliers of the plurality of devices based on the mean or median and the updated test limit value.

9. The apparatus according to claim 8, wherein the method further comprises a step of determining important measurement values for testing by performing at least one of principal component analysis (PCA); autoencoding; or machine learning.

10. The apparatus according to claim 8, wherein calculating the multivariate statistics using the test results includes at least one of forming bivariate pairs; forming bivariate ratios; and forming bivariate deltas.

11. The apparatus according to claim 8, wherein calculating the multivariate statistics using the test results includes clustering the test results according to result types.

12. The apparatus according to claim 8, wherein the method further comprises a step of transforming the mean or median into a residual space using a non-linear, non-monotonic transformation to amplify the outlier results.

13. The apparatus according to claim 8, wherein the method further comprises a step of excluding outlier results before updating the test limit value based on the mean or median.

14. The apparatus according to claim 8, wherein the method further comprises a step of performing reweighted least squares regression before updating the test limit value based on the mean or median.

15. To a processor, A procedure for determining a test limit value based on past test result data; A procedure for testing a plurality of devices according to the test limit value and obtaining test results; A procedure for calculating multivariate statistics using the test results; A procedure for calculating the mean or median of the multivariate statistics; A procedure for calculating the difference between the multivariate statistics and the mean or median; A procedure for updating the test limit value based on the mean or median and generating an updated test limit value; and A procedure for identifying outliers of the plurality of devices based on the mean or median and the updated test limit value A computer program for causing execution.

16. The processor is further caused to perform a procedure for determining important measurement values for testing by executing at least one of principal component analysis (PCA); independent component analysis (ICA); autoencoding; or machine learning. The computer program according to claim 15.

17. The computer program according to claim 15, wherein the calculating the multivariate statistics using the test results includes at least one of forming a pair of bivariate variables; forming a ratio; and forming a delta.

18. The computer program according to claim 15, wherein the calculating the multivariate statistics using the test results includes clustering the test results according to a result type.

19. The processor is further caused to perform a procedure for converting the mean value or the median into a residual space using a non-linear, non-monotonic transformation to amplify the result of the outlier. The computer program according to claim 15.

20. The computer program according to claim 15, wherein the determining the test limit value based on past test result data includes accessing at least one of e-test data; data generated during a previous manufacturing stage; statistical correlation data; or phase correlation data. ​ ​

Images