Verification method and module of test data and related equipment

Abstract

The invention discloses a test data verification method and module and related equipment, and the method comprises the steps: obtaining the test data of a tested device; obtaining a test verification model according to the verification attention information; and verifying the test data by adopting the test verification model to obtain a verification result. According to the technical scheme, the test data verification efficiency and accuracy can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of testing and verification, and in particular to a method, module and related equipment for verifying test data. Background Technology

[0002] With the development of electronic technology, there are more and more types of chips. In order to ensure the quality of chips, it is often necessary to conduct multiple functional tests on the chips to check the integrity of the chip functions, screen out defective products, ensure the quality of chip manufacturing, and reduce redundant manufacturing costs.

[0003] Generally speaking, verifying the test data of the same test item obtained by testing the device under test with different measurement systems can help understand the stability of the measurement system, thereby selecting a measurement system that meets the requirements to meet the needs of chip mass production.

[0004] However, existing methods for verifying test data suffer from low efficiency and accuracy. Summary of the Invention

[0005] The problem solved by the embodiments of the present invention is to provide a method, module and related equipment for verifying test data, which can improve the efficiency and accuracy of test data verification.

[0006] To address the above problems, embodiments of the present invention provide a method for verifying test data, comprising:

[0007] Acquire test data for the device under test;

[0008] Based on the verification concerns, obtain the test verification model;

[0009] The test data is verified using the aforementioned test verification model to obtain verification results.

[0010] Optionally, if the verification concern information includes at least one of the following: the repeatability of the test procedure, the difference of the test circuit board or test probe card at different test sites, and the reasonableness of setting the test limit values ​​of the test items, then the test verification model includes the GRR model.

[0011] Optionally, the step of using the test verification model to verify the test data and obtain verification results includes:

[0012] The test data of the test item to be verified is processed using the GRR model to obtain the equipment variation and evaluator variation of the test item.

[0013] Based on the equipment variation and evaluator variation of the test item, obtain the first repeatability and reproducibility values ​​of the test item;

[0014] Based on the validity information of the acceptable upper and lower limits of the test item, obtain the first total variation type of the test item;

[0015] Based on the first total variation type of the test item, obtain the first total variation value of the test item;

[0016] The first repeatability and reproducibility index of the test item is obtained based on the first repeatability and reproducibility value of the test item and the first total variation value of the test item.

[0017] Optionally, when the user inputs the first total variation type of the test item, the test data verification method further includes: obtaining the first total variation value of the test item based on the user input of the first total variation type of the test item.

[0018] Optionally, obtaining the first total variation type of the test item based on the validity information of the acceptable upper and lower limits of the test item includes: if both the acceptable upper and lower limits of the test item are invalid, then the first total variation type of the test item is type one; if only the acceptable lower limit of the test item is valid, then the first total variation type of the test item is type two; if only the acceptable upper limit of the test item is valid, then the first total variation type of the test item is type three; if both the acceptable upper and lower limits of the test item are valid, then the first total variation type of the test item is type four.

[0019] The step of obtaining the first total variation value of the test item based on the first total variation type of the test item includes: if the first total variation type of the test item is type 1, then using the formula... Calculate the first total variation of the test item, where TV1 represents the first total variation value of the test item, GRR1 represents the first repeatability and reproducibility value of the test item, and PV represents the sample variation; if the first total variation type of the test item is type II, then the formula is used. Calculate the first total variation value of the test item, where Mean1 represents the average value of the test data for the test item, and LSL1 represents the acceptable lower limit value of the test item; if the first total variation type of the test item is type III, then the formula is used. Calculate the first total variation value of the test item, where USL1 represents the acceptable upper limit of the test item; if the first total variation type of the test item is type IV, then the formula is used. Calculate the first total variation value of the test item, where Min(.) represents the minimum value operation.

[0020] Optionally, if the verification concern information includes differences between measurement systems, then the test verification model includes a T-verification model.

[0021] Optionally, if the measurement systems to be compared use at least some of the same devices under test to obtain test data for the same test item, then the T-verification model includes a paired sample T-verification model.

[0022] The step of verifying the test data using the acquired test verification model and obtaining the verification results includes: using the paired sample T-verification model to perform a second verification process on the test data of the same test item obtained by testing the same device under test on the measurement systems to be compared, and obtaining the T-bias value, P-value and effect size of the test item.

[0023] Optionally, the step of using the acquired test verification model to verify the test data and obtain verification results further includes:

[0024] The range method GRR model is used to verify the test data of the same test item obtained by testing the same device under test on the measurement system to be compared, and to obtain the second repeatability and reproducibility index of the test item.

[0025] Optionally, the verification of the same test item obtained by using the range method GRR model to test the measurement systems under comparison with the same device under test, and obtaining the second repeatability and reproducibility index of the test item, includes:

[0026] The range method GRR model is used to perform a second verification process on the test data of the same test item obtained by testing the same test device on the measurement system to be compared, and to obtain the second repeatability and reproducibility values ​​of the test item.

[0027] Based on the validity information of the acceptable upper and lower limits of the test item, obtain the second total variation type of the test item;

[0028] Based on the second total variation type of the test item, obtain the second total variation value of the test item;

[0029] The second repeatability and reproducibility index of the test item is obtained based on the second repeatability and reproducibility value of the test item and the second total variation value of the test item.

[0030] Optionally, based on the validity information of the acceptable upper and lower limits of the test item, the total variation of the test data for the test item is obtained, including: if both the acceptable upper and lower limits of the test item are invalid, then the second total variation type of the test item is the first type; if only the acceptable lower limit of the acceptable upper and lower limits of the test item is valid, then the second total variation type of the test item is the second type; if only the acceptable lower limit of the acceptable upper and lower limits of the test item is valid, then the second total variation type of the test item is the third type; if both the acceptable upper and lower limits of the test item are valid, then the second total variation type of the test item is the fourth type.

[0031] The step of obtaining the second total variation value of the test item based on the second total variation type of the test item includes: if the second total variation type of the test item is a first type, then the second total variation value of the test item is calculated using the formula TV2 = S, where TV2 represents the second total variation value of the test item and S represents the sample standard deviation; if the second total variation type of the test item is a second type, then the formula... Calculate the second total variation value of the test item, where Mean2 represents the mean of the test data for the test item, and LSL2 represents the acceptable lower limit of the test item; if the second total variation type of the test item is type 3, then the formula is used. Calculate the second total variation value of the test item, where USL2 represents the acceptable upper limit of the test item; if the second total variation type of the test item is type four, then the formula is used. Calculate the second total variation value of the test item, where Min(.) represents the minimum value operation.

[0032] Optionally, if the measurement systems to be compared are tested using different devices under test to obtain the test data, then the T-calibration model includes an independent T-calibration model.

[0033] The step of verifying the test data using the acquired test verification model and obtaining verification results includes: using the independent T-test model to perform a third verification process on the test data of the same test item obtained by testing the measurement system to be compared with different devices under test, and obtaining the P-value and effect size of the test item.

[0034] Optionally, after verifying the test data using the test verification model and obtaining the verification result, the test data verification method further includes: outputting the verification result.

[0035] Optionally, after verifying the test data using the test verification model and obtaining the verification result, the test data verification method further includes: outputting a graphical display of the test data.

[0036] Accordingly, embodiments of the present invention also provide a method for verifying test data, including:

[0037] The data acquisition unit acquires the test data of the device under test;

[0038] The model acquisition unit is suitable for acquiring test and verification models based on verification interest information.

[0039] The test verification unit is adapted to use the acquired test verification model to verify the test data and obtain verification results.

[0040] Accordingly, embodiments of the present invention also provide a computer program product, including a computer program / instructions, which, when executed by a processor, are used to implement the test data verification method as described in any of the preceding claims.

[0041] Accordingly, embodiments of the present invention also provide an apparatus including at least one memory and at least one processor, wherein the memory stores one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the test data verification method as described in any of the preceding claims.

[0042] Accordingly, embodiments of the present invention also provide a storage medium storing one or more computer instructions, the one or more computer instructions being used to implement the test data verification method as described in any of the preceding claims.

[0043] Compared with the prior art, the technical solution of the embodiments of the present invention has the following advantages:

[0044] The test data verification method provided in this embodiment of the invention includes: acquiring test data of the device under test; acquiring a test verification model based on verification interest information; verifying the test data using the test verification model; and obtaining verification results.

[0045] In the test data verification method provided in this embodiment of the invention, the test data of the device under test is verified using a corresponding test verification model based on the verification information of concern. This enables automated verification of test data, reduces manual intervention, and consequently improves the verification efficiency and accuracy of test data. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an embodiment of the test data verification method provided by the technical solution of the present invention;

[0047] Figure 2 It is a dot plot of the test data for a test item;

[0048] Figure 3 It is a histogram and kernel density estimation plot of the test data for a test item;

[0049] Figure 4 It is a range control chart of test data;

[0050] Figure 5 It is a mean control chart of test data;

[0051] Figure 6 It is a difference distribution diagram of the test data of the same test item obtained by the two measurement systems to be compared;

[0052] Figure 7 It is a distribution diagram of the test data of the same test item obtained by the two measurement systems to be compared;

[0053] Figure 8 It is a box plot of the test data of the same test item obtained by the two measurement systems to be compared at each test point;

[0054] Figure 9 This is a schematic diagram of the linear fitting line Y = aX + b for the test data of the same test item obtained by the two measurement systems to be compared;

[0055] Figure 10 This is a schematic diagram of the structure of an embodiment of the test data verification module provided by the technical solution of the present invention;

[0056] Figure 11 This is a schematic diagram of an optional hardware structure of the device provided by the technical solution of the present invention. Detailed Implementation

[0057] As can be seen from the background technology, the efficiency and accuracy of current test data verification methods still need to be improved.

[0058] To address the aforementioned technical problem, the present invention provides a test data verification method, comprising: acquiring test data of the device under test; acquiring a test verification model based on verification interest information; verifying the test data using the test verification model; and obtaining verification results.

[0059] In the test data verification method provided in this embodiment of the invention, the test data of the device under test is verified using a corresponding test verification model based on the verification information of concern. This enables automated verification of test data, reduces manual intervention, and consequently improves the verification efficiency and accuracy of test data.

[0060] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0061] Figure 1 This diagram illustrates a flowchart of an embodiment of the test data verification method provided by the technical solution of the present invention. (See reference...) Figure 1 A method for verifying test data may specifically include the following steps:

[0062] Step S110: Obtain test data of the device under test;

[0063] Step S120: Obtain the test verification model based on the verification concern information;

[0064] Step S130: Use the test verification model to verify the test data and obtain the verification results.

[0065] Please continue to refer to the reference. Figure 1 Execute step S110 to obtain the test data of the device under test.

[0066] The test data of the device under test is obtained, which provides a basis for subsequent verification of the test data using the test verification model and obtaining the verification results.

[0067] In this embodiment, a measurement system is used to test multiple test items in the device under test (DUT) to obtain test data of the DUT.

[0068] Specifically, the measurement system includes a tooling circuit board and an automatic test equipment (ATE). When testing the device under test (DUT), the DUT is placed one-to-one into one or more test sockets on the tooling circuit board, and connected to the ATE via multiple data interfaces. The tooling circuit board reads the test commands output by the ATE and outputs the test results in parallel from the multiple data interfaces. The tooling circuit board can have one or more test sockets. Where there is only one test socket, one tooling circuit board can test one DUT at a time; where there are multiple test sockets, one tooling circuit board can test multiple DUTs simultaneously. In other words, the number of simultaneous tests is the same as the number of test sockets on a tooling circuit board.

[0069] Please continue to refer to the reference. Figure 1 Execute step S120 to obtain the test verification model based on the verification concern information.

[0070] Based on the verification information, a test verification model is obtained, which provides a basis for subsequently using the test verification model to verify the test data and obtain verification results.

[0071] In this embodiment, the verification concern information, or application scenario, has a corresponding relationship with the test verification model. Therefore, the corresponding test verification model can be obtained through the verification concern information.

[0072] In this embodiment, if the verification of information of concern includes the repeatability of the test procedure, the differences of the test circuit board or test probe card at different test sites, and the reasonableness of setting the test limits for the test items, then the test verification model includes a Gauge Repeatability and Reproducibility (GRR) model. The reasonableness of setting the test limits includes whether the test limits are set too loosely, too tightly, or too biased.

[0073] The GRR model is a model in Measurement Systems Analysis (MSA) used to characterize the repeatability and reproducibility of measurement systems.

[0074] The repeatability of a measurement system is often referred to as equipment variation (EV). In automated testing, repeatability refers to the measurement variation obtained by performing multiple loop tests on a single parameter of the same test item for the same chip under test at a single test site on a test board using the same measuring instrument. Correspondingly, the stability of the test program, i.e., the stability of multiple loop tests, can be characterized using equipment variation.

[0075] The reproducibility of a measurement system is often referred to as appraiser variation (AV). In automated testing, the operators of the test equipment and handler mechanisms typically only act as clickers during the testing process, and their impact on the measurement system is negligible. Accordingly, in automated testing, reproducibility refers to the variation in the average value obtained by measuring the same parameter of the same chip under test using the same measuring instrument at different test sites on the test circuit board. Consequently, the site unbalance between test sites on the test circuit board can be characterized using appraiser variation.

[0076] In this embodiment, the GRR model includes the mean-range GRR model, the ANOVA analysis-of-variance GRR model, and the clustered standard deviation GRR model. In other embodiments, the GRR model may also include more or fewer types of GRR models, and those skilled in the art can select them according to actual needs, without limitation.

[0077] The verification focus information corresponding to the GRR model includes both a single test board and a single test program itself. Accordingly, the corresponding test data includes test data from the loop tests of the test program itself and test data from the rotation tests of the same chip under test at different test sites.

[0078] When acquiring test data for the same chip under test (DUT) at different test sites, the number of DUTs is usually the same as the number of test sites on the test circuit board. For example, for a test circuit board with 8 test sites, 8 DUTs are usually selected and rotated across the 8 test sites on the test circuit board. Specifically, in the first test, DUTs 1 through 8 are placed one-to-one on test sites 1 through 8; in the second test, DUTs 1 through 8 are placed one-to-one on test sites 1 through 8 in the order of DUT 8, DUTs 1 through 7; in the third test, DUTs 1 through 8 are placed one-to-one on test sites 1 through 8 in the order of DUT 7, DUT 8, DUTs 1 through 6; and so on; this process is repeated 8 times to achieve testing of each of the DUTs 1 through 8 at each test site on the test circuit board.

[0079] Accordingly, by repeating the above 8 tests multiple times, such as 10 to 50 times, the test program can be looped, thereby obtaining the test data of the loop test of the test program itself.

[0080] In this embodiment, if the verification information of interest includes differences between measurement systems, then the test verification model includes the T-calibration model.

[0081] The T-validation model is another test and validation model defined in Measurement Systems Analysis (MSA) besides the GRR model. It treats the measurement system as a whole and is used to characterize the differences between different measurement systems.

[0082] In this embodiment, the T-verification model includes a paired-sample T-verification model and an independent T-verification model. Specifically, when the measurement systems to be compared use at least partially identical devices under test to obtain test data for the same test item, the T-verification model is a paired-sample T-verification model; when the measurement systems to be compared use different devices under test to obtain test data for the same test item, the T-verification model is an independent T-verification model.

[0083] Please continue to refer to the reference. Figure 1 Step S130 is executed, in which the test data is verified using the test verification model, and the verification result is obtained.

[0084] The test data is validated using the aforementioned test verification model to obtain verification results, thereby enabling the analysis based on the verification results.

[0085] In this embodiment, the verification information of concern includes the repeatability of the test procedure, the differences of the test circuit board or test probe card at different test sites, and the rationality of setting the test limit values ​​of the test items. The corresponding test verification model includes the GRR model.

[0086] Accordingly, the step of verifying the test data using the test verification model and obtaining verification results includes: performing a first verification process on the test data of the test item to be verified using the GRR model to obtain the equipment variation and evaluator variation of the test item; obtaining the first repeatability and reproducibility values ​​of the test item based on the equipment variation and evaluator variation of the test item; obtaining the first total variation type of the test item based on the validity information of the acceptable upper limit and acceptable lower limit of the test item; obtaining the first total variation value of the test item based on the first total variation type of the test item; and obtaining the first repeatability and reproducibility index of the test item based on the first repeatability and reproducibility values ​​and the first total variation value of the test item.

[0087] In this embodiment, the repeatability of the measurement system is characterized by equipment variation (EV), and the reproducibility of the measurement system is characterized by evaluator variation (AV). Accordingly, based on the equipment variation and evaluator variation of the test item, the first repeatability and reproducibility values ​​of the test item are obtained using the following formula:

[0088]

[0089] Wherein, GRR1 represents the first repeatability and reproducibility value of the corresponding test item.

[0090] In this embodiment, the step of obtaining the first total variation type of the test item based on the validity information of the acceptable upper limit and acceptable lower limit of the test item includes the following scenarios:

[0091] (I) If both the acceptable upper limit and the acceptable lower limit of the test item are invalid, then the first total variation type of the test item is type I;

[0092] (II) If only the lower acceptable limit is valid among the upper and lower acceptable limits of the test item, then the first total variation type of the test item is type II;

[0093] (III) If only the upper acceptable value is valid among the upper acceptable value and the lower acceptable value of the test item, then the first total variation type of the test item is type III;

[0094] (III) If both the acceptable upper limit and the acceptable lower limit of the test item are valid, then the test item is of type four for the first total variation type.

[0095] Accordingly, the step of obtaining the first total variation value of the test item according to the first total variation type of the test item includes: calculating the first total variation value of the test item using the corresponding total variation calculation formula according to the first total variation type of the test item.

[0096] Specifically, when the first total variation type of the test item is type I, process difference applies, and the first total variation value of the test item is calculated using the following formula:

[0097]

[0098] Wherein, TV1 represents the first total variation value of the test item, GRR1 represents the first repeatability and reproducibility value of the test item, and PV represents the sample variation.

[0099] It should be noted that the above formula (2) is applicable when the samples are not completely randomly selected, but are as diverse as possible to cover the differences in the whole process.

[0100] If the first total variation type of the test item is type II, then the first total variation value of the test item is calculated using the following formula:

[0101]

[0102] Wherein, Mean1 represents the average value of the test data for the test item, and LSL1 represents the acceptable lower limit value for the test item.

[0103] If the first total variation type of the test item is type III, then the first total variation value of the test item is calculated using the following formula:

[0104]

[0105] USL1 represents the acceptable upper limit of the test item.

[0106] If the first total variation type of the test item is type four, then the first total variation value of the test item is calculated using the following formula:

[0107]

[0108] Min(.) represents the minimum value operation.

[0109] For test items where both the upper and lower acceptable limits are valid, the first total variation value of the test item described in the above formula (5) can be used to clearly reflect the situation where the test data of the test item is close to the upper or lower acceptable limit in the first repeatability and reproducibility index of the test item generated subsequently, without having to combine it with other indicators, such as the process deviation rate (CA) index in the Complex Process Capability index (CPK).

[0110] In this embodiment, upon receiving the total variation type of the test item input by the user, the first total variation value of the test item is obtained according to the first total variation type of the test item input by the user.

[0111] In other words, the first total variation type of the test item input by the user has a higher priority than the first total variation type of the test item automatically generated based on the validity information of the acceptable upper and lower limits of the test item.

[0112] The total variation type of the test item input by the user has a higher priority than the first total variation type of the test item automatically generated based on the validity information of the acceptable upper and lower limits of the test item. This can improve the flexibility of obtaining the first total variation type of the test item and meet the diverse needs of users for setting the first total variation type of the test item.

[0113] In this embodiment, the first total variation type of the test item input by the user includes the first to fourth types mentioned above, and also includes the fifth type.

[0114] Specifically, when the total variation type of the test item is received as type 5 from user input, the first total variation value of the test item is calculated using the following formula:

[0115] TV1 = sigma (6)

[0116] Wherein, sigma represents the variance of the chip throughout the entire mass production process or the variance of the test data of the test item of the device under test obtained from the mass production data.

[0117] In this embodiment, the first repeatability and reproducibility index of the test item is calculated using the following formula based on the first repeatability and reproducibility value and the first total variation value of the test item:

[0118]

[0119] Here, GRR1% represents the first repeatability and reproducibility index of the test item.

[0120] In this embodiment, if the verification concern information includes differences between measurement systems, then the test verification model includes a T-verification model.

[0121] In this embodiment, when the measurement systems to be compared use at least some of the same devices under test to obtain test data for the same test item, the T-verification model includes a paired sample T-verification model.

[0122] Accordingly, the step of verifying the test data using the acquired test verification model and obtaining the verification results includes: using the paired sample T-verification model to perform a second verification process on the test data of the same test item obtained by testing the same device under test on the measurement system to be compared, and obtaining the T-bias value, P value and effect size (Cohen's D) of the test item.

[0123] The inventors of this application discovered in practical applications that if the traditional 95% confidence interval, i.e., a P-value >= 0.05, is used as the standard for no significant bias between two measurement systems being compared, the number of test items that need to be screened out will be relatively large. However, after analysis, it will be found that the differences of most test items are within the normal range. The reason for this is that the larger the T-bias value, the smaller the P-value. Since the T-bias value is proportional to the square root of the sample size, the P-value will become very small when the sample size is large (e.g., sample size greater than or equal to 1000).

[0124] In the formula for calculating the effect size (Cohen's D), the numerator is the difference between the average test data of the same test item obtained by testing the chip under test using two measurement systems to be compared, and the denominator is the standard deviation of the differences between the test data of the same test item obtained by testing each chip under test using two measurement systems to be compared. Therefore, the effect size does not change significantly with the sample size.

[0125] Therefore, by obtaining the T-bias and P-value of the test item, adding the effect size (Cohen's D) as a judgment indicator can eliminate the interference caused by changes in the sample size, which is beneficial to improving the verification accuracy of the test data.

[0126] In this embodiment, when the verification concern information includes differences between measurement systems, and the measurement systems to be compared use at least partially identical devices under test to obtain test data for the same test item, the test verification model further includes the range method GRR model.

[0127] Accordingly, the step of verifying the test data using the acquired test verification model and obtaining the verification results further includes: verifying the test data of the same test item obtained by testing the same device under test with the range method GRR model for the measurement system to be compared, and obtaining the second repeatability and reproducibility index of the test item.

[0128] Specifically, the step of verifying the test data of the same test item obtained by testing the same measurement system under test using the range method GRR model with the same device under test, and obtaining the second repeatability and reproducibility index of the test item, includes: performing a second verification process on the test data of the same test item obtained by testing the same measurement system under test using the range method GRR model with the same device under test, and obtaining the second repeatability and reproducibility value of the test item; obtaining the second total variation type of the test item based on the validity information of the acceptable upper limit and acceptable lower limit of the test item; obtaining the second total variation value of the test item based on the second total variation type of the test item; and obtaining the second repeatability and reproducibility index of the test item based on the second repeatability and reproducibility value and the second total variation value of the test item.

[0129] In this embodiment, the step of obtaining the total variation of test data for a test item based on the validity information of the acceptable upper limit and acceptable lower limit of the test item includes: if both the acceptable upper limit and acceptable lower limit of the test item are invalid, then the second total variation type of the test item is the first type; if only the acceptable lower limit of the acceptable upper limit and acceptable lower limit of the test item are valid, then the second total variation type of the test item is the second type; if only the acceptable lower limit of the acceptable upper limit and acceptable lower limit of the test item are valid, then the second total variation type of the test item is the third type; if both the acceptable upper limit and acceptable lower limit of the test item are valid, then the second total variation type of the test item is the fourth type.

[0130] Accordingly, the step of obtaining the second total variation value of the test item based on the second total variation type of the test item includes: calculating the second total variation value of the test item using the corresponding total variation calculation formula based on the second total variation type of the test item. Specifically:

[0131] If the second total variation type of the test item is the first type, then the second total variation value of the test item is calculated using the following formula:

[0132] TV2 = S (8)

[0133] Where TV2 represents the second total variation value of the test item, and S represents the sample standard deviation.

[0134] If the second total variation type of the test item is type 2, then the second total variation value of the test item is calculated using the following formula:

[0135]

[0136] Where Mean2 represents the mean of the test data for the test item, and LSL2 represents the acceptable lower limit of the test item.

[0137] If the second total variation type of the test item is the third type, then the second total variation value of the test item is calculated using the following formula:

[0138]

[0139] USL2 represents the acceptable upper limit of the test item.

[0140] If the second total variation type of the test item is the third type, then the second total variation value of the test item is calculated using the following formula:

[0141]

[0142] Min(.) represents the minimum value operation.

[0143] As can be seen from the above description, except that the first type in the second total variation type of the test item is different from the first type in the second total variation type of the test item, the second type, third type and fourth type in the second total variation type of the test item are all in one-to-one correspondence with the second type, third type and fourth type in the first total variation type of the test item described in the foregoing section and the calculation rules are the same.

[0144] For the first type of the second total variation type of the test item, the existing approach is to apply the process performance (PP) or process performance index (PPK) target value, and the corresponding calculation formula is as follows:

[0145]

[0146] In ATE mass production testing, almost all test items cannot specify a process performance target value PP, nor can the test limit value of the test item be set according to the process performance target value PP. Therefore, in this embodiment, the process performance target value PP of the sample is adopted, thereby obtaining the above calculation formula (8).

[0147] For the second, third, and fourth types of the second total variation type of the test item, please refer to the description of the second, third, and fourth types of the first total variation type of the test item in the foregoing section, which will not be repeated here.

[0148] In this embodiment, if the measurement systems to be compared use different chips under test to obtain test data for the same test item, then the T-verification model includes an independent T-verification model.

[0149] Accordingly, the step of verifying the test data using the acquired test verification model and obtaining the verification results includes: using the independent T-verification model to perform a third verification process on the test data of the same test item obtained by testing the measurement system to be compared with different devices under test, and obtaining the P-value and effect size of the test item.

[0150] It should be noted that when the measurement systems to be compared are tested using different chips under test to obtain test data for the same test item, the independent t-test model is used for third verification processing of the test data for the same test item obtained by testing the measurement systems to be compared using different devices under test. This only obtains the p-value and effect size of the test item, and it is impossible to calculate repeatability and reproducibility indices. Compared to the paired-samples t-test model, the verification results obtained using the independent t-test model are relatively coarse, but have a wider range of applications.

[0151] In this embodiment, after verifying the test data using the test verification model and obtaining the verification result, the test data verification method further includes: outputting the verification result.

[0152] In this embodiment, when the test verification model includes the GRR model, the corresponding verification results include: the first repeatability and reproducibility index GRR1% for each test item and the equipment variation EV and evaluator variation AV associated with the first repeatability and reproducibility index GRR1%.

[0153] Among them, the first repeatability and reproducibility metric (GRR1%) for each test item is an important verification indicator that helps users locate test items with problems. As an example, if the first repeatability and reproducibility metric (GRR1%) for a test item exceeds 30%, it indicates that the verification result of that test item is abnormal. In this case, the test item, as a test item that requires special attention from the user, will be placed separately, such as in the corresponding first warning sheet (Warning Sheet1), so that the user can focus on it.

[0154] The equipment variation (EV) and evaluator variation (AV) associated with the first repeatability and reproducibility metric, GRR1%, can help users pinpoint the cause of discrepancies in test items that are causing problems, such as imbalances between test procedures or test sites.

[0155] In this embodiment, when the test validation model includes a paired-samples T-validation model and a range GRR model, the corresponding validation results include: the T-bias value, P-value, confidence interval, effect size, and second repeatability and reproducibility value GRR2% for each test item.

[0156] Test items that exceed the limits for P-value, effect size, second repeatability, and reproducibility GRR2% will be placed separately, such as in the corresponding second warning sheet (Warning Sheet2), for users to observe in detail.

[0157] In this embodiment, when the test verification model includes a paired sample T-verification model and a range method GRR model, the corresponding verification result data includes: the average and standard deviation of the test data of the same test item obtained by the two measurement systems to be compared on the same chip under test, and the average and standard deviation of the differences between the test data of the same test item obtained by the two measurement systems to be compared on the same chip under test.

[0158] In this embodiment, when the test verification model includes a paired sample T-validation model and a range method GRR model, the corresponding verification result data also includes: for each measurement system to be compared, outputting the values ​​of parameters a and b in the linear fitting line Y = aX + b of the test data for each test item, as well as the null hypothesis H0 and the goodness-of-fit parameter R2 when parameters a and b are each equal to 0.

[0159] In addition, when the measurement system to be compared is a measurement system that includes chip probing, the corresponding verification result data also includes: a bin shift map of the test data for the test items and a kappa coefficient of consistency.

[0160] The bin shift map is used to mark the die under test that changes during the final binning process for the two measurement systems being compared. When both measurement systems being compared include chip probing and the wafer probing is performed on the entire wafer, the samples are considered to conform to the random sampling principle. In this case, the corresponding consistency coefficient kappa is output, allowing the user to quantify the bin shift of the test data for the same test item obtained from the two measurement systems being compared.

[0161] In this embodiment, when the test validation model includes an independent T-test model, the corresponding validation result data includes: the p-value, confidence interval, and effect size for each test item.

[0162] Test items that exceed both the P-value and effect size will be placed separately, such as in the corresponding third warning sheet (Warning Sheet3), so that users can focus on observing them.

[0163] In this embodiment, after verifying the test data using the test verification model and obtaining the verification results, the test data verification method further includes: outputting a graphical display of the test data of the device under test.

[0164] Specifically, when the test verification model includes the GRR model, the graphical display of the test data of the device under test includes a dot plot, histogram, and kernel density estimation plot (KDE) of the test data for each test item of the device under test.

[0165] Reference Figure 2 This shows a dot plot of the test data for the test items. For example... Figure 2As shown, the horizontal axis represents different test sites, and the same color represents the same chip under test. Taking a test circuit board with 10 test sites as an example, and using 10 chips under test for 20 test cycles, the number of test data points for this test item is 10 * 10 * 20 = 2000 data points. Among them, the signal-to-noise ratio (SNR) of test sites 4 and 7 is lower than that of other test sites, and the SNR difference of the test data from the cyclic test of the same test site is approximately 1.5 dB to 2 dB.

[0166] Using a dot plot of the test data for each test item can visually demonstrate whether the differences in the test data are due to differences between test sites or differences in the test procedure, helping users to intuitively pinpoint the cause of the differences in the test data.

[0167] Reference Figure 3 This includes histograms and KDE plots of the test data for each test item, allowing users to observe the distribution of the test data. For example... Figure 3 As shown, the test data are basically normally distributed and exhibit a certain degree of bimodal phenomenon.

[0168] also, Figure 3 The system also uses charts to list the acceptable upper limit, acceptable lower limit, standard deviation (Stdev), mean (Mean), maximum (Max), minimum (Min), 75% of the test data (Q3), median (Median), 25% of the test data (Q1), process precision (CP), process deviation (Ca), and process capability index (Cpk) of the test items, so that users can judge the rationality of setting the test limit values ​​of the test items based on these parameters.

[0169] For example, a process capability index (Cpk) that is too low indicates that the test limits for the test items are too tight, which may result in low yield; a process capability index (Cpk) that is too high may mean that the test limits for the test items are too lenient. The test data indicators for each test item can be filtered and sorted for display according to actual needs.

[0170] Reference Figure 4 This includes a range control graph of the test data for each test item, allowing users to visually quantify the differences in cyclic testing of the test procedure. For example... Figure 4 As shown, Sample 0 has the largest difference in the cycle test at test site SITE11, specifically, the difference in the cycle test of Sample 0 at test site SITE11 exceeds 2dB.

[0171] Reference Figure 5 The chart shows the mean control (X-Control) plot of the test data for each test item. Figure 5 As shown, test sites SITE12 (marked in blue) and SITE6 (marked in purple) have the best performance, while test site SITE4 (marked in green) has the worst performance, with a difference of approximately 2 dB. This helps users intuitively quantify the differences in test data between test sites.

[0172] In this embodiment, when the test verification model includes a paired sample T-verification model and a range method GRR model, the graphical display of the test data of the device under test includes: a distribution map, a difference distribution map, a linear fitting line Y = aX + b, and a box plot of each test point of the test data of the same test item obtained by the two measurement systems to be compared.

[0173] Reference Figure 6 This diagram illustrates the difference distribution of test data for the same test item acquired by two measurement systems being compared. The horizontal axis represents the difference (diffRESULT) between the test data of the same test item acquired by the two measurement systems being compared, and the vertical axis represents the number of test data points (count).

[0174] Outputting a difference distribution chart of the test data of the same test item obtained by the two measurement systems to be compared helps users intuitively understand the differences between the test data of the same test item obtained by the two measurement systems to be compared, and whether the differences conform to a normal distribution.

[0175] Reference Figure 7 The graph shows the distribution of test data for the same test item acquired by two measurement systems being compared, marked in yellow and blue respectively. The horizontal axis represents the test data (RESULT) for the same test item acquired by the two measurement systems being compared, and the vertical axis represents the number of test data (count).

[0176] Outputting a distribution chart of the test data for the same test item obtained by the two measurement systems to be compared helps users intuitively understand whether the test data for the same test item obtained by the two measurement systems to be compared conforms to a normal distribution.

[0177] Reference Figure 8 This diagram shows box plots of test data for the same test item acquired by two measurement systems being compared, at each test site. The horizontal axis represents the test site (SITE_NUM), and the vertical axis represents the test data (RESULT) for the same test item acquired by the two measurement systems being compared. Figure 8The outputs show that the test data of the same test item obtained by the two measurement systems to be compared have no significant differences between the various test sites.

[0178] The system outputs box plots of the test data for the same test item obtained by the two measurement systems to be compared, corresponding to each test point. This allows users to intuitively understand the differences or imbalances between the test points on the test circuit board.

[0179] Reference Figure 9 The diagram illustrates the linear fitting line Y = aX + b for the test data of the same test item acquired by two measurement systems to be compared. The horizontal axis represents the test data of the chip under test for a single test item acquired by one of the two measurement systems, while the vertical axis represents the difference between the test data of the same test item for the same chip under test acquired by the two measurement systems.

[0180] The linear fitting line Y = aX + b of the test data of the same test item acquired by the two measurement systems being compared allows users to intuitively understand whether there is a stable linear offset between the two measurement systems. The closer the goodness-of-fit R² of the linear fitting line Y = aX + b is to 1, the better the linearity of the test data acquired by the measurement system.

[0181] In this embodiment, when the test verification model includes an independent T-calibration model, the graphical display of the test data of the device under test includes: a distribution map of the test data of the same test item obtained by the two measurement systems to be compared and a box plot of each test point.

[0182] For the distribution plots of the test data of the same test item obtained by the two measurement systems being compared and the box plots of each test point, please refer to the aforementioned graphical display of the test data of the device under test when the test verification model includes an independent T-calibration model, and will not be repeated here.

[0183] The graphical representation of the test data above is merely an example. It is understood that, depending on actual needs, test data can be displayed using more or fewer graphical methods, and this is not a limitation.

[0184] Accordingly, embodiments of the present invention also provide a test data verification module.

[0185] Figure 10 A schematic diagram of an embodiment of the test data verification module provided by the technical solution of the present invention is shown. See also Figure 10A test data verification module 100 includes: a data acquisition unit 1001 for acquiring test data of the device under test; a model acquisition unit 1002 adapted to acquire a test verification model based on verification interest information; and a test verification unit 1003 adapted to verify the test data using the acquired test verification model and obtain verification results.

[0186] The test data verification module in this embodiment of the invention can be used to execute the aforementioned test data verification method, or other functional modules can be used to execute the aforementioned test data verification method. For a detailed description of the test data verification method, please refer to the foregoing section; it will not be repeated here.

[0187] Accordingly, embodiments of the present invention also provide a device that can implement the test data verification method provided in the embodiments of the present invention by loading the above-described test data verification method in the form of a program.

[0188] refer to Figure 11 The diagram illustrates an optional hardware structure of a device provided in an embodiment of the present invention. The device in this embodiment includes: at least one processor 01, at least one communication interface 02, at least one memory 03, and at least one communication bus 04.

[0189] In this embodiment, the number of processor 01, communication interface 02, memory 03 and communication bus 04 is at least one, and processor 01, communication interface 02 and memory 03 communicate with each other through communication bus 04.

[0190] Communication interface 02 can be an interface for a communication module used for network communication, such as the interface for a GSM module.

[0191] Processor 01 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the verification method for the test data of this embodiment.

[0192] Memory 03 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device. Memory 03 stores one or more computer instructions, which are executed by processor 01 to implement the test data verification method provided in the foregoing embodiments.

[0193] It should be noted that the above-described implementing electronic device may also include other devices (not shown) that may not be essential to understanding the content disclosed in the embodiments of the present invention; given that these other devices may not be essential to understanding the content disclosed in the embodiments of the present invention, the embodiments of the present invention will not describe them one by one.

[0194] Accordingly, embodiments of the present invention also provide a computer program product, including a computer program / instruction, which, when executed by a processor, is used to implement the test data verification method described in the embodiments of the present invention.

[0195] This invention also provides a storage medium storing one or more computer instructions for implementing the test data verification method provided in the foregoing embodiments.

[0196] The embodiments of the present invention described above are combinations of elements and features of the present invention. Unless otherwise stated, elements or features may be considered optional. Individual elements or features may be practiced without combination with other elements or features. Furthermore, embodiments of the present invention may be constructed by combining some elements and / or features. The order of operations described in the embodiments of the present invention may be rearranged. Some constructions of any embodiment may be included in another embodiment and may be replaced by corresponding constructions of another embodiment. It will be apparent to those skilled in the art that claims in the appended claims that are not expressly referenced in each other may be combined to form embodiments of the present invention, or may be included as new claims in amendments made after the filing of this application.

[0197] Embodiments of the present invention can be implemented by various means, such as hardware, firmware, software, or combinations thereof. In a hardware configuration, the method according to an exemplary embodiment of the present invention can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0198] In firmware or software configuration, embodiments of the present invention can be implemented in the form of modules, processes, functions, etc. Software code can be stored in a memory unit and executed by a processor. The memory unit is located inside or outside the processor and can send data to and receive data from the processor via various known means.

[0199] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is accorded the widest scope consistent with the principles and novel features disclosed herein.

[0200] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method of verifying test data, characterized by, The method comprises the following steps: acquiring test data of a device under test; acquiring a test verification model according to verification attention information; verifying the test data by using the test verification model to acquire a verification result.

2. The method of claim 1, wherein, If the verification attention information comprises at least one of repeatability of a test program, difference of a test circuit board or a test probe card at different test sites, and rationality of setting of a test limit value of a test item, the test verification model comprises a GRR model.

3. The method of claim 2, wherein, The verification of the test data by using the test verification model to acquire a verification result comprises: performing first verification processing on test data of a test item to be verified by using the GRR model to acquire device variation and evaluator variation of the test item; acquiring first repeatability and reproducibility values of the test item according to the device variation and the evaluator variation of the test item; acquiring a first total variation type of the test item according to effectiveness information of an acceptable upper limit value and an acceptable lower limit value of the test item; acquiring a first total variation value of the test item according to the first total variation type of the test item; acquiring a first repeatability and reproducibility index of the test item according to the first repeatability and reproducibility values of the test item and the first total variation value of the test item.

4. The method of claim 3, wherein, When the first total variation type of the test item input by a user is received, the method further comprises: acquiring the first total variation value of the test item according to the first total variation type of the test item input by the user.

5. The method of claim 3, wherein the test data is verified by, If the acceptable upper limit value and the acceptable lower limit value of the test item are both invalid, the first total variation type of the test item is a first type; if only the acceptable lower limit value of the test item is valid, the first total variation type of the test item is a second type; if only the acceptable upper limit value of the test item is valid, the first total variation type of the test item is a third type; and if the acceptable upper limit value and the acceptable lower limit value of the test item are both valid, the first total variation type of the test item is a fourth type. The first total variation value of the test item is obtained according to the first total variation type of the test item, including: if the first total variation type of the test item is the first type, a formula The first total variation of the test item is calculated, wherein TV1 represents the first total variation value of the test item, GRR1 represents the first repeatability and reproducibility value of the test item, and PV represents sample variation; if the first total variation type of the test item is the second type, a formula The first total variation value of the test item is calculated, wherein Mean1 represents the average value of the test data of the test item, and LSL1 represents the acceptable lower limit value of the test item; if the first total variation type of the test item is the third type, a formula The first total variation value of the test item is calculated, wherein USL1 represents the acceptable upper limit value of the test item; if the first total variation type of the test item is the fourth type, a formula The first total variation value of the test item is calculated, wherein Min(.) represents a minimum value operation.

6. The method of claim 1, wherein, If the verification attention information comprises difference between measurement systems, the test verification model comprises a T test model.

7. The method of claim 6, wherein the test data is verified by, If the measurement systems to be compared acquire test data of a same test item by testing at least partially same devices under test, the T test model comprises a paired sample T test model. The verification of the test data by using the test verification model to acquire a verification result comprises: performing second verification processing on test data of a same test item acquired by testing at least partially same devices under test by using the paired sample T test model to acquire a T bias value, a P value and an effect size of the test item.

8. The method of claim 7, wherein the test data is verified by, The verification of the test data by using the test verification model to acquire a verification result further comprises: The GRR model is used to verify the test data of the same test item obtained by testing the same device by the measurement systems to be compared, and second repeatability and reproducibility indexes of the test item are obtained.

9. The method of claim 8, wherein, The GRR model is used to verify the test data of the same test item obtained by testing the same device by the measurement systems to be compared, and second repeatability and reproducibility indexes of the test item are obtained. The GRR model is used to verify the test data of the same test item obtained by testing the same device by the measurement systems to be compared, and second repeatability and reproducibility indexes of the test item are obtained. According to the effectiveness information of the acceptable upper limit value and the acceptable lower limit value of the test item, a second total variation type of the test item is obtained. According to the second total variation type of the test item, a second total variation value of the test item is obtained. According to the second repeatability and reproducibility value of the test item and the second total variation value of the test item, a second repeatability and reproducibility index of the test item is obtained.

10. The method of claim 9, wherein the test data is verified by, According to the effectiveness information of the acceptable upper limit value and the acceptable lower limit value of the test item, a total variation of the test data of the test item is obtained, including: if the acceptable upper limit value and the acceptable lower limit value of the test item are both invalid, the second total variation type of the test item is the first type; if only the acceptable lower limit value of the acceptable upper limit value and the acceptable lower limit value of the test item is valid, the second total variation type of the test item is the second type; if only the acceptable lower limit value of the acceptable upper limit value and the acceptable lower limit value of the test item is valid, the second total variation type of the test item is the third type, and if the acceptable upper limit value and the acceptable lower limit value of the test item are both valid, the second total variation type of the test item is the fourth type; The second total variation value of the test item is obtained according to the second total variation type of the test item, including: if the second total variation type of the test item is the first type, the second total variation value of the test item is calculated by using a formula TV2=S, wherein TV2 represents the second total variation value of the test item, and S represents a sample standard deviation; if the second total variation type of the test item is the second type, the second total variation value of the test item is calculated by using a formula The second total variation value of the test item is calculated by using a formula The second total variation value of the test item is calculated by using a formula The second total variation value of the test item is calculated by using a formula, wherein Min(.) represents a minimum value operation.

11. The method of claim 6, wherein the test data is verified by, If the measurement systems to be compared use different devices to test and obtain the test data, the T-checking model includes an independent T-checking model; The test data is verified by using the obtained test verification model, and a verification result is obtained, including: the independent T-checking model is used to perform third verification processing on the test data of the same test item obtained by testing different devices by the measurement systems to be compared, and a P value and an effect size of the test item are obtained.

12. The method of claim 1, wherein, After the test data is verified by using the test verification model and a verification result is obtained, the verification result is outputted.

13. The method of claim 1, wherein, After the test data is verified by using the test verification model and a verification result is obtained, a graphical display of the test data is outputted.

14. A validation module for testing data, characterized by The data acquisition unit acquires the test data of the device under test. The model acquisition unit is adapted to acquire a test verification model according to the verification attention information. The test verification unit is adapted to verify the test data by using the acquired test verification model, and obtain a verification result. The computer program / instructions are executed by the processor to implement the test data verification method of any one of claims 1-13.

15. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the test data verification method of any one of claims 1-13.

16. An apparatus, comprising: A computer program product including at least one storage medium having stored thereon one or more computer instructions that, when executed by at least one processor, implement the method of verifying test data according to any one of claims 1-13.

17. A storage medium, characterized by The storage medium stores one or more computer instructions for implementing the method of verifying test data according to any one of claims 1-13.

Images