Method for analyzing the performance of multiple test instruments measuring parts of the same type

Abstract

The present invention discloses a method for operating a data processing system to analyze a data set for grouping and a computer-readable medium having instructions for executing the method. The method includes causing the data processing system to receive a plurality of data sets, each data set including a plurality of values ​​characterized by a statistical distribution and a label. The method also includes causing the data processing system to calculate a plurality of statistical parameters for each of the plurality of data sets, generating a data set vector having components equal to the plurality of statistical parameters for each of the plurality of data sets, assigning each data set to a cluster based on the data set vector using a clustering algorithm, and generating a display of the statistical distribution as a function of the label, wherein statistical distributions belonging to the same cluster are grouped together.

Description

Background Art

[0001] Consider a production line of a particular part that is tested by an instrument that measures one or more signals from the part. Part testing may involve multiple test instruments of the same type to speed up the testing process. In addition, the same part may be tested multiple times. For example, a part may be found to be defective in the first test and sent to a repair station. The repaired part is then tested again. In an ideal situation, all test instruments would measure the same value for any given part. However, in reality, the set of measurements provided by any test instrument for multiple theoretically identical parts will have a certain statistical distribution. Similarly, even good parts will show certain differences between each other. When the number of test instruments is large, identifying the faulty test instrument becomes challenging, especially when the errors in the output values ​​are small, which vary from part to part and are therefore hidden in the statistical noise. Finally, these test instruments may vary in the exact model and the exact date when the instrument was put into service. Therefore, inspecting the production line in the face of these differences poses a significant challenge. Summary of the invention

[0002] The present invention includes a method for operating a data processing system to analyze a data set for grouping and a computer-readable medium having instructions for causing the data processing system to perform the method. The method includes causing the data processing system to receive a plurality of data sets, each data set including a plurality of values ​​characterized by a statistical distribution and a label. The method also includes causing the data processing system to calculate a plurality of statistical parameters for each of the plurality of data sets, generating a data set vector having components equal to the plurality of statistical parameters for each of the plurality of data sets, assigning each data set to a cluster based on the data set vector using a clustering algorithm, and generating a display of the statistical distribution as a function of the label, wherein statistical distributions belonging to the same cluster are grouped together.

[0003] In one aspect of the invention, the display includes a graph of a symbol associated with a statistical parameter as a function of a label, the symbol being located at the median or mean of the values ​​of a data set for the label.

[0004] In another aspect, the data sets are further sorted by median or mean within each cluster.

[0005] In another aspect, the data sets each include a plurality of measurement values ​​of a first physical quantity measured by a first test probe associated with one of a plurality of test instruments, each label corresponding to one of the test instruments.

[0006] In another aspect, the display includes a graph of a symbol associated with a statistical parameter as a function of the instrument ID, the symbol being located at the median or mean of the measurements taken by the instrument.

[0007] In another aspect, the symbol comprises a box plot.

[0008] In another aspect, each of the instruments includes a second test probe and provides a measurement value of a second physical quantity through the second test probe, and wherein the data processing system generates a plurality of statistical parameters characterizing a statistical distribution of a set of measurement values, the set of measurement values ​​corresponding to each of the plurality of instruments and the second test probe associated with the instrument, and the instrument vector further includes a second plurality of statistical parameters.

[0009] The present invention also includes the following:

[0010] Item 1. A method for operating a data processing system to analyze a data set for grouping, the method comprising causing the data processing system to:

[0011] receiving a plurality of data sets, each data set comprising a plurality of values ​​characterized by a statistical distribution;

[0012] calculating a plurality of statistical parameters for each of the plurality of data sets;

[0013] generating a data set vector for each of the plurality of data sets, the data set vector having components equal to the plurality of statistical parameters;

[0014] Assigning each data set to a cluster based on the data set vectors using a clustering algorithm; and generating a display of the statistical distributions in which the statistical distributions belonging to the same cluster are grouped together.

[0015] Item 2. The method of Item 1, wherein the display includes a chart of a symbol associated with the statistical parameter, the symbol being located at the median or mean of the values ​​of the data set.

[0016] Item 3. The method of Item 1, wherein the data set is further sorted by the median or the mean within each cluster.

[0017] Item 4. The method of Item 1, wherein the data sets each include multiple measurement values ​​of a first physical quantity, the first physical quantity being measured by a first test probe associated with one of a plurality of test instruments, and each data set is characterized by a label, the label including an instrument ID corresponding to one of the test instruments.

[0018] Item 5. The method of Item 4, wherein the display includes a graph of a symbol associated with the statistical parameter as a function of the instrument ID, the symbol being located at the median or mean of the plurality of measurements taken by the test instrument.

[0019] Item 6. The method of Item 5, wherein the symbols include box plots.

[0020] Item 7. The method of Item 4, wherein the test instruments each include a second test probe and provide a measurement value of a second physical quantity through the second test probe, and wherein the data processing system generates a plurality of statistical parameters characterizing the statistical distribution of a set of measurement values, the set of measurement values ​​corresponding to each of the plurality of test instruments and the second test probe associated with the test instrument, and the instrument vector further includes the second plurality of statistical parameters.

[0021] Item 8. A computer-readable medium comprising instructions for causing a data processing system to perform a method, the method causing the data processing system to:

[0022] receiving a plurality of data sets, each data set comprising a plurality of values ​​characterized by a statistical distribution;

[0023] calculating a plurality of statistical parameters for each of the plurality of data sets;

[0024] generating a data set vector for each of the plurality of data sets, the data set vector having components equal to the plurality of statistical parameters;

[0025] Assigning each data set to a cluster based on the data set vectors using a clustering algorithm; and

[0026] A display of the statistical distributions is generated wherein the statistical distributions belonging to the same cluster are grouped together.

[0027] Item 9. The computer-readable medium of Item 8, wherein the display includes a chart of a symbol associated with the statistical parameter, the symbol being located at a median or mean of the data set values.

[0028] Item 10. The computer-readable medium of Item 8, wherein the data set is further sorted within each cluster by the median value or the mean value.

[0029] Item 11. Computer-readable medium of Item 8, wherein the data sets each include multiple measurement values ​​of a first physical quantity, the first physical quantity being measured by a first test probe associated with one of a plurality of test instruments, and each data set is characterized by a label, the label including an instrument ID corresponding to one of the test instruments.

[0030] Item 12. The computer-readable medium of Item 11, wherein the display comprises a graph of a symbol associated with the statistical parameter as a function of the instrument ID, the symbol being located at a median or average of a plurality of the measurements taken by the test instrument.

[0031] Item 13. Computer-readable medium of Item 12, wherein the symbol comprises a box plot.

[0032] Item 14. Computer-readable medium of Item 11, wherein the test instruments each include a second test probe and provide a measurement value of a second physical quantity through the second test probe, and wherein the data processing system generates a plurality of statistical parameters characterizing a statistical distribution of a set of measurement values, the set of measurement values ​​corresponding to each of the plurality of test instruments and the second test probe associated with the test instrument, and the instrument vector further includes the second plurality of statistical parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A production line is shown which may advantageously comprise the present invention.

[0034] Figure 2 Multiple 1D scatter plots sorted by instrument ID are shown for specific test probes on each instrument.

[0035] Figure 3 The scatter plots overlaid with box plots show Figure 2 A scatter plot of .

[0036] Figure 4 Shows that after reordering the scatter plot Figure 3 , so that data for instruments grouped together are displayed adjacent to each other.

[0037] Figure 5 Using box plot symbols plotted on these data, we show Figure 4 The data shown in . DETAILED DESCRIPTION

[0038] The manner in which the present invention provides its advantages may be more readily understood with reference to a simple example. Figure 1 , which shows a production line that can advantageously include the present invention. The production line produces integrated circuit boards 14, where the final circuit boards are tested by multiple test instruments 12 to increase the yield of the production test stage. In this example, the production line is also divided into multiple sub-production lines 21 to further increase the production rate of integrated circuit boards. For the purpose of this example, each instrument uses N t The test probes 13 measure the same N on the circuit board tested by the instrument. t A set of test points 15, one test probe 13 corresponding to one of the test points 15. It is assumed that each instrument reports its measurement values ​​to a controller 22, which, among other things, stores the raw test results in a table or similar structure and provides a user interface 23 for displaying data and graphs required for a user to better understand the operation of the test instrument. Each row of the table has Nt +1 entry, one for the instrument ID, N t entries for measurements. In the following discussion, this table will be referred to as the raw measurement table. Each measurement in each column is a data value obtained by a specific test probe and instrument. The columns define the test probes, and the instrument ID identifies the instrument that made the measurement. The set of measurements taken for multiple boards by any given instrument and one of its test probes will be referred to as the measurement set (MSITP) for that instrument / test probe.

[0039] For each test instrument, a scatter plot can be generated for each MSITP. Figure 2 , which shows multiple one-dimensional scatter plots of specific test probes on each instrument sorted by instrument ID. The MSITP of any given instrument is characterized by a statistical distribution. In one aspect of the invention, one or more statistical parameters are calculated for each MSITP. For the purposes of this discussion, a statistical parameter is defined as any parameter derived from the difference between the median or mean of each measurement value and each measurement value. Examples of such statistical parameters based on the average of the data values ​​include a function of the standard deviation of the data values ​​of the scatter plot, a function of the average of the absolute differences of each data value from the mean. The mean or median of the measurement values ​​is also included in the statistical parameter.

[0040] Statistical parameters can also be based on the median of the data values ​​and other values ​​related to the median of the distribution. These summary statistics can be displayed with a relatively simple symbol that shows five values. To calculate these values, the values ​​of the set are sorted by their values. The first value of the summary statistic is the median of the MSITP. The remaining quantities are the quartiles and outliers of the distribution. The quartiles are calculated by dividing the sorted list of distribution values ​​into four parts of approximately equal length. The quantity in the middle of the list (or the value interpolated between the two middle values ​​of the list) will be called "Q2" or "median". The first quartile "Q1" divides the first half of the list into two equal parts, while the third quartile "Q3" divides the second half of the list into two equal parts. The middle half of the list (the part between Q1 and Q3) is called the interquartile range or "IQR". The IQR is also equal to the value between the 25th and 75th percentiles of the data. The IQR is used to find outliers in the set. Outliers are usually defined as values ​​below Q1-1.5IQR or above Q3+1.5IQR.

[0041] The symbol representing these summary statistics has a box with the Q1 and Q3 points represented by horizontal lines. Q1 is represented by the horizontal line within the box. Outliers are shown by small bars connected to the box by vertical lines. Exemplary symbols will be shown in the subsequent figures. In the following discussion, this symbol will be referred to as a box plot. Figure 3The scatter plots overlaid with box plots show Figure 2 It should be noted that even with the box plots overlaid on the individual scatter plots, it is still difficult to discern whether there is any form of systematic grouping of the instruments.

[0042] The present invention groups the results for any given test probe by finding clusters in the individual instrument data across all test points and then displaying the results for the test probes reordered so that the instrument results for that test probe are grouped together in the display of results for that test probe. To cluster the test probe data, a vector is constructed for each instrument based on statistical parameters that represent the statistical distribution measured by the instrument for each test probe.

[0043] The manner in which clustering is achieved can be more easily understood with reference to an exemplary embodiment of the present invention. For each instrument / test probe, the statistical distribution of the measurements taken by the instrument / test probe is simplified to a number of statistical parameters such as in a box plot. That is, parameters Q1 to Q5 are calculated for each instrument / test probe. Thus, a 5*N distribution of measurements is generated for each instrument. t The clustering vector of components. Each of the vectors in the obtained vector is in 5N t A probe is defined in a dimensional space. In one aspect of the invention, the data processing system determines the locations and boundaries of clusters of these points. Ideally, if all instruments are operating correctly and there are no systematic differences between the parts or boards being tested, there would be only one such cluster. However, in practice, there are multiple clusters or large clusters with a single outlier, that is, clusters with only one member. By sorting the data based on clusters, users can more easily understand the raw data table and gain insight into the production line.

[0044] Algorithms for identifying clusters in such a space are known in the art and, therefore, will not be discussed in detail here. For example, one class of these algorithms defines a density function (representing the number of points per unit volume in the space) and then attempts to find peaks in the density function. Each peak represents a cluster. Points surrounding any given peak are then defined as belonging to the cluster assigned to the peak. Other examples of clustering algorithms utilize multidimensional averaging methods (such as K-Means) or "ensemble" methods that combine multiple clustering techniques.

[0045] Once the instruments are assigned to clusters, the data from the raw data table can be presented in a way that allows the operator to more easily understand the differences between the instruments / boards in each cluster. Figure 4 , which shows that after reordering the scatter plot Figure 3, so that the data for instruments grouped together are displayed next to each other. A vertical dividing line separates the two clusters found in the clustering. It should be noted that this plot shows data for only one test probe. Similar plots can be generated for other test points.

[0046] Although Figure 4 The data shown in are useful when viewing the raw data, but including the display of statistical parameters can provide greater insight into the data. Figure 5 , which shows the box plot symbols plotted on these data. Figure 4 . Those elements of the original data located under the symbol are not shown. The box plot symbol clearly shows the clustered nature of the data. The remaining outliers are shown as individual points. The combined data display also provides a deeper understanding of the differences between the clusters by including outliers. In particular, the data in cluster 31 is characterized by significantly more high outliers than the data in cluster 32, which indicates that the statistical distribution has a "tail" that extends to higher data values. Similarly, the data in cluster 32 has significantly more low outliers, which indicates that the statistical distribution has a tail that extends to lower data values.

[0047] Clusters can be the result of instrument-to-instrument variation or of differences between components of the board being tested. If all test probe cluster plots show the same clusters, the clusters are most likely the result of instrument-to-instrument variation. However, if different test probe cluster plots show different clusters, the clusters are more likely the result of differences between components of the board. For example, one test may measure the capacitance of a specific capacitor on a board being tested. If different boards have capacitors from different batches of capacitors, the resulting clusters may be the result of systematic differences between the capacitors.

[0048] In the above embodiment, the clustering vectors utilize the same statistical parameters for each test probe. However, embodiments may also be constructed that use different statistical parameters for different test points. The only requirement of the clustering algorithm is that the same statistical parameters are used for each instrument / test probe. Because each test probe measures a different parameter on the circuit board, statistical parameters that better characterize the resulting distribution are preferred. For example, the first test probe data may be characterized by a box plot parameter, while the second test probe data may be characterized by the median and standard deviation of the statistical distribution of the second test probe data.

[0049] Clustering the measurements based on statistical parameters rather than raw data allows the invention to be applied to data sets where different probes have different numbers of data points. That is, the data set corresponding to one of the probes may be missing measurements. Furthermore, using statistical parameters instead of raw data significantly reduces the computational effort of performing cluster analysis.

[0050] In the above example, the clustering vector includes components of the statistical parameters from all test probes. However, embodiments can also be constructed to cluster data based on a subset of test probes. For example, data can be clustered based on a clustering vector having only statistical parameters derived from one of the test probes.

[0051] The data processing system performing the clustering and display functions may be the controller that collects data from the various instruments, or a separate data processing system that receives the collected data and performs analysis in response to commands entered by a user through a user interface associated with the data processing system.

[0052] The above-mentioned embodiment of the present invention is directed to analyzing a data set generated by a physical instrument having one or more test probes for measuring physical quantities. However, the method of the present invention can be applied to other data sets to detect and display the grouping in those data sets. In a more general case, a data processing system receives multiple data sets, each of which includes multiple values ​​characterized by a statistical distribution and a label. The data processing system calculates multiple statistical parameters for each of the multiple data sets. Then a data set vector is generated for each of the multiple data sets, the data set vector having components equal to multiple statistical parameters, and the data set vectors are clustered using a clustering algorithm. Then a display of the statistical distribution as a function of the label is generated, wherein the data sets belonging to the same cluster are grouped together. The display can also include a chart of symbols associated with the statistical parameters as a function of the label. The symbol is preferably located at the median or mean value of the data set value of the label. Within a given cluster, the statistical distribution and / or symbol can be sorted by one of the statistical parameters (such as the median or mean value of the values ​​in the data set).

[0053] The present invention also includes a computer readable medium storing instructions that cause a data processing system to perform the method of the present invention. A computer readable medium is defined as any medium that constitutes patentable subject matter under 35 U.S.C. 101 and excludes any medium that does not constitute patentable subject matter under 35 U.S.C. 101. Examples of patentable media include non-transitory media such as computer storage devices that store information in a format readable by a computer or data processing system.

[0054] The above-described embodiments of the present invention have been provided to demonstrate various aspects of the present invention. However, it should be understood that the different aspects of the present invention shown in different specific embodiments can be combined to provide other embodiments of the present invention. In addition, various modifications to the present invention will become apparent from the foregoing description and the accompanying drawings. Therefore, the present invention is limited only by the scope of the appended claims.

Claims

1. A method for operating a data processing system to analyze a data set for grouping, the method comprising causing the data processing system to: receiving a plurality of data sets, each data set comprising a plurality of measurements of a first physical quantity made by a corresponding test instrument and a label identifying the test instrument and characterized by a statistical distribution; calculating a plurality of statistical parameters for each of the plurality of data sets; generating a data set vector for each of the plurality of data sets, the data set vector having components equal to the plurality of statistical parameters; assigning each data set to a cluster based on the data set vector using a clustering algorithm; as well as A graph of the statistical distribution is generated on a display of the data processing system, the statistical distribution being a function of the label for viewing by a user to determine whether systematic differences in the data set are caused by differences in the test instrument or by differences in components measured by the test instrument, wherein the statistical distributions belonging to the same cluster are grouped together along a first axis of the graph, the label is displayed on the first axis, and the statistical distribution is displayed on a second axis of the graph, wherein each statistical distribution comprises a one-dimensional scatter plot of the measured values ​​of the data set corresponding to the label, and adjacent clusters are separated by a dividing line parallel to the second axis. 2 . The method of claim 1 , wherein the display includes a graph of a symbol associated with the statistical parameter, the symbol being located at a median or mean of the data set values.

3. The method according to claim 1, wherein the data set is further sorted by median or mean within each cluster.

4. The method of claim 1 , wherein the data sets each comprise a plurality of measured values ​​of a first physical quantity, the first physical quantity being measured by a first test probe associated with one of a plurality of test instruments, and each data set is characterized by a tag comprising an instrument ID corresponding to one of the test instruments.

5. A method according to claim 1, wherein the data sets each include multiple measurement values ​​of a first physical quantity, the first physical quantity being measured by a first test probe associated with one of a plurality of test instruments, and each data set is characterized by a label, the label including an instrument ID corresponding to one of the test instruments, and wherein the display includes a graph of a symbol associated with the statistical parameter as a function of the instrument ID, the symbol being located at the median or average of the multiple measurement values ​​taken by the test instrument.

6. A method according to claim 1, wherein the data sets each include multiple measurement values ​​of a first physical quantity, the first physical quantity is measured by a first test probe associated with a test instrument among a plurality of test instruments, and each data set is characterized by a label, the label includes an instrument ID corresponding to one of the test instruments, and wherein the display includes a chart of symbols associated with the statistical parameter as a function of the instrument ID, the symbols are located at the median or mean of the multiple measurement values ​​taken by the test instrument, and wherein the symbols include box plots.

7. A method according to claim 1, wherein the data sets each include multiple measurement values ​​of a first physical quantity, the first physical quantity is measured by a first test probe associated with a test instrument among a plurality of test instruments, and each data set is characterized by a label, the label includes an instrument ID corresponding to a test instrument among the test instruments, wherein the test instruments each include a second test probe and provide measurement values ​​of a second physical quantity through the second test probe, and wherein the data processing system generates a plurality of statistical parameters characterizing the statistical distribution of a set of measurement values, the set of measurement values ​​corresponds to each test instrument among the plurality of test instruments and the second test probe associated with the test instrument, and the instrument vector further includes a second plurality of statistical parameters.

8. A computer readable medium comprising instructions for causing a data processing system to perform a method, the method causing the data processing system to: receiving a plurality of data sets, each data set comprising a plurality of measurements of a first physical quantity made by a corresponding test instrument and a label identifying the test instrument and characterized by a statistical distribution; calculating a plurality of statistical parameters for each of the plurality of data sets; generating a data set vector for each of the plurality of data sets, the data set vector having components equal to the plurality of statistical parameters; assigning each data set to a cluster based on the data set vector using a clustering algorithm; as well as A graph of the statistical distribution is generated on a display of the data processing system, the statistical distribution being a function of the label for viewing by a user to determine whether systematic differences in the data set are caused by differences in the test instrument or by differences in components measured by the test instrument, wherein the statistical distributions belonging to the same cluster are grouped together along a first axis of the graph, the label is displayed on the first axis, and the statistical distribution is displayed on a second axis of the graph, wherein each statistical distribution comprises a one-dimensional scatter plot of the measured values ​​of the data set corresponding to the label, and adjacent clusters are separated by a dividing line parallel to the second axis.

9. The computer-readable medium of claim 8, wherein the display includes a graph of a symbol associated with the statistical parameter, the symbol being located at a median or mean of the data set values.

10. The computer-readable medium of claim 8, wherein the data set is further sorted by median or mean within each cluster.

11. The computer-readable medium of claim 8, wherein the data sets each comprise a plurality of measured values ​​of a first physical quantity measured by a first test probe associated with one of a plurality of test instruments, and each data set is characterized by a tag comprising an instrument ID corresponding to one of the test instruments.

12. The computer-readable medium of claim 8 , wherein the data sets each comprise a plurality of measured values ​​of a first physical quantity, the first physical quantity being measured by a first test probe associated with one of a plurality of test instruments, and each data set is characterized by a label comprising an instrument ID corresponding to one of the test instruments, and wherein the display comprises a graph of a symbol associated with the statistical parameter as a function of the instrument ID, the symbol being located at a median or average of a plurality of the measured values ​​taken by the test instrument.

13. A computer-readable medium according to claim 8, wherein the data sets each include multiple measurement values ​​of a first physical quantity, the first physical quantity is measured by a first test probe associated with one of a plurality of test instruments, and each data set is characterized by a label, the label including an instrument ID corresponding to one of the test instruments, and wherein the display includes a chart of symbols associated with the statistical parameter as a function of the instrument ID, the symbols are located at the median or mean of multiple measurement values ​​taken by the test instrument, and wherein the symbols include box plots.

14. A computer-readable medium according to claim 8, wherein the data sets each include multiple measurement values ​​of a first physical quantity, the first physical quantity is measured by a first test probe associated with a test instrument among a plurality of test instruments, and each data set is characterized by a label, the label includes an instrument ID corresponding to a test instrument among the test instruments, wherein the test instruments each include a second test probe and provide measurement values ​​of a second physical quantity through the second test probe, and wherein the data processing system generates a plurality of statistical parameters characterizing the statistical distribution of a set of measurement values, the set of measurement values ​​corresponds to each test instrument among the plurality of test instruments and the second test probe associated with the test instrument, and the instrument vector further includes a second plurality of statistical parameters.

Images