Method and apparatus for determining information about characteristics of one or more devices under test (DUTs) using statistically significant dissimilarity values ​​- Patents.com

Abstract

Embodiments include methods and apparatus for determining information regarding characteristics of one or more devices under test (DUTs) using measurement data from automatic test equipment (ATE), where the measurement data includes a plurality of measurement results and information describing corresponding measurement conditions of the devices under test (DUTs).

Description

[Technical field]

[0001] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments in accordance with the present invention relate to methods and apparatus for using statistically significant dissimilarity values ​​to determine information about characteristics of one or more devices under test, DUTs. [Background technology]

[0002] Often, technology devices must be tested before being used in their intended applications. To provide cost-effective and reliable test results, Automatic Test Equipment, ATE, may be used.

[0003] However, test results can be difficult to process and / or evaluate, especially in test scenarios where multiple parameters and large numbers of devices are tested. Summary of the Invention [Problem to be solved by the invention]

[0004] It is therefore desirable to have a concept that is applicable in such scenarios and offers a better compromise between reliability, efficiency, and cost. [Means for solving the problem]

[0005] This is achieved by the subject matter of the independent claims of the present application.

[0006] Further embodiments according to the invention are defined by the subject matter of the dependent claims of the present application.

[0007] An embodiment according to the invention includes a method for determining information about characteristics of one or more devices under test DUTs (e.g., information that "the device under test behaves differently under these conditions with statistical significance") using measurement data from automatic test equipment (ATE), the measurement data including a plurality of measurement results (e.g., a target variable, e.g. T, e.g. "test pass", e.g. "test fail", e.g. a continuous value, e.g. voltage or current or signal characteristic) and information describing corresponding measurement conditions of the device under test (DUT) (e.g. describing the condition of a corresponding independent variable, e.g. V=v, e.g. describing the condition that the independent variable V has value v; e.g. describing the measurement conditions under which the measurement results were obtained).

[0008] The method may, for example, compute a conditional distribution (e.g., in the case of a non-Boolean target variable, a local conditional target distribution, e.g., p v (t), or, for example, in the case of a Boolean target variable, the true target value p v The method includes determining information about the information that represents or describes the information (a percentage or fraction of the information).

[0009] The method may, for example, include selecting at least one sample distribution (e.g., in the case of a non-Boolean target variable, multiple empirical distributions q s Or q s (t), e.g., in the case of a Boolean target variable, a probability q'), where at least one sample distribution (e.g., in the case of a Boolean target variable, one analytically determined representative sample distribution, or in the case of a non-Boolean target variable, multiple empirical sample distributions) is comparable to, e.g., a conditional distribution (e.g., associated with a wider range of measurement conditions than the conditional distribution, e.g., including an equal or similar number of measurement outcomes).

[0010] Further, the method may use information, e.g., regarding the information representing or describing the conditional distribution, and information, e.g., regarding the information representing or describing at least one sample distribution, to generate a statistically significant dissimilarity value (e.g., which indicates how the behavior of the DUT depends on the measurement conditions; e.g., d(v); e.g., which characterizes the behavior of the DUT for a given measurement condition; e.g., describing a statistically significant dissimilarity between the conditional distribution and one or more sample distributions; e.g., determining a dissimilarity such that only a predetermined portion, e.g., 5%, of the sample distributions contain dissimilarities with respect to the conditional distribution that are below a dissimilarity threshold. determining a similarity threshold, e.g., d(v); determining a dissimilarity threshold such that only a predetermined portion, e.g., 5%, of all cases of the sample distribution, e.g., binomial binomial distribution, include dissimilarities with respect to the conditional distribution that are below the dissimilarity threshold; determining a dissimilarity threshold that indicates whether the conditional distribution includes a predetermined percentage of the sample distribution that are most similar to the conditional distribution to indicate a minimum dissimilarity threshold between the conditional distribution and the sample distribution), wherein the information about the characteristics of the one or more DUTs includes a statistically significant dissimilarity value or is based on a statistically significant dissimilarity value.

[0011] The inventors have recognised that DUT characteristics may be robustly and efficiently determined or assessed based on measurement data including a plurality of measurement results and information describing corresponding measurement conditions, for example by comparing information relating to a conditional distribution with information relating to at least one sample distribution.

[0012] Briefly, as an example, the inventors have recognized that in order to evaluate or classify a measurement result corresponding to a first subset of measurement conditions, information about its conditional distribution can be determined. To enable a robust and reliable evaluation, this conditional distribution can be compared, for example, with at least one sample distribution. Briefly, from a plurality of ATE test cases (e.g., including results and corresponding conditions), a test case can be selected, for example randomly, one or more times, and its one or more sample distributions can be determined. To keep the one or more sample distributions comparable to the conditional distribution, each sample distribution can be associated with a similar or equal number of test cases as the conditional distribution.

[0013] Here, one or each of the sample distributions can be compared with, for example, a conditional distribution. In simple terms, the comparison can answer the question of whether the conditional distribution is similar to the sample distribution or significantly diverges (as an example, the sample distribution is based on randomly drawn test cases). Thus, based on the comparison, for example, not only can differences be detected, but also information about the statistical significance of the differences can be provided.

[0014] The inventors have recognized that a dissimilarity value may enable the provision of a good metric for such a comparison. Thus, a statistically significant dissimilarity value may be determined based on one or more comparisons between a conditional distribution and one or more sample distributions.

[0015] According to a further embodiment of the invention, the sample distribution is an empirical distribution, and determining information relating to, e.g. expressing or describing, at least one sample distribution comprises generating a plurality of empirical distributions, e.g. q s This includes determining (e.g., expressing or describing) information about.

[0016] Further, the number of measurement results taken into account to determine the information about each empirical distribution, e.g., the information about each of the empirical distributions, is related to (e.g., corresponds to and deviates from by no more than 10% or no more than 20%; e.g., is equal to) the number of measurement results taken into account to determine the information about the conditional distribution, and the measurement results of each subset of the measurement data are a randomly or pseudo-randomly, e.g., or arbitrarily, selected set of measurement results of the measurement data.

[0017] The method further includes using information about the conditional distributions (e.g., the conditional distributions themselves) and using information about each of the empirical distributions (e.g., the empirical distributions themselves) to determine a plurality of dissimilarity measures, e.g., Kullback-Leibler divergence, between the conditional distributions and each of the empirical distributions, e.g., each of the empirical distributions. Additionally, the method further includes using the plurality of dissimilarity measures to determine a statistically significant dissimilarity value, e.g., d(v).

[0018] According to further embodiments of the invention, a statistically significant dissimilarity value (e.g., d(v), e.g., which indicates how the behavior of the DUT depends on the measurement conditions; e.g., d(v), which characterizes the behavior of the DUT for given measurement conditions; e.g., describing a statistically significant dissimilarity between a conditional distribution and one or more sample distributions; determining a dissimilarity threshold, e.g., d(v), such that only a predetermined portion, e.g., 5%, of the sample distributions include dissimilarities with respect to the conditional distribution that are below the dissimilarity threshold; e.g., determining a dissimilarity threshold such that only a predetermined portion, e.g., 5%, of all cases of the sample distribution, e.g., binomial binomial distribution, includes a dissimilarity with respect to the conditional distribution that is below the dissimilarity threshold; determining a dissimilarity threshold indicating whether the conditional distribution includes excluding a predetermined percentage of the sample distribution that is most similar to the conditional distribution to indicate a minimum dissimilarity threshold between the conditional distribution and the sample distribution) is associated with a quantile (e.g., percentile, e.g., the lower 1-C percentile, e.g., C=95%) of the distribution of the dissimilarity measure.

[0019] According to a further embodiment of the invention, the method further comprises the steps of determining a cumulative distribution, e.g. a CDF, of the plurality of dissimilarity measures and determining a statistically significant dissimilarity value, e.g. d(v), such that only a predetermined portion of the empirical distributions comprises dissimilarity measures for the conditional distributions that are below the statistically significant dissimilarity value, e.g. such that only a predetermined portion of the dissimilarity measures between each empirical distribution and the conditional distributions is below the statistically significant dissimilarity value.

[0020] According to a further embodiment of the invention, the method further comprises the step of extrapolating the cumulative distribution and / or interpolating the cumulative distribution to determine a statistically significant dissimilarity value, e.g. d(v).

[0021] According to a further embodiment of the invention, the method further comprises the step of: s’wherein the selected dissimilarity measure comprises a statistically significant dissimilarity value, e.g., a minimum deviation for d(v), from the plurality of dissimilarity measures, and the method further comprises determining a mean of an empirical distribution corresponding to the selected dissimilarity measure.

[0022] Further, the method includes calculating a mean value of a conditional distribution of the measurement results corresponding to a first subset of measurement conditions, e.g.

number

number

number

[0023] The method uses an average alignment dissimilarity measure, e.g.

number

number

[0024] Furthermore, information about the characteristics of one or more DUTs can be obtained using normalized dissimilarity ratios, e.g.

number

[0025] According to a further embodiment of the invention, the method further comprises the step of: s’ The method further includes a step of selecting a dissimilarity measure, wherein the selected dissimilarity measure comprises a statistically significant dissimilarity value, e.g., a minimum deviation for d(v), from the plurality of dissimilarity measures, and the information regarding the characteristics of the one or more DUTs includes information regarding the selected dissimilarity measure, e.g., the selected dissimilarity measure.

[0026] According to a further embodiment of the invention, the method further comprises: determining a mean value of a conditional distribution of the measurement results corresponding to a first subset of measurement conditions, e.g.

number

number

[0027] Furthermore, the method includes a mean aligned dissimilarity measure (e.g.,

number

number

number

[0028] Additionally, the method uses an average alignment dissimilarity measure, e.g.

number

number

[0029] Additionally, the information regarding the characteristics of the one or more DUTs includes information regarding a normalized dissimilarity ratio, eg, μ.

[0030] According to a further embodiment of the invention, the method further comprises binning of the measured data values, the binning being performed based on a parameter α, which parameter α influences how many measured data values ​​(e.g. hits, e.g. values ​​of a target variable, e.g. target variable t, e.g. values ​​of an independent variable, e.g. independent variable v) are quantized, e.g. binned, into value bins, e.g. quantization levels.

[0031] Furthermore, the measurement data, e.g. the measurement conditions and / or the measurement results, comprise continuous variables, and the method further comprises quantizing values ​​of the continuous variables into value bins.

[0032] Alternatively or additionally, the measurement data includes categorical variables, and each categorical variable value is associated, eg, binned, eg, assigned to one respective value bin.

[0033] Alternatively or additionally, the measurement data includes a discrete variable, the discrete variable including M distinct values, and the values ​​of the discrete variable being quantized into up to M value bins.

[0034] According to a further embodiment of the invention, the measurement data comprises several data sets (e.g. several measurement results and corresponding measurement conditions; e.g. several test cases) of the DUT test. Furthermore, the number of value bins, e.g. quantization levels, of the measurement data is determined using the number of data sets and using a partition based on the number of combinations of categorical variable values ​​(e.g. splitting by the number of combinations of categorical variable values), and / or using the number of data sets and using a partition based on the parameter α (e.g. splitting by the parameter α); and / or using the number of data sets and using a partition based on the number of continuous and discrete variables (e.g. using a square root); and / or using several data sets and using a partition based on several continuous variables (e.g. using a square root), e.g. several partitions can be combined in a joint or combinatorial process.

[0035] According to a further embodiment of the invention, the measurement data comprises measurement results (e.g. target variable values; e.g. a target variable and its value) and values ​​of one or more corresponding independent variables, and the method further comprises a step of determining a first subset of measurement conditions such that the measurement conditions of the first subset are, e.g. adjacent to a neighborhood of the first measurement condition; e.g. nearest neighbors of the first measurement condition; e.g. adjacent to the independent variable value.

[0036] According to a further embodiment of the invention, the measurement data comprises measurement results, e.g. a target variable and its value, as well as values ​​of one or more corresponding independent variables, and the method further comprises binning values ​​of the measurement data, wherein the measurement results and / or the independent variable values ​​of the measurement data are quantized, e.g. binned, into value bins.

[0037] The method further includes determining information, eg, probability; eg, Q(t), regarding the overall distribution of the measurement results based on the measurement results for each, eg, all, value bins.

[0038] Further, the method includes determining information regarding a conditional distribution of measurement results for a first subset of measurement conditions, the first subset of measurement conditions being associated with independent variable values ​​for each value bin, and the method includes determining a number of measurement results, e.g., n(v), corresponding to the first subset of measurement conditions.

[0039] Further, the method includes generating a plurality of empirical distributions of the measurement outcomes, e.g., q s wherein the number of measurement results considered for determining the information about the respective empirical distribution is equal to the number of measurement results of (e.g. corresponding to, e.g. associated with) the first subset of measurement conditions, e.g. n(v), and the measurement results of each subset of measurement data is a randomly or pseudo-randomly selected set of measurement results of the measurement data.

[0040] The method further includes using information about the conditional distributions, e.g., the conditional distributions themselves, and using information about the respective empirical distributions to determine a plurality of dissimilarity measures, e.g., Kullback-Leibler divergence, between the conditional distributions and respective empirical distributions, e.g., each of the empirical distributions.

[0041] The method further includes determining a cumulative distribution, e.g., a CDF, of the plurality of dissimilarity measures and determining a statistically significant dissimilarity value, e.g., d(v), such that only a predetermined portion of the empirical distribution includes dissimilarity measures for a conditional distribution that are less than or equal to the statistically significant dissimilarity value, and the method further includes determining a statistically significant dissimilarity value, e.g., d(v), of the plurality of dissimilarity measures. s wherein the selected dissimilarity measure comprises a statistically significant dissimilarity value, e.g., a minimum deviation for d(v), from a dissimilarity measure among the plurality of dissimilarity measures.

[0042] Further, the method includes the steps of determining a mean of an empirical distribution corresponding to the selected dissimilarity measure, and determining a mean of a conditional distribution of the measurement results corresponding to a first subset of measurement conditions, e.g.

number

number

number

[0043] Additionally, the method uses an average alignment dissimilarity measure, e.g.

number

number

[0044] Furthermore, information about the characteristics of the DUT can be obtained by normalized dissimilarity ratios, e.g.

number

[0045] According to a further embodiment of the invention, the measurement data comprises measurement results, for example a target variable and its value as well as values ​​of one or more corresponding independent variables.

[0046] Further, the method includes a step of binning values ​​of the measurement data, wherein measurement results and / or independent variable values ​​of the measurement data are quantized, e.g. binned, into value bins, and the method includes a step of determining information about the overall distribution of the measurement results, e.g. Q(t), based on the measurement results for each, e.g. all, value bins.

[0047] Further, the method includes determining information regarding a conditional distribution of measurement results for a first subset of measurement conditions, the first subset of measurement conditions being associated with independent variable values ​​for each value bin, and the method includes determining a number of measurement results, e.g., n(v), corresponding to the first subset of measurement conditions.

[0048] Further, the method includes generating a plurality of empirical distributions of the measurement results for a plurality of subsets of the measurement data, e.g., q s wherein the number of measurement results considered for determining the information on the respective empirical distribution is equal to the number of measurement results of (e.g. corresponding to, e.g. associated with) the first subset of measurement conditions. Further, the measurement results of each subset of measurement data is a randomly or pseudo-randomly selected set of measurement results of the measurement data.

[0049] The method further includes using information about the conditional distributions, e.g., the conditional distributions themselves, and using information about the respective empirical distributions to determine a plurality of dissimilarity measures, e.g., Kullback-Leibler divergence, between the conditional distributions and the respective empirical distributions, e.g., each of the empirical distributions, and the method includes determining a cumulative distribution, e.g., a CDF, of the plurality of dissimilarity measures.

[0050] Furthermore, the method includes determining a statistically significant dissimilarity value, such as d(v), such that only a predetermined portion of the empirical distribution is included in the dissimilarity measure for the conditional distribution that is below the statistically significant dissimilarity value, and the method includes selecting one of the plurality of dissimilarity measures, such as D s’ where the selected dissimilarity measure includes the minimum deviation from the statistically significant dissimilarity value, such as d(v), of the dissimilarity measures among the plurality of dissimilarity measures, and the information regarding the characteristics of the DUT includes the selected dissimilarity measure.

[0051] According to a further embodiment of the invention, a plurality of statistically significant dissimilarity values, such as d(v), and / or a plurality of normalized dissimilarity ratios, such as μ'; for example, μ is determined for different subsets of measurement conditions corresponding to each value bin.

[0052] According to a further embodiment of the invention, the method includes obtaining, such as receiving or determining, information, such as Q, regarding the overall distribution of the measurement results based on the measurement data, and comparing the information regarding the overall distribution with the information regarding the conditional distribution, such as p v <Q or p v further including determining whether p v v <Q holds, and determining at least one sample distribution information, such as q', based on the comparison between the overall distribution information, such as Q, and the conditional distribution information, such as p (for example, when p v <Q, determining q' according to the largest k' that satisfies Pr(X≦k')≦1-C, or, for example, when p

[0053] The method includes comparing the information regarding the overall distribution, the information regarding the conditional distribution, and the information regarding at least one sample distribution (for example, whether p v <q'<Q or q'≦p v <Q; or whether Q≦p​v whether q’ ≤ or Q < q’ < p v a step of determining whether), and based on a comparison of information about the overall distribution, information about the conditional distribution, and information about at least one sample distribution, e.g., depending thereon, (e.g., p v when < q’ < Q

Number

Number

[0054] According to a further embodiment of the present invention, the measurement data includes measurement results, e.g., a target variable and / or its value, and corresponding independent variable values (e.g., independent variable value v; e.g., of the corresponding independent variable V; e.g., and further the definition of the corresponding independent variable; e.g., represented together by V), the method further includes binned values of the measurement data, and the measurement results and / or independent variable values of the measurement data are binned into value bins, e.g., quantized, and the measurement results are boolean target variables having true or false measurement results.

[0055] The method further includes a step of determining information Q about the overall distribution of the measurement results, where information Q is the probability that the measurement result is true or the probability that the measurement result is false, based on the overall measurement results of the measurement data, and information Q may represent, e.g., a predetermined proportion of the measurement results including "true" values within the total amount of the measurement results.

[0056] Furthermore, the method includes determining the number n(v) of measurement results corresponding to a first subset of measurement conditions, for example corresponding to the independent variable value v, where the first subset of measurement conditions is associated with the independent variable values of the respective value bins, and the method determines information p regarding the conditional distribution of the measurement results corresponding to the first subset of measurement conditions v including determining information p v which is the probability that the measurement result is true based on the measurement results corresponding to the first subset of measurement conditions, or information p v is the probability that the measurement result is false based on the measurement results corresponding to the first subset of measurement conditions.

[0057] Furthermore, the method includes determining whether p v < Q holds, and / or whether p v > Q holds, and / or whether p v = Q holds, and / or whether p v ≧ Q holds.

[0058] If p v < Q holds, the method includes determining information q' regarding the sample distribution using q' = k' / n(v), where k' is the maximum number of true test cases satisfying Pr(X ≦ k') ≦ 1 - C, C is the confidence value, and Pr(X ≦ k') ≦ 1 - C means that the probability Pr of having k' or fewer positive measurement results X when selecting n(v) arbitrary measurement results from the measurement data (for example, results from the overall set of measurement results; for example, considering the above information Q; the method can determine k' using, for example, the evaluation of the binomial distribution using the number of samples n(v) and the individual probabilities defined by Q) is less than or equal to 1 minus the confidence level C, and p v If < q' < Q holds

Number

[0059] Alternatively or additionally, p v When p≥Q holds, the method includes the step of determining information q’ about the sample distribution using q’=k’ / n(v) according to the smallest k’ that satisfies Pr(X≤k’)≤C, where Pr(X≤k’)≤C is the condition that the probability Pr of having k’ or fewer positive measurement results X when selecting n(v) arbitrary measurement results from the measurement data (e.g., results from the overall set of measurement results; e.g., considering the above information Q; the method can determine k’ using the evaluation of the binomial distribution using, for example, the number of samples n(v) and the individual probabilities defined by Q), and when Q≤p v ≤q’ holds, the step of setting the statistically significant dissimilarity value d(v) to d(v)=0, or when Q<q’<p v holds,

Number

[0060] Alternatively or additionally, p v When p = Q holds, the method includes the step of setting the statistically significant dissimilarity value d(v) to d(v)=0.

[0061] According to a further embodiment of the present invention, the method further includes the step of determining a plurality of respective statistically significant dissimilarity values, such as d(v), for different subsets of measurement conditions, for example, those binned into respective value bins.

[0062] According to a further embodiment of the invention, the method further comprises a step of augmenting the measurement data using a statistically significant dissimilarity value, e.g. d(v), where the augmented measurement data is a combination of the measurement data itself and associated measurement data meta information, e.g. augmented information, where the associated measurement data meta information is determined based on the statistically significant dissimilarity value, e.g. d(v), where the measurement data meta information comprises information on a statistically significant dissimilarity between the measurement results corresponding to the first subset of measurement conditions and the overall measurement results or between the measurement results corresponding to the first subset of measurement conditions and the measurement results of the selected sample distribution.

[0063] According to a further embodiment of the invention, the method further comprises the step of:

number

[0064] Furthermore, the measurement data meta information includes information on normalized dissimilarity ratios (thus including, for example, information on the amount of dissimilarity measures associated with different means between the conditional distribution and the overall distribution, or between the conditional distribution and the sample distribution).

[0065] According to a further embodiment of the invention, the method further comprises a visualising step, e.g. plotting step, of the measured data and the measured data meta-information, which may also be considered as extended information, e.g. based on statistically significant dissimilarity values, e.g. d(v).

[0066] Further, e.g., the visualization, e.g., plot, of the extended measurement data may include a representation of the measurement results across measurement conditions or across corresponding independent variables, e.g. across corresponding independent variable values, and the visualization (e.g., the representation of the measurement results) may include highlighting, such that measurement results of the corresponding conditional distribution that deviate statistically significantly from the overall distribution of the measurement results or that deviate from a sample distribution, e.g., an empirical distribution, of the measurement results are highlighted; and / or may depend on (e.g., be based on; e.g., be associated with) the measurement data meta information (e.g., corresponding measurement data meta information of the respective measurement result), such that areas of the visualization associated with measurement results of the corresponding conditional distribution that deviate statistically significantly from the overall distribution of the measurement results or that deviate from a sample distribution, e.g., an empirical distribution, of the measurement results are highlighted.

[0067] According to a further embodiment of the invention, the highlighting comprises colouring the background of the plot, the saturation of the colouring being determined according to the statistically significant dissimilarity values.

[0068] According to a further embodiment of the invention, the color of the coloring is determined according to a normalized dissimilarity ratio. Alternatively or additionally, the color of the coloring is determined according to a comparison of information about the global distribution of the measurement results with information about the conditional distribution. Alternatively or additionally, the color of the coloring is determined according to a comparison of information about the sample distribution of the measurement results with information about the conditional distribution.

[0069] According to a further embodiment of the invention, the method further comprises the step of visualising the measurement data using statistically significant dissimilarity values.

[0070] A further embodiment according to the invention comprises a computer program for carrying out the method according to any one of the preceding claims when the computer program is run on a computer.

[0071] Further embodiments according to the invention include an apparatus for determining information about characteristics of one or more devices under test (DUTs) (e.g., information that "the device under test behaves differently under these conditions with statistical significance") using measurement data from an automatic test equipment (ATE), the measurement data including a plurality of measurement results (e.g., a target variable, e.g. T, e.g. "test pass", e.g. "test fail", e.g. a continuous value, e.g. a voltage or current or a signal characteristic) and information describing corresponding measurement conditions of the device under test (DUT) (e.g. describing a condition of a corresponding independent variable, e.g. V=v, e.g. describing the condition that the independent variable V has value v; e.g. describing the measurement conditions under which the measurement results were obtained).

[0072] The apparatus may, for example, compute a conditional distribution (e.g., in the case of a non-Boolean target variable, a local conditional target distribution, e.g., p v (t), or, for example, in the case of a Boolean target variable, the true target value p v The method is configured to determine information relating to the information representing or describing the ratio of the number of pixels to the number of pixels.

[0073] Further, the apparatus may, for example, select at least one sample distribution (e.g., in the case of a non-Boolean target variable, a plurality of empirical distributions q s Or q s(t), e.g., in the case of a Boolean target variable, a probability q'), and wherein at least one sample distribution (e.g., in the case of a Boolean target variable, one analytically determined representative sample distribution, or in the case of a non-Boolean target variable, multiple empirical sample distributions) is comparable, e.g., to a conditional distribution (e.g., associated with a wider range of measurement conditions than the conditional distribution, e.g., including an equal or similar number of measurement outcomes).

[0074] Further, the apparatus is configured to determine, e.g., using information about the information representing or describing the conditional distribution, and, e.g., using information about the information representing or describing at least one sample distribution, a statistically significant dissimilarity value (e.g., which indicates how the behavior of the DUT depends on measurement conditions; e.g., d(v); e.g., which characterizes the behavior of the DUT for given measurement conditions; e.g., describing a statistically significant dissimilarity between the conditional distribution and one or more sample distributions; e.g., determining a dissimilarity threshold, e.g., d(v), such that only a predetermined portion, e.g., 5%, of the sample distributions include dissimilarities for the conditional distribution that are below the dissimilarity threshold; e.g., determining a dissimilarity threshold such that only a predetermined portion, e.g., 5%, of all cases of the sample distribution, e.g., binomial binomial distribution, include dissimilarity for the conditional distribution that are below the dissimilarity threshold; e.g., determining a dissimilarity threshold indicating whether the conditional distribution includes excluding a predetermined percentage of the sample distributions that are most similar to the conditional distribution, indicating a minimum dissimilarity threshold between the conditional distribution and the sample distribution).

[0075] Further, the information regarding the characteristics of the one or more DUTs includes a statistically significant dissimilarity value or is based on a statistically significant dissimilarity value (e.g., the apparatus is configured to determine a statistically significant dissimilarity value as information regarding the characteristics of the one or more DUTs, or the apparatus is configured to determine information regarding the characteristics of the one or more DUTs based on the statistically significant dissimilarity value).

[0076] The above described device is based on the same considerations as the above described method, and the device can also be completed with all the features and functions that are also described with respect to the device.

[0077] The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which: [Brief description of the drawings]

[0078] [Figure 1] FIG. 1 shows a schematic block diagram of a method according to an embodiment of the present invention. [Diagram 2] FIG. 2 shows a schematic block diagram of a method having optional features according to an embodiment of the present invention. [Diagram 3] 4 shows a schematic block diagram of a method according to an embodiment of the present invention with further optional features. [Figure 4] FIG. 1 shows a schematic block diagram of another method having optional features according to an embodiment of the present invention. [Diagram 5] 4 shows a block diagram of a further method according to an embodiment of the invention; [Figure 6] 1 shows a block diagram of a method including a comparison step according to an embodiment of the present invention; [Figure 7] 4 shows a block diagram of a further method including a comparison step according to an embodiment of the present invention; [Figure 8] FIG. 1 shows a schematic block diagram for measurement data enhancement according to an embodiment of the present invention. [Figure 9] 1 shows a schematic diagram of an apparatus according to an embodiment of the present invention. [Figure 10] Here is an example of a scatter plot with a Boolean target variable (true: error, blue: no error). [Figure 11] 1 illustrates an example of a statistically significant divergence according to an embodiment of the present invention. [Figure 12] 1 shows an example of a probability density according to an embodiment of the present invention. [Figure 13] 13A-13C show examples of highlighted regions of independent variables for various plot types where the local target distribution differs from its global distribution in accordance with embodiments of the present invention. [Figure 14] 1 illustrates an example of a scatter plot using a Boolean target variable in accordance with an embodiment of the present invention. [Figure 15] 4 illustrates an example of saturation highlighting according to an embodiment of the present invention. [Figure 16] 1 shows some example plots according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0079] Equivalent or equivalent elements, or elements having equivalent or equivalent functionality, are designated in the following description with the same or equivalent reference numbers, even if they occur in different figures.

[0080] In the following description, numerous details are set forth to provide a more thorough description of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring the embodiments of the present invention. Furthermore, features of different embodiments described hereinafter may be combined with each other unless otherwise specified.

[0081] 1 shows a schematic block diagram of a method according to an embodiment of the present invention. Method 100 is for determining information about characteristics of one or more devices under test (DUTs) using measurement data from automatic test equipment (ATE), the measurement data including a plurality of measurement results and information describing corresponding measurement conditions of the devices under test (DUTs).

[0082] The method 100 includes determining 110 information about a conditional distribution of measurement results corresponding to a first subset of measurement conditions, determining 120 information about at least one sample distribution that is comparable to the conditional distribution, and determining 130 a statistically significant dissimilarity value using the information about the conditional distribution and using the information about the at least one sample distribution. Furthermore, the information about the characteristic of the one or more DUTs includes or is based on the statistically significant dissimilarity value.

[0083] 2 shows a schematic block diagram of a method having optional features according to an embodiment of the present invention. Based on measurement data 210 from an automatic test equipment (ATE), measurement data is determined that includes a plurality of measurement results, information describing corresponding measurement conditions, information on a conditional distribution of the measurement results corresponding to a first subset of the measurement conditions (block 220), and information on at least one sample distribution that is comparable to the conditional distribution (block 230).

[0084] As an optional feature, determining the information regarding the at least one sample distribution may comprise determining information regarding a plurality of empirical distributions of measurement results for a plurality of subsets of the measurement data, the number of measurement results taken into account for the determination of the information regarding the respective empirical distribution being related to the number of measurement results taken into account for the determination of the information regarding the conditional distribution.

[0085] Furthermore, the measurements of each subset of the measurement data may be, for example, a randomly or pseudo-randomly selected set of the measurement data.

[0086] As another optional feature, the method may include using information about the conditional distributions and using information about the respective empirical distributions to determine a number of dissimilarity measures between the conditional distributions and the respective empirical distributions (block 240).

[0087] As an example, the method further includes determining a statistically significant dissimilarity value using multiple dissimilarity measures (Block 260).

[0088] Optionally, statistically significant dissimilarity values ​​may be associated with quantiles of the distribution of dissimilarity measures.

[0089] As another optional feature, the method may further include determining a cumulative distribution of the plurality of dissimilarity measures and determining a statistically significant dissimilarity value (block 250) such that only a predetermined portion of the empirical distribution includes dissimilarity measures for the conditional distribution that are less than or equal to the statistically significant dissimilarity value.

[0090] Optionally, the method may further include extrapolating the cumulative distribution and / or interpolating the cumulative distribution to determine statistically significant dissimilarity values ​​(e.g., at block 250).

[0091] FIG. 3 shows a schematic block diagram of a method according to an embodiment of the present invention, with further optional features.

[0092] As shown in FIG. 3, an optional binning step 310 may be performed based on measurement data 210 from an automatic test equipment (ATE) that includes multiple measurement results.

[0093] The binning may be performed based on a parameter α (not shown), which affects how many measurement data values ​​are quantized into value bins. Furthermore, the measurement data 310 may optionally include continuous variables, and the method may further include quantizing values ​​of the continuous variables into value bins. Alternatively or additionally, the measurement data 310 may include categorical variables, and each categorical variable value may be associated with a respective value bin. Alternatively or additionally, the measurement data may include discrete variables, and the discrete variables may include M distinct values, and the values ​​of the discrete variables are quantized into up to M value bins.

[0094] Optionally, the measurement data 210 may include several data sets of DUT tests, and the number of value bins of the measurement data may be determined using the number of data sets and using a partitioning based on the number of combinations of categorical variable values, and / or using the number of data sets and using a partitioning based on the parameter α, and / or using the number of data sets and using a partitioning based on the number of continuous and discrete variables, and / or using the number of data sets and using a partitioning based on the number of continuous variables.

[0095] For example, based on the optionally binned measurement data, information may be determined regarding a conditional distribution of measurement results corresponding to a first subset of measurement conditions (block 220), and information regarding at least one sample distribution that is comparable to the conditional distribution (block 230).

[0096] As another optional feature corresponding to the example of FIG. 2, determining information about at least one sample distribution may include determining information about a plurality of empirical distributions.

[0097] Using information about the conditional distributions and using information about at least one sample distribution, as an example, a statistically significant dissimilarity value may be determined (block 250) based on determining multiple dissimilarity measures (block 240).

[0098] Thus, information regarding the characteristics of one or more DUTs may include or be based on a statistically significant dissimilarity value.

[0099] Further, optionally, as shown, the method further comprises selecting one dissimilarity measure among a plurality of dissimilarity measures, e.g., D s’ A selected dissimilarity measure, e.g., D s’may contain a statistically significant dissimilarity value, e.g., the minimum deviation for d(v). This is indicated by block 320.

[0100] Optionally, the information about the characteristics of the one or more DUTs may include information about a selected dissimilarity measure. Thus, a selected dissimilarity measure, e.g., D s’ may provide information about one or more DUTs or allow information about one or more DUTs to be evaluated.

[0101] Further, optionally, a selected dissimilarity measure (e.g., with index s′), e.g., D s ' may be determined (block 330), optionally using the mean of the conditional distribution of the measurement results corresponding to the first subset of measurement conditions, e.g.

number

[0102] Additionally, as optionally shown, an average alignment dissimilarity measure, e.g.

number

number

number

[0103] The chosen dissimilarity measure, e.g. D s’ , and the average alignment dissimilarity measure, e.g.

number

number

[0104] Thus, information regarding the characteristics of one or more DUTs may include information regarding a normalized dissimilarity ratio, eg, μ′.

[0105] FIG. 4 shows a schematic block diagram of another method having optional features according to an embodiment of the present invention.

[0106] 1, the method of the present invention may include determining 220, 430 information regarding at least one sample distribution and information regarding the conditional distribution. Additionally, the method may include determining (block 450) a statistically significant dissimilarity value, e.g., d(v), using the information regarding the conditional distribution and using the information regarding the at least one sample distribution.

[0107] As optionally shown in FIG. 4 , the method further comprises: computing a mean value of a conditional distribution of the measurement results corresponding to a first subset of measurement conditions, e.g.

number

number

[0108] Further, the method optionally includes calculating a mean of a conditional distribution of the measurement results corresponding to a first subset of measurement conditions, e.g.,

number

number

number

[0109] Further, the method may optionally include determining (470) a dissimilarity measure, e.g., D, between the conditional distribution and the global distribution.

[0110] Further, optionally, the method comprises calculating an average alignment dissimilarity measure, e.g.

number

number

[0111] Thus, information regarding the characteristics of one or more DUTs may include information regarding a normalized dissimilarity ratio, eg, μ.

[0112] Optionally (not shown), the measurement data may include several data sets of the DUT tests, and the number of value bins of the measurement data is determined using the number of data sets and using a partitioning based on the number of combinations of categorical variable values, and / or using the number of data sets and using a partitioning based on the parameter α, and / or using the number of data sets and using a partitioning based on the number of continuous and discrete variables, and / or using the number of data sets and using a partitioning based on the number of continuous variables.

[0113] Optionally, the measurement data may include measurement results and values ​​of one or more corresponding independent variables, and the method may further include determining a first subset of measurement conditions such that the measurement conditions of the first subset are contiguous.

[0114] 5 shows a block diagram of a further method according to an embodiment of the invention. The method 500 comprises a step 501 of binning values ​​of measurement data, whereby measurement results and / or independent variable values ​​of the measurement data are quantized, e.g. binned, into value bins.

[0115] The method 500 further includes determining 502 information, eg, probability; eg, Q(t), about the overall distribution of the measurements based on the measurements of each, eg, all, value bins.

[0116] Further, the method 500 includes step 503 of determining information regarding a conditional distribution of measurement results for a first subset of measurement conditions, where the first subset of measurement conditions is associated with an independent variable value for each value bin, and step 504 of determining a number of measurement results, e.g., n(v), corresponding to the first subset of measurement conditions.

[0117] Additionally, the method 500 may include generating multiple empirical distributions of measurement outcomes, e.g., q swherein the number of measurement results taken into account for determining the information about the respective empirical distributions is equal to the number of associated measurement results, e.g. n(v), e.g. corresponding to a first subset of measurement conditions, and the measurement results of each subset of measurement data are a randomly or pseudo-randomly selected set of measurement results of the measurement data.

[0118] Further, the method 500 includes a step 506 of determining, using information about the conditional distributions, e.g., the conditional distributions themselves, and using information about the respective empirical distributions, a plurality of dissimilarity measures, e.g., Kullback-Leibler divergence, between the conditional distributions and the respective empirical distributions, e.g., each of the empirical distributions.

[0119] The method 500 further includes step 507 of determining a cumulative distribution, e.g., a CDF, of the multiple dissimilarity measures, and step 508 of determining a statistically significant dissimilarity value, e.g., d(v), such that only a predetermined portion of the empirical distribution includes dissimilarity measures for the conditional distribution that are less than or equal to the statistically significant dissimilarity value.

[0120] Additionally, the method 500 may further comprise: s’ The method includes step 509 of selecting a dissimilarity measure that includes a statistically significant dissimilarity value, e.g., the smallest deviation for d(v), from a dissimilarity measure among the plurality of dissimilarity measures, and step 510 of determining a mean of an empirical distribution corresponding to the selected dissimilarity measure.

[0121] Additionally, the method 500 may further include calculating a mean of a conditional distribution of the measurement results corresponding to the first subset of measurement conditions, e.g.

number

[0122] Additionally, the method 500 may include a mean alignment dissimilarity measure, e.g.

number

number

[0123] 5 should be considered optional, it being noted that, according to an embodiment, certain steps 509 to 512 may not be performed, e.g., may be skipped. Thus, the information regarding the characteristics of the DUT may include a selected dissimilarity measure.

[0124] With reference to all of the aforementioned methods, optionally, multiple statistically significant dissimilarity values, e.g., d(v), and / or multiple normalized dissimilarity ratios, e.g., μ′; e.g., μ, may be determined for different subsets of measurement conditions corresponding to each value bin.

[0125] Fig. 6 shows a block diagram of a method including a comparison step according to an embodiment of the present invention. Fig. 6 shows a method 600, which includes a step 601 of obtaining information about an overall distribution of measurement results based on measurement data, and a step 602 of comparing the information about the overall distribution with information about a conditional distribution, and information about at least one sample distribution is determined based on the comparison of the information about the overall distribution with the information about the conditional distribution.

[0126] Furthermore, the method includes a step 603 of comparing information regarding the overall distribution, information regarding the conditional distribution, and information regarding at least one sample distribution, and a step 604 of determining a statistically significant dissimilarity value based on the comparison of the information regarding the overall distribution, the information regarding the conditional distribution, and the information regarding at least one sample distribution.

[0127] FIG. 7 shows a block diagram of a further method including a comparison step according to an embodiment of the present invention. FIG. 7 shows a method 700, which includes a step 701 of binning the values of measurement data, where the measurement results and / or independent variable values of the measurement data are binned into value bins, and the measurement results are boolean target variables having true or false measurement results.

[0128] The method 700 further includes a step 702 of determining information Q regarding the overall distribution of the measurement results, where the information Q is the probability that the measurement result is true or the probability that the measurement result is false based on the overall measurement results of the measurement data, and a step 703 of determining the number n(v) of measurement results corresponding to a first subset of measurement conditions, where the first subset of measurement conditions is associated with the independent variable values of the respective value bins.

[0129] Furthermore, the method 700 includes a step 704 of determining information p regarding the conditional distribution of the measurement results corresponding to the first subset of measurement conditions v where the information p v is the probability that the measurement result is true based on the measurement results corresponding to the first subset of measurement conditions, or the information p v is the probability that the measurement result is false based on the measurement results corresponding to the first subset of measurement conditions.

[0130] Furthermore, the method 700 includes whether p v < Q holds, and / or whether p v > Q holds, and / or whether p v = Q holds, and / or whether p vIt includes step 705 of determining whether ≧Q holds.

[0131] p v If Q holds, the method includes step 706 of determining information q' about the sample distribution using q' = k' / n(v), where k' is the maximum number of true test cases satisfying Pr(X≦k')≦1 - C, C is the confidence value, and Pr(X≦k')≦1 - C is the condition that the probability Pr of having k' or fewer positive measurement results X when selecting n(v) arbitrary measurement results from the measurement data is less than or equal to 1 minus the confidence level C.

[0132] Furthermore, p v If q' < Q holds, the method

Number

[0133] Alternatively or additionally, if q' ≦ p v If Q holds, the method includes step 708 of setting the statistically significant dissimilarity value d(v) to d(v) = 0.

[0134] Alternatively or additionally, p v If ≧Q holds, the method includes step 709 of determining information q' about the sample distribution using q' = k' / n(v) according to the smallest k' satisfying Pr(X≦k')≦C, where Pr(X≦k')≦C is the condition that the probability Pr of having k' or fewer positive measurement results X when selecting n(v) arbitrary measurement results from the measurement data is less than or equal to the confidence level C.

[0135] Furthermore, if Q ≦ p v ≦ q' holds, the method includes step 710 of setting the statistically significant dissimilarity value d(v) to d(v) = 0, or if Q < q' < p v holds, the method

Number

[0136] Alternatively or additionally, p v If .gamma.=Q holds, the method includes a step 712 of setting the statistically significant dissimilarity value d(v) to d(v)=0.

[0137] It should be noted that method 700 may optionally include determining a plurality of respective statistically significant dissimilarity values ​​for different subsets of measurement conditions corresponding to respective value bins.

[0138] A schematic block diagram for augmenting measured data according to an embodiment of the present invention is shown in Figure 8. As shown in Figure 8, a method according to an embodiment can include augmenting measured data 810 based on or using measured data 820 and a statistically significant dissimilarity value 830, e.g., d(v).

[0139] The extended measurement data may be a combination of the measurement data 820 itself and associated measurement data meta-information, which is determined based on the statistically significant dissimilarity values ​​930. Furthermore, the measurement data meta-information may include information on a statistically significant dissimilarity between the measurement results corresponding to the first subset of measurement conditions and the overall measurement results, or between the measurement results corresponding to the first subset of measurement conditions and the measurement results of the selected sample distribution. The determination of the measurement data meta-information may be performed in block 810.

[0140] Furthermore, it should be noted that, optionally, to determine a normalized dissimilarity ratio, e.g. μ or μ′, a mean aligned dissimilarity measure and a dissimilarity measure between the conditional distribution and the overall distribution of measurement results may be used, where the normalized dissimilarity ratio contains information about the amount of dissimilarity measure associated with the different means between the conditional distribution and the overall distribution.

[0141] Alternatively, a dissimilarity measure between the conditional distribution and a sample distribution of the measurement results, e.g. an empirical distribution, may be used to determine a normalized dissimilarity ratio, e.g. μ or μ′, the normalized dissimilarity ratio comprising information on the amount of dissimilarity measure associated with the different means between the conditional distribution and the sample distribution. Furthermore, the measurement data meta information may comprise information on the normalized dissimilarity ratio.

[0142] Fig. 9 shows a schematic diagram of an apparatus according to an embodiment of the present invention. Fig. 9 shows an apparatus 900 for determining information about characteristics of one or more devices under test (DUTs).

[0143] The apparatus 900 may optionally be configured to receive measurement data 902 from automatic test equipment (ATE), the measurement data including a plurality of measurement results and information describing corresponding measurement conditions of a device under test (DUT).

[0144] The apparatus further comprises a conditional distribution information determination unit 910 configured to determine information regarding a conditional distribution of measurement results corresponding to a first subset of measurement conditions, and a sample distribution information determination unit 920 configured to determine information regarding at least one sample distribution that is comparable to the conditional distribution.

[0145] Furthermore, the apparatus 900 comprises a statistically significant dissimilarity value determination unit 930 configured to determine a statistically significant dissimilarity value using information on the conditional distribution and using information on at least one sample distribution, and the information on the characteristic of the one or more DUTs includes or is based on the statistically significant dissimilarity value. Thus, as optionally shown, the apparatus 900 may comprise a characteristic information determination unit 940.

[0146] In the following, further embodiments of the present invention are disclosed. It should be noted that any of the features, functions, and details thereof can be used individually or in combination with any of the above-mentioned embodiments. Moreover, the embodiments of the present invention can be described in different words.

[0147] In general, embodiments in accordance with the present invention may include or enable the provision of plots with informational highlighting, and thus the following section may be entitled, for example, "Plots with Informational Highlighting."

[0148] The following may provide a brief overview of some embodiments of the present invention.

[0149] Plots should often convey how device characteristics depend on operating conditions. Traditional plots suffer from several challenges to this task, namely, for example, (1) obfuscation, (2) statistical variation can masquerade as non-existent dependencies, (3) visual bias due to uneven distribution of test cases, and (4) difficulty in visually estimating percentages or distributions.

[0150] Embodiments according to the invention include or suggest highlighting regions of the independent variables where the target distribution differs, e.g., statistically significantly, from, e.g., the overall target distribution. According to embodiments, for example, two highlighting techniques are disclosed or suggested that can be applied to all common plot types. Calculation of highlighting parameters is described.

[0151] Section 1: In the following, motivation for embodiments in accordance with the present invention may be presented.

[0152] In post-silicon validation, plots may be used frequently, for example, to attempt to understand how a target variable depends on an independent variable without being misled by statistical fluctuations.

[0153] As an example, if the target variable is a Boolean error variable, one may, for example, want to know how the error probability depends on the independent variables. Note that the error probability may be very low, for example 1% or even 0.1%. As an example, if the target variable is a continuous performance variable, one may, for example, want to know how the distribution of the target variable depends on the independent variables, for example not only in terms of its mean but also in terms of the distribution shape. The independent variables may, for example, be a continuous supply voltage, or a discrete numeric register setting or a categorical device mode.

[0154] Traditional plots, for example, may face several challenges for this task (note that any of the following challenges may be addressed in embodiments of the present invention): obfuscation; statistical variation may appear to be dependencies where none exist; uneven distribution of test cases may lead to visual bias; visual estimation of proportions or distributions may be difficult.

[0155] In the following discussion, we refer to Figure 10, which shows an example of a scatter plot with a Boolean target variable (True: Error, Blue: No Error). Rare errors can be hard to see.

[0156] We next illustrate these challenges for some common plot types (note that any of the following plot types may be addressed or used in embodiments of the present invention).

[0157] Scatter plot with Boolean target as color. Red dots for error cases as in Figure 10. Figure 10 shows plot of voltage test cases against temperature. Obfuscation: A large number of good test cases may hide, for example, rare error cases. If the problem is rare, it may be difficult to visually determine, for example, whether a clustering of a small number of error cases is statistically significant or simply an artifact of statistical fluctuations. If the test cases are unevenly distributed, it may be difficult to visually determine, for example, whether an (x,y) region appears more red because of a higher proportion of error cases in that region or simply because there are more test cases in this region. For example, it may be difficult to estimate and compare error percentages across (x,y).

[0158] Scatter plot with continuous target on y-axis: If test cases are unevenly distributed along x, it may be difficult to visually determine, for example, whether a wider spread of y values ​​in some x-regions is due to higher variance or simply due to more test cases in this x-region. For example, it may be difficult to visually determine which value differences are due to statistical outliers and which differences are due to statistically significant deviations. Target distributions may be difficult to estimate and compare, for example, across x. If test cases are dense, obfuscation may make this estimation difficult, for example. If test cases are sparse, statistical variation may make this estimation difficult as well, for example.

[0159] Scatter plot with continuous target as color and independent variables along x and y. Rare outliers may be hidden by many good test cases, for example. If test cases are unevenly distributed along (x,y), it may be difficult to visually determine, for example, whether a wider spread of color values ​​is due to higher variance or simply more test cases. It may be difficult to visually determine, for example, which color differences are due to statistical outliers or statistically significant deviations. The distribution of color values ​​may be extremely difficult to grasp, let alone compare them across (x,y), for example. Even the average target value (color) may be difficult to visually estimate, for example.

[0160] Stacked bars with error / non-error stacks, one bar per value of one independent variable. For example, it may be difficult to compare error rates across various bar heights. On the other hand, full bars, which may be normalized to equal heights, may not convey the number of test cases, and therefore may not indicate how reliable the rate estimates are, for example. For example, it may be difficult to determine whether different error rates can be explained by statistical variation or may or even should be assumed to represent statistically significant deviations.

[0161] Stacked distributions, for example, can have the same challenges as stacked bars.

[0162] Box plots may facilitate comparison of, for example, means and quantiles, but may hide, for example, details of the distribution.

[0163] Section 2: In the following, objectives according to the embodiments are disclosed. In other words, the embodiments according to the present invention may satisfy some or even all of the following characteristics:

[0164] One high level objective according to the embodiment may be to help the user to better understand how the target distribution depends on the independent variables, e.g., statistically significantly. We may assume, or according to the embodiment, for example, that the global target distribution may already be known. Regions of the independent variables that follow this global target distribution may not be of interest, e.g., of interest. We may wish to direct the user's attention, or as an example according to the embodiment, the user's attention is directed, e.g., using highlighting, to regions where the local target distribution is, e.g., statistically significantly different from the global target distribution, or from the sample distribution, e.g., of interest. The local target distribution may have, e.g., a different mean than the global distribution, or the same mean but different variance. For example, the difference between both or a combination of both may be of interest.

[0165] The user's attention requirements may be, for example:

[0166] Suppress statistical fluctuations and show only statistically significant deviations from the overall target distribution; i.e., reflect uncertainty due to a small number of local test cases (optional); independence of test case distributions along the independent variables; i.e., use conditional distributions (optional); detect different target distributions even when the mean is unchanged (optional); avoid masking effects that hide information (optional); the highlighting scheme may support many plot types, or even all plot types as an example (optional); the highlighting scheme is consistent across plot types (optional); the highlighting scheme is compatible with brushing (optional).

[0167] Embodiments of the present invention may be used to address or even meet some or even all of these requirements, or any combination.

[0168] Section 3: In the following, an optional nomenclature for better understanding of embodiments according to the present invention is disclosed.

[0169] For example, a target variable may be represented as T, e.g., with value t. Independent variables may be represented as V, e.g., with combined value v. For example, a probability distribution that may be estimated from a sufficiently large number of test cases may be represented as uppercase letters, e.g., P(t) or Q(t), e.g., to sufficiently approximate the true distribution, while an empirical distribution based on a potentially small number of test cases may be represented as lowercase letters, e.g., p(t), q(t). Value sets may be represented as uppercase letters, e.g., in script type, e.g.,

number

[0170] Section 4: In the following, concepts according to embodiments of the present invention are disclosed.

[0171] Further features, functions, and aspects according to embodiments of the present invention are described below, it being noted that all features, functions, and aspects may be used alone or in combination with any other features, functions, and / or aspects of other embodiments.

[0172] Consider, for example, a plot of N test cases in which one can visualize a target variable T as a function of one or more independent variables jointly represented by V. For example, to avoid confusion with the plot axes, one may avoid calling these variables y and x. In the following, or according to an example embodiment, discrete values

number

[0173] The proposal, or for example, the idea according to the embodiment of the present invention, is to calculate an empirical local distribution of a target variable t at a position v, p v (t)=p(t|v)=Pr(T=t|V=v)···(1) and, for example, the overall target distribution Q(t) = Pr(T = t) (2) Based for example on the statistically significant dissimilarity d(v) between the values ​​of the independent variables

number

[0174] Therefore, the position

number

[0175] Section 4.1: In the following sections, the optional use of Kullback-Leibler divergence in accordance with embodiment concepts is disclosed.

[0176] Kullback-Libra Divergence

number

[0177] [Cover-2006, section 2.3] may be the most commonly used dissimilarity measure between two probability distributions P(t) and Q(t). As an example, for a discrete distribution, e.g.,

number

[0178] The Kullback-Leibler divergence is non-negative, i.e., D KL ≧0, and in the discrete case, it is 0 if and only if the two distributions are equal.

[0179]

number

[0180] A local empirical target distribution p(t|v) at, for example, a position v can be compared with a global target distribution Q(t) using Kullback-Leibler divergence or, for example, using Kullback-Leibler divergence according to an embodiment. The resulting divergence D(v) at position v can be calculated as, for example,

number

[0181] The local empirical target distribution p(t|v) may, for example, be a conditional distribution that can constrain the global target distribution Q(t), so that p(t|v) may, for example, be 0 when Q(t) is 0, which may, for example, satisfy the above requirements.

[0182] Section 4.2: In the following section, an example of a relationship with mutual information according to the concept of the embodiment is disclosed.

[0183] For a plot (T,V), the mutual information I(T,V) [Cover-2006, section 2.3] can quantify the amount of information in, say, an independent variable V about, say, a target variable T. For discrete variables, it can be calculated, for example, as follows:

number

[0184] As an example, we use here the well-known formula for conditional probability.

number

number

[0185] As an example, mutual information can be calculated using a conditional target distribution P TV (t|v) and the overall target distribution P, denoted as Q(t) for example. T(t) can be seen to be a weighted sum of the Kullback-Leibler divergences between

number

[0186] Divergence may also be, for example, the normalized local contribution to the total information, for example, in all independent variables with respect to the target variable.

[0187] Section 4.3: In the following section, an example of a statistically significant divergence according to the concept of the embodiment is disclosed.

[0188] The overall target distribution Q(t) may, for example, be calculated from all test cases and can thus be treated as an accurately estimated distribution, for example without statistical fluctuations.

[0189] However, the local conditional target distribution p(t|v) may, for example, be optionally estimated only from a subset of test cases where the independent variable takes the value V=v. As an example, let n(v) be the number of such test cases. As an example, N(v) may, for example, be quite small, so that p(t|v) may, for example, be subject to considerable statistical variation.

[0190] For example, it seems most natural to ask how close a varying empirical local distribution p(t|v) comes to a global target distribution Q(t) and treat the remaining divergence as the desired statistically significant divergence. Unfortunately, this may not be possible, for example, since the statistics of p(t|v), and therefore its variance, may be unknown.

[0191] Instead, a common technique in statistics may be to express a null hypothesis about, say, a known distribution, in this case Q(t). As a null hypothesis, one may assume that, say, a local distribution p(t|v) can be explained as random fluctuations, say, when drawing a sample of one of n(v) test cases from a known global distribution Q(t). As an example, say that q s Let (t) be the empirical target distribution of a random sample s of n(v) test cases, which may be chosen at random from a known overall distribution Q(t). As an example, we can take a large number S of such samples, each consisting of n(v) test cases, i.e., s = 1...S, and thus have S empirical distributions q s (t), we obtain s = 1···S (see Fig. 11a).

[0192] FIG. 11 shows an example of a statistically significant divergence according to an embodiment of the present invention: a) the empirical target distribution q(t|v) from a random sample of the global target distribution compared to the local target distribution p(t|v). s (t), s = 1 S, b) D from a random sample s The divergence D between (v) and the local target distribution p(t|v) s (v) Probability density distribution of,c) statistically significant divergence d is,D s Mark the bottom 1-C percentile of the cumulative distribution of (v), where C is a confidence level, e.g., C=95%.

[0193] Each empirical distribution q sFor example, for (t), we can calculate the divergence D from the empirical local distribution p(t|v) at position v. s (v) can be calculated,

number

[0194] As an example, for a confidence level, e.g., C=95%, a statistically significant divergence d(v) may be, e.g., a fraction 1-C of all divergences may be smaller, i.e., optionally, a divergence value marking the lower 1-C percentile. d(v):Pr(D s (v) <d(v))=1-C···(11)

[0195] Section 4.4: In the following section, the optional use of mean alignment divergence for continuous target variables according to embodiment concepts is disclosed.

[0196] Divergence

number

number

number

number

[0197] In our case, the local distribution p v When comparing (t) with the overall distribution Q(t), this results, for example, in the following:

number

[0198] By comparing the two means, it is possible to determine, for example, whether a local mean is higher than the global mean, i.e.

number

number

number

[0199] Section 4.5: In the following section, an example of a Boolean target variable according to the concept of the embodiment is disclosed.

[0200] The Boolean target variable may, for example, follow a binomial distribution, optionally with the fraction of the true target value as the only parameter. Let us say that for the global distribution, we denote this fraction by Q, and for the local distribution at location V=v, we denote by p v As an example, the goal variable can take only two values, namely, probability Q or p v then true, or probability 1-Q or 1-p v is false, so equation (5) for the divergence D(v) can be, for example,

number

[0201] Since we know that the target value follows a binomial distribution with known probability Q, we can calculate the distribution of the empirical probability q, e.g., analytically, without actually drawing a random sample of n(v) test cases, as outlined in Section 4.3 as an example.

[0202] The probability of drawing exactly k test cases with the true target value among n(v) randomly selected test cases may be given, for example, by a binomial probability mass. Pr(X=k)=Q k (1-Q) n(v)-k (16)

[0203] The probability of drawing up to k' such test cases may be given, for example, by the cumulative binomial distribution.

number

[0204] The inverse cumulative distribution can, for example, find k' for a given probability. The corresponding empirical probability can, for example, simply be q' = k' / n(v), which is the local empirical distribution p v This can result in the following divergence from

number

[0205] As an example, p v <q’およびq’<p v In either case, the divergence D is, for example, vIt may monotonically decrease as it moves away from

[0206] Figure 12 shows an example of probability density according to an embodiment of the present invention: The blue curve represents the estimated proportion q of test cases having the target value t = true for random samples from the overall distribution of the test cases s of the distribution, shown as the red line, of the local proportion p v is shown in comparison. The distribution mean is Q. The probability q' is shown as the black line and represents the 1 - C quantile of the distribution q v with respect to p s Four examples of scenarios are shown: a) p v is smaller than both q s and Q, and is thus statistically significantly smaller than Q, b) and c) p v is between q s and Q and is thus not statistically significantly different from Q, d) p v is larger than both q s and Q and is thus statistically significantly larger than Q.

[0207] p v < Q, for example, Pr(X ≦ k') ≦ 1 - C ··· (19) when it holds, one may be interested in the 1 - C quantile and the largest k'. The empirical probability q may be larger than q' = k' / n(v) with a confidence level C, for example. When p v < q' < Q, referring to Figure 12a), the statistically significant divergence d can thus be, for example, as follows.

Equation

[0208] Otherwise, q' ≦ pv <In the case of Q, for example, referring to FIG. 12b), there may be no statistically significant divergence, and optionally, d(v) = 0 can be set. Q < p v In the case of, for example, Pr(X ≤ k') ≤ C ··· (21) There may be an interest in the C quantile and the minimum k' when the following does not hold. Q ≤ p v ≤ q' In the case of, for example, referring to FIG. 12c), there may be no statistically significant divergence, and d(v) = 0 can be set. Otherwise, Q < q' < p v In the case of, for example, referring to FIG. 12d), the statistically significant divergence d can be given, for example, by Equation (20) again.

[0209] Section 5: In the following section, an example or visualization of the visualization of the results of the method of the present invention according to an embodiment is disclosed.

[0210] Section 5.1: First, an example of a plot of the object of interest by visualization according to an embodiment is disclosed.

[0211] Note that any or all of the following plot types can be used in embodiments of the present invention or can be addressed in embodiments of the present invention.

[0212] FIG. 13 shows an example according to an embodiment for highlighted regions of independent variables of various plot types where the local target distribution is different from its overall distribution according to an embodiment of the present invention.

[0213] Bar graph: The target variable can always select a stack, for example. Refer to FIG. 13a, for example. The series, horizontal or vertical split variable may not be perceptual as the target variable, for example.

[0214] Box plots: The target variable can always be shown, for example, along the distribution axis. See, for example, FIG. 13b. A series, horizontal or vertical splitting variable, for example, may not be sensible as the target variable.

[0215] Distribution plots: The target variable may be a discrete stack variable, e.g., as shown in Fig. 13c, or may be a continuous distribution axis (e.g., preferably vertical), e.g., as shown in Fig. 13d and Fig. 13e. Horizontal or vertical split variables may not be sensible, e.g., as the target variable.

[0216] Scatter plots: the target variable can be assigned to the y-axis, as shown for example in Fig. 13f, or to the color, as shown for example in Fig. 13g. The x-axis may not be conventional as the target variable, for example.

[0217] Grid Plots: The target variable can always define the grid cell content, as shown for example in Fig. 13g. The x- and y-axes may not be as sensible as the target variable, for example.

[0218] Section 5.2: In the following, the use of optional highlighting-based divergence by visualization according to an embodiment is disclosed.

[0219] A first proposal, e.g. a first idea according to an embodiment, is to highlight regions of an independent variable where the target distribution is, e.g. statistically significantly different from the overall target distribution. As a highlighting, a yellow background is proposed as an example, since, e.g., yellow may be commonly used to draw attention. This may not conflict with, e.g., brushing, where the used color may be saturated. However, adopting this proposal may require, e.g., avoiding yellow as a regular plot color, which may not be recommended anyway, e.g., due to its low contrast against a white plot background.

[0220] As described above, the yellow background can indicate a locally deviated target distribution, for example, measured by a statistically significant divergence, according to, for example, equations (11) and (10).

[0221] Optionally, for comparable highlighting between plots with the same target variable, many or for example all currently visible plots should use or can use the same mapping from divergence to saturation.

[0222] When hovering over the plot area, for example, a histogram of the local target distribution may be shown and compared to the histogram of the overall target distribution.

[0223] Section 5.3: Below, an example of visualizing a boolean target variable according to an embodiment is disclosed.

[0224] In the case of a boolean error variable as the target variable, the target distribution can be fully described, for example, by the proportion of true values. To provide more information, when the local error rate exceeds the average, for example, when p v >Q, a red background (e.g., the background of the first color) can be used, and when the local error rate is below the average, for example, when p v <Q, a blue background (e.g., the background of the second color) can be used.

[0225] Figure 14 shows an example of a scatter plot using a boolean target variable (true = error, blue = no error) according to an embodiment of the present invention. The red / blue background highlights areas where the error density has increased / decreased. This is to be contrasted with the standard scatter plot of Figure 10 visualizing the same data.

[0226] Section 5.4: Below, the use of optional highlighting based on divergence and different means in visualization according to an embodiment is disclosed.

[0227] Optionally, if a local distribution of a continuous target variable diverges from its global distribution, it may be interesting to know, e.g., up to which extent the divergence can be explained by a higher or lower mean and / or up to which extent the distribution shape may differ independently of, e.g., a shifted mean.

[0228] As an example for such a case, the following saturation highlight coloring is proposed, see e.g. FIG. 15 (please note that the following color choices are only examples; embodiments may include other colors and / or color ranges for visualization):

[0229] FIG. 15 shows an example of saturation highlighting according to an embodiment of the invention: the divergence value controls the saturation (vertical axis), while the percentage of the divergence that can be explained by the different averages controls the color, interpolating a) between yellow and blue when the local average is below the global average, or b) between yellow and red when the local average is above the global average (example).

[0230] Optionally, if the local mean is above the mean, i.e.

number

number

[0231] The statistically significant divergence value d(v) can, for example, control the saturation of the highlighting color, ranging, for example, from white at zero divergence to a fully saturated color, see the vertical axis of FIG. 15 as an example.

[0232] For example, the proportion μ of divergence that can be explained by a shift towards a higher or lower mean of the distribution can control the highlight color, e.g., interpolated between red / blue and yellow, see the horizontal axis of FIG. 15 as an example.

[0233] This description can assume, for example, a blue to red color map for a continuous target variable, but can be adapted to other colors as well. Optionally, however, a color map may be recommended that interpolates a continuous target variable only between two colors, for example from blue to red for white. Preferably, a pure white color can code, for example, the mean value of the target variable.

[0234] The legend may, for example, indicate the use of red, blue, and yellow colors. For example, to make the highlighting comparable between plots with the same target variable, all visible plots should or can use the same color map, for example according to FIG. 15.

[0235] FIG. 16 shows some example plots according to an embodiment of the present invention.

[0236] Figure 16 is an example plot that highlights areas where the target value is higher than the average with a red background, areas where the target value is lower than the average with a blue background, and areas where the distribution shape is different but the average is equal to the overall average with a yellow background.

[0237] Note that in plots where blue and red do not code target values, the appearance of blue or red may be unexpected, see, for example, Figures 16a-c. In these cases, it may be better to simply use yellow highlighting (e.g., one-color highlighting), e.g., as described in Section 5.2.

[0238] Thus, in general, a method according to an embodiment of the invention may comprise a step of visualising the measurement data and the measurement data meta information, which may also be considered as extended information based, for example, on a statistically significant dissimilarity value, e.g. d(v), the visualisation of the measurement data may comprise a representation of the measurement results across measurement conditions or across corresponding independent variables, e.g. across corresponding independent variable values, and the visualisation, e.g. the representation of the measurement results, may comprise a highlighting, which depends on, e.g. is based on; e.g. is associated with, the measurement data meta information, e.g. the corresponding measurement data meta information of the respective measurement result, such that measurement results of the corresponding conditional distribution that deviate statistically significantly from the overall distribution of the measurement results or that deviate from a sample distribution, e.g. an empirical distribution, of the measurement results are highlighted and / or such that areas of the visualisation associated with measurement results of the corresponding conditional distribution that deviate statistically significantly from the overall distribution of the measurement results or that deviate from a sample distribution, e.g. an empirical distribution, of the measurement results are highlighted.

[0239] As explained above, the highlighting may optionally be done by colouring the background of the plot, with the saturation of the colouring being determined according to the statistically significant dissimilarity values.

[0240] As another optional feature, the color of the coloring may be determined according to a normalized dissimilarity ratio. Alternatively or additionally, the color of the coloring may be determined according to a comparison of information about the overall distribution of the measurement results and information about the conditional distribution. Alternatively or additionally, the color of the coloring may be determined according to a comparison of information about the sample distribution of the measurement results and information about the conditional distribution.

[0241] Optionally, as explained above, but generally speaking, methods according to embodiments may include visualising the measurement data using statistically significant dissimilarity values.

[0242] Section 6: In the following section, an example algorithm according to an embodiment is disclosed.

[0243] Sections 6.1 and 6.2 describe two alternatives for processing continuous variables according to an embodiment. Section 6.3 summarizes example options for the computation of statistically significant divergences. Finally, Section 6.4 describes example options for fast computation of Boolean target variables.

[0244] Section 6.1: In the following, the use of optional binning of continuous variables by algorithms according to embodiments is disclosed.

[0245] Binning can be, for example, a simple and fast method for discretizing continuous variables.

[0246] It tries to balance the number of value bins between all plot variables, target and independent variables, and the average number of hits within a value bin. The user can increase or decrease the number of hits per value bin, for example, by using a factor α, 0.1≦α≦10, with an initial value of α=1. Two obvious exceptions may have to be made: for categorical variables, one may, for example, always reserve one value bin per categorical value; for discrete numeric variables that can take on, for example, M distinct values, one may, for example, limit the number of value bins to M or less.

[0247] The following example illustrates an embodiment of the approach: Assume one categorical variable with 5 values, one integer variable with 8 values, two continuous variables, N=100,000 test cases, and a user-specified coefficient α=2. Optionally, the test cases are first sorted, for example, by the number of categorical value combinations, B cat = 5. Each division is N1 = N / B cat = 20,000 test cases. To optionally double the number of hits per value bin, we can divide the pool of test cases further defining the number of value bins, e.g., α = 2, N2 = N1 / B cat =10,000.

[0248] These test cases can be spread, for example, along the remaining three variables (one discrete, two continuous) and the number of hits per value bin, i.e., along, for example, four dimensions.

number

number

number

[0249] Section 6.2: In the following, the use of local neighborhoods of continuous variables by the algorithm according to the embodiment is disclosed.

[0250] Binning may not match test case density, for example. If one optionally desires smooth background shading along one or more continuous axes that matches test case density, one can, for example, collect local statistics from test cases closest to the points of a relatively fine grid, as shown by way of example in FIG.

[0251] For each grid point v, we determine, say, its n nearest neighbors and estimate the empirical local target distribution p of these neighbors, as described as an example in Section 6.3. v (t) can be compared to the overall distribution Q(t).

[0252] For example, the number of nearest neighbors n can be determined, optionally following a similar purport as described in Section 6.2. Let N be the number of all test cases, and let B be the product of the number of value bins of all plot variables, e.g., even if they are not subsequently discretized, e.g., according to Section 6.1. T Let H be the number of bins of the target variable. Then the average number of hits per virtual bin can be, for example, H=N / B. For example, optionally, the following can be chosen: n = H B T ···(twenty two) As a result, similar to binning, but in a local neighborhood, we have full information about the local target distribution.

[0253] In one continuous dimension, such as a distribution plot, finding nearest neighbors may be very fast, for example: all that may be required is to sort the target values ​​and select, for example, a range of n test cases around each grid point.

[0254] Two continuous dimensions, such as a scatter plot, may require, for example, finding the n nearest neighbors of each grid point, which may, for example, be computationally more demanding.

[0255] Compared to simple binning, this method can, for example, adapt to local test case density, can, for example, provide higher resolution, and can, for example, be more visually appealing. Compared to binning, as an example, for example, only the method for obtaining local test cases may be different. For example, the subsequent calculations may be the same.

[0256] Section 6.3: Below, the use of the optional calculation of statistically significant divergence by the algorithm according to the embodiment is disclosed.

[0257] Embodiments may include any combination of the following steps, and thus, as an example, the method may include a subset of the following steps.

[0258] Binning on the goal variable

number

number

number

number

[0259] Section 6.4: In the following, an example of a Boolean target variable according to an algorithm according to an embodiment is disclosed.

[0260] Embodiments may include any combination of the following steps, and thus, as an example, the method may include a subset of the following steps.

[0261] Optionally, calculate the proportion Q of test cases that have the target value t=true, for example across all test cases. The following calculation can be done, for example, for each value bin:

number

number

[0262] As an example, p v<For the case of Q: When drawing n(v) randomly selected test cases, when Q is the probability of the true target value, calculate the lower (1 - C) quantile k' for the number of test cases with the target value t = true. For example, refer to equations (19) and (17) (optional). Note that the inverse CDF of the binomial distribution may be calculated using the Python function stats.binom.ppf (optional). Calculate q' = k' / n(v) (optional). p v <If q' < Q, calculate the statistically significant divergence d(v) using, for example, equation (20), otherwise optionally set d(v) = 0 (optional).

[0263] Otherwise, for example, Q < p v <In the case of, calculate the C quantile k' (optional). Calculate q' = k' / n(v) (optional). Q < q' < p v <If the case of, calculate the statistically significant divergence d(v) using, for example, equation (20), otherwise optionally set d(v) = 0 (optional).

[0264] <As an example, p v <In the case of = Q, set d(v) = 0 (optional).

[0265] <Section 7: Below, embodiments according to the present invention are further described.

[0266] <Section 7.1: First, an example of a recommended or preferred embodiment according to the present invention is disclosed.

[0267] Table 1 shows, for example, examples of recommended highlighting for different plot types and data types of target variables. Yellow highlighting can refer, for example, to the scheme described in section 5.2, which is based only optionally on divergence. Red / blue highlighting can refer, for example, to the scheme for Boolean target variables described in section 5.3. Red / blue / yellow highlighting can refer, for example, to the scheme described in section 5.4, which is based, for example, on a combination of divergence and different means. Smooth highlighting can refer, for example, to using local neighborhoods from a fine grid, as described in section 6.2. If smoothness is not mentioned for a continuous target variable, binning can be assumed, for example, as described in section 6.1.

[0268] Table 1: Recommended highlighting for various plot types (examples) [Table 1]

[0269] Note (1): The red / blue colors, for example, may not be natural colors in this plot.

[0270] Note (2): As an example, the target variable may be directly visible along, for example, the y-axis, so that the differences between, for example, shifted means or different distribution shapes may already be visible. More challenging red / blue / yellow highlighting may not be necessary, unless, for example, consistency is desired.

[0271] As an example, if a combined red / blue / yellow highlighting is considered complex, one could consider two separate views, e.g., one view with yellow highlighting (e.g., first color highlighting) and another view with only red and blue (e.g., second and third colors). According to a preferred embodiment, for example, a simple yellow highlighting with a marker tooltip that compares (or can enable comparing) histograms of local and global target distributions could be used.

[0272] Section 7.2: Below, optional finer grained highlighting according to embodiments of the present invention is disclosed.

[0273] An alternative according to an embodiment may be, for example, to highlight each graph plot element based on the statistically significant contribution to the mutual information from the test cases represented by those graph plot elements.

[0274] A graph plot element can be, for example, a single stack of a bar graph, a stacked slice of a distribution plot, a single dot of a scatter plot, or a quantile box of a box plot. Each test case can, for example, contribute a so-called point mutual information (PMI). A statistically significant PMI can be estimated, for example, at a desired confidence level, as the lowest PMI from a random data subset. If a graph plot element aggregates values ​​from multiple test cases, their PMIs are also aggregated.

[0275] As an advantage, this method can provide, for example, more fine-grained user prompting.

[0276] A main advantage of the inventive approach described above, for example the approach according to the section preceding 7.2, may be, for example, to mitigate or overcome the following problems: for example, in a bar chart, if a stack disappears in one of the bars or in a distribution plot, this should certainly be brought to the user's attention or may be brought to the user's attention, but there may be no way to highlight the corresponding graph plot element, for example, because the corresponding graph plot element has disappeared completely. Secondly, the highlighted graph element may be too small for effective highlighting. In other words, according to the embodiment described above, highlighting can be achieved even in such cases where a graph plot element disappears by using an alternative or approach. Moreover, effective highlighting can be achieved.

[0277] In Section 7.3: The following features, namely, embodiments of the invention are disclosed:

[0278] Embodiments according to the invention may provide the following functions and / or features: Insights in this specification Highlighted Insights Pinpoint Insights Boosted Insights View Insights

[0279] References Cover-2006 Thomas M.Cover,Joy A.Thomas:Elements of Information Theory,Wiley 2006

[0280] Further observations: Embodiments in accordance with the present invention can accommodate plots that are used to understand how a target variable depends on one or more independent variables.

[0281] Moreover, embodiments in accordance with the present invention can overcome several challenges associated with conventional plots.

[0282] Traditional plots can face several challenges with this task. *A large number of good cases may mask rare error cases (as in this example). *Statistical variations may suggest dependencies to the user that do not exist. *Visual bias due to uneven distribution of samples *Proportions (in this example) or distributions are difficult to estimate visually

[0283] The embodiments according to the present invention may have the following advantages, effects, or intents. * Direct the user's attention to the area of ​​interest * Making clearly visible (subtle) effects that would otherwise be difficult or impossible to see

[0284] An embodiment according to the present invention may include the following techniques, or the techniques according to the embodiment may include the following aspects. * Augment existing plot types *Highlights regions of the independent variable (here x / y) where the target distribution is statistically significantly different from the overall target distribution.

[0285] It is noted that any embodiment defined by the claims can be supplemented by any of the details (features and functions) described in the above sections.

[0286] Also, the embodiments described in the above sections can be used individually or can be supplemented by any of the features of the other sections or by any features included in the claims.

[0287] It should also be noted that the individual aspects described herein may be used individually or in combination, and thus details may be added to each of the individual aspects without adding details to another one of the aspects.

[0288] Moreover, features and functions disclosed herein in relation to a method may also be used in an apparatus (configured to perform such functions). Moreover, any feature and function disclosed herein in relation to an apparatus may also be used in the corresponding method. In other words, a method disclosed herein may be captured by any of the features and functions described in relation to an apparatus.

[0289] Additionally, any of the features and functionality described herein can be implemented in hardware or software, or using a combination of hardware and software, as described in the "Implementation Alternatives" section.

[0290] Implementation alternatives: While some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, with blocks or devices corresponding to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0291] Depending on specific implementation requirements, embodiments of the invention can be implemented in hardware or software. Implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Thus, the digital storage medium can be computer readable.

[0292] Some embodiments according to the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0293] Generally, embodiments of the present invention can be implemented as a computer program product having program code which operates to perform one of the methods when the computer program product is run on a computer. The program code can for example be stored on a machine readable carrier.

[0294] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0295] In other words, therefore, an embodiment of the inventive methods is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0296] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0297] A further embodiment of the inventive method is therefore a data stream or a signal sequence representing the computer program for performing one of the methods described herein, the data stream or the signal sequence being for example arranged to be transferred via a data communication connection, for example via the Internet.

[0298] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0299] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0300] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0301] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to implement some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to implement one of the methods described herein. In general, the methods are preferably implemented by any hardware apparatus.

[0302] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0303] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0304] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0305] Any of the methods described herein, or components of the apparatus described herein, may be implemented at least in part by hardware and / or software.

[0306] The described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented by the description and illustration of the embodiments herein.

Claims

1. 1. A method for determining information regarding characteristics of one or more devices under test (DUTs) using measurement data from automatic test equipment (ATE), the measurement data including a plurality of measurement results and information describing corresponding measurement conditions of the devices under test (DUTs), the method comprising: determining information regarding a conditional distribution of measurement results corresponding to a first subset of measurement conditions; and determining information about at least one sample distribution that is comparable to the conditional distribution; and determining a statistically significant dissimilarity value using said information about said conditional distribution and using said information about said at least one sample distribution; Including, the information regarding the characteristics of the one or more DUTs includes or is based on the statistically significant dissimilarity values; method.

2. the sample distribution is an empirical distribution; determining the information about the at least one sample distribution comprises determining information about a plurality of empirical distributions of measurement results of a plurality of subsets of the measurement data; the number of measurements taken into account for the determination of the information regarding each empirical distribution is related to the number of measurements taken into account for the determination of the information regarding the conditional distribution; the measurement results for each subset of the measurement data being a randomly or pseudo-randomly selected set of measurement results of the measurement data; the method further comprising: determining a plurality of dissimilarity measures between the conditional distributions and the respective empirical distributions using the information about the conditional distributions and using the information about the respective empirical distributions; the method further comprising determining the statistically significant dissimilarity value using the plurality of dissimilarity measures. The method of claim 1.

3. the statistically significant dissimilarity value is associated with a quantile of a distribution of the dissimilarity measures; The method according to claim 1 or 2.

4. determining a cumulative distribution of the plurality of dissimilarity measures; and determining the statistically significant dissimilarity value such that only a predetermined portion of the empirical distributions include dissimilarity measures for the conditional distributions that are less than or equal to the statistically significant dissimilarity value; 4. The method according to any one of claims 1 to 3.

5. To determine the statistically significant dissimilarity value, extrapolating said cumulative distribution; and / or interpolating the cumulative distribution. The method according to claim 4.

6. selecting a dissimilarity measure from the plurality of dissimilarity measures, the selected dissimilarity measure comprising a minimum deviation for the statistically significant dissimilarity value from the plurality of dissimilarity measures; and determining the mean of the empirical distribution corresponding to the selected dissimilarity measure; and determining a mean value of the conditional distribution of the measurement results corresponding to the first subset of measurement conditions; and determining a mean aligned dissimilarity measure between the conditional distribution and the empirical distribution corresponding to the selected dissimilarity measure; and determining information regarding a normalized dissimilarity ratio using the average aligned dissimilarity measure and the dissimilarity measure between the conditional distribution and the empirical distribution corresponding to the selected dissimilarity measure; Further comprising: the information regarding the characteristics of the one or more DUTs includes the information regarding the normalized dissimilarity ratio.

6. The method according to any one of claims 1 to 5.

7. selecting a dissimilarity measure from the plurality of dissimilarity measures, the selected dissimilarity measure comprising a minimum deviation for the statistically significant dissimilarity value from the plurality of dissimilarity measures; Further comprising: the information regarding the characteristics of the one or more DUTs includes information regarding the selected dissimilarity measure; 7. The method according to any one of claims 1 to 6.

8. determining a mean value of the conditional distribution of the measurement results corresponding to the first subset of measurement conditions; and determining an average alignment dissimilarity measure between the conditional distribution and the overall distribution of measurements; and determining a dissimilarity measure between the conditional distribution and the global distribution; and determining information regarding a normalized dissimilarity ratio using the mean aligned dissimilarity measure and using the dissimilarity measure between the conditional distribution and the global distribution; Further comprising: the information regarding the characteristics of the one or more DUTs includes the information regarding the normalized dissimilarity ratio.

8. The method according to any one of claims 1 to 7.

9. the method further comprising binning the measured data values, the binning being performed based on a parameter α, the parameter α influencing how many measured data values ​​are quantized into value bins; the measurement data comprises a continuous variable, the method further comprising quantizing values ​​of the continuous variable into value bins; and / or the measurement data includes categorical variables, each categorical variable value being associated with a respective value bin; and / or the measurement data includes a discrete variable, the discrete variable including M distinct values, and the values ​​of the discrete variable being quantized into up to M value bins; 9. The method according to any one of claims 1 to 8.

10. the measurement data includes several data sets of tests of the DUT; The number of value bins of the measurement data is using the number of data sets and using partitioning based on the number of combinations of categorical variable values; and / or using a number of data sets and using a partitioning based on the parameter α; and / or using the number of data sets and using partitioning based on the number of continuous and discrete variables; and / or Using the number of data sets and partitioning based on the number of continuous variables, To be determined, 10. The method according to any one of claims 1 to 9.

11. the measurement data includes the measurement results and values ​​of one or more corresponding independent variables, and the method further comprises: determining the first subset of measurement conditions such that the measurement conditions in the first subset are contiguous; 11. The method according to any one of claims 1 to 10.

12. the measurement data includes the measurement results and values ​​of one or more corresponding independent variables, and the method further comprises: binning values ​​of the measurement data, wherein measurements and / or independent variable values ​​of the measurement data are quantized into value bins; and determining information about the overall distribution of the measurements based on each value bin; and determining information regarding a conditional distribution of measurement outcomes for the first subset of measurement conditions, the first subset of measurement conditions being associated with the independent variable values ​​for respective value bins; and determining a number of measurements corresponding to said first subset of measurement conditions; and determining information regarding a plurality of empirical distributions of measurement outcomes for a plurality of subsets of the measurement data, the number of measurements taken into account for determining said information on each empirical distribution is equal to the number of measurements in said first subset of measurement conditions; the measurement results for each subset of the measurement data are a randomly or pseudo-randomly selected set of measurement results for the measurement data. steps; and using the information about the conditional distribution and using the information about the respective empirical distributions to determine a plurality of dissimilarity measures between the conditional distribution and each empirical distribution; and determining a cumulative distribution of the plurality of dissimilarity measures; and determining the statistically significant dissimilarity value such that only a predetermined portion of the empirical distribution contains dissimilarity measures for the conditional distributions that are less than or equal to the statistically significant dissimilarity value; and selecting a dissimilarity measure from the plurality of dissimilarity measures, the selected dissimilarity measure comprising a minimum deviation for the statistically significant dissimilarity value from a dissimilarity measure from the plurality of dissimilarity measures; and determining the mean of the empirical distribution corresponding to the selected dissimilarity measure; and determining a mean value of the conditional distribution of the measurement results corresponding to the first subset of measurement conditions; and determining a mean aligned dissimilarity measure between the conditional distribution and the empirical distribution corresponding to the selected dissimilarity measure; and determining information regarding a normalized dissimilarity ratio using the mean aligned dissimilarity measure and using the dissimilarity measure between the conditional distribution and the empirical distribution corresponding to the selected dissimilarity measure; Further comprising: the information regarding the characteristics of the DUT includes the information regarding the normalized dissimilarity ratio.

12. The method according to any one of claims 1 to 11.

13. the measurement data includes the measurement results and values ​​of one or more corresponding independent variables, and the method further comprises: binning values ​​of the measurement data, wherein measurements and / or independent variable values ​​of the measurement data are quantized into value bins; and determining information about the overall distribution of the measurements based on each value bin; and determining information regarding a conditional distribution of measurement outcomes for the first subset of measurement conditions, the first subset of measurement conditions being associated with the independent variable values ​​for respective value bins; and determining a number of measurements corresponding to said first subset of measurement conditions; and determining information regarding a plurality of empirical distributions of measurement outcomes for a plurality of subsets of the measurement data, the number of measurements taken into account for determining said information on each empirical distribution is equal to the number of measurements in said first subset of measurement conditions; the measurement results for each subset of the measurement data are a randomly or pseudo-randomly selected set of measurement results for the measurement data. steps; and using the information about the conditional distribution and using the information about the respective empirical distributions to determine a plurality of dissimilarity measures between the conditional distribution and each empirical distribution; and determining a cumulative distribution of the plurality of dissimilarity measures; and determining the statistically significant dissimilarity value such that only a predetermined portion of the empirical distribution contains dissimilarity measures for the conditional distributions that are less than or equal to the statistically significant dissimilarity value; and selecting a dissimilarity measure from the plurality of dissimilarity measures, the selected dissimilarity measure comprising a minimum deviation for the statistically significant dissimilarity value from a dissimilarity measure from the plurality of dissimilarity measures; Further comprising: the information regarding the characteristics of the DUT includes the selected dissimilarity measure; 13. The method according to any one of claims 1 to 12.

14. A plurality of statistically significant dissimilarity values ​​and / or a plurality of normalized dissimilarity ratios are determined for different subsets of measurement conditions corresponding to respective value bins.

14. The method according to any one of claims 1 to 13.

15. obtaining information about the overall distribution of the measurement results based on the measurement data; and comparing the information regarding the global distribution with the information regarding the conditional distribution; The information regarding the at least one sample distribution is determined based on the comparison of the information regarding the overall distribution and the information regarding the conditional distribution; and comparing the information regarding the global distribution, the information regarding the conditional distribution, and the information regarding the at least one sample distribution; and determining the statistically significant dissimilarity value based on the comparison of the information regarding the overall distribution, the information regarding the conditional distribution, and the information regarding the at least one sample distribution.

15. The method according to any one of claims 1 to 14.

16. the measurement data includes the measurement results and corresponding independent variable values, and the method further comprises: binning values ​​of the measurement data, wherein measurements and / or independent variable values ​​of the measurement data are quantized into value bins; the measurement is a Boolean target variable having a measurement of true or false; and Determining information Q about the overall distribution of the measurement results based on the overall measurement results of the measurement data, the information Q being the probability that the measurement result is true or the probability that the measurement result is false; and determining a number n(v) of measurement results corresponding to the first subset of measurement conditions, the first subset of measurement conditions being associated with the independent variable values ​​of respective value bins; and Information p on the conditional distribution of measurement results corresponding to the first subset of measurement conditions ｖ determining a The information p ｖ is the probability that a measurement result is true based on the measurement results corresponding to the first subset of measurement conditions, or ｖ is the probability that a measurement result is false based on the measurement results corresponding to the first subset of measurement conditions; and p ｖ Q holds true, and / or p ｖ > Whether Q holds, and / or p ｖ = Q holds true, and / or p ｖ determining whether ≧Q holds; and p ｖ If Q holds, then determining (706) information q′ about the sample distribution using q′=k′ / n(v), where k′ is the maximum number of true test cases for which Pr(X≦k′)≦1−C, and C is a confidence value; Pr(X≦k′)≦1−C is a condition that the probability Pr of having k′ or less positive measurement results X when selecting n(v) arbitrary measurement results from the measurement data is equal to or less than 1 minus the confidence level C; and p ｖ If Q′ is satisfied, [0010] and / or determining the statistically significant dissimilarity value d(v) using q′≦p ｖ If Q holds, then setting the statistically significant dissimilarity value d(v) to d(v)=0; and / or p ｖ If ≧Q holds, determining information q' about the sample distribution using q'=k' / n(v) according to the smallest k' that satisfies Pr(X≦k')≦C, where Pr(X≦k')≦C is the condition that the probability Pr of having k' or fewer positive measurements X when selecting n(v) arbitrary measurements from the measurement data is less than or equal to a confidence level C (the method can, for example, determine k' using an evaluation of a binomial distribution using the number of samples n(v) and individual probabilities defined by Q); and Q≦p ｖ If ≦q′ holds, setting the statistically significant dissimilarity value d(v)=0; or Q<q′<p ｖ If holds, then [0025] and / or determining the statistically significant dissimilarity value d(v) using p ｖ If v = Q, then setting the statistically significant dissimilarity value d(v) to d(v) = 0; 16. The method according to any one of claims 1 to 15.

17. determining a plurality of respective statistically significant dissimilarity values ​​for different subsets of measurement conditions corresponding to respective value bins; 17. The method of claim 16.

18. augmenting the measurement data using the statistically significant dissimilarity values; the augmented measured data being a combination of the measured data itself and associated measured data meta-information, the associated measured data meta-information being determined based on the statistically significant dissimilarity values; the measurement data meta-information comprises information regarding a statistically significant dissimilarity between the measurement results corresponding to the first subset of measurement conditions and the overall measurement results, or between the measurement results corresponding to the first subset of measurement conditions and the measurement results of a selected sample distribution; 18. The method according to any one of claims 1 to 17.

19. Determining a normalized dissimilarity ratio, Using the average alignment dissimilarity measure; and using a dissimilarity measure between the conditional distribution and the global distribution of measurements, where the normalized dissimilarity ratio contains information about the amount of the dissimilarity measure associated with a different mean between the conditional distribution and the global distribution; or using a dissimilarity measure between the conditional distribution and the sample distribution of the measurement outcomes, determining a normalized dissimilarity ratio, where the normalized dissimilarity ratio contains information about the amount of the dissimilarity measure associated with a difference mean between the conditional distribution and the sample distribution; the measurement data meta-information includes information regarding the normalized dissimilarity ratio; 19. The method according to any one of claims 1 to 18.

20. The method further comprises visualizing the measured data and measured data meta-information based on the statistically significant dissimilarity values; the visualization of the measurement data comprises a representation of the measurement results across the measurement conditions or across corresponding independent variables; The visualization includes highlighting, the highlighting comprising: such that measurements of corresponding conditional distributions that deviate statistically significantly from the overall distribution of said measurements or that deviate from the sample distribution of said measurements are highlighted; and / or such that areas of the visualization associated with measurements of the corresponding conditional distribution that deviate statistically significantly from the overall distribution of the measurements or deviate from the sample distribution of the measurements are highlighted. Dependent on the measurement data meta information, 20. The method according to any one of claims 1 to 19.

21. said highlighting including coloring a background of said plot, the saturation of said coloring being determined according to said statistically significant dissimilarity values; 21. The method of claim 20.

22. the color of the coloring is determined according to a normalized dissimilarity ratio; and / or the color of the coloring is determined according to a comparison of information about the overall distribution of the measurements with the information about the conditional distribution; and / or the color of the coloring is determined according to a comparison of information about a sample distribution of the measurement results and the information about the conditional distribution.

22. The method of claim 20 or 21.

23. visualizing the measurement data using the statistically significant dissimilarity values.

23. The method of any one of claims 1 to 22.

24. A computer program for carrying out the method according to any one of claims 1 to 23 when the computer program is run on a computer.

25. 1. An apparatus for determining information relating to a characteristic of one or more devices under test (DUTs) using measurement data from automatic test equipment (ATE), the measurement data including a plurality of measurement results and information describing corresponding measurement conditions of the devices under test (DUTs), the apparatus comprising: determining information regarding a conditional distribution of measurement outcomes corresponding to a first subset of measurement conditions; and determining information about at least one sample distribution that is comparable to the conditional distribution; and determining a statistically significant dissimilarity value using said information about said conditional distribution and using said information about said at least one sample distribution; Consists of the information regarding the characteristics of the one or more DUTs includes or is based on the statistically significant dissimilarity values; Device.

Images