Data similarity analysis method and system, electronic equipment, medium and product

Abstract

The invention provides a data similarity analysis method and system, an electronic device, a medium and a product, the similarity of two arrays is determined by using at least one of operation 1-operation 3, the operation 1 is to judge whether the similarity of the two arrays exceeds an expected range or not according to the difference of medians of the two arrays, and the operation 2 is to judge whether the similarity of the two arrays exceeds the expected range or not according to the difference of medians of the two arrays; 2, judging whether the similarity of the two arrays exceeds an expected range or not according to the difference of the data widths of the two first specific quantiles of the two arrays; and 3, judging whether the similarity of the two arrays exceeds an expected range or not according to the difference of the quantile cumulative probabilities of the two arrays. Namely, the similarity of the two arrays can be measured from at least one dimension of the center position, the fluctuation range and the distribution overlap ratio of the two arrays, the method is not affected by the distribution mode and the abnormal value of the data in the arrays, the method can be applied to the arrays in normal distribution and non-normal distribution, and the analysis result is very accurate.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data similarity analysis method, system, electronic equipment, medium and product. Background Art

[0002] Currently, similarity analysis of data in an array is required in many scenarios. The distribution of data in the array can be normal distribution or non-normal distribution.

[0003] There are usually two existing data similarity analysis methods: one is the variance analysis method, which mainly uses the difference in the mean between the arrays to determine the similarity of two arrays; the other is the HC table analysis method, which mainly uses the difference in the mean between the arrays and the difference in the probability area to determine the similarity of two arrays. These two similarity analysis methods have high accuracy when the data in the array is normally distributed, but when the data in the array is non-normally distributed, the accuracy of both similarity analysis methods is low. Summary of the invention

[0004] The purpose of the present application is to provide a data similarity analysis method, system, electronic device, medium and product to solve the problem of low accuracy of existing similarity analysis methods.

[0005] In order to achieve the above object, the present invention provides a data similarity analysis method, comprising:

[0006] Provide two arrays;

[0007] Perform at least one of the following operations to determine the similarity of the two arrays:

[0008] Operation 1: judging whether the similarity of the two arrays exceeds an expected range according to the difference between the medians of the two arrays;

[0009] Operation 2: judging whether the similarity of the two arrays exceeds an expected range according to the difference in data widths of the two first specific quantiles of the two arrays;

[0010] Operation 3: judging whether the similarity of the two arrays exceeds an expected range according to the difference in the quantile cumulative probabilities of the two arrays.

[0011] Optionally, at least operation 2 and / or operation 3 are performed.

[0012] Optionally, perform operation 1, operation 2, and operation 3.

[0013] Optionally, the steps of operation 1 include:

[0014] Obtaining the absolute value of the difference between the medians of the two arrays and normalizing the difference to obtain a normalized value; and,

[0015] Determine the size relationship between the standardized value and the first threshold value. When the standardized value is greater than the first threshold value, determine that the similarity between the two arrays exceeds the expected range. When the standardized value is less than or equal to the first threshold value, determine that the similarity between the two arrays does not exceed the expected range.

[0016] Optionally, the absolute value of the difference between the medians of the two arrays is normalized according to the following formula:

[0017]

[0018] Where D is the standardized value, ΔMedian1 is the absolute value of the difference between the two medians, and ΔDATA M is the absolute value of the difference between the data corresponding to the two second specific quantiles, and C is a constant related to the two second specific quantiles.

[0019] Optionally, the ranges of the two second specific quantiles are 0-50% and 50%-100% respectively.

[0020] Optionally, the data width of two first specific quantiles is the absolute value of the difference between the data corresponding to the two first specific quantiles of the array.

[0021] Optionally, the ranges of the two first specific quantiles are 75% to 99.865% and 0.135% to 25%, respectively.

[0022] Optionally, the steps of operation 2 include:

[0023] One of the two arrays is used as a baseline array, and the other is used as a comparison array;

[0024] Obtaining a ratio of the data width of the two first specific quantiles of the comparison array to the data width of the two first specific quantiles of the baseline array; and,

[0025] The relationship between the ratio and a second threshold is determined. When the ratio is greater than the second threshold, it is determined that the similarity between the two arrays exceeds the expected range. When the ratio is less than or equal to the second threshold, it is determined that the similarity between the two arrays does not exceed the expected range.

[0026] Optionally, the steps of operation 3 include:

[0027] One of the two arrays is used as a baseline array, and the other is used as a comparison array;

[0028] Obtaining fourth specific quantiles corresponding to two third specific quantiles of the baseline array in the comparison array;

[0029] Obtaining a ratio of an absolute value of a difference between two fourth specific quantiles to an absolute value of a difference between two third specific quantiles;

[0030] The relationship between the ratio and a third threshold is determined. When the ratio is less than the third threshold, it is determined that the similarity between the two arrays exceeds the expected range. When the ratio is greater than or equal to the third threshold, it is determined that the similarity between the two arrays does not exceed the expected range.

[0031] Optionally, the ranges of the two third specific quantiles are 75% to 99.865% and 0.135% to 25%, respectively.

[0032] Optionally, the steps of operation 3 include:

[0033] One of the two arrays is used as a baseline array, and the other is used as a comparison array;

[0034] dividing the interval between two third specific quantiles of the baseline array into at least two non-overlapping quantile intervals;

[0035] Obtain the quantiles corresponding to the maximum quantile and the minimum quantile of each quantile interval in the comparison array;

[0036] Obtaining the absolute value of the difference between the maximum quantile and the minimum quantile in each quantile interval and the difference between the two corresponding quantiles in the comparison array, and calculating the cumulative difference of all the quantile intervals;

[0037] Obtaining the ratio of the cumulative difference to the absolute value of the difference between the two third specific quantiles; and,

[0038] The relationship between the ratio and a fourth threshold is determined. When the ratio is less than the fourth threshold, it is determined that the similarity between the two arrays exceeds the expected range. When the ratio is greater than or equal to the fourth threshold, it is determined that the similarity between the two arrays does not exceed the expected range.

[0039] Optionally, the cumulative difference of the quantile interval is obtained according to the following formula:

[0040]

[0041] Among them, ΔDIF totel is the cumulative difference, ΔDIF i is the absolute value of the ith difference, and n is the number of quantile intervals.

[0042] Optionally, the ranges of the two third specific quantiles are 75% to 99.865% and 0.135% to 25%, respectively.

[0043] Optionally, the distribution of the data in the array is normal distribution or non-normal distribution.

[0044] Optionally, the data in the array is greater than or equal to 300.

[0045] The present application also provides a data similarity analysis system, comprising:

[0046] A data supply module, used for providing two arrays;

[0047] An execution module is used to perform at least one of the following operations to determine the similarity between the two arrays:

[0048] Operation 1: judging whether the similarity of the two arrays exceeds an expected range according to the difference between the medians of the two arrays;

[0049] Operation 2: judging whether the similarity of the two arrays exceeds an expected range according to the difference in data widths of the two first specific quantiles of the two arrays;

[0050] Operation 3: judging whether the similarity of the two arrays exceeds an expected range according to the difference in the quantile cumulative probabilities of the two arrays.

[0051] The present application also provides an electronic device, characterized in that it includes:

[0052] one or more actuators; and,

[0053] a memory for storing one or more programs; and,

[0054] When the one or more programs are executed by the one or more executors, the one or more executors implement the data similarity analysis method.

[0055] The present application also provides a computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and the computer program implements the data similarity analysis method when executed by a processor.

[0056] The present application also provides a computer program product, characterized in that the computer program product includes instructions, and when the instructions are executed by a processor, the data similarity analysis method is implemented.

[0057] In the data similarity analysis method, system, electronic device, medium and product provided in the present application, at least one of operations 1 to 3 is used to determine the similarity of the two arrays, wherein operation 1 is to judge whether the similarity of the two arrays exceeds the expected range based on the difference in the medians of the two arrays; operation 2 is to judge whether the similarity of the two arrays exceeds the expected range based on the difference in the data width of the two first specific quantiles of the two arrays; operation 3 is to judge whether the similarity of the two arrays exceeds the expected range based on the difference in the quantile cumulative probability of the two arrays. That is, the present application can measure the similarity of the two arrays from at least one dimension of the center position, fluctuation range and distribution overlap of the two arrays, is not affected by the distribution mode of the data in the arrays and the outliers, can be applied to arrays with normal distribution and non-normal distribution, the analysis is more comprehensive, the application range is wider, and the analysis results are very accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart of a data similarity analysis method provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of the medians of two arrays and data corresponding to two specific quantiles provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of data of two specific quantiles of two arrays provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the quantile cumulative probability of two arrays provided in an embodiment of the present application;

[0062] Figure 5 for Figure 4 The relationship between the data and probability density of the two arrays in ;

[0063] Figure 6 A schematic diagram of dividing the area between two third specific quantiles of the baseline array into three non-overlapping quantile intervals is provided for the embodiment of the present application;

[0064] Figure 7 A structural block diagram of a data similarity analysis system provided in an embodiment of the present application;

[0065] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0066] Wherein, the accompanying drawings are marked as follows:

[0067] 10 - data supply module; 20 - execution module; 301 - processor; 302 - memory; 303 - communication interface; 304 - communication bus. DETAILED DESCRIPTION

[0068] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0069] Figure 1 Flow chart of the data similarity analysis method provided in this embodiment. Figure 1 As shown, the data similarity analysis method includes:

[0070] Step S100: providing two arrays;

[0071] Step S200: performing at least one of the following operations to determine the similarity of two arrays:

[0072] Operation 1: Determine whether the similarity of the two arrays exceeds the expected range based on the difference in the medians of the two arrays;

[0073] Operation 2: Determine whether the similarity of the two arrays exceeds the expected range based on the difference in the data width of the first specific quantiles of the two arrays;

[0074] Operation 3: Determine whether the similarity of the two arrays exceeds the expected range based on the difference in the quantile cumulative probabilities of the two arrays.

[0075] Specifically, firstly execute step S100 to provide two arrays that need to be analyzed for similarity. Each array may contain multiple data. To ensure the accuracy of the similarity analysis, the data in the array may be greater than 300, but it should not be limited thereto. The number of data in the two arrays may be the same or different.

[0076] Furthermore, the array can be a set of data captured from a production machine. The two arrays that need to be analyzed for similarity can be, for example, two sets of data captured from two different production machines when the production machine is replaced; or they can be two sets of data captured from the same production machine when the product is iterated or the production conditions are changed. The distribution of the data in each array can be normal distribution or non-normal distribution.

[0077] It is understandable that one of the two arrays can be used as a baseline array, and the other array can be used as a comparison array to perform similarity analysis with the baseline array. For example, in order to analyze the similarity of two or more arrays, one array is usually used as a baseline array, and the other arrays are comparison arrays, and each comparison array is analyzed for similarity with the baseline array; but this should not be limited to this, and the present invention can also be applied to the similarity analysis of two random arrays.

[0078] Further, at least one of operation 1, operation 2, and operation 3 is performed to determine the similarity of the two arrays. In some embodiments, operation 1, operation 2, and operation 3 may also be performed. When any one of operation 1, operation 2, and operation 3 determines that the similarity of the two arrays exceeds the expected range, it can be determined that the similarity of the two arrays exceeds the expected range. Operation 1, operation 2, and operation 3 may be performed sequentially or synchronously, and the present invention does not limit the execution order of operation 1, operation 2, and operation 3.

[0079] In some embodiments, only operation 2 and / or operation 3 may be executed, or only one or two of operation 1, operation 2 and operation 3 may be executed. When any of the executed operations determines that the similarity of the two arrays exceeds the expected range, it can be determined that the similarity of the two arrays exceeds the expected range.

[0080] Specifically, when performing operation 1, the difference between the medians of the two arrays is used to determine whether the similarity of the two arrays exceeds the expected range. It can be understood that the median of the array can represent the center position of the array, and the similarity of the two arrays can be evaluated by the difference between the medians of the two arrays. The positioning is accurate, and it is not affected by the distribution mode of the data in the array and the outliers. It can be applied to arrays with normal distribution and non-normal distribution, has a wide range of applications, and the analysis results are relatively accurate.

[0081] Furthermore, when executing operation 1, the absolute value of the difference between the medians of the two arrays is first obtained and standardized to obtain a standardized value, and then the relationship between the standardized value and the first threshold is determined. When the standardized value is greater than the first threshold, it can be determined that the similarity of the two arrays exceeds the expected range. When the standardized value is less than or equal to the first threshold, it can be determined that the similarity of the two arrays does not exceed the expected range.

[0082] It is understandable that since an array may contain a variety of data, for example, an array may contain dimensional data (such as thickness, width, key dimensions, etc.), electrical parameter data (such as current, voltage, etc.), process data (such as ion implantation concentration, temperature, pressure, etc.) and quantity data (such as the number of chips in a wafer, etc.). The physical units of these data may not be the same. After standardizing the difference in the medians of the two arrays, the impact of different physical units of the data on the similarity analysis can be eliminated, thereby improving accuracy.

[0083] Specifically, the difference between the medians of two arrays can be standardized according to the following formula:

[0084]

[0085] Where D is the standardized value, ΔMedian1 is the absolute value of the difference between the medians of the two arrays, and ΔDATA M is the absolute value of the difference between the data corresponding to the two second specific quantiles, and C is a constant related to the two second specific quantiles, which can usually be calculated by multiples of the standard deviation corresponding to the cumulative percentage of the normal distribution.

[0086] Optionally, the ranges of the two second specific quantiles are 0-50% and 50%-100%, respectively. The two second specific quantiles are preferably 99.865% and 0.135%. The data range covered by the quantiles 99.865% and 0.135% is equivalent to the probability area of ​​6 times the standard deviation of the normal distribution (99.73%), which includes most of the data in the array. Even if the data in the array is non-normally distributed, the analysis result is relatively accurate. In addition, when the two second specific quantiles are 99.865% and 0.135%, C is 6. Of course, the two second specific quantiles can also be other quantiles, in which case C will also change accordingly, and examples will not be given here one by one.

[0087] Figure 2 The median of two arrays and the data corresponding to two specific quantiles provided in this embodiment are shown in FIG. Figure 2 As shown, the medians of the two arrays are 8.73 and 9.28 (either 1 or 2 decimal places are acceptable, and will not be repeated later), the two second specific quantiles are 99.865% and 0.135%, and their corresponding data are 13.6 and 7.3, respectively. The first threshold is 0.5. According to formula 1, the standardized value can be obtained Since 0.52 is greater than 0.5, it can be determined that the similarity between the two arrays exceeds the expected range.

[0088] When using the HC table analysis method, the ratio of the mean difference to the standard deviation (baseline array) is used to compare with the first threshold. Figure 2 It can be seen that the means of the two arrays are 9.42 and 9.1 respectively, and the standard deviation of the baseline array is 1.33, then (9.42-9.1) / 1.33=0.24. Since 0.24 is less than 0.5, it is determined that the similarity of the two arrays does not exceed the expected range (in fact, the difference in the center positions of the two arrays is large, and the similarity is not high).

[0089] It can be seen that using the difference in the medians of two arrays to judge whether the similarity of two arrays exceeds the expected range is more accurate than the conventional similarity analysis method.

[0090] When performing operation 2, it is determined whether the similarity of the two arrays exceeds the expected range based on the difference in the data widths of the two first specific quantiles of the two arrays. Among them, the data widths of the two first specific quantiles are the absolute values ​​of the difference in the data corresponding to the two first specific quantiles of the array. It can be understood that the data width of the first specific quantile can characterize the fluctuation range of the array between the two first specific quantiles. Therefore, the similarity of the two arrays can be evaluated by the difference in the data widths of the first specific quantiles of the two arrays, which is not affected by the distribution mode of the data in the array and the outliers, and can also have a higher accuracy when the data in the array is non-normally distributed. In particular, when the two first specific quantiles are 99.865% and 0.135%, the data range covered by the quantiles 99.865% and 0.135% is equivalent to the probability area of ​​6 times the standard deviation of the normal distribution (99.73%), which includes most of the data in the array. Even if the data in the array is non-normally distributed, the analysis result is relatively accurate.

[0091] In some embodiments, one of the two first specific quantiles may be 75% to 99.865%, and the other may be 0.135% to 25%, but the present invention should not be limited thereto. The two first specific quantiles are preferably 99.865% and 0.135%.

[0092] Furthermore, when executing operation 2, first, one of the two arrays is used as the baseline array and the other is used as the comparison array; the ratio of the data width of the two first specific quantiles of the comparison array to the data width of the two first specific quantiles of the baseline array is obtained; then the relationship between the ratio and the second threshold is determined, and when the ratio is greater than the second threshold, it is determined that the similarity of the two arrays exceeds the expected range, and when the ratio is less than or equal to the second threshold, it is determined that the similarity of the two arrays does not exceed the expected range.

[0093] Figure 3 A schematic diagram of data of two specific quantiles of two arrays provided in this embodiment. Figure 3 As shown, the two first specific quantiles are 99.865% and 0.135%, respectively, and the corresponding data in one array are 3.3 and 2.77, respectively, and the corresponding data in the other array are 3.36 and 2.69, respectively. The second threshold is 1.2, and the data widths of the two first specific quantiles of the two arrays are 3.36-2.69=0.67 and 3.3-2.77=0.53, respectively. The ratio of the data widths of the two first specific quantiles of the comparison array to the data widths of the two first specific quantiles of the baseline array is 0.67 / 0.53=1.26. Since 1.26 is greater than 1.2, it can be determined that the similarity of the two arrays exceeds the expected range.

[0094] When using the HC table analysis method, the ratio of the standard deviation (comparison array) to the standard deviation (baseline array) is used to compare with the second threshold. Figure 3 It can be seen that the standard deviations of the two arrays are 0.14 and 0.118 respectively, so 0.14 / 0.118=1.18. Since 1.18 is less than 1.2, it is determined that the similarity of the two arrays does not exceed the expected range (in fact, the fluctuation range of the two arrays is quite different, and the similarity is not high).

[0095] It can be seen that using the difference in data widths of the two first specific quantiles of the two arrays to judge whether the similarity of the two arrays exceeds the expected range is more accurate than the conventional similarity analysis method.

[0096] When performing operation 3, the difference in the quantile cumulative probabilities of the two arrays is used to determine whether the similarity of the two arrays exceeds the expected range. It can be understood that the quantile cumulative probability is the probability that the data in the array falls between two specific data, which can be applied to all data distribution methods (normal distribution or non-normal distribution) and can characterize the distribution overlap of the array. Therefore, the similarity of the two arrays can be evaluated by the difference in the quantile cumulative probabilities of the two arrays. It is not affected by the distribution method of the data in the array and the outliers, has a wide range of applications, and can also have high accuracy when the data in the array is non-normally distributed.

[0097] Further, when performing operation 3, firstly, one of the two arrays is used as the baseline array and the other is used as the comparison array; then, the fourth specific quantiles corresponding to the two third specific quantiles of the baseline array in the comparison array are obtained; then, the ratio of the absolute value of the difference between the two fourth specific quantiles and the absolute value of the difference between the two third specific quantiles is obtained; the magnitude relationship between the ratio and the third threshold is determined, and when the ratio is less than the third threshold, it is determined that the similarity of the two arrays exceeds the expected range, and when the ratio is greater than or equal to the third threshold, it is determined that the similarity of the two arrays does not exceed the expected range. In particular, when the two third specific quantiles are 99.865% and 0.135%, the data range covered by the quantiles 99.865% and 0.135% is equivalent to the probability area of ​​6 times the standard deviation of the normal distribution (99.73%), which includes most of the data in the array, and even if the data in the array is non-normally distributed, the analysis result is relatively accurate.

[0098] In some embodiments, one of the two third specific quantiles may be 75% to 99.865%, and the other may be 0.135% to 25%, but the present invention should not be limited thereto. The two third specific quantiles are preferably 99.865% and 0.135%.

[0099] Figure 4 This is a schematic diagram of the quantile cumulative probability of the two arrays provided in this embodiment. Figure 4 As shown, the two third specific quantiles of the baseline array are 99.865% and 0.135% respectively, and the two third specific quantiles in the baseline array are 0.50 and 0.52 respectively. 0.50 and 0.52 are brought into the comparison array, and the fourth specific quantiles of the two third specific quantiles 99.865% and 0.135% in the comparison array are calculated to be 96.1% and 0.8% respectively, and the third threshold is 97%. The absolute value of the difference between the two fourth specific quantiles is 95.3%, the absolute value of the difference between the two third specific quantiles is 99.73%, and the ratio of the absolute value of the difference between the two fourth specific quantiles to the absolute value of the difference between the two third specific quantiles is 95.5%. Since 95.5% is less than 97%, it can be determined that the similarity of the two arrays exceeds the expected range.

[0100] When using the HC table analysis method, the probability area overlap of the two arrays is calculated by the mean and standard deviation and compared with the third threshold. Figure 5 for Figure 4 The relationship between the data (x) and the probability density of the two arrays in Figure 5 It can be calculated that Figure 4 The probability area overlap of the two arrays is 98.4%. Since 98.4% is greater than 97%, it is determined that the similarity of the two arrays does not exceed the expected range (in fact, the difference in the distribution overlap of the two arrays is large, and the similarity is not high).

[0101] It can be seen that using the difference in the quantile cumulative probabilities of two arrays to judge whether the similarity of two arrays exceeds the expected range is more accurate than the conventional similarity analysis method.

[0102] In some embodiments, when performing operation 3, one of the two arrays can be first used as a baseline array and the other as a comparison array; then the area between the two third specific quantiles of the baseline array is divided into at least two non-overlapping quantile intervals; then the quantiles corresponding to the maximum quantile and the minimum quantile of each quantile interval in the comparison array are obtained; the absolute value of the difference between the maximum quantile and the minimum quantile in each quantile interval and the difference between the corresponding two quantiles in the comparison array is obtained, and the cumulative difference of all quantile intervals is calculated; then the ratio of the cumulative difference to the absolute value of the difference between the two third specific quantiles is obtained; the relationship between the ratio and the fourth threshold is determined, and when the ratio is less than the fourth threshold, it is determined that the similarity of the two arrays exceeds the expected range, and when the ratio is greater than or equal to the fourth threshold, it is determined that the similarity of the two arrays does not exceed the expected range.

[0103] The cumulative difference of the quantile interval can be obtained according to the following formula:

[0104]

[0105] Among them, ΔDIF totel is the cumulative difference, ΔDIF i is the absolute value of the difference between the maximum quantile and the minimum quantile in the i-th quantile interval and the difference between the two corresponding quantiles in the comparison array, n is the number of quantile intervals, n is greater than or equal to 2.

[0106] Figure 6 A schematic diagram is provided for dividing the area between two third specific quantiles of the baseline array into three non-overlapping quantile intervals for this embodiment. Figure 6 As shown, the two third specific quantiles of the baseline array are 99.865% and 0.135%, and the three non-overlapping quantile intervals are 99.865%-75%, 75%-25% and 25%-0.135%, respectively. The data of 99.865%, 75%, 25%, 0.135% are 0.52, 0.517, 0.511, 0.50, respectively. Bringing 0.52, 0.517, 0.511, 0.50 into the comparison array, it is calculated that the corresponding quantiles of 99.865%, 75%, 25%, 0.135% in the comparison array are 96.1%, 66.9%, 33.2% and 0.8%, respectively. The fourth threshold is 97%. The absolute value of the difference between the two third specific quantiles is 99.73%, the absolute value of the difference between the maximum quantile and the minimum quantile of 99.865% to 75% and the difference between the corresponding two quantiles in the comparison array is (33.2%-0.8%)-(25%-0.135%)=7.5%, the absolute value of the difference between the maximum quantile and the minimum quantile of 75% to 25% and the difference between the corresponding two quantiles in the comparison array is (75%-25%)-(66.9%-33.2%)=13.3%, and the absolute value of the difference between the maximum quantile and the minimum quantile of 25% to 0.135% and the difference between the corresponding two quantiles in the comparison array is (96.1%-66.9%)-(98.865%-75%)=1.3%. According to formula (2), the cumulative difference is 100-(7.5%+13.3%+1.3%) / 2=88.95%, and the ratio of the cumulative difference to the absolute value of the difference between the two third specific quantiles is 88.95% / 99.73%=89%. Since 89% is less than 97%, it can be determined that the similarity of the two arrays exceeds the expected range.

[0107] By calculating the cumulative difference of quantile intervals, we can refer to the probability overlap of multiple quantile intervals of two arrays. The result is more accurate and has a wider range of applicability.

[0108] Corresponding to the above-mentioned embodiment of the data similarity analysis method, this embodiment also provides a data similarity analysis system. Figure 7 This is a structural block diagram of the data similarity analysis system provided in this embodiment. Figure 7 As shown, the data similarity analysis system includes:

[0109] A data supply module 10, for providing two arrays;

[0110] The execution module 20 is configured to perform at least one of the following operations to determine the similarity between two arrays:

[0111] Operation 1: Determine whether the similarity of the two arrays exceeds the expected range based on the difference in the medians of the two arrays;

[0112] Operation 2: Determine whether the similarity of the two arrays exceeds the expected range based on the difference in the data widths of the two first specific quantiles of the two arrays;

[0113] Operation 3: Determine whether the similarity of the two arrays exceeds the expected range based on the difference in the quantile cumulative probabilities of the two arrays.

[0114] Furthermore, this embodiment provides an electronic device that can be used for data similarity analysis. Figure 8 The structural block diagram of the electronic device provided in this embodiment is as follows: Figure 8 As shown, the electronic device includes:

[0115] One or more processors 301;

[0116] Memory 302, used to store one or more programs;

[0117] When one or more programs are executed by one or more of the processors 301 , the one or more processors 301 implement the data similarity analysis method proposed in the above embodiment.

[0118] In this embodiment, the processor 301 and the memory 302 are both one, and the electronic device further includes a communication interface 303. The processor 301, the memory 302, and the communication interface 303 communicate with each other through a communication bus 304. The communication bus 304 mentioned in the above electronic device can be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The communication bus 304 can be divided into an address bus, a data bus, a control bus, etc.

[0119] The memory 302 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the problem explanation method in the embodiment of the present invention. The processor 301 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 302, that is, realizing the above-mentioned problem explanation method.

[0120] The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 302 may further include a memory remotely arranged relative to the processor 301, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0121] The electronic device proposed in this embodiment and the data similarity analysis method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0122] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by the processor 301, the data similarity analysis method proposed in the above embodiment is implemented.

[0123] This embodiment also provides a computer program product, which includes instructions. When the instructions are executed by a processor, the above-mentioned data similarity analysis method is implemented.

[0124] Through the above description of the implementation mode, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation mode. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0125] In summary, in the data similarity analysis method, system, electronic device, medium and product provided in this embodiment, at least one of operations 1 to 3 is used to determine the similarity of the two arrays, wherein operation 1 is to judge whether the similarity of the two arrays exceeds the expected range based on the difference in the medians of the two arrays; operation 2 is to judge whether the similarity of the two arrays exceeds the expected range based on the difference in the data width of the two first specific quantiles of the two arrays; operation 3 is to judge whether the similarity of the two arrays exceeds the expected range based on the difference in the quantile cumulative probability of the two arrays. That is, the present application can measure the similarity of the two arrays from at least one dimension of the center position, fluctuation range and distribution overlap of the two arrays, is not affected by the distribution mode of the data in the arrays and the outliers, can be applied to arrays with normal distribution and non-normal distribution, the analysis is more comprehensive, the application range is wider, and the analysis results are very accurate.

[0126] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0127] It should also be noted that, although the present invention has been disclosed as a preferred embodiment, the above embodiment is not intended to limit the present invention. For any technician familiar with the art, without departing from the scope of the technical solution of the present invention, the technical content disclosed above can be used to make many possible changes and modifications to the technical solution of the present invention, or modified into equivalent embodiments of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

[0128] It should also be understood that, unless otherwise specified or indicated, the terms "first", "second", "third", etc. in the specification are merely used to distinguish between the various components, elements, steps, etc. in the specification, and are not used to indicate the logical relationship or sequential relationship between the various components, elements, steps, etc.

[0129] It should also be recognized that the terms described herein are only used to describe specific embodiments and are not intended to limit the scope of the invention. It should be noted that the singular forms "a" and "an" used herein and in the appended claims include plural references unless the context clearly indicates otherwise. For example, a reference to "a step" or "a device" means a reference to one or more steps or devices, and may include secondary steps and secondary devices. All conjunctions used should be understood in the broadest sense. And, the word "or" should be understood to have the definition of a logical "or", rather than a logical "exclusive or", unless the context clearly indicates otherwise. In addition, the implementation of the method and / or device in the embodiments of the present invention may include performing the selected task manually, automatically, or in combination.

Claims

1. A data similarity analysis method, characterized in that: include: Provide two arrays; Perform at least one of the following operations to determine the similarity of the two arrays: Operation 1: judging whether the similarity of the two arrays exceeds an expected range according to the difference between the medians of the two arrays; Operation 2: judging whether the similarity of the two arrays exceeds an expected range according to the difference in data widths of the two first specific quantiles of the two arrays; Operation 3: judging whether the similarity of the two arrays exceeds an expected range according to the difference in the quantile cumulative probabilities of the two arrays.

2. The data similarity analysis method according to claim 1, characterized in that: At least perform operation 2 and / or operation 3.

3. The data similarity analysis method according to claim 2, characterized in that: The operation 1, the operation 2, and the operation 3 are performed.

4. The data similarity analysis method according to any one of claims 1 to 3, characterized in that: The steps of operation 1 include: Obtaining the absolute value of the difference between the medians of the two arrays and normalizing the difference to obtain a normalized value; and, Determine the size relationship between the standardized value and the first threshold value. When the standardized value is greater than the first threshold value, determine that the similarity between the two arrays exceeds the expected range. When the standardized value is less than or equal to the first threshold value, determine that the similarity between the two arrays does not exceed the expected range.

5. The data similarity analysis method according to claim 4, characterized in that: The absolute value of the difference between the medians of the two arrays is normalized according to the following formula: Where D is the standardized value, ΔMedian1 is the absolute value of the difference between the two medians, and ΔDATA M is the absolute value of the difference between the data corresponding to the two second specific quantiles, and C is a constant related to the two second specific quantiles.

6. The data similarity analysis method according to claim 5, characterized in that: The ranges of the two second specific quantiles are 0-50% and 50%-100% respectively.

7. The data similarity analysis method according to claim 1, characterized in that: The data width of the two first specific quantiles is the absolute value of the difference between the data corresponding to the two first specific quantiles of the array.

8. The data similarity analysis method according to claim 7, characterized in that: The ranges of the two first specific quantiles are 75% to 99.865% and 0.135% to 25% respectively.

9. The data similarity analysis method according to any one of claims 1, 2, 3, 7 or 8, characterized in that: The steps of operation 2 include: One of the two arrays is used as a baseline array, and the other is used as a comparison array; Obtaining a ratio of the data width of the two first specific quantiles of the comparison array to the data width of the two first specific quantiles of the baseline array; and, The relationship between the ratio and a second threshold is determined. When the ratio is greater than the second threshold, it is determined that the similarity between the two arrays exceeds the expected range. When the ratio is less than or equal to the second threshold, it is determined that the similarity between the two arrays does not exceed the expected range.

10. The data similarity analysis method according to any one of claims 1 to 3, characterized in that: The steps of operation 3 include: One of the two arrays is used as a baseline array, and the other is used as a comparison array; Obtaining fourth specific quantiles corresponding to two third specific quantiles of the baseline array in the comparison array; Obtaining a ratio of an absolute value of a difference between two fourth specific quantiles to an absolute value of a difference between two third specific quantiles; The relationship between the ratio and a third threshold is determined. When the ratio is less than the third threshold, it is determined that the similarity between the two arrays exceeds the expected range. When the ratio is greater than or equal to the third threshold, it is determined that the similarity between the two arrays does not exceed the expected range.

11. The data similarity analysis method according to claim 10, characterized in that: The ranges of the two third specific quantiles are 75% to 99.865% and 0.135% to 25%, respectively.

12. The data similarity analysis method according to any one of claims 1 to 3, characterized in that: The steps of operation 3 include: One of the two arrays is used as a baseline array, and the other is used as a comparison array; dividing the interval between two third specific quantiles of the baseline array into at least two non-overlapping quantile intervals; Obtain the quantiles corresponding to the maximum quantile and the minimum quantile of each quantile interval in the comparison array; Obtaining the absolute value of the difference between the maximum quantile and the minimum quantile in each quantile interval and the difference between the two corresponding quantiles in the comparison array, and calculating the cumulative difference of all the quantile intervals; Obtaining the ratio of the cumulative difference to the absolute value of the difference between the two third specific quantiles; and, The relationship between the ratio and a fourth threshold is determined. When the ratio is less than the fourth threshold, it is determined that the similarity between the two arrays exceeds the expected range. When the ratio is greater than or equal to the fourth threshold, it is determined that the similarity between the two arrays does not exceed the expected range.

13. The data similarity analysis method according to claim 12, characterized in that: The cumulative difference of the quantile interval is obtained according to the following formula: Among them, ΔDIF totel is the cumulative difference, ΔDIF i is the absolute value of the ith difference, and n is the number of quantile intervals.

14. The data similarity analysis method according to claim 12, characterized in that: The ranges of the two third specific quantiles are 75% to 99.865% and 0.135% to 25%, respectively.

15. The data similarity analysis method according to claim 1, characterized in that: The distribution mode of the data in the array is normal distribution or non-normal distribution.

16. The data similarity analysis method according to claim 1, characterized in that: The data in the array is greater than or equal to 300.

17. A data similarity analysis system, characterized in that: include: A data supply module, used for providing two arrays; An execution module is used to perform at least one of the following operations to determine the similarity between the two arrays: Operation 1: judging whether the similarity of the two arrays exceeds an expected range according to the difference between the medians of the two arrays; Operation 2: judging whether the similarity of the two arrays exceeds an expected range according to the difference in data widths of the two first specific quantiles of the two arrays; Operation 3: judging whether the similarity of the two arrays exceeds an expected range according to the difference in the quantile cumulative probabilities of the two arrays.

18. An electronic device, characterized in that: include: one or more actuators; as well as, A memory for storing one or more programs; as well as, When the one or more programs are executed by the one or more executors, the one or more executors implement the data similarity analysis method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data similarity analysis method according to any one of claims 1 to 16 is implemented.

20. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are executed by a processor, the data similarity analysis method according to any one of claims 1 to 16 is implemented.

Images