Calculation method for improving precision of calculation of sum of squares of deviations from mean

Abstract

A calculation method for improving precision of calculation of a sum of squares of deviations from the mean, comprising: setting successive differences of two variables used for calculation, which are respectively sdx and sdy, the two variables being separately recorded and values of the successive differences of same being separately obtained, cancellation errors that may be generated during calculation of the successive differences being avoided by means of a rounding function, and then calculations being iteratively performed on sums of squares of deviations from the mean. The method can avoid cancellation errors that may be generated during calculation, thus increasing the precision of calculation of a software package that uses the sum of squares of deviations from the mean as the basic statistics, which can be precise up to the 16th digit of significant digits in each step of the iterative calculations of successive difference.

Description

A calculation method to improve the accuracy of the sum of squared deviations from the mean. Technical Field

[0001] This invention relates to the fields of statistical analysis and computer technology, and in particular to a calculation method for improving the accuracy of the sum of squared deviations from the mean. Background Technology

[0002] The sum of squares of the deviation from the mean is calculated by squareding the difference between each observation and the mean, and then summing the squares. It is one of the most important indicators in statistics for describing the dispersion of data distribution. The greater the dispersion of the population distribution, the larger the sum of squares of the deviation from the mean, and the greater the variance.

[0003] The sum of squares of deviations from the mean is an important concept in statistics. It represents the sum of the squared differences between each observation and the mean. This concept measures the dispersion of data and is a key indicator describing the tendency of data distribution to disperse. Specifically, if the sample data are x1, x2, ..., x... n The average is Then the sum of squared deviations from the mean can be calculated. The smaller this value, the more stable the data, meaning the data points are closer to the mean; conversely, a larger value indicates a greater degree of dispersion in the data distribution.

[0004] The formula for calculating the sum of squares of deviations from the mean in Excel is based on the above definition. First, the mean of the data needs to be calculated, which can be done using Excel's AVERAGE function. Then, the difference between each data point and the mean is squared, and all these squared values ​​are summed to obtain the sum of squares of deviations from the mean. This process can be automated using Excel's formula functionality, thereby improving the efficiency of data processing and analysis.

[0005] The mathematical expression for the sum of squared deviations from the mean is:

[0006] ,in

[0007]

[0008] SS n It is the foundation of numerical data analysis, and the unbiased estimator of the sample variance is: ,

[0009] Generalizing this to two variables, we get the covariance, which is calculated using the following formula:

[0010] ,

[0011] It is well known that the binary computer does decimal real number calculation, which inevitably causes calculation error and results in inaccuracy of calculation result. n In addition to the problem of possible calculation inaccuracy, the calculation of SS n is also inefficient and inaccurate. The publication date is March 21, 2015, and the patent application number is TWI477984B, and the name is: a calculation method for improving the accuracy of computer hardware calculation correction product and improving the calculation speed, which discloses the improvement of the error caused by the binary computer for decimal calculation. This method not only improves the calculation speed of binary calculation of decimal, but also reduces the calculation error of such conversion, thereby improving the calculation accuracy of a large number of stacked data.

[0012] This scheme also improves the lack of calculation efficiency under high calculation accuracy, a large number of data accumulation, and the disadvantage of unable to do real-time fast update calculation; the CSP calculation method can also be extended to covariance, correlation coefficient, multivariate covariance matrix and correlation matrix calculation, and the problem that the accuracy and efficiency of CSP calculation cannot be considered at the same time is improved.

[0013] This calculation method mentions 8 items, which are described as follows: the first item introduces CSP and its special case, correction square sum. The second item introduces CSP algorithm. The third item introduces CSS algorithm. The fourth item points out the algorithm used to compare the calculation accuracy of the application. The fifth item is a new CSP calculation method proposed by the inventor of the application. The sixth item explains the application requirement of update calculation. The seventh item explains that the algorithm listed in the fifth item can be further developed into a new algorithm. The eighth item explains that the function of CSP algorithm in several application software produces inaccurate calculation.

[0014] Although this calculation method has improved a lot compared with the previous software calculation scheme, it is found that this method still has some errors under some different calculation methods, especially after the eighth significant digit, there is still a calculation error, which shows that the completely accurate data of this method can only support the eighth significant digit after the eighth significant digit. The method will record the decimal after the eighth digit in other simplified way, and the smaller number will be operated by approximation, at this time, the error after the eighth digit will be caused, which will be amplified after the statistical calculation of a large number of orders, especially when the data is large, the average number of data is small and the variance is large, the error is more obvious. Now the published scheme needs to be improved for further optimization to meet the higher requirement of calculation accuracy.

[0015] Therefore, the present application designs a calculation method for improving the calculation accuracy of the deviation from the mean square sum to solve the above problems. Technical problem

[0016] The purpose of the present application is to provide a calculation method for improving the calculation accuracy of the sum of squares of deviations from the mean, which is improved on the basis of the prior art patent method cited above, to obtain a calculation scheme with higher accuracy, so as to reduce the errors generated by the computer software during calculation, and further improve the calculation accuracy of multi-digit numbers in tables and calculation software. In particular, the method can accurately reach the 16th digit of the significant digits during successive difference calculation, and still maintain the calculation accuracy at the 15th digit, and only needs to add one step of calculating the integer part length of the value during successive difference calculation and rounding operation. The calculation method still maintains high calculation efficiency. Technical solution

[0017] The present application is implemented as follows: a calculation method for improving the calculation accuracy of the sum of squares of deviations from the mean, comprising:

[0018] comprising the following steps:

[0019] Step S1, setting the successive differences of two variables for calculation as sdx and sdy, respectively;

[0020] wherein the successive difference of the first variable is sdx i , the starting value sdx0 of sdx i is 0, x0 is defined as 0, the first value x1 is read, sdx1 is equal to x1 minus 0, when the second value x2 is read, sdx2 is equal to x2 minus x1, when the third value x3 is read, sdx3 is equal to x3 minus x2, and so on.

[0021] The successive difference of the second variable is sdy i , the starting value sdy0 of sdy i is 0, y0 is defined as 0, the first value y1 is read, sdy1 is equal to y1 minus 0, when the second value y2 is read, sdy2 is equal to y2 minus y1, when the third value y3 is read, sdy3 is equal to y3 minus y2, and so on.

[0022] The other variables are defined as follows: the superimposed sum of the successive difference values of the first variable sdx is set as isum_sdx, and the superimposed sum of the successive difference values of the second variable sdy is set as isum_sdy, wherein isum_sdy is defined as 0 during the first calculation, n=2 during the second calculation, and so on.

[0023] The update variable of the corrected product sum is recorded as sp variable during each calculation, and the update variable of the n times corrected product sum is set as ncsp variable.

[0024] Step S2, the length of the integer part of the input value x is calculated, denoted by the symbol len(str(int(x i )); i ));

[0025] The length of the integer part of the input value y is calculated, denoted by the symbol len(str(int(y i )); i ));

[0026] The first variable is subjected to a rounding operation of the successive difference calculation, the 16th - len(str(int(x i ))) bit after the decimal point is taken, and then stored in sdx i ;

[0027] The second variable is subjected to a rounding operation of the successive difference calculation, the 16th - len(str(int(y i ))) bit after the decimal point is taken, and then stored in sdy i ;

[0028] Step S3, (i-1) is multiplied by sdx i , and then multiplied by sdy i , the sum obtained is spx i ;

[0029] ism_sdx is multiplied by sdy, and ism_sdy is multiplied by sdx, the sum obtained is spy i ;

[0030] spx i and spy i are all added together, and the sum sp obtained last time is added, and then stored in sp;

[0031] Step S4, sp is added to ncsp, and then stored in ncsp;

[0032] Step S5, the recursive calculation of ism_sdx and ism_sdy is performed;

[0033] Step S6, (i-1) is multiplied by isum_sdx after adding isum_sdx, and then stored in isum_sdx;

[0034] (i-1) is multiplied by sdyi after adding isum_sdy, and then stored in isum_sdy;

[0035] When the (i+1)th data is added, return to step S1. Advantages

[0036] The advantages of the present application are: the present application extracts x i and yi The integer part length is recorded in a function form, and the two variables are separately subjected to a successive difference calculation, and the separately calculated data is subjected to a rounding operation, so that cancellation error that can be generated in the successive difference calculation is avoided, subsequent calculation is performed, and a more accurate calculation result is obtained. Embodiments of the present application

[0037] [Corrected according to Rule 91 on 11.09.2024] The present application provides a calculation method for improving the calculation accuracy of the deviation from the mean square sum. In order to better understand the above technical solution, the above technical solution is described in detail through the following specific embodiments.

[0038] In one specific embodiment of the technical solution of the present application:

[0039] comprises the following steps:

[0040] Step S1, set the successive difference of two variables for calculation, respectively sdx and sdy;

[0041] Wherein, the successive difference of the first variable is sdxi, the initial value sdx0 of sdxi is 0, x0 is defined as 0, the first value x1 is read, sdx1 is equal to x1 minus 0, when the second value x2 is read, sdx2 is equal to x2 minus x1, when the third value x3 is read, sdx3 is equal to x3 minus x2, and so on.

[0042] The successive difference of the second variable is sdy i , the initial value sdy0 of sdy i is 0, y0 is defined as 0, the first value y1 is read, sdy1 is equal to y1 minus 0, when the second value y2 is read, sdy2 is equal to y2 minus y1, when the third value y3 is read, sdy3 is equal to y3 minus y2, and so on.

[0043] Define other variables: the sum of the successive difference values of the first variable sdx is set as isum_sdx, and the sum of the successive difference values of the second variable sdy is set as isum_sdy, wherein isum_sdy is defined as 0 in the first calculation, n=2 in the second calculation, and so on.

[0044] The update variable of the correction product sum is recorded as sp variable each time, and the update variable of n times of the correction product sum is set as ncsp variable each time.

[0045] Step S2, calculate the input value x iThe length of the integer part, expressed in terms of the symbol len(str(int(x)). i )))express;

[0046] Calculate the input value y i The length of the integer part, expressed in terms of the symbol len(str(int(y)). i )))express;

[0047] Perform a rounding operation on the successive differences of the first variable, and get the 16th decimal place. i Then store it in sdx. i middle;

[0048] Perform a rounding operation on the successive differences of the second variable, rounding to the 16th decimal place. i ))) positions, then store in sdy i middle;

[0049] Step S3, (i-1) multiplied by sdx i Multiply by sdy i The calculated sum is spx i ;

[0050] Multiply ism_sdx by sdy, and then multiply ism_sdy by sdx. The sum of the results is spy. i ;

[0051] SPX i with spy i Add all the values ​​together, then add the sum sp from the previous sum and store it in sp.

[0052] Step S4: After adding ncsp to sp, store it in ncsp;

[0053] Step S5: Perform iterative calculations of ism_sdx and ism_sdy;

[0054] Step S6: Multiply (i-1) by the sum of isum_sdx and store the result in isum_sdx;

[0055] Multiply (i-1) by sdyi, add isum_sdy, and store the result in isum_sdy;

[0056] When the (i+1)th data is added, return to step S1.

[0057] The above steps can be written as the following algorithm that can be executed on a computer, in Python, C++, R, etc., or in an Excel spreadsheet:

[0058] csp:= 0;

[0059] sp := 0;

[0060] isum_sdx := 0;

[0061] isum_sdy := 0;

[0062] x0 = 0;

[0063] y0 = 0;

[0064]

[0065] The above is a calculation method for inputting computer program language, which can be directly applied to computer bottom layer calculation logic.

[0066] It should be noted that:

[0067] 1. The previous scheme does not record the integer part and the decimal part of each data separately, but calculates them together, which leads to the so-called cancellation error. This is the regularity of floating point calculation of computer, especially for data with large constant leading digits length, subtraction will produce larger cancellation error and more serious calculation error. This is disclosed in patent application TW I477984B, named: An algorithm for improving the accuracy of computer hardware calculation and the speed of calculation, the publication date is March 21, 2015. All previous calculations of sum of squares of deviations from the mean have not been disclosed. The biggest breakthrough of this invention is to avoid such cancellation error and improve the calculation accuracy.

[0068] The calculation results and LRE values of the new calculation method proposed in this case are compared with the LRE of US patent: N. US 8,719,303 B2 as shown in the following table. The comparison data in the following table shows that the new method of this case greatly improves the calculation accuracy of the verification data set, especially NumAcc3, NumAcc4 and SmLs07~09.

[0069] Comparison of LRE values of univariate summary statistics data sets, as shown in the following table:

[0070]

[0071] While the foregoing description has described specific embodiments of the application, one ordinary skill in the art will appreciate that various modifications and changes can be made thereto without departing from the spirit and scope of the application, as set forth in the appended claims.

Claims

1. A calculation method for improving the accuracy of the calculation of the sum of squared deviations from the mean, characterized in that, The method comprises the following steps: Step S1, setting the two variables for calculation, sdx and sdy, as the successive difference; wherein the successive difference of the first variable is sdx i , the initial value sdx0 of sdx i is 0, x0 is defined as equal to 0, the first value x1 is read, sdx1 is equal to x1 minus 0, the second value x2 is read, sdx2 is equal to x2 minus x1, the third value x3 is read, sdx3 is equal to x3 minus x2, and so on. The successive difference of the second variable is sdy i The starting value sdy0 of sdy i is 0, y0 is defined as 0, the first value y1 is read, sdy1 is equal to y1 minus 0, the second value y2 is read, sdy2 is equal to y2 minus y1, the third value y3 is read, sdy3 is equal to y3 minus y2, and so on. Defining other variables: the sum of the successive difference values of the first variable sdx, isum_sdx, and the sum of the successive difference values of the second variable sdy, isum_sdy, wherein isum_sdy is defined as 0 in the first calculation, n=2 in the second calculation, and so on; Calculating the updating variable of the correction product sum each time, and recording it as the sp variable; calculating the updating variable of the n times correction product sum each time, and setting it as the ncsp variable; Step S2, compute the integer part length of the input value x i , denoted by the symbol len(str(int(x i ))) ; The integer part of the input value y is computed i with the length of the string len(str(int(y i ))) The rounding operation is performed on the successive difference calculation of the first variable, the 16 - len(str(int(x i ))) bit after the decimal point is taken, and then stored in sdx i ​ The rounding operation is performed on the successive difference calculation of the second variable, and the 16 - len(str(int(y i )))th digit after the decimal point is taken, and then stored in sdy i ​ Step S3, (i-1) is multiplied by sdx i Again multiplied by sdy i The resulting sum is spx i ; Multiply ism_sdx by sdy, ism_sdy by sdx, and compute the sum as spy i ; add sp, sp, #0 i add sp, sp, #0 i add sp, sp, #0 Step S4, storing sp plus ncsp in ncsp; Step S5, performing iterative calculation of ism_sdx and ism_sdy; Step S6, multiplying (i-1) by isum_sdx plus isum_sdx, and storing it in isum_sdx; multiply (i-1) by sdy i add isum_sdy to isum_sdy; When the (i+1)th data is added, returning to step S1 to recalculate the (i+1)th data.

Images