Calculation method for improving calculation accuracy of sum of squares of mean deviation

Abstract

The invention discloses a calculation method for improving the calculation accuracy of the sum of squares of mean deviation, which comprises the following steps of: setting successive differences of two variables for calculation, namely sdx and sdy; wherein the two variables are independently recorded, a successive difference value is independently taken, elimination errors possibly generated during successive difference calculation are avoided through a rounding-off function, and then calculation of the sum of squares of mean deviation is iteratively carried out, after the method is improved, the elimination errors possibly generated during calculation can be avoided, and the calculation accuracy is improved. According to the method, each step of calculation of successive difference iterative calculation can be accurate to the sixteenth digit of an effective number, finally, the calculation accuracy is still kept at the fifteenth digit, and the calculation accuracy of the software package is still kept at the sixteenth digit. And the calculation method can still keep high calculation efficiency only by increasing the calculation cost for calculating the constant leader digit length and the rounding function once in the iterative calculation process.

Description

Technical Field

[0001] The invention relates to the field of statistical analysis and computer technology, and in particular to a calculation method for improving the calculation accuracy of the sum of squares of deviations from the mean. Background Art

[0002] The meaning of the sum of squares of the deviation from the mean is to calculate the difference between each observation and the mean, square them and add them up. It is one of the most important indicators in describing the discrete trend of data distribution in statistics. The greater the dispersion of the overall distribution, the greater the sum of squares of the deviation from the mean, and the greater the variance.

[0003] The sum of squared deviations from the mean is an important concept in statistics. It represents the sum of the squares of the differences between each observation and the mean. This concept is used to measure the dispersion of data and is a key indicator for describing the discrete trend of data distribution. Specifically, if the sample data is x1, x2, ..., x n , the average is Then the sum of squared deviations from the mean can be calculated by The smaller this value is, the more stable the data is, that is, the closer the data point is to the mean, and vice versa, the more dispersed the data distribution is.

[0004] The formula for calculating the sum of squared deviations from the mean in Excel software is set based on the above definition. First, you need to calculate the average of the data, which can be achieved through Excel's AVERAGE function. Then, square the difference between each data point and the mean, and add all these squared values ​​to get the sum of squared deviations from the mean. This process can be automated through Excel's formula function, thereby improving the efficiency of data processing and analysis.

[0005] The mathematical expression for the sum of squared deviations from the mean is:

[0006] in

[0007] SS n It is the basis of numerical data analysis. The unbiased estimator of sample variance is:

[0008] The generalization of two variables is covariance, and its calculation formula is:

[0009]

[0010] As we all know, using a binary computer to perform decimal real number calculations will inevitably produce calculation errors and lead to inaccurate calculation results. In addition to the problem of inaccurate calculations, SSn The calculation of the binary system also has the problem that efficiency and accuracy cannot be taken into account at the same time. The publication date is March 21, 2015, the patent application number is TWI477984B, and the name is: A method for improving the accuracy of the correction product and the calculation speed of computer hardware calculation, which discloses the improvement of the error generated by the decimal calculation of the binary computer. The method not only improves the calculation speed of the binary calculation to the decimal, but also reduces the calculation error of the conversion, thereby improving the calculation accuracy of a large number of stacked data runs.

[0011] This solution also improves the shortcomings of lack of computational efficiency, accumulation of large amounts of data, and inability to perform instant and rapid updates of calculations while having high computational accuracy. The CSP calculation method can also be extended to the calculation of covariance, correlation coefficient, covariance matrix and correlation matrix of multiple variables, improving the problem that the accuracy and efficiency of CSP calculations could not be taken into account at the same time.

[0012] The calculation method mentions 8 statements, which are described as follows: Item 1 introduces CSP and its special case - Corrected Sum of Squares. Item 2 introduces the CSP algorithm. Item 3 introduces the CSS algorithm. Item 4 points out the algorithm used by the present invention to compare calculation accuracy. Item 5 is a new CSP calculation method proposed by the inventor of the present invention. Item 6 explains the application requirements for updating calculations. Item 7 explains that the algorithm listed in Item 5 can be further developed into a space for improvement of new algorithms. Item 8 explains the problem of inaccurate calculations caused by functions related to the CSP algorithm in several application software.

[0013] Although this calculation method has been improved a lot compared to the previous software calculation scheme, it was found that this method still has some errors under some different calculation methods, especially after the 8th significant digit, there are still calculation errors. This shows that the completely accurate data of this calculation method can only support the 8th significant digit. This method will record the decimals after the 8th digit in other simplified ways and perform divisor operations on smaller numbers, which will cause errors in the numbers after the 8th digit. This error will be accumulated and amplified after the sum of statistical calculations of more orders of magnitude, especially when the data volume is large, the average of the data is small and the variance is large. The error is more obvious. Now it is necessary to improve the published scheme for further optimization to meet higher requirements for calculation accuracy.

[0014] Based on this, the present invention designs a calculation method for improving the calculation accuracy of the sum of squares of deviations from the mean to solve the above problem. Summary of the invention

[0015] The object of the present invention is to provide a calculation method for improving the calculation accuracy of the sum of squares of deviations from the mean. The method is further improved on the basis of the previously cited patent method to obtain a calculation scheme with higher accuracy, so as to reduce the error generated by computer software when performing calculations, and further improve the calculation accuracy of multi-digit numbers in tables and calculation software. In particular, the method can be accurate to the 16th significant digit when performing successive difference calculations, and still maintain the calculation accuracy at the 15th digit. Moreover, only one more step of calculating the length of the integer part of the value and performing rounding operations are required when performing successive difference calculations. The calculation method can still maintain high calculation efficiency.

[0016] The present invention is achieved in that a calculation method for improving the calculation accuracy of the sum of squares of deviations from the mean comprises:

[0017] The following steps are involved:

[0018] Step S1, setting the successive differences of two variables used for calculation, sdx and sdy respectively;

[0019] Among them, the successive differences of the first variable are sdx i ,sdx i The starting value sdx0 is 0, and x0 is defined as 0. When the first value x1 is read, sdx1 is equal to x1 minus 0. When the second value x2 is read, sdx2 is equal to x2 minus x1. When the third value x3 is read, sdx3 is equal to x3 minus x2, and so on.

[0020] The successive differences of the second variable are sdy i ,sdy i The starting value sdy0 is 0, y0 is defined as 0, the first value y1 is read, sdy1 is equal to y1 minus 0, when the second value y2 is read, sdy2 is equal to y2 minus y1, when the third value y3 is read, sdy3 is equal to y3 minus y2, and so on;

[0021] Define other variables: the sum of the successive difference values ​​of the first variable sdx is set as isum_sdx, and the sum of the successive difference values ​​of the second variable sdy is set as isum_sdy. In the first calculation, isum_sdy is defined as 0, the second as n=2, and so on;

[0022] Each time the update term variable of the corrected product sum is calculated, it is recorded as the sp variable, and each time the update term variable of the corrected product sum is calculated n times, it is set as the ncsp variable;

[0023] Step S2, calculate the input value x iThe length of the integer part of the string, using the notation len(str(int(x i )))express;

[0024] Calculate the input value y i The length of the integer part of the string, using the notation len(str(int(y i )))express;

[0025] Round the successive differences of the first variable to the 16th - len(str(int(x i ))) bit, and then store it in sdx i middle;

[0026] Round the successive differences of the second variable to the 16th - len(str(int(y i ))) bit, and then store it in sdy i middle;

[0027] Step S3, (i-1) multiplied by sdx i Multiply by sdy i , the calculated sum is spx i ;

[0028] Multiply ism_sdx by sdy, ism_sdy by sdx, and the sum is spy i ;

[0029] spx i With spy i Add all the values ​​of , add them to the previous sum to get the total sp, and store it in sp;

[0030] Step S4, after adding ncsp to sp, store it in ncsp;

[0031] Step S5, recursive calculation of ism_sdx and ism_sdy;

[0032] Step S6, multiplying (i-1) by adding isum_sdx and storing it in isum_sdx;

[0033] Multiply (i-1) by sdyi and add isum_sdy and store in isum_sdy;

[0034] When the i+1th data is added, return to step S1.

[0035] The beneficial effects of the present invention are as follows: the present invention captures x i and iThe integer part length is calculated and accurately recorded in the form of a function. At the same time, the two variables are individually calculated for successive differences, and the individually calculated data are rounded off to avoid the cancellation error that may occur during the successive difference calculations. Subsequent calculations are performed to obtain more accurate calculation results. This method effectively improves the problem that the error becomes larger after a large amount of data is superimposed and then calculated. DETAILED DESCRIPTION

[0036] The present invention provides a calculation method for improving the calculation accuracy of the sum of squared deviations from the mean. In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the specific implementation methods of the specification.

[0037] In a specific embodiment of the technical solution of the present invention:

[0038] The following steps are involved:

[0039] Step S1, setting the successive differences of two variables used for calculation, sdx and sdy respectively;

[0040] Among them, the successive difference of the first variable is sdxi, the starting value sdx0 of sdxi is 0, x0 is defined to be equal to 0, when the first value x1 is read, sdx1 is equal to x1 minus 0, when the second value x2 is read, sdx2 is equal to x2 minus x1, when the third value x3 is read, sdx3 is equal to x3 minus x2, and so on;

[0041] The successive differences of the second variable are sdy i ,sdy i The starting value sdy0 is 0, y0 is defined as 0, the first value y1 is read, sdy1 is equal to y1 minus 0, when the second value y2 is read, sdy2 is equal to y2 minus y1, when the third value y3 is read, sdy3 is equal to y3 minus y2, and so on;

[0042] Define other variables: the sum of the successive difference values ​​of the first variable sdx is set as isum_sdx, and the sum of the successive difference values ​​of the second variable sdy is set as isum_sdy. In the first calculation, isum_sdy is defined as 0, the second as n=2, and so on;

[0043] Each time the update term variable of the corrected product sum is calculated, it is recorded as the sp variable, and each time the update term variable of the corrected product sum is calculated n times, it is set as the ncsp variable;

[0044] Step S2, calculate the input value x i The length of the integer part of the string, using the notation len(str(int(x i)))express;

[0045] Calculate the input value y i The length of the integer part of the string, using the notation len(str(int(y i )))express;

[0046] Round the successive differences of the first variable to the 16th - len(str(int(x i ))) bit, and then store it in sdx i middle;

[0047] Round the successive differences of the second variable to the 16th - len(str(int(y i ))) bit, and then store it in sdy i middle;

[0048] Step S3, (i-1) multiplied by sdx i Multiply by sdy i , the calculated sum is spx i ;

[0049] Multiply ism_sdx by sdy, ism_sdy by sdx, and the sum is spy i ;

[0050] spx i With spy i Add all the values ​​of , add them to the previous sum to get the total sp, and store it in sp;

[0051] Step S4, after adding ncsp to sp, store it in ncsp;

[0052] Step S5, iterative calculation of ism_sdx and ism_sdy;

[0053] Step S6, multiplying (i-1) by adding isum_sdx and storing it in isum_sdx;

[0054] Multiply (i-1) by sdyi and add isum_sdy and store in isum_sdy;

[0055] When the i+1th data is added, return to step S1.

[0056] The above steps can be written as the following algorithm that can be executed on a computer, in Python, C++, R, etc., or in an Excel spreadsheet:

[0057] csp:=0;

[0058] sp:=0;

[0059] isum_sdx:=0;

[0060] isum_sdy:=0;

[0061] x0=0;

[0062] y0=0;

[0063] fori:=1to n;

[0064] sdx:=x i -x i-1

[0065] sdy:=y i -y i-1

[0066] sdx:=round(sdx,16-len(str(int(x i ))));

[0067] sdy:=round(sdy,16-len(str(int(y i ))));

[0068] sp:=sp+isum_sdx*sdy+isum_sdy*sdx+(i-1)*sdx*sdy;

[0069] ncsp:=ncsp+sp;

[0070] isum_sdx:=isum_sdx+(i-1)*sdx;

[0071] isum_sdy:=isum_sdx+(i-1)*sdy;

[0072] csp:=ncsp / n.

[0073] The above is a calculation method for inputting computer program language, which can be directly applied to the underlying calculation logic of the computer.

[0074] It should be noted that:

[0075] 1. The previous scheme does not record the integer part and the decimal part of each data separately, but calculates them together as a whole, which leads to the so-called cancellation error. This is a common practice of floating-point calculations in computers. In particular, when subtracting data with a large constant leading digits length, a larger cancellation error will be generated, resulting in a more serious calculation error. This was published on March 21, 2015, and the patent application number is TWI477984B, entitled: A method for improving the accuracy of computer hardware calculations of corrected products and increasing the calculation speed. All previous calculations of the sum of squares of deviations from the mean have not been disclosed. The biggest breakthrough of the present invention is to avoid this cancellation error and improve the calculation accuracy.

[0076] The calculation results and LRE values ​​of the new calculation method proposed in this case are compared with the LRE of US patent: N.US 8,719,303B2 and are shown in the following table. From the comparison data in the following table, it is found that the new method in this case has greatly improved the calculation accuracy of the verification data set, especially the five data sets of NumAcc3, NumAcc4 and SmLs07~09.

[0077] The comparison of LRE values ​​for the univariate summary statistics data sets is shown in the following table:

[0078]

[0079] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for improving the accuracy of calculating the sum of squared deviations from the mean, characterized in that: The following steps are involved: Step S1, setting the successive differences of two variables used for calculation, sdx and sdy respectively; Among them, the successive differences of the first variable are sdx i ,sdx i The starting value sdx0 is 0, and x0 is defined as 0. When the first value x1 is read, sdx1 is equal to x1 minus 0. When the second value x2 is read, sdx2 is equal to x2 minus x1. When the third value x3 is read, sdx3 is equal to x3 minus x2, and so on. The successive differences of the second variable are sdy i ,sdy i The starting value sdy0 is 0, y0 is defined as 0, the first value y1 is read, sdy1 is equal to y1 minus 0, when the second value y2 is read, sdy2 is equal to y2 minus y1, when the third value y3 is read, sdy3 is equal to y3 minus y2, and so on; Define other variables: the sum of the successive difference values ​​of the first variable sdx is set as isum_sdx, and the sum of the successive difference values ​​of the second variable sdy is set as isum_sdy. In the first calculation, isum_sdy is defined as 0, the second as n=2, and so on; Each time the update term variable of the corrected product sum is calculated, it is recorded as the sp variable, and each time the update term variable of the corrected product sum is calculated n times, it is set as the ncsp variable; Step S2, calculate the input value x i The length of the integer part of the string, using the notation len(str(int(x i )))express; Calculate the input value y i The length of the integer part of the string, using the notation len(str(int(y i )))express; Round the successive differences of the first variable to the 16th - len(str(int(x i ))) bit, and then store it in sdx i middle; Round the successive differences of the second variable to the 16th - len(str(int(y i ))) bit, and then store it in sdy i middle; Step S3, (i-1) multiplied by sdx i Multiply by sdy i , the calculated sum is spx i ; Multiply ism_sdx by sdy, ism_sdy by sdx, and the sum is spy i ; spx i With spy i Add all the values ​​of , add them to the previous sum to get the total sp, and store it in sp; Step S4, after adding ncsp to sp, store it in ncsp; Step S5, iterative calculation of ism_sdx and ism_sdy; Step S6, multiplying (i-1) by adding isum_sdx and storing it in isum_sdx; Multiply sdy by (i-1) i Add isum_sdy and store it in isum_sdy; When the i+1th data is added, return to step S1 to recalculate the i+1 data.

Images