Big data skewness increment calculation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field related to data security, in particular to a big data skewness increment calculation method and device, electronic equipment and a storage medium. The method comprises the following steps: carrying out statistics on incremental data to obtain basic statistical information for skewness calculation; acquiring basic statistical information of the original data; calculating the skewness of the current data based on the basic statistical information of the original data and the basic statistical information of the incremental data; wherein the basic statistical information comprises cubic sum, quadratic sum, mean value and quantity. Through the setting, in the scheme provided by the invention, only the incremental data needs to be counted, and the whole data set does not need to be reprocessed, so that the load of a CPU (Central Processing Unit) and a memory can be greatly reduced, a large amount of computing resources are avoided, the processing time consumption is also reduced, and the requirement on equipment performance is reduced.

Description

Technical Field

[0001] The present application relates to the technical field related to data security, and specifically to a method, device, electronic device and storage medium for calculating the skewness increment of big data. Background Art

[0002] With the acceleration of digital transformation, fields ranging from financial risk control, social media, e-commerce to smart manufacturing are experiencing a surge in data volume and facing many challenges in processing massive data sets.

[0003] Skewness is one of the statistics that describes the shape of data distribution. It is mainly used to measure the symmetry of data distribution. The skewness of data distribution affects the shape of the distribution and can be positively skewed (right-skewed), zero-skewed (symmetrical), or negatively skewed (left-skewed). In data science and statistical analysis, the calculation and understanding of skewness is crucial. Through comprehensive skewness analysis, companies and researchers can more accurately identify and interpret asymmetric features in data and develop more effective strategies and decisions.

[0004] In the existing solution, the calculation of skewness depends on the full amount of data, that is, it is necessary to scan the full amount of data. No matter how much the data changes, even if only a small amount of data is added, the entire data set needs to be reprocessed, which will greatly increase the CPU and memory load, consume a lot of computing resources, and increase processing time. As the amount of data grows, performance issues will become more and more obvious. Summary of the invention

[0005] In view of this, the embodiments of the present application are directed to providing a method, device, electronic device and storage medium for calculating the skewness increment of big data.

[0006] The present application provides a method for calculating the incremental skewness of big data, including:

[0007] Perform statistics on incremental data to obtain basic statistical information for skewness calculation;

[0008] Get basic statistical information of raw data;

[0009] Calculating the skewness of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data;

[0010] The basic statistical information includes: cube sum, square sum, mean and quantity.

[0011] In some embodiments, the calculating the skewness of the current data based on the basic statistical information of the original data and the basic statistical information of the incremental data includes:

[0012] Calculate basic statistical information of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data;

[0013] Based on the basic statistical information of the current data, the skewness of the current data is calculated.

[0014] In some embodiments, the calculating the basic statistical information of the current data based on the basic statistical information of the original data and the basic statistical information of the incremental data includes:

[0015] Calculating the cubic sum of current data based on the cubic sum of the original data and the cubic sum of the incremental data;

[0016] Calculating the sum of squares of current data based on the sum of squares of the original data and the sum of squares of the incremental data;

[0017] Calculate the mean of current data based on the mean of the original data and the mean of the incremental data;

[0018] The amount of current data is calculated based on the amount of the original data and the amount of the incremental data.

[0019] In some embodiments, the calculating the skewness of the current data based on the basic statistical information of the current data includes:

[0020] Based on the basic statistical information of the current data, the third-order center distance of the current data is calculated;

[0021] Based on the basic statistical information of the current data, a standard deviation of the current data is calculated;

[0022] The skewness of the current data is calculated based on the standard deviation and the third-order center distance.

[0023] The present application also provides a large data skewness increment calculation device, comprising:

[0024] The statistical module is used to perform statistics on incremental data and obtain basic statistical information for skewness calculation;

[0025] Acquisition module, used to obtain basic statistical information of raw data;

[0026] A calculation module, used for calculating the skewness of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data;

[0027] The basic statistical information includes: cube sum, square sum, mean and quantity.

[0028] The present application also provides an electronic device, comprising:

[0029] A processor, and a memory for storing a program executable by the processor;

[0030] The processor is used to implement the above-mentioned large data skewness increment calculation method by running the program in the memory.

[0031] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the above-mentioned large data skewness increment calculation method.

[0032] The present application provides a method for calculating the incremental skewness of big data. First, the incremental data is counted to obtain basic statistical information for skewness calculation; the basic statistical information of the original data is obtained; based on the basic statistical information of the original data and the basic statistical information of the incremental data, the skewness of the current data is calculated; wherein the basic statistical information includes: the sum of cubes, the sum of squares, the mean and the quantity. With such a setting, only part of the incremental data is counted, and the skewness of the current data is calculated by using the previously known basic statistical information of the original data and the basic statistical information of the incremental data obtained by statistics; that is, the solution provided by the present application only needs to count the incremental data, and there is no need to reprocess the entire data set, which will greatly reduce the load on the CPU and memory, avoid a large amount of computing resources, reduce processing time, and reduce the demand for device performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0034] Figure 1 It is a flowchart of a method for calculating the skewness increment of big data provided by an embodiment of the present application.

[0035] Figure 2 It is a structural diagram of a big data skewness increment calculation device provided by an embodiment of the present application.

[0036] Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] Application Overview

[0039] With the acceleration of digital transformation, fields ranging from financial risk control, social media, e-commerce to smart manufacturing are experiencing a surge in data volume and facing many challenges in processing massive data sets.

[0040] Skewness is one of the statistics that describes the shape of data distribution. It is mainly used to measure the symmetry of data distribution. The skewness of data distribution affects the shape of the distribution and can be positively skewed (right-skewed), zero-skewed (symmetrical), or negatively skewed (left-skewed). In data science and statistical analysis, the calculation and understanding of skewness is crucial. Through comprehensive skewness analysis, companies and researchers can more accurately identify and interpret asymmetric features in data and develop more effective strategies and decisions.

[0041] The calculation formula for skewness is: third-order center distance / cubic standard deviation. The mathematical formula is as follows

[0042]

[0043] where x i For each value in the data, μ is the mean of the data, n is the number of data, and σ is the standard deviation.

[0044] Existing technical solutions

[0045] Apache Hive is a data warehouse software built on the Hadoop ecosystem, especially suitable for processing large-scale data sets. Hive provides some built-in functions and optimizations to make it easier to process and analyze distributed data sets. In this scenario, you can use Hive's standard deviation function, mean function, sum function, count function, and power function to calculate skewness.

[0046] The implementation steps are as follows:

[0047] 1. Use the AVG mean function to obtain the overall mean, then use the SUM sum function to obtain the sum of the differences between each data value and the mean, use the POW power function to obtain the cube of the sum of the differences, and then divide it by the number of fields to obtain the numerator of the skewness formula.

[0048] 2. Use the STDDEV_POP standard deviation function to get the standard deviation of the data set, and then use the POW power function to get the cube of the standard deviation, which is the denominator of the skewness formula.

[0049] 3. Divide the numerator and denominator to get the final skewness calculation result.

[0050] Directly using the above method to calculate the full amount of data requires scanning the full amount of data. No matter how much the data changes, even if only a small amount of data is added, the entire data set needs to be reprocessed. In particular, the STDDEV_POP standard deviation function relies on the full amount of fields and cannot perform incremental calculations. This will greatly increase the CPU and memory load, consume a lot of computing resources, and increase processing time. As the amount of data grows, performance issues will become more and more obvious.

[0051] The built-in standard deviation function in Hive does not mean that other database systems have such functions that can be used directly. If the database changes and the new database system does not have the corresponding function, the SQL code needs to be adjusted, which greatly increases the maintenance cost.

[0052] The purpose of this application is to provide a method for incremental calculation of the skewness coefficient indicator in a big data environment, so as to avoid calculating the full amount of data in a period each time, maintain several basic statistical results required for skewness calculation, integrate the existing historical statistical results with the incremental statistical results each time, and then substitute them into the final formula derived from the skewness calculation formula, and use the basic function to calculate the results, thereby reducing resource consumption and reducing calculation time.

[0053] Furthermore, based on the principle of being more general and easy to implement, the method of the present application should be able to be seamlessly integrated with different versions of data processing frameworks, reduce the usage and maintenance costs caused by platform or system reasons, and meet a wider range of incremental computing needs.

[0054] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0055] Exemplary Methods

[0056] Figure 1 FIG. 1 is a flow chart of a method for calculating the skewness increment of big data provided by an embodiment of the present application. Figure 1 As shown, the method includes the following contents.

[0057] Step S110, performing statistics on the incremental data to obtain basic statistical information for skewness calculation;

[0058] In actual applications, some existing functions of the database can be called to perform statistics on incremental data and calculate basic statistical information.

[0059] Step S120, obtaining basic statistical information of original data;

[0060] It should be noted that when the skewness is calculated last time, the basic statistical information corresponding to the original number can be saved for subsequent calls.

[0061] In some embodiments, basic statistical information of the original data may also be counted and calculated in advance to facilitate subsequent calls.

[0062] Step S130, calculating the skewness of the current data based on the basic statistical information of the original data and the basic statistical information of the incremental data;

[0063] The basic statistical information includes: cube sum, square sum, mean and quantity.

[0064] It should be noted that the selection of basic statistical information plays a very important role in subsequent calculations. In order to better determine the basic statistical information, this application also performs the following operations:

[0065] First, the third-order center distance and standard deviation need to be further converted into formulas.

[0066] The third-order center distance formula is as follows

[0067]

[0068] By calculating the cube value, adding consecutively, combining similar terms, etc., we can get the following convenient calculation formula:

[0069]

[0070] The mathematical symbol formula for standard deviation is

[0071]

[0072] By calculating square values, adding together, and combining similar terms, we can get the following convenient calculation formula:

[0073]

[0074] Through the formula derived from the third-order center distance and standard deviation, it is found that only the following four basic values ​​need to be obtained, and the skewness coefficient result can be obtained through the basic function, but the core purpose is to realize the incremental calculation integration of these four values.

[0075] μ、n

[0076] Furthermore, in order to better call the above information on the computer, you can use variable names to represent these four values

[0077] Cubic sum:

[0078] Sum of squares:

[0079] Mean: var_avg = μ

[0080] Quantity: var_cnt = n

[0081] With such a setting, only part of the incremental data is counted, and the skewness of the current data is calculated by using the basic statistical information of the previously known original data and the basic statistical information of the incremental data obtained by statistics; that is, the solution provided in the present application only needs to count the incremental data, and there is no need to reprocess the entire data set. This will greatly reduce the load on the CPU and memory, avoid a large amount of computing resources, reduce processing time, and reduce the demand for device performance.

[0082] It should be noted that in various database systems, there are basic functions such as summation, counting, mean, power function, and square root, which are represented by SUM, CNT, AVG, POW, and SQRT respectively. tbc represents incremental data, tbh represents original data, and the skewness coefficient of the field col is incrementally calculated.

[0083] The solution provided in this application specifically includes the following steps:

[0084] First, corresponding to step S110: data that may come from multiple data sources are collected in an incremental collection manner, and stored in a database system after unifying the structure.

[0085] The basic value of the newly added incremental data is aggregated and calculated to obtain the following data:

[0086] tbc.var_sum_p3=SUM(POW(col,3))

[0087] tbc.var_sum_p2=SUM(POW(col,2))

[0088] tbc.var_avg = AVG(col)

[0089] tbc.var_cnt = CNT(col)

[0090] Among them, tbc.var_sum_p3 is the sum of the cubes of the incremental data; tbc.var_sum_p2 is the sum of the squares of the incremental data; tbc.var_avg = AVG (col) is the mean of the incremental data; tbc.var_cnt = CNT (col) quantity.

[0091] Integrate the var_sum_p3 of the original data with the var_sum_p3 of the incremental data. Since it is an addition operation, they can be directly added.

[0092] Then, corresponding to step S120, basic statistical information of the original data is obtained;

[0093] Then, corresponding to step S130, the skewness of the current data is calculated based on the basic statistical information of the original data and the basic statistical information of the incremental data, specifically including:

[0094] Calculate basic statistical information of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data;

[0095] Based on the basic statistical information of the current data, the skewness of the current data is calculated.

[0096] The basic statistical information of the current data is calculated based on the basic statistical information of the original data and the basic statistical information of the incremental data, including:

[0097] Based on the cubic sum of the original data and the cubic sum of the incremental data, the cubic sum of the current data is calculated; that is, var_sum_p3 of the original data and var_sum_p3 of the incremental data are integrated. Because it is an addition operation, they can be directly added.

[0098] The sql code is as follows:

[0099] var_sum_p3=tbh.var_sum_p3+tbc.var_sum_p3

[0100] Based on the square sum of the original data and the square sum of the incremental data, the square sum of the current data is calculated; that is, var_sum_p2 of the original data and var_sum_p2 of the incremental data are integrated. Because it is an addition operation, they can be directly added.

[0101] The sql code is as follows:

[0102] var_sum_p2=tbh.var_sum_p2+tbc.var_sum_p2

[0103] Based on the number of the original data and the number of the incremental data, the number of current data is calculated. The var_cnt of the original data and the var_cnt of the incremental data are integrated. Since the total is to be calculated, they can be directly added.

[0104] The sql code is as follows:

[0105] var_cnt=tbh.var_cnt+tbc.var_cnt

[0106] Based on the mean of the original data and the mean of the incremental data, the mean of the current data is calculated; that is, the var_avg of the original data and the var_avg of the incremental data are integrated. Here, the total sum of the col field is divided by the total count result to obtain the mean, without using the AVG function to calculate the average of the entire amount. The total count result can be obtained through the content provided above, and the total sum result can be obtained by multiplying the historical mean and the historical count plus the product of the incremental data mean and the incremental data.

[0107] The sql code is as follows:

[0108] var_avg=(tbh.var_cnt x tbh.var_avg+tbc.var_cnt x tbc.var_avg) / (tbh.var_cnt+tbc.var_cnt)

[0109] It should be noted that if there is more incremental data, the above steps can be repeated to obtain the integration result of the original data and the four basic values ​​of the incremental data. Later, the third-order center distance and standard deviation can be obtained by substituting them into the previously derived formula, and thus the skewness coefficient can be obtained.

[0110] Specifically, based on the basic statistical information of the current data, calculating the skewness of the current data includes: based on the basic statistical information of the current data, calculating the third-order center distance of the current data; that is, obtaining the third-order center distance

[0111]

[0112] Change to sql code as follows

[0113] (var_sum_p3 / var_cnt)–(3x var_avg x var_sum_p2 / var_cnt)+(2xPOW(var_avg,3))

[0114] Based on the basic statistical information of the current data, the standard deviation of the current data is calculated:

[0115] That is: find the standard deviation

[0116]

[0117] The sql code is as follows:

[0118] SQRT(var_sum_p2 / var_cnt-POW(var_avg,2))

[0119] The skewness of the current data is calculated based on the standard deviation and the third-order center distance.

[0120] That is, by dividing the result value of the third-order center distance by the cube of the standard deviation, the skewness coefficient value of the original data of the col field can be obtained.

[0121] In summary, the solution provided by this application does not directly use complex functions to obtain key indicators, but further decomposes the skewness formula into a basic formula that can be implemented using basic functions in the database system. By maintaining the historical aggregation results of four basic indicators that can achieve incremental calculations, integrating them with the aggregation results of incremental data, and then substituting them into the basic formula, the incremental calculation strategy method of the skewness coefficient is further obtained. It only needs to count the incremental data, and there is no need to reprocess the entire data set, which will greatly reduce the load on the CPU and memory and avoid a large amount of computing resources.

[0122] Exemplary Devices

[0123] The device embodiments of the present application can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0124] Figure 2 FIG. 1 is a block diagram of a large data skewness increment calculation device provided by an embodiment of the present application. Figure 2 As shown, the device comprises:

[0125] A statistical module 21 is used to perform statistics on the incremental data to obtain basic statistical information for skewness calculation;

[0126] An acquisition module 22 is used to acquire basic statistical information of raw data;

[0127] A calculation module 23, configured to calculate the skewness of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data;

[0128] The basic statistical information includes: cube sum, square sum, mean and quantity.

[0129] In some embodiments, the calculating the skewness of the current data based on the basic statistical information of the original data and the basic statistical information of the incremental data includes:

[0130] Calculate basic statistical information of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data;

[0131] Based on the basic statistical information of the current data, the skewness of the current data is calculated.

[0132] In some embodiments, the calculating the basic statistical information of the current data based on the basic statistical information of the original data and the basic statistical information of the incremental data includes:

[0133] Calculating the cubic sum of current data based on the cubic sum of the original data and the cubic sum of the incremental data;

[0134] Calculating the sum of squares of current data based on the sum of squares of the original data and the sum of squares of the incremental data;

[0135] Calculate the mean of current data based on the mean of the original data and the mean of the incremental data;

[0136] The amount of current data is calculated based on the amount of the original data and the amount of the incremental data.

[0137] In some embodiments, the calculating the skewness of the current data based on the basic statistical information of the current data includes:

[0138] Based on the basic statistical information of the current data, the third-order center distance of the current data is calculated;

[0139] Based on the basic statistical information of the current data, a standard deviation of the current data is calculated;

[0140] The skewness of the current data is calculated based on the standard deviation and the third-order center distance.

[0141] Exemplary Electronic Devices

[0142] Below, reference Figure 3 To describe an electronic device according to an embodiment of the present application. Figure 3 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0143] like Figure 3 As shown, electronic device 300 includes one or more processors 310 and memory 320 .

[0144] The processor 310 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 300 to perform desired functions.

[0145] The memory 320 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 310 may run the program instructions to implement the big data skewness increment calculation method of each embodiment of the present application described above and / or other desired functions. Various contents such as category correspondences may also be stored in the computer-readable storage medium.

[0146] In one example, the electronic device 300 may further include: an input device 330 and an output device 340 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0147] In addition, the input device 330 may also include, for example, a keyboard, a mouse, an interface, etc. The output device 340 may output various information to the outside, including analysis results, etc. The output device 340 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0148] Of course, to simplify, Figure 3 Only some of the components in the electronic device related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0149] Exemplary computer program products and computer-readable storage media

[0150] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the big data skewness increment calculation method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0151] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0152] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the processor executes the steps of the method for calculating the skewness increment of large data according to various embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0153] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for calculating the skewness increment of big data, characterized in that: include: Perform statistics on incremental data to obtain basic statistical information for skewness calculation; Get basic statistical information of raw data; Calculating the skewness of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data; The basic statistical information includes: cube sum, square sum, mean and quantity.

2. The method for calculating the skewness increment of big data according to claim 1, characterized in that: The calculating the skewness of the current data based on the basic statistical information of the original data and the basic statistical information of the incremental data includes: Calculate basic statistical information of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data; Based on the basic statistical information of the current data, the skewness of the current data is calculated.

3. The method for calculating the skewness increment of big data according to claim 2, characterized in that: The calculating the basic statistical information of the current data based on the basic statistical information of the original data and the basic statistical information of the incremental data includes: Calculating the cubic sum of current data based on the cubic sum of the original data and the cubic sum of the incremental data; Calculating the sum of squares of current data based on the sum of squares of the original data and the sum of squares of the incremental data; Calculate the mean of current data based on the mean of the original data and the mean of the incremental data; The amount of current data is calculated based on the amount of the original data and the amount of the incremental data.

4. The method for calculating the skewness increment of big data according to claim 1, characterized in that: The calculating the skewness of the current data based on the basic statistical information of the current data includes: Based on the basic statistical information of the current data, the third-order center distance of the current data is calculated; Based on the basic statistical information of the current data, a standard deviation of the current data is calculated; The skewness of the current data is calculated based on the standard deviation and the third-order center distance.

5. A big data skewness increment calculation device, characterized in that: include: The statistical module is used to perform statistics on incremental data and obtain basic statistical information for skewness calculation; Acquisition module, used to obtain basic statistical information of raw data; A calculation module, used for calculating the skewness of current data based on the basic statistical information of the original data and the basic statistical information of the incremental data; The basic statistical information includes: cube sum, square sum, mean and quantity.

6. An electronic device, characterized in that: include: A processor, and a memory for storing a program executable by the processor; The processor is used to implement the large data skewness increment calculation method according to any one of claims 1 to 4 by running the program in the memory.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the large data skewness increment calculation method according to any one of claims 1 to 4.

Images