Method for judging distribution asymmetry of big data or streaming data in real time

Abstract

The skewness can be used for judging the degree of asymmetry of big data or streaming data distribution. The invention discloses a method, a system and a computing device program product for calculating the skewness of a subset through incremental computation so as to judge the distribution asymmetry degree of big data or streaming data in real time. The method disclosed in an embodiment of the present invention includes the following steps: incrementally calculating, based on two or more components that calculate a skew of a subset before adjustment, two or more components that calculate a skew of a subset after adjustment; then generating, based on the two or more components that require incremental calculation, the skew of the subset after adjustment. The incremental computation skewness can update the computation result in real time based on the latest data and avoid accessing all data elements in the adjusted computation subset and executing repeated computation, so that the computation efficiency is improved, the computation resources are saved, and the energy consumption of the computation system is reduced. Therefore, real-time judgment of the asymmetry degree of big data or streaming data distribution is efficient and low in consumption, and some real-time judgment of the asymmetry degree of big data or streaming data distribution becomes possible.

Description

technical field [0001] Big data or streaming data analytics. Background technique [0002] The Internet, mobile communications, navigation, online games, sensor technology and large-scale computing infrastructure generate massive amounts of data every day. Big data is data that exceeds the processing capability of traditional database systems and the analysis capabilities of traditional analysis methods due to its huge scale, rapid change and growth rate. [0003] Streaming data is data that is continuously sent by a sender and received by a receiver. Streaming data can be real-time data collected by sensors and continuously transmitted to computing devices or electronic devices. It usually includes successively receiving time-spaced, similarly formatted data elements. Streaming data may also be data that is continuously read from memory, for example, a large data set that exists in the memory of multiple computing devices. [0004] Streaming data processing algorithms c...

AI Technical Summary

Problems solved by technology

After the data has changed, recalculating the skewness on big data or stream data is time-consuming and wastes resources

Images