Method and system for estimating average value of data flow under sliding window

Abstract

The invention discloses a method for estimating the average value of data flow under a sliding window. The method comprises the following steps that (1) an exponential histogram is initialized, the exponential histogram divides data elements under the sliding window into multiple buckets of different levels, the buckets are used for storing data set information in the data flow in a period of time, and the information comprises timestamps of the data element closest to the current moment in a data set and the sum of data set elements; (2) the information of the data flow is stored in the buckets, the positions of the buckets in the exponential histogram are determined through the timestamps, and when new data are added and old data are deleted, combination and decomposition of the buckets are carried out with the volumes of the buckets as criteria; (3) the exponential histogram is inquired, and an approximate average value is output. The invention further provides a corresponding system for estimating the average value of the data flow under the sliding window. According to the method and system for estimating the average value of the data flow under the sliding window, not all historic data need to be stored, the historic data do not need to be scanned when the average value in the sliding window is calculated, and errors are controllable.

Description

technical field [0001] The invention belongs to the technical field of data flow analysis, and more specifically relates to a method and system for estimating the average value of data flow under a sliding window. Background technique [0002] In recent years, data flow analysis technology has been widely used in sensor network communication, web server click logs, network security monitoring and anomaly detection, call records, securities and stock transactions and other fields. Data flow analysis has become a research hotspot at home and abroad. [0003] Data flow is different from traditional data. It is not a limited collection of data stored on a disk, but an infinite, continuous, fast, real-time collection of data in the form of a stream. In practical applications, the storage space is limited, but the data stream arrives continuously. The application scenarios of data streams have high requirements for real-time performance, but traditional static data mining often n...

AI Technical Summary

Problems solved by technology

However, the current solution needs to scan the historical data in the window and save all the data in a section of the window, which requires high memory and computing power, and is far from meeting the continuous arrival of unlimited data streams and real-time application requirements. Therefore, it is necessary to propose a real-time calculation method based on a single scan

Images