Outlier data discovery method and system based on low water level sliding time window

Abstract

The invention discloses an outlier data discovery method and system based on a low-water mark sliding time window. The method comprises the following steps: (1) data distribution, that is, external data streams are received and then distributed to various data processing nodes; (2) data processing, that is, the data processing nodes process the received external data streams; the low-water mark sliding time window is defined, a time stamp is used as a horizontal coordinate axis, and the low-water mark sliding time window constantly moves from left to right on the time stamp horizontal coordinate axis as time passes by; at any time point, data in the low-water mark sliding time window above the horizontal coordinate axis is unprocessed data, and data in the low-water mark sliding time window below the horizontal coordinate axis is processed data; and whether currently processed data is outlier data or not is discovered according to a position of a current data processing time stamp in the low-water mark sliding time window; and (3) data aggregation, that is, data processing results are aggregated and output. The method and system has the advantages that discardable data, the outlier data and data to be normally processed can be distinguished, so that the data processing reliability is improved, and failure recovery is accelerated.

Description

technical field [0001] The invention relates to a method for discovering outlier data, in particular to a method and system for discovering outlier data based on a low water level sliding time window. Background technique [0002] Stream processing is the real-time computation of constantly changing streams of data. In order to meet the challenges brought by users' real-time processing of massive data, and to solve the bottleneck problem of real-time processing represented by traditional MapReduce batch processing, the emerging stream processing method can be used in risk management, marketing management, advertising, social recommendation, etc. All aspects have important application value. [0003] The source of data for stream processing is due to network delays, internal concurrency of the system, etc., and the same type of data cannot be guaranteed to arrive at the data processing node in strict order of time stamps, and there are outlier data that are inconsistent in t...

AI Technical Summary

Problems solved by technology

This method provides a judgment method for missing data, but does not give a judgment method for outlier data, and only using time points to indicate low water levels cannot strictly distinguish outlier data

The existing technology Trident avoids outlier data through the strict and orderly requirements of the data to be processed. This method relies on the transaction framework and generates a lot of additional overhead.

Images