Abnormal data flow online calibration system heurized by brain-like hierarchical memory mechanism

Abstract

The invention relates to an abnormal data flow online calibration system heurized by a brain-like hierarchical memory mechanism. The system comprises a missing data filling module (I), a dimension reduction module (II), a multi-dimensional counting Bloom filter module (III), a hierarchical memory library module (IV), an experience knowledge base module (V) and a calibrated data block module (VI).The I is used for filling up missing data in data blocks in a data stream batch processing link. The II is used for performing low-dimensional representation on the high-dimensional data in the data block. The III is used for judging whether the new data sample is abnormal . The IV is used for storing the historical data sample processed in the step II. The V is used for storing historical data samples which are processed by I but not processed by II. The III and theIV are matched with each other to be replaced, and the IV and the V are matched with each other to be updated; and the VI is usedstoring the replaced and updated new data sample. According to the online calibration system, under the condition that original data distribution is not changed, outliers, noise, missing values and the self-adaptive concept drift phenomenon are corrected in real time.

Description

technical field [0001] The invention belongs to the technical field of big data, and relates to an online calibration system for abnormal data streams inspired by a brain-like hierarchical memory mechanism. Background technique [0002] Due to factors such as external environmental interference, aging physical facilities, network transmission packet loss, etc., there are more or less outliers in the real-time collected data, such as noise, outliers, missing, etc. Data calibration is the first link in the process of data mining and knowledge discovery, and it is also the key to ensure the acquisition of valuable information. Some reports point out that the time spent on data calibration in a project often takes up more than 50% of the overall project time. Traditional data cleaning methods are mainly for static, offline, and non-data flow data sample sets, such as: [0003] (1) Noise smoothing: There are mainly model-based methods and data-driven methods. Model-based method...

AI Technical Summary

Problems solved by technology

[0009] The purpose of the present invention is to solve the problem that there is no data stream online calibration system in the prior art, and to provide an abnormal data stream online calibration system inspired by the brain-like hierarchical memory mechanism

Images