Information gain computation based mass data abnormality detecting method

Abstract

The invention discloses an information gain computation based mass data abnormality detecting method, which comprises an off-line stage processing step and an on-line processing step. The off-line processing step is used for generating statistical information, and the on-line processing step is used for quickly generating analysis results in an approximate process manner on the basis of the statistical information. Accordingly, the problems of memory outflow, long processing time and high expense and the like caused when mass data are analyzed by the convention algorithm are solved, and analysis performance is improved.

Description

technical field [0001] The invention relates to a method for solving key classifications only by traversing a database once, and belongs to the technical field of data mining and knowledge discovery. Background technique [0002] Data mining technology obtains useful knowledge from various data collections. Since the mid-1990s, data mining technology has been deeply applied in many fields, such as finance, logistics, transportation, scientific research and other fields. Typical data mining algorithms include classification, clustering, association rules, regression analysis, etc. Since the beginning of the 21st century, the scale of data to be processed in many fields has become larger and larger, and often cannot be solved directly by applying traditional data mining algorithms. It is necessary to develop new algorithms and improve some key steps to solve related problems. [0003] The present invention researches an anomaly detection technology for massive data, and its ...

AI Technical Summary

Problems solved by technology

[0004] The present invention overcomes the defects of memory overflow and excessive processing time overhead caused by traditional algorithms in the prior art when analyzing massive scale data, and proposes a massive data anomaly detection method based on information gain calculation

The invention proposes a new method to process massive information through two different stages, that is, the offline stage and the online stage, and solves the problems caused by traditional algorithms when analyzing massive-scale data in view of the large-scale data and the relatively insufficient memory of the computer system. problems such as memory overflow and excessive processing time overhead, thereby improving analysis performance

Images