Abnormal data diagnosis method and system based on two-step clustering algorithm

Abstract

The invention provides an abnormal data diagnosis method and system based on a two-step clustering algorithm, and the method comprises a diagnosis class definition step which is used for carrying out user-defined class division of data with a diagnosis requirement, and determining a diagnosis class, a data clustering step used for carrying out clustering processing by applying a two-step clustering algorithm according to a set strategy on the basis of the determined diagnosis class; and am abnormality diagnosis step used for analyzing and calculating the clustered data, determining an abnormality index of each data and discovering target abnormal data. By adopting the scheme, the abnormal data is diagnosed on the basis of the classification formed by clustering and the data distance corresponding to the classification on the basis of the two-step clustering algorithm, the limitation of application data types and data scenes in the existing diagnosis technology can be overcome, the method can be suitable for all users with diagnosis requirements, practicability is better. A more stable and more accurate data diagnosis result is provided for each user, and a reliable resource basis is provided for data analysis and processing work.

Description

technical field [0001] The invention relates to the technical field of data cleaning and processing, in particular to a method and system for diagnosing abnormal data based on a two-step clustering algorithm. Background technique [0002] Data cleaning is an important step in data analysis, and the identification and processing of abnormal data is the cornerstone of effective data cleaning and data analysis. Existing researchers mostly diagnose abnormal data based on the following types of abnormal data identification methods: [0003] 1. The method of identification based on data characteristics. Commonly used identification methods of this type include box plot analysis, standardization and distance identification. Among them, the box plot analysis identifies the data beyond a certain range beyond the upper and lower quartiles through analysis. Data, define satisfied data as abnormal data. The normalization method calculates the normal normalization value of each data, a...

AI Technical Summary

Problems solved by technology

[0006] 1). The selection of the prediction model is not a simple yes / no option. In many cases, the selection of the user model algorithm and the selection of parameters are different, and the evaluation index is usually more than one, so different users may choose different The abnormal data diagnosis results are too dependent on the selection of the prediction model, which leads to relatively high requirements for the user's data processing and analysis professional ability, and poor applicability;

[0007] 2). The reliability of the prediction model cannot be guaranteed, that is, the quality of the prediction model corresponding to each type of data needs to be guaranteed by massive data for calculation and fitting. The data processing volume is large and the calculation process is cumbersome, otherwise the model cannot be truly represent the overall trend of the data;

[0008] 3). For unsupervised data diagnosis scenarios, this type of method is difficult to apply effectively. Usually, this type of method is applied to data with two types of parameters: independent variable (cause) and dependent variable (result). If for unsupervised When there is no obvious and effective trend envoy in the scene or original data, the prediction model cannot be established. It can be seen that the stability and consistency of the method of model identification are not good, which greatly limits its practicability for different data scenarios

Images