Abnormal data mining method and device, equipment and medium

Abstract

The invention discloses an abnormal data mining method and device, equipment and a medium, and the method comprises the steps: collecting first to-be-detected data, metadata of the first to-be-detected data and log data related to the first to-be-detected data, removing invalid data meeting a preset invalid condition from the log data, and obtaining target log data; determining first abnormal data of the first to-be-detected data, removing the first abnormal data from the first to-be-detected data to obtain second to-be-detected data, clustering the second to-be-detected data to obtain a plurality of clusters, and calculating center data of each cluster, determining outlier candidate data based on the center data and a first distance between the second to-be-detected data in the corresponding cluster; and determining the outlier candidate data meeting a preset abnormal condition as second abnormal data, and then taking the first abnormal data and the second abnormal data as target abnormal data. Abnormal data mining with high adaptability, low cost and high efficiency is realized.

Description

technical field [0001] The invention relates to the technical field of data quality management, in particular to an abnormal data mining method, device, equipment and medium. Background technique [0002] Data quality is a very important part of data management, and the level of data quality governance has a profound impact on the availability of massive data assets. When carrying out abnormal data mining at present, most of the conventional mining modes are based on the requirements of business specifications. By setting the rules of data quality audit and substituting the pre-warning threshold standard of abnormal data, the abnormal data can be monitored by establishing regular and online data quality monitoring tasks. Data mining, this mode has the advantages of strong pertinence and high mining accuracy. It has relatively high usability when facing data quality problems with key topics, but at the same time, it also has poor adaptability and artificial There are many fa...

AI Technical Summary

Problems solved by technology

When carrying out abnormal data mining at present, most of the conventional mining modes are based on the requirements of business specifications. By setting the rules of data quality audit and substituting the pre-warning threshold standard of abnormal data, the abnormal data can be monitored by establishing regular and online data quality monitoring tasks. Data mining, this mode has the advantages of strong pertinence and high mining accuracy. It has relatively high usability when facing data quality problems with key topics, but at the same time, it also has poor adaptability and artificial Many fatal problems such as high cost and the need to adjust the fixed threshold at any time

Images