Data exception monitoring method, device and system

Abstract

The invention discloses a data exception monitoring method, device and system. The method comprises the following steps: loading a data exception detection rule; monitoring a data message queue in real time, and if a data production notification message is monitored, obtaining the produced original data in real time, wherein the data production notification message is written into a data message queue by the data production end; detecting the original data by utilizing the loaded data exception detection rule; if the data exception is detected, recording a data exception result. According to the technical scheme of the invention, the original data is produced at the data production end. Original data produced by the data production end is obtained in time for data anomaly detection. According to the data exception monitoring method and system, the real-time performance of data exception monitoring is greatly improved, data is more accurate and reliable, data guarantee is provided for subsequent data statistical processing, and the problems that an existing data exception monitoring method is poor in real-time performance, data stored in a database has deviation, and service use isaffected are solved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data anomaly monitoring method, device and system. Background technique [0002] For the data statistics system, the accuracy of the data is very important. If the data is inaccurate, the statistical results of the data will be incorrect. The small ones may not be accounted for, and the large ones may cause wrong decisions. For a large-scale distributed Internet system, because the business is very complicated, no one can know it well; and because it is a distributed system, there are many subsystems, and the calls between subsystems are very complicated, and different teams To be responsible, it is also greatly affected by various environmental factors, and abnormal data will appear from time to time. [0003] At present, the monitoring and alarming of abnormal data is mainly detected by setting timed tasks. For example, all the data generated in the previous ...

AI Technical Summary

Problems solved by technology

[0002] For the data statistics system, the accuracy of the data is very important. If the data is inaccurate, the statistical results of the data will be incorrect. The small ones may not be accountable, and the large ones may cause wrong decisions.

For a large-scale distributed Internet system, because the business is very complicated, no one can know it well; and because it is a distributed system, there are many subsystems, and the calls between subsystems are very complicated, and different teams To be responsible, affected by various environmental factors is also greater, abnormal data will appear from time to time

[0003] At present, the monitoring and alarming of abnormal data is mainly detected by setting timed tasks. For example, all the data generated in the previous day are detected every morning. This detection method usually first stores the generated data in the database. In other words, regardless of whether the data is abnormal or not, it will be stored in the database, resulting in the use of abnormal data in subsequent businesses, resulting in data deviation, resulting in poor real-time detection

Images