Method, system, and medium for detecting data collection anomalies in real-time data transmission

Abstract

The application belongs to the technical field of real-time data collection, and discloses a method, system and medium for detecting data collection abnormity in a real-time data transmission process. Business data is obtained by analyzing database operation logs. The business data is stored in a distributed publish-subscribe message system to obtain a real-time data warehouse. A detection time interval is set, and offset information of the real-time data warehouse is obtained according to the time interval. Offset information obtained at a current time node is compared with offset information obtained at a previous time node to obtain a comparison result. If the comparison result is that the offset information has changed, the database operation logs are continuously analyzed to obtain the cause of data collection abnormity. Otherwise, it is determined that data collection is normal, thereby realizing detection of data collection abnormity in the real-time data transmission process. The application can also call a preset intelligent script to repair a data collection program when the collection program is dead.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of real-time data collection, in particular to a method and system for detecting data collection abnormality in real-time data transmission process and a medium. BACKGROUND

[0002] Currently, the core principle of real-time data collection technology is to realize real-time collection of business data by analyzing ORACLE REDO log and MYSQLBINLOG database operation log. However, during real-time data collection using the existing real-time data collection technology, data collection abnormality may occur occasionally. The reasons for data collection abnormality include non-updated upstream data, upstream data master-slave delay and program false death. However, the existing real-time data collection technology can only detect task-level abnormality and cannot detect data-level abnormality. When collecting data using the existing real-time data collection technology, once data collection abnormality occurs, the reason for data collection abnormality can only be determined by human experience, which slows down the data collection efficiency, affects the timeliness and business continuity of data, and also has the risk of data loss.

[0003] Therefore, the present application is proposed. SUMMARY

[0004] The present application solves the technical problem that the existing real-time data collection technology cannot detect the reason for data collection abnormality. The purpose is to provide a method and system for detecting data collection abnormality in real-time data transmission process and a medium, which analyzes the offset information between the database and the distributed publish-subscribe message system at intervals, analyzes the change of adjacent two offset information, determines the reason for data collection abnormality, and realizes the detection of data collection abnormality in real-time data transmission process.

[0005] The present application is realized by the following technical scheme:

[0006] On the one hand, the present application provides a method for detecting data collection abnormality in real-time data transmission process, comprising the following steps: analyzing database operation log to obtain business data; storing the business data into a distributed publish-subscribe message system to obtain a real-time data warehouse; setting a detection time interval, and obtaining offset information of the real-time data warehouse according to the time interval; comparing the offset information obtained at the current time node with the offset information obtained at the last time node to obtain a comparison result; if the comparison result is that the offset information has not changed, continue to analyze the database operation log to obtain the reason for data collection abnormality; otherwise, it is determined that data collection is normal.

[0007] Further, the cause of the data collection anomaly includes no change in upstream data within the time interval, master-slave delay of upstream data within the time interval and no change in slave database data, and a data collection program hang.

[0008] Further, the continuing parsing of the database operation log includes the following steps: checking whether there is data update in the database operation log within the time interval; if there is no data update, determining that the cause of the data collection anomaly is no change in upstream data within the time interval or master-slave delay of upstream data within the time interval and no change in slave database data; otherwise, determining that the cause of the data collection anomaly is a data collection program hang.

[0009] Further, after determining that the cause of the data collection anomaly is a data collection program hang, the method includes the following steps: calling a preset intelligent script to repair the data collection program.

[0010] In another aspect, the present application provides a system for detecting data collection anomaly in real-time data transmission, including an operation log parsing module for parsing a database operation log to obtain business data; a data warehouse generation module for storing the business data into a distributed publish-subscribe message system to obtain a real-time data warehouse; a time interval presetting module for setting a detection time interval; an offset information parsing module for obtaining offset information of the real-time data warehouse according to the time interval; an offset information comparison module for comparing offset information obtained at a current time node with offset information obtained at a previous time node to obtain a comparison result; a comparison result judging module for judging whether the offset information has changed according to the comparison result; if the offset information has not changed, calling the operation log parsing module to continue parsing the database operation log and outputting a cause of the data collection anomaly.

[0011] Further, the operation log parsing module includes a first data parsing unit for first parsing the database operation log to obtain the business data; and a second data parsing unit for continuing to check whether there is data update in the database operation log within the time interval if the offset information has not changed according to the output of the comparison result judging module; if there is no data update, determining that the cause of the data collection anomaly is no change in upstream data within the time interval or master-slave delay of upstream data within the time interval and no change in slave database data; otherwise, determining that the cause of the data collection anomaly is a data collection program hang.

[0012] Further, the system further includes a program repair module for calling a preset intelligent script to repair the data collection program.

[0013] Further, the program repair module comprises a current offset resolution unit, configured to acquire offset information of a current data collection program; and a data collection resumption unit, configured to resume data collection according to the offset information of the current collection program.

[0014] In another aspect, the application provides a medium for detecting data collection anomalies in real-time data transmission, the medium being loaded with a computer program for the above method.

[0015] Compared with the prior art, the application has the following advantages and beneficial effects: 1. By intermittently resolving offset information between a database and a distributed publish-subscribe message system at preset times, and by comparing the changes in the offset information of adjacent two times, the cause of data collection anomalies can be determined, which makes up for the defect that the prior real-time data collection technology can only find task-level anomalies and cannot detect data-level anomalies, and thus solves the problem that when data is collected by using the prior real-time data collection technology, once data collection anomalies occur, the cause of the data collection anomalies can only be determined by human experience, which slows down the data collection efficiency. 2. After the cause of the data collection anomalies is identified, a preset intelligent script is used to acquire the collection offset when the data collection anomalies occur, and the latest offset is submitted for collection, which guarantees the business continuity of real-time data, and solves the problems of data not being updated due to program death, affecting data timeliness and business continuity, and data loss. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the example embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the application, and therefore should not be considered as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0017] Figure 1 A logic diagram of a real-time data transmission data collection anomaly detection method provided for the embodiment 1 of the application;

[0018] Figure 2 A flowchart of a real-time data transmission data collection anomaly detection method provided for the embodiment 1 of the application;

[0019] Figure 3 A structure diagram of a real-time data transmission data collection anomaly detection system provided for the embodiment 2 of the application. DETAILED DESCRIPTION

[0020] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given below in combination with embodiments and drawings. The schematic embodiments and their descriptions are only used to explain the present application, and do not limit the present application.

[0021] Embodiment 1

[0022] The embodiment provides a method for detecting data collection anomaly in real-time data transmission, and the basic principle is referred to the reference Figure 1 Firstly, the architecture of the prior art solution is analyzed, and the database sources are ORACLE REDO logs and MYSQL BINLOG logs; business data is obtained by analyzing the database operation logs, and the business data is output to the message queue KAFKA in real time, which is used for a big data real-time data warehouse system; secondly, the defects existing in the prior art framework are analyzed and the solutions are analyzed. The method is divided into two parts, the first part is to detect the offset of the message queue KAFKA at a certain interval, and compare it with the offset detected last time, if the offset does not change, it can be initially determined that the reason for the data collection anomaly is that the upstream data does not change or the upstream data master-slave delay, and the collection slave database has no data change. When the offset does not change, the upstream ORACLE REDO log or MYSQL BINLOG log is analyzed, if the log changes normally at this time, it is determined that the reason for the real-time data collection anomaly is that the data collection program is dead. When the collection program is dead, the preset intelligent script is called to realize fault self-healing.

[0023] According to the above basic principle, the embodiment provides an implementation process of a method for detecting data collection anomaly in real-time data transmission, as shown in Figure 2 , including the following steps:

[0024] Step 1: Analyzing the ORACLE REDO log and the MYSQL BINLOG log to obtain business data.

[0025] Step 2: The business data obtained by analysis is output to the message queue KAFKA in real time, which is used for a big data real-time data warehouse system.

[0026] Step 3: According to the real-time demand of the business, the detection frequency of the data consistency dynamic verification detection program in the real-time data transmission process is set, for example, the detection frequency is set to be detected and checked once every 10 minutes.

[0027] Step 4: Call the offset interface output by the real-time acquisition application to obtain the offset information of the acquisition table, and compare the currently obtained offset information with the previously obtained offset information. If the comparison result shows that the offset information obtained before and after is inconsistent, it can be determined that the acquisition program is executing normally; if the comparison result shows that the offset information obtained before and after is consistent, continue to parse the database operation log to find out the reason for the data acquisition anomaly; otherwise, it is determined that the data acquisition is normal.

[0028] Step 5: If the offset information obtained in the two consecutive steps is consistent, continue to parse the upstream ORACLE REDO log or MySQL BINLOG log to analyze whether there are any data updates in the log within 10 minutes. If there are no data updates, it means that the upstream data has not changed within 10 minutes, or the upstream data master-slave delay and the data collection slave database has not changed within 10 minutes. In this case, it is determined that the cause of the data collection anomaly is that the upstream data has not changed within the time interval or the upstream data master-slave delay and the data collection slave database has not changed within the time interval. Otherwise, it is determined that the cause of the data collection anomaly is that the data collection program has frozen.

[0029] Step 6: When the acquisition program is determined to be frozen, obtain the offset information of the current acquisition program, and re-acquire data using the offset of the current frozen state to achieve continuous dynamic detection and dynamic reconnection of services.

[0030] In summary, the method for detecting data acquisition anomalies in real-time data transmission provided in this embodiment overcomes the shortcomings of existing technologies that can only detect task-level anomalies, lack self-healing mechanisms, and cannot detect data-level anomalies. This method addresses the current deficiency of acquisition programs appearing to freeze or experiencing abnormal task states, where data is not actually being acquired. By analyzing the ORACLE and KAFKA offset information of the real-time acquisition application and comparing the changes in the nearest neighbor offsets, it determines whether the upstream acquisition source data has not been updated or the acquisition program has frozen and failed to acquire data normally. In the case of data acquisition anomalies, a pre-set intelligent script submits the acquisition with the latest offset, ensuring the business continuity of real-time data and solving the problem of not being able to determine whether the lack of data acquisition is due to no data updates or a freeze in the program itself.

[0031] It should be further noted that the method for detecting data acquisition anomalies in real-time data transmission provided in this embodiment can also be stored in a medium as a computer program, forming a tool-based dynamic detection and handling program. It has the function of real-time acquisition application to fill in the gaps, and it can also be integrated into the real-time acquisition application to complete the monitoring and self-healing functions. At the same time, it provides a monitoring and handling method and fault self-healing means for dynamic verification of data consistency in real-time data warehouse transmission process for similar data acquisition products in the industry.

[0032] Example 2

[0033] Corresponding to Example 1, this example provides a... Figure 3 The system shown is for detecting data acquisition anomalies in real-time data transmission. The system includes: an operation log parsing module for parsing database operation logs to obtain business data; a data warehouse generation module for storing the business data in a distributed publish-subscribe messaging system to obtain a real-time data warehouse; a time interval preset module for setting the detection time interval; an offset information parsing module for obtaining offset information of the real-time data warehouse based on the time interval; an offset information comparison module for comparing the offset information obtained at the current time node with the offset information obtained at the previous time node to obtain a comparison result; a comparison result judgment module for judging whether the offset information has changed based on the comparison result; if the offset information has not changed, the operation log parsing module is called to continue parsing the database operation logs and output the reason for the data acquisition anomaly; and a program repair module for calling a preset intelligent script to repair the data acquisition program.

[0034] The operation log parsing module includes a first data parsing unit for initially parsing the database operation log to obtain business data; and a second data parsing unit for further checking whether the database operation log has been updated within the time interval if the offset information output by the comparison result judgment module has not changed. If no data is updated, the cause of the data acquisition anomaly is determined to be that the upstream data has not changed within the time interval or that the upstream data master-slave delay and the slave database data has not changed within the time interval; otherwise, the cause of the data acquisition anomaly is determined to be that the data acquisition program has frozen. The program repair module includes a current offset parsing unit for obtaining the offset information of the current data acquisition program; and a data acquisition continuation unit for re-acquiring data based on the offset information of the current acquisition program.

[0035] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting data acquisition anomalies in real-time data transmission, characterized in that, Includes the following steps: Parse the database operation logs to obtain business data; The business data is stored in a distributed publish-subscribe messaging system to obtain a real-time data warehouse; Set a detection time interval, and obtain the offset of the message queue from the distributed publish-subscribe messaging system according to the time interval to obtain the offset information of the real-time data warehouse; The offset information obtained at the current time node is compared with the offset information obtained at the previous time node to obtain the comparison result. If the comparison result shows that the offset information has not changed, then continue to parse the database operation log to find the reason for the abnormal data collection. Otherwise, the data collection is considered normal; The continued parsing of the database operation log includes the following steps: Check the database operation log for any data updates within the specified time interval; If there is no data update, the cause of the abnormal data acquisition is determined to be that the upstream data has not changed during the time interval or that the upstream data master-slave delay and the slave data has not changed during the time interval; otherwise, the cause of the abnormal data acquisition is determined to be that the data acquisition program has frozen.

2. The method for detecting data acquisition anomalies in real-time data transmission according to claim 1, characterized in that, The reasons for abnormal data acquisition include no change in upstream data during the time interval, master-slave delay in upstream data and no change in slave data during the time interval, and data acquisition program freezing.

3. The method for detecting data acquisition anomalies in real-time data transmission according to claim 1, characterized in that, If the cause of the abnormal data acquisition is determined to be a freeze in the data acquisition program, the following steps are taken: call a pre-set intelligent script to repair the data acquisition program.

4. The method for detecting data acquisition anomalies in real-time data transmission according to claim 3, characterized in that, Repairing the data acquisition program includes the following steps: Obtain the offset information of the current data acquisition program; Data acquisition will be performed again based on the offset information of the current acquisition program.

5. A system for detecting abnormal data acquisition during real-time data transmission, characterized in that, include The operation log parsing module is used to parse database operation logs to obtain business data; The data warehouse generation module is used to store the business data into a distributed publish-subscribe messaging system to obtain a real-time data warehouse; The time interval preset module is used to set the detection time interval; The offset information parsing module is used to obtain the offset of the message queue from the distributed publish-subscribe messaging system according to the time interval, and obtain the offset information of the real-time data warehouse. The offset information comparison module is used to compare the offset information obtained at the current time node with the offset information obtained at the previous time node to obtain the comparison result. The comparison result judgment module is used to determine whether the offset information has changed based on the comparison result; if the offset information has not changed, the operation log parsing module is called to continue parsing the database operation log and output the reason for the abnormal data collection. The operation log parsing module includes: The first data parsing unit is used to parse the database operation log for the first time to obtain business data; The second data parsing unit is used to continue checking whether the database operation log has been updated within the time interval if the offset information output by the comparison result judgment module has not changed. If there is no data update, it is determined that the cause of the data acquisition anomaly is that the upstream data has not changed within the time interval or that the upstream data master-slave delay and the slave data has not changed within the time interval. Otherwise, it is determined that the cause of the data acquisition anomaly is that the data acquisition program has frozen.

6. A system for detecting abnormal data acquisition during real-time data transmission according to claim 5, characterized in that, It also includes a program repair module, which is used to call pre-set intelligent scripts to repair the data acquisition program.

7. A system for detecting abnormal data acquisition during real-time data transmission according to claim 6, characterized in that, The program repair module includes The current offset parsing unit is used to obtain the offset information of the current data acquisition program; The data acquisition continuation unit is used to re-acquire data based on the offset information of the current acquisition program.

8. A medium for detecting abnormal data acquisition during real-time data transmission, characterized in that, The medium carries a computer program for performing the method of any one of claims 1-4.

Images