A monitoring and alarm system for monitoring airflow scheduling tasks

Abstract

The present invention discloses a monitoring and alarming system for monitoring airflow scheduling tasks, characterized in that the system realizes the timed monitoring of the execution state of airflow tasks, and sends alarm information through multiple channels when data anomalies are detected, so that relevant personnel can respond quickly and deal with problems; the system includes a user configuration module, a DAG data acquisition module, a TASK_ID state analysis module, a data anomaly detection module, an alarm notification module, and a data storage module; the DAG data acquisition module is used to collect task data related to the user configuration module scheduling, and the TASK_ID state analysis module and the data anomaly detection module analyze and detect the task data collected by the DAG data acquisition module, and when anomalies are found, the alarm notification module is called, and the relevant information is stored in the data storage module. The present invention can realize the timed alarming function, support multi-dimensional data anomaly detection, simplify difficult things, improve operation and maintenance efficiency, and enhance system reliability and scalability.

Description

Technical Field

[0001] The invention belongs to the field of information technology, and in particular relates to a monitoring and alarm system for monitoring airflow scheduling tasks. Background Art

[0002] With the development of the information age, more and more enterprises and users use airflow for automatic scheduling and management of data. The present invention provides a monitoring and alarming system for monitoring airflow scheduling tasks, aiming to solve the problem that airflow can only alarm for successful, failed, or timed tasks or dags in the prior art. The system monitors the data during the operation of airflow, alarms for abnormal data, helps users to discover and solve problems in a timely manner, and improves the efficiency and accuracy of data processing. Summary of the invention

[0003] The present invention provides a monitoring and alarm system for monitoring airflow scheduling tasks, aiming to realize regular monitoring of the execution status of airflow tasks and send alarm information through multiple channels when data anomalies are detected so that relevant personnel can respond and handle problems quickly.

[0004] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0005] A monitoring and alarm system for monitoring airflow scheduling tasks includes a user configuration module, a DAG data collection module, a TASK_ID state analysis module, a data anomaly detection module, an alarm notification module, and a data storage module.

[0006] The DAG data collection module is used to collect task data related to the user configuration module scheduling. The TASK_ID status analysis module and the data anomaly detection module analyze and detect the task data collected by the DAG data collection module. When an anomaly is found, the alarm notification module is called and the relevant information is stored in the data storage module.

[0007] The user configuration module provides an intuitive user interface that allows users to configure, manage monitoring tasks, and view historical alarm records; the user configuration unit includes: configuration of DAG_ID monitoring tasks, configuration of TASK_ID monitoring tasks, configuration of interval duration, configuration of check item SQL, data comparison rules, and configuration of data comparison thresholds. The following are some explanations of terms:

[0008]

[0009] The user configuration module process is as follows:

[0010] Step (1): Create a DAG_ID monitoring task. Add a new DAG_ID monitoring task on the DAG page, including the tenant instance, DAG_ID, and whether it is activated.

[0011] Step (2): Create a TASK_ID monitoring task under the DAG_ID monitoring task, including TASK_ID, interval time, and alarm level.

[0012] Step (3): Create a check item task under the TASK_ID monitoring task, including the check item SQL, data comparison rules, and data comparison threshold.

[0013] Step (4): Add / edit alarm recipients.

[0014] Step (5): Assign alarm recipients to the DAG_ID monitoring task.

[0015] DAG data collection module: This module reads the DAG_ID in the user configuration module from the monitoring alarm database, and then collects the running DAG_ID from the airflow database.

[0016] TASK_ID status analysis module: This module reads the target TASK_ID from the monitoring alarm database, obtains the running status of the target TASK_ID from the airflow API interface, and identifies the TASK_ID that has not been successfully run after the DAG_ID running time exceeds the set time threshold, and triggers an alarm;

[0017] The duration threshold of the DAG_ID operation setting is the "interval duration", and the "interval duration" can be adaptively adjusted according to actual conditions.

[0018] The failure to run successfully includes any non-successful state, and the non-successful state includes: not starting to run, waiting for retry, failure, etc.

[0019] The specific implementation of the TASK_ID status analysis module is as follows:

[0020] Step (1): After the DAG data collection module collects the running DAG_ID, it reads the TASK_ID and "interval duration" to be monitored in the DAG_ID from the monitoring alarm database.

[0021] Step (2): Wait for DAG_ID to run. When the running time of DAG_ID reaches the "interval time", check whether the target TASK_ID runs successfully. If it does not run successfully (including the following states: not started, waiting for retry, failed), an alarm is triggered.

[0022] Step (3): In step (2), if TASK_ID runs successfully, proceed to the next step: data anomaly detection.

[0023] Data anomaly detection module: Checks whether there is abnormal data according to the preset inspection item rules, including data quality anomalies (such as data anomalies, internal task calculation failures, etc.) and data volume anomalies (such as data volume is too large or too small). If there is an anomaly, an alarm is triggered. The specific implementation of the data anomaly detection module is as follows:

[0024] Step (1): Read the check item SQL, data comparison rules, and data comparison thresholds previously configured in the user configuration module from the monitoring alarm database.

[0025] Step (2): The data anomaly detection module checks the data according to the configured inspection item rules. The inspection item rules are configured according to actual needs. The specific configurable inspection item rules include but are not limited to the following situations:

[0026] ① Whether the data volume is lower than the set threshold, the inspection item rules are as follows:

[0027] Check item SQL:

[0028] select

[0029] count(1)

[0030] from

[0031] table_A

[0032] where xx ;

[0034] Data comparison rules: less then

[0035] Data comparison threshold: 0

[0036] ②Whether the data volume is higher than the set threshold, the inspection item rules are as follows:

[0037] Check item SQL:

[0038] select

[0039] count(1)

[0040] from

[0041] table_A

[0042] where xx ;

[0044] Check item comparison rules: more then

[0045] Check item data threshold: 30

[0046] ③Track the calculation status, the check item rules are as follows:

[0047] Check item SQL:

[0048] selectstep as step

[0049] case

[0050] when state = 1 then 'not running'

[0051] when state = 2 then 'running'

[0052] when state = 3 then 'success'

[0053] when state = 4 then 'failed'

[0054] end asCalculation status

[0055] from table_B where day_date='{day_date}'and state=4;check item comparison rule: null

[0056] Check item data threshold: 0

[0057] ④ Check whether there is duplicate data. The check item rules are as follows:

[0058] select

[0059] column_a

[0060] ,count(*)

[0061] from table_A

[0062] group by

[0063] column_a

[0064] having count(*)>1

[0065] Check item comparison rule: null

[0066] Check item data threshold: 0

[0067] After receiving the alarm task from the TASK_ID status analysis module or the data anomaly detection module, the alarm notification module reads the alarm level, alarm contact person and alarm contact information of the alarm item from the monitoring alarm database, and sends an email or phone alarm. The alarm methods are as follows:

[0068] Alarm level = 0, the alarm method is to make a phone call and send an email.

[0069] Alert level = 1, alert method is email.

[0070] Beneficial effects of the present invention:

[0071] (1) Scheduled alarm function: The alarm function of airflow can only alarm for running TASKs. However, in actual situations, many important TASK_IDs will not run for a long time because the predecessor nodes run slowly, resulting in the TASK_ID not running to completion at the critical time point, and airflow does not alarm it. The present invention solves this problem very well. After [interval] after the DAG starts running, if the TASK_ID does not run successfully (including the following states: not started, waiting for retry, failed), an alarm will be issued immediately;

[0072] (2) Support multi-dimensional data anomaly detection: The present invention can not only monitor the running status of airflow TASK_ID, but also add anomaly detection of data quality and data volume;

[0073] (3) Unify and simplify difficult things: The data anomaly detection module supports multiple check item rules, including but not limited to the data volume being lower or higher than the threshold, tracking the calculation status, tracking abnormal data, etc. The present invention integrates the more complex data verification into one system, and the data anomaly verification can be completed through multiple check item SQLs;

[0074] (4) User-friendly configuration interface: Provides an intuitive user interface that allows users to easily configure and manage monitoring tasks. Users can customize parameters such as DAG_ID, TASK_ID, interval duration, check item SQL, data comparison rules, and data comparison thresholds to meet personalized monitoring needs.

[0075] (5) Improved operation and maintenance efficiency: By introducing a monitoring and alarm system, the workload of manual monitoring is reduced and the operation and maintenance efficiency is improved. The present invention can automatically identify and report abnormal situations, allowing operation and maintenance personnel to devote more energy to solving problems rather than discovering them.

[0076] (6) Enhanced system reliability: Timely warning and response mechanisms help to quickly resolve potential problems and avoid system failures caused by task failures or data anomalies, thereby enhancing system reliability and stability.

[0077] (7) Scalability: The system is designed with good scalability and can be expanded and optimized as business needs change to adapt to changing monitoring needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is the overall flow chart of the present invention;

[0079] Figure 2 It is a flow chart of the DAG data acquisition module of the present invention;

[0080] Figure 3 It is a flow chart of the TASK_ID state analysis module of the present invention;

[0081] Figure 4 It is a flow chart of the data anomaly detection module of the present invention;

[0082] Figure 5 This is a screenshot of some monitoring alarm logs of the present invention;

[0083] Figure 6 This is a screenshot of the warning email that TASK_ID of the present invention has timed out and failed to run successfully;

[0084] Figure 7 This is a screenshot of the data anomaly (data duplication) detection alarm email of the present invention;

[0085] Figure 8 This is a screenshot of the data anomaly (data volume is higher than the set threshold) detection alarm email of the present invention;

[0086] Fig. 9 This is a screenshot of the data anomaly (data volume is lower than the set threshold) detection alarm email of the present invention;

[0087] Fig.10 This is a screenshot of the data anomaly (no output from the model) detection alarm email of the present invention;

[0088] Fig.11 This is a screenshot of the data anomaly (there are calculation failure steps within the model) detection alarm email of the present invention. DETAILED DESCRIPTION

[0089] The present invention will be described below in conjunction with actual cases, and the purpose and results of the present invention will become more apparent.

[0090] like Figure 1-Figure 5 As shown, a monitoring and alarm system for monitoring airflow scheduling tasks includes a user configuration module, a DAG data collection module, a TASK_ID status analysis module, a data anomaly detection module, an alarm notification module, and a data storage module.

[0091] The user configuration module provides an intuitive user interface that allows users to configure and manage monitoring tasks and view historical alarm records, such as Figure 5 As shown in the figure, the user configuration unit includes: configuration of DAG_ID monitoring task, configuration of TASK_ID monitoring task, configuration of interval duration, configuration of check item SQL, configuration of data comparison rule and configuration of data comparison threshold. The following are some explanations of terms:

[0092]

[0093]

[0094] The user configuration module process is as follows:

[0095] Step (1): Create a DAG_ID monitoring task. Add a new DAG_ID monitoring task on the DAG page, including the tenant instance, DAG_ID, and whether it is activated.

[0096] Step (2): Create a TASK_ID monitoring task under the DAG_ID monitoring task, including TASK_ID, interval time, and alarm level.

[0097] TASK_ID Interval time Alarm level adm.adm__skc_org_basic 30 1 plan_task__py 60 0

[0098] Step (3): Create a check item task under the TASK_ID monitoring task, which specifically includes the check item SQL, data comparison rules, and data comparison thresholds. The specific implementation is as follows:

[0099] TASK_ID: adm.adm__skc_org_basic

[0100] Check item 1: Check whether the data volume of the table is too large. Here, it is set to be greater than or equal to 10,000,000. The configuration is as follows:

[0101] Check item SQL:

[0102] select

[0103] count(1)

[0104] from skc_org_basic

[0105] where day_date='{day_date}'

[0106] and etl_time::date=current_date

[0107] Data comparison rule: more than

[0108] Data comparison threshold: 10000000

[0109] Check Item 2: Check whether there is duplicate data in the table. The configuration is as follows:

[0110] Check item SQL:

[0111] select

[0112] stockorg_sk

[0113] ,skc_sk

[0114] ,count(1)

[0115] from skc_org_basic

[0116] where day_date='{day_date}'

[0117] and etl_time::date=current_date

[0118] group by stockorg_sk

[0119] ,skc_sk having count(1)>1

[0120] Data comparison rule: null

[0121] Data comparison threshold: 0

[0122] TASK_ID: plan_task__py

[0123] Check Item 1: Check whether the model has output. The configuration is as follows:

[0124] Check item SQL:

[0125] select count(1)

[0126] frominstruction where day_date='{day_date}'

[0127] Data comparison rule: less than

[0128] Data comparison threshold: 0

[0129] Check Item 2: Check whether there are any calculation failure steps in the model. The configuration is as follows: Check Item SQL:

[0130] select

[0131] step as

[0132] casewhen state=1then'not running'

[0133] when state = 2 then 'running'

[0134] when state = 3 then 'success'

[0135] when state = 4 then 'failed'

[0136] end asCalculation status

[0137] from step_record where day_date='{day_date}'and state=4;

[0138] Data comparison rule: less than

[0139] Data comparison threshold: 0

[0140] Step (4): Add / edit alarm recipients.

[0141] Step (5): Assign an alarm recipient to the DAG_ID monitoring task: Xiao Ming.

[0142] DAG data collection module: This module reads the DAG_ID in the user configuration module from the monitoring alarm database, and then collects the running DAG_ID from the airflow database.

[0143] TASK_ID status analysis module: This module reads the target TASK_ID from the monitoring alarm database, obtains the running status of the target TASK_ID from the airflow API interface, and identifies the TASK_ID that has not been successfully run after the DAG_ID running time exceeds the set time threshold, and triggers an alarm;

[0144] The duration threshold of the DAG_ID operation setting is the "interval duration", and the "interval duration" can be adaptively adjusted according to actual conditions.

[0145] The failure to run successfully includes any non-successful state, and the non-successful state includes: not starting to run, waiting for retry, failure, etc.

[0146] For TASK_ID: adm.adm__skc_org_basic, the specific implementation is as follows:

[0147] Step (1): After the DAG data collection module collects the running DAG_ID, it reads the TASK_ID and "interval duration" to be monitored in the DAG_ID from the monitoring alarm database.

[0148] Step (2): Wait for DAG_ID to run. When the running time of DAG_ID reaches "30" minutes, check whether adm.adm__skc_org_basic runs successfully. If it does not run successfully (including the following states: not started, waiting for retry, failure), an alarm is triggered.

[0149] Step (3): In step (2), if adm.adm__skc_org_basic runs successfully, the next step is to proceed: data anomaly detection.

[0150] Or for TASK_ID: the specific implementation of plan_task__py is as follows:

[0151] Step (1): After the DAG data collection module collects the running DAG_ID, it reads the TASK_ID and "interval duration" to be monitored in the DAG_ID from the monitoring alarm database.

[0152] Step (2): Wait for DAG_ID to run. When DAG_ID runs for 60 minutes, check whether plan_task__py runs successfully. If it does not run successfully (including the following states: not started, waiting for retry, failed), Figure 6 As shown, an alarm is triggered.

[0153] Step (3): In step (2), if plan_task__py runs successfully, proceed to the next step: data anomaly detection.

[0154] Data anomaly detection module: Checks whether there is abnormal data according to the preset inspection item rules, including data quality anomalies (such as data anomalies, internal task calculation failures, etc.) and data volume anomalies (such as data volume is too large or too small). If there is an anomaly, an alarm is triggered, such as Figure 7-Figure 11 The data anomaly detection module is specifically implemented as follows:

[0155] Step (1): Read the check item SQL, data comparison rules, and data comparison thresholds previously configured in the user configuration module from the monitoring alarm database.

[0156] Step (2): The data anomaly detection module checks the data according to the configured inspection item rules. The inspection item rules are configured according to actual needs. The specific configurable inspection item rules include but are not limited to the following situations:

[0157] According to the above configuration, when the running time of DAG_ID reaches "30" minutes, adm.adm__skc_org_basic runs successfully, and data anomaly detection is performed on it. If any of the following situations occurs, an alarm will be issued:

[0158] ①The amount of data in the skc_org_basic table is greater than or equal to 10,000,000;

[0159] ②The skc_org_basic table contains duplicate data;

[0160] When the running time of DAG_ID reaches "60" minutes, if plan_task__py runs successfully, data anomaly detection will be performed on it, and an alarm will be issued if any of the following situations occur:

[0161] ①Instruction data is empty;

[0162] ②step_record has a calculation failure state;

[0163] After receiving the alarm task from the TASK_ID status analysis module or the data anomaly detection module, the alarm notification module reads the alarm level, alarm contact person and alarm contact information of the alarm item from the monitoring alarm database, and sends an email or phone alarm. The alarm methods are as follows:

[0164] The warning level of adm.adm__skc_org_basic is 1, so Xiao Ming will receive a warning email.

[0165] The warning level of plan_task__py is 0, and Xiao Ming will receive both a warning phone call and a warning email.

[0166] Those skilled in the art will appreciate that the embodiments in the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt a complete hardware embodiment, a complete software embodiment, or a form of an embodiment combining software and hardware. Moreover, the present application may adopt a form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0170] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0171] Obviously, those skilled in the art can make various changes and modifications to the embodiments in the present application without departing from the spirit and scope of the embodiments in the present application. Thus, if these modifications and variations of the embodiments in the present application fall within the scope of the claims and their equivalents in the embodiments of the present application, the embodiments of the present application are also intended to include these modifications and variations.

Claims

1. A monitoring and alarm system for monitoring airflow scheduling tasks, characterized in that: Realize the scheduled monitoring of the execution status of airflow tasks, and send alarm information through multiple channels when data anomalies are detected, so that relevant personnel can respond and handle problems quickly; the system includes user configuration module, DAG data collection module, TASK_ID status analysis module, data anomaly detection module, alarm notification module, and data storage module; the DAG data collection module is used to collect task data related to the user configuration module scheduling, the TASK_ID status analysis module and the data anomaly detection module analyze and detect the task data collected by the DAG data collection module, and when an anomaly is found, the alarm notification module is called and the relevant information is stored in the data storage module.

2. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 1, characterized in that: The user configuration module provides an intuitive user interface that allows users to configure, manage monitoring tasks, and view historical alarm records; the user configuration module includes: configuration of DAG_ID monitoring tasks, configuration of TASK_ID monitoring tasks, configuration of interval duration, configuration of check item SQL, data comparison rules, and configuration of data comparison thresholds.

3. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 2, characterized in that: The user configuration module process is as follows: Step (1): Create a DAG_ID monitoring task. Add a new DAG_ID monitoring task on the DAG page, including the tenant instance, DAG_ID, and whether it is activated; Step (2): Create a TASK_ID monitoring task under the DAG_ID monitoring task, including TASK_ID, interval time, and alarm level; Step (3): Create a check item task under the TASK_ID monitoring task, including the check item SQL, data comparison rules, and data comparison thresholds; Step (4): Add / edit alarm recipients. Step (5): Assign alarm recipients to the DAG_ID monitoring task.

4. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 2, characterized in that: The DAG data collection module reads the DAG_ID in the user configuration module from the monitoring alarm database, and then collects the running DAG_ID from the airflow database.

5. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 2, characterized in that: The TASK_ID status analysis module reads the target TASK_ID from the monitoring alarm database, obtains the running status of the target TASK_ID from the airflow API interface, and identifies the TASK_ID that has not been successfully run after the DAG_ID running time exceeds the set time threshold, and triggers an alarm; The duration threshold of the DAG_ID operation setting is the "interval duration", and the "interval duration" can be adaptively adjusted according to actual conditions.

6. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 5, characterized in that: The specific implementation of the TASK_ID status analysis module is as follows: Step (1): After the DAG data collection module collects the running DAG_ID, it reads the TASK_ID and "interval duration" to be monitored in the DAG_ID from the monitoring alarm database; Step (2): Wait for DAG_ID to run. When the running time of DAG_ID reaches the "interval time", check whether the target TASK_ID runs successfully. If it does not run successfully, trigger an alarm. The failure to run successfully includes any non-successful state, which includes: not started running, waiting for retry, failure, etc. Step (3): In step (2), if TASK_ID runs successfully, proceed to the next step: data anomaly detection.

7. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 6, characterized in that: The data anomaly detection module is used to check whether there is abnormal data according to the preset inspection item rules, including data quality anomalies and data quantity anomalies. If there is an anomaly, an alarm will be triggered.

8. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 6, characterized in that: The data anomaly detection module is specifically implemented as follows: Step (1): Read the check item SQL, data comparison rules, and data comparison thresholds previously configured in the user configuration module from the monitoring alarm database; Step (2): The data anomaly detection module checks the data according to the configured inspection item rules, and the inspection item rules are configured according to actual needs.

9. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 8, characterized in that: Specific configurable inspection item rules include but are not limited to the following situations: ① Whether the data volume is lower than the set threshold, the inspection item rules are as follows: Check item SQL: select count(1) from table_A where xx ； Data comparison rules: less then Data comparison threshold: 0 ②Whether the data volume is higher than the set threshold, the check item rule example is as follows: Check item SQL: select count(1) from table_A where xx ； Check item comparison rules: more then Check item data threshold: 30 ③Track the calculation status, the check item rules are as follows: Check item SQL: select step as case when state = 1 then 'not running' when state = 2 then 'running' when state = 3 then 'success' when state = 4 then 'failed' end asCalculation status from table_B where day_date='{day_date}'and state=4;check item comparison rule: null Check item data threshold: 0 ④ Check whether there is duplicate data. The check item rule example is as follows: select column_a ,count(*) from table_A group by column_a having count(*)>1 Check item comparison rule: null Check item data threshold:

0.

10. A monitoring and alarm system for monitoring airflow scheduling tasks according to claim 9, characterized in that: After receiving the alarm task from the TASK_ID status analysis module or the data anomaly detection module, the alarm notification module reads the alarm level, alarm contact person and alarm contact information of the alarm item from the monitoring alarm database, and sends an email or phone alarm. The alarm methods are as follows: Alarm level = 0, the alarm method is to make a phone call and send an email; Alert level = 1, alert method is email.

Images