Real-time precise monitoring and intelligent alarm recovery methods, devices, media and equipment

Abstract

This invention discloses a method, apparatus, medium, and device for precise real-time task monitoring and intelligent alarm recovery. The method includes: performing alarm rule configuration operations; collecting operational metrics data of Flink real-time tasks; managing alarms based on the operational metric data; and performing task recovery operations when the task status information of the current Flink real-time task is confirmed as a task failure. This invention configures personalized alarm rules for alarm metrics of different tasks, providing excellent flexibility. In alarm management, by setting multi-level judgments, key alarm information can be effectively filtered out, improving alarm accuracy. In acquiring operational metric data, by exposing the operational metrics managed internally by the Flink real-time task jobmanager in a multi-process manner, the real-time performance and accuracy of data collection are improved. It also features an automatic restart function based on alarm priority, which can quickly determine and automatically restart the task according to a preset strategy when a task fails.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, medium, and device for real-time precise monitoring of tasks and intelligent alarm recovery. Background Technology

[0002] In big data processing, Flink tasks refer to real-time data processing jobs running within a distributed stream processing framework. Existing task monitoring for Flink tasks typically uses Pushgateway (a temporary metric relay component in the Prometheus ecosystem) as a relay station for Flink metrics (quantitative metrics generated during Flink task execution). This single-machine model has significant scalability issues, and as the task volume increases, Pushgateway becomes a performance bottleneck, leading to Pushgateway anomalies and a high risk of single-point failure. A Pushgateway outage will cause the entire monitoring system to fail. Furthermore, alarm management lacks flexibility, with various alarms lacking effective management, resulting in low operational efficiency. When task failures are caused by YARN cluster anomalies (a general resource management and job scheduling system responsible for the elastic scaling of Flink tasks), there are also no corresponding recovery methods.

[0003] Therefore, there is an urgent need for a real-time, precise monitoring method for tasks and an intelligent alarm recovery method. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a method, apparatus, medium and device for real-time task precision monitoring and intelligent alarm recovery that overcomes or at least partially solves the above problems.

[0005] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0006] According to a first aspect of the present invention, a method for real-time precise monitoring and intelligent alarm recovery of tasks is provided, the method comprising:

[0007] S1. Perform alarm rule configuration operation to generate alarm configuration information corresponding to the alarm rules of Flink real-time tasks. The alarm rules include at least task indicator alarm rules, data quality alarm rules and alarm trigger time rules, and store the configured alarm configuration information in the database.

[0008] S2. After exposing the running metrics corresponding to the current Flink real-time task in the form of multiple processes, the running metrics of the Flink real-time task are collected to obtain the running metric data, and the real-time output table of the current Flink real-time task is obtained.

[0009] S3. Within a preset interval, determine whether the real-time output table is within the data quality alarm rule range of the alarm configuration information. If the real-time output table is within the data quality alarm rule range, then execute step S4. If the real-time output table is not within the data quality alarm rule range, then execute step S7.

[0010] S4. Based on the task indicator alarm rules in the alarm configuration information, perform threshold judgment on the collected operation indicator data, and confirm that the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules. If the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules, generate alarm information and execute step S5. If the indicator parameters in the operation indicator data do not exceed the threshold range of the task indicator alarm rules, execute step S7.

[0011] S5. Confirm whether the triggering time of generating alarm information and the triggering interval between two adjacent alarm information are within the range of the alarm triggering time rule. If the triggering time and the triggering interval between two adjacent alarm information are within the range of the alarm triggering time rule, then execute step S6. If the triggering time and the triggering interval between two adjacent alarm information are not within the range of the alarm triggering time rule, then execute step S7.

[0012] S6. Confirm whether an alarm notification policy is configured in the alarm configuration information. The alarm notification policy includes the alarm level corresponding to the Flink real-time task and the notification channel corresponding to each alarm level. If a notification policy is configured in the alarm configuration information, the alarm information is notified according to the notification policy. If no notification policy is configured in the alarm configuration information, the alarm information is notified in the default way.

[0013] S7. Monitor the task status of the current Flink real-time task in real time, obtain the task status information of the current Flink real-time task, confirm whether the task status information of the current Flink real-time task indicates task failure, and perform a task recovery operation when it is confirmed that the task status information of the current Flink real-time task indicates task failure.

[0014] In some embodiments of the present invention, the step of real-time monitoring of the task status of the current Flink real-time task, obtaining the task status information of the current Flink real-time task, and confirming whether the task status information of the current Flink real-time task indicates task failure includes:

[0015] The task status query results are obtained from the YARN cluster and / or the Flink real-time task execution interface based on the unique identifier of the current Flink real-time task through the xxl-job task scheduling module.

[0016] If any of the task status query results corresponding to the YARN cluster and the Flink real-time task execution interface indicate that the task has failed, then the task status information of the current Flink real-time task is confirmed as a task failure.

[0017] In some embodiments of the present invention, the task recovery operation includes:

[0018] Obtain the most recent restart time of the Flink real-time task, and obtain the current restart interval based on the restart time and the current time, and determine whether the current restart interval is greater than the preset restart interval;

[0019] If the current restart interval is not greater than the preset restart interval, the process ends;

[0020] If the current restart interval is greater than the preset restart interval, then a task recovery operation is performed to obtain the cached data of the Flink real-time task in the database and restart the task.

[0021] After the Flink real-time task restarts, the task status information of the current Flink real-time task is obtained again. If the task status information of the Flink real-time task is in the task execution state, the process ends. If the task status information of the Flink real-time task is in the task failure state, step S2 is executed.

[0022] In some embodiments of the present invention, the method further includes: after generating the alarm information, storing the operating indicator data and the alarm information in a database.

[0023] According to a second aspect of the present invention, a real-time task precision monitoring and intelligent alarm recovery device is provided, the real-time task precision monitoring and intelligent alarm recovery device comprising:

[0024] The task alarm rule configuration module is used to perform alarm rule configuration operations, generate alarm configuration information corresponding to the alarm rules of Flink real-time tasks, and the alarm rules include at least task indicator alarm rules, data quality alarm rules and alarm trigger time rules, and store the configured alarm configuration information in the database.

[0025] The real-time monitoring module is used to expose the running metrics corresponding to the current Flink real-time task in the form of multiple processes, collect the running metrics data of the Flink real-time task, and obtain the real-time output table of the current Flink real-time task.

[0026] The alarm management module is used to determine, within a preset interval, whether the real-time output table is within the data quality alarm rules of the alarm configuration information; to perform threshold judgment on the collected operation indicator data based on the task indicator alarm rules in the alarm configuration information, confirming that the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules; to confirm whether the trigger time of generating alarm information and the trigger interval between two adjacent alarm information are within the range of the alarm trigger time rules; and to confirm whether an alarm notification policy is configured in the alarm configuration information, wherein the alarm notification policy includes the alarm level corresponding to the Flink real-time task and the notification channel corresponding to each alarm level. If a notification policy is configured in the alarm configuration information, the alarm information is notified according to the notification policy; if no notification policy is configured in the alarm configuration information, the alarm information is notified in the default manner.

[0027] The intelligent recovery module is used to monitor the task status of the current Flink real-time task in real time, obtain the task status information of the current Flink real-time task, confirm whether the task status information of the current Flink real-time task indicates task failure, and perform task recovery operation when it is confirmed that the task status information of the current Flink real-time task indicates task failure.

[0028] In some embodiments of the present invention, the intelligent recovery module monitors the task status of the current Flink real-time task in real time, obtains the task status information of the current Flink real-time task, and confirms whether the task status information of the current Flink real-time task indicates task failure, including:

[0029] The task status query results are obtained from the YARN cluster and / or the Flink real-time task execution interface based on the unique identifier of the current Flink real-time task through the xxl-job task scheduling module.

[0030] If any of the task status query results corresponding to the YARN cluster and the Flink real-time task execution interface indicate that the task has failed, then the task status information of the current Flink real-time task is confirmed as a task failure.

[0031] In some embodiments of the present invention, the intelligent recovery module performs task recovery operations including:

[0032] Obtain the most recent restart time of the Flink real-time task, and obtain the current restart interval based on the restart time and the current time, and determine whether the current restart interval is greater than the preset restart interval;

[0033] If the current restart interval is not greater than the preset restart interval, the process ends;

[0034] If the current restart interval is greater than the preset restart interval, then a task recovery operation is performed to obtain the cached data of the Flink real-time task in the database and restart the task.

[0035] After the Flink real-time task restarts, the task status information of the current Flink real-time task is obtained again. If the task status information of the Flink real-time task is in the task execution state, the process ends. If the task status information of the Flink real-time task is in the task failure state, step S2 is executed.

[0036] In some embodiments of the present invention, the alarm management module is further configured to: after generating the alarm information, store the operating indicator data and the alarm information in a database.

[0037] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein computer program instructions are stored therein, the computer program instructions being loaded and executed by a processor to perform the operations performed by the method described in any of the preceding claims.

[0038] According to a fourth aspect of the present invention, an electronic device is provided, including a processor and a memory, the memory storing computer program instructions executable by the processor, wherein when the processor executes the computer program instructions, it implements the instructions of any of the methods described above.

[0039] The technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0040] This invention provides a method, apparatus, medium, and device for precise real-time task monitoring and intelligent alarm recovery. The method configures personalized alarm rules for different task alarm indicators based on the criticality of Flink real-time tasks, offering excellent flexibility. In alarm management, multi-level judgments effectively filter out key alarm information, improving alarm accuracy and operational efficiency. Regarding the acquisition of operational indicator data, it bypasses the traditional Pushgateway by exposing operational indicators managed internally by the Flink real-time task jobmanager in a multi-process manner, achieving efficient collection and transmission of monitoring data and improving the real-time performance and accuracy of data collection. It also features an automatic restart function based on alarm priority, quickly identifying and automatically restarting tasks according to preset strategies when a task fails.

[0041] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a real-time task precision monitoring and intelligent alarm recovery method provided in an embodiment of the present invention;

[0044] Figure 2 This is a reference pseudocode for exposing operational metrics in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram illustrating the principle structure of a real-time task precision monitoring and intelligent alarm recovery device provided in an embodiment of the present invention. Detailed Implementation

[0046] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings.

[0047] The accompanying drawings illustrate various structural schematics according to embodiments of the present disclosure. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0048] In the context of this disclosure, when a layer / component is referred to as being "above" another layer / component, that layer / component may be directly above the other layer / component, or there may be an intermediate layer / component between them. Additionally, if a layer / component is "above" another layer / component in one orientation, then when the orientation is reversed, that layer / component may be "below" the other layer / component. In the context of this disclosure, similar or identical components may be denoted by the same or similar reference numerals.

[0049] To better understand the above technical solutions, the following will describe the above technical solutions in detail with reference to specific implementation methods. It should be understood that the embodiments of this disclosure and the specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0050] Figure 1 This is a flowchart illustrating a real-time task precision monitoring and intelligent alarm recovery method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, this real-time task precision monitoring and intelligent alarm recovery method includes the following steps:

[0051] S1. Perform alarm rule configuration operation to generate alarm configuration information corresponding to the alarm rules of Flink real-time tasks. The alarm rules include at least task indicator alarm rules, data quality alarm rules and alarm trigger time rules, and store the configured alarm configuration information in the database.

[0052] In this embodiment of the invention, the alarm rules are configured through a user interface on a terminal, such as a PC or a smart device. Users can perform alarm rule configuration operations through the user interface, configuring task indicator alarm rules, data quality alarm rules, and alarm trigger time rules, etc. After the alarm configuration information is configured, it is stored in a database in the format of a structured table. The task indicator alarm rules include, for example, threshold ranges for various indicator parameters; the data quality alarm rules include, for example, whether the update time of the real-time output table of the Flink real-time task matches the update time of the business source table; and the alarm trigger time rules include, for example, whether the trigger time point for generating alarm information and the trigger interval between two adjacent alarm information meet preset requirements (the trigger time point is within the alarm time period and the trigger interval between two adjacent alarm information is within the alarm frequency interval).

[0053] In this embodiment of the invention, the alarm rule may further include an alarm notification strategy. The alarm notification strategy may include, for example, the alarm level corresponding to the Flink real-time task and the notification channel corresponding to each alarm level. The alarm level is related to the abnormal state of the Flink real-time task. The larger and more frequent the abnormality, the higher the corresponding alarm level. It can be configured according to actual application needs or automatically configured based on existing industry classification standards, etc. This embodiment of the invention does not limit this.

[0054] In this embodiment of the invention, the configured alarm configuration information is stored in a relational database (e.g., MySQL), which can effectively ensure data availability and fast access. In other embodiments of the invention, the alarm configuration information is verified for threshold range (e.g., threshold ≥ 0) and time period validity (valid_time_end > valid_time_start) before being stored in the database, to ensure the security and accuracy of the configuration information.

[0055] S2. After exposing the running metrics corresponding to the current Flink real-time task in the form of multiple processes, the running metrics of the Flink real-time task are collected to obtain the running metric data, and the real-time output table of the current Flink real-time task is obtained.

[0056] This embodiment of the invention exposes the runtime metrics managed internally by the Flink real-time task's JobManager in a multi-process manner. The exposed runtime metric data is processed and saved to a database or collected to a local server via Prometheus. This data is then compared with the alarm configuration information in step S1 to confirm whether an alarm rule has been triggered. The pseudocode for exposing runtime metrics in this embodiment is as follows: Figure 2 See reference.

[0057] In this embodiment of the invention, the real-time output table of the current Flink real-time task is obtained simultaneously to execute step S3.

[0058] S3. Within a preset interval, determine whether the real-time output table is within the data quality alarm rule range of the alarm configuration information. If the real-time output table is within the data quality alarm rule range, then execute step S4. If the real-time output table is not within the data quality alarm rule range, then execute step S7.

[0059] The data quality alarm rules described in this embodiment of the invention can be monitoring fields or monitoring data volume. For example, the monitoring field is update_time (update time). For obtaining the real-time output table, the pseudocode is as follows: SELECT MAX(update_time) FROM sink_table1 where condition; For obtaining the business source table, the pseudocode is as follows: SELECT MAX(update_time) FROM source_table1 where condition; Calculate the monitoring difference between the two. For example, for the monitoring field update_time, if diff > 60s, then mark dataLag = true.

[0060] S4. Based on the task indicator alarm rules in the alarm configuration information, perform threshold judgment on the collected operation indicator data, and confirm that the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules. If the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules, generate alarm information and execute step S5. If the indicator parameters in the operation indicator data do not exceed the threshold range of the task indicator alarm rules, execute step S7.

[0061] In this embodiment of the invention, the indicator thresholds in the task indicator alarm rules include instantaneous thresholds and window thresholds, and the indicator parameters in the running indicator data correspond to one of the instantaneous thresholds and window thresholds. For instantaneous thresholds, when performing alarm rule configuration operations, it is necessary to configure operators and threshold ranges, where the operator is a comparison between the current instantaneous value and the threshold range, such as an alarm for the number of pending consumptions > the threshold. For window thresholds, when performing alarm rule configuration operations, it is necessary to configure the window time, operator, and threshold range. This indicator threshold is used to compare the indicator value of the indicator parameter within the window time period with the threshold range, such as an alarm for the number of checkpoints in 5 minutes <= 2.

[0062] S5. Confirm whether the triggering time of generating alarm information and the triggering interval between two adjacent alarm information are within the range of the alarm triggering time rule. If the triggering time and the triggering interval between two adjacent alarm information are within the range of the alarm triggering time rule, then execute step S6. If the triggering time and the triggering interval between two adjacent alarm information are not within the range of the alarm triggering time rule, then execute step S7.

[0063] To avoid frequent alarms forming an alarm storm and affecting alarm processing efficiency, this embodiment of the invention compares the trigger time of generating alarm information and the trigger interval between two adjacent alarm information with the alarm trigger time rule to confirm whether the trigger time of generating alarm information and the trigger interval between two adjacent alarm information are within the range of the alarm trigger time rule. If the trigger time of generating alarm information is within the alarm period and the trigger interval between two adjacent alarm information is not greater than the alarm frequency interval, then step S6 is executed; otherwise, step S7 is executed.

[0064] S6. Confirm whether an alarm notification policy is configured in the alarm configuration information. The alarm notification policy includes the alarm level corresponding to the Flink real-time task and the notification channel corresponding to each alarm level. If a notification policy is configured in the alarm configuration information, the alarm information is notified according to the notification policy. If no notification policy is configured in the alarm configuration information, the alarm information is notified in the default way.

[0065] In this embodiment of the invention, the alarm level can be divided into multiple levels. For example, based on the urgency, it can be divided into four levels: critical, important, alarm, and notification. When the alarm level is critical, the notification channels include, for example, telephone notification, SMS notification, email notification, and alarm group notification. When the alarm level is important, the notification channels include, for example, telephone notification, SMS notification, and alarm group notification. When the alarm level is alarm, the notification channels include, for example, SMS notification, email notification, and alarm group notification. When the alarm level is notification, the notification channel includes, for example, alarm group notification. The default notification method refers to alarm group notification. In other embodiments, other notification methods or combinations of multiple notification methods can also be used, depending on the actual application requirements. This embodiment of the invention does not limit this.

[0066] S7. Monitor the task status of the current Flink real-time task in real time, obtain the task status information of the current Flink real-time task, confirm whether the task status information of the current Flink real-time task indicates task failure, and perform a task recovery operation when it is confirmed that the task status information of the current Flink real-time task indicates task failure.

[0067] The real-time monitoring of the task status of the current Flink real-time task, the acquisition of the task status information of the current Flink real-time task, and the confirmation of whether the task status information of the current Flink real-time task indicates task failure, as described in this embodiment of the invention, includes: obtaining task status query results from the YARN cluster and / or the Flink real-time task's execution interface based on the unique identifier of the current Flink real-time task through the XXL-Job task scheduling module; and confirming that the task status information of the current Flink real-time task indicates task failure when either the task status query result corresponding to the YARN cluster or the Flink real-time task's execution interface indicates task failure.

[0068] In this embodiment of the invention, a task failure is indicated by a FAILED result obtained from the YARN cluster and a 404 error result obtained from the Flink real-time task execution interface.

[0069] In this embodiment of the invention, the task recovery operation includes: obtaining the most recent restart time of the Flink real-time task, and obtaining the current restart interval based on the restart time and the current time, and determining whether the current restart interval is greater than a preset restart interval; if the current restart interval is not greater than the preset restart interval, the process ends; if the current restart interval is greater than the preset restart interval, the task recovery operation is performed, and the cached data of the Flink real-time task in the database is obtained to restart the task; after the Flink real-time task is restarted, the task status information of the current Flink real-time task is obtained again, and if the task status information of the Flink real-time task is a task execution status, the process ends; if the task status information of the Flink real-time task is a task failure, step S2 is executed.

[0070] In this embodiment of the invention, the automatic restart switch on the task configuration page is used to determine whether the Flink real-time task should be restarted. The automatic restart switch corresponds to the enable_auto_restart field in the Flink task output table (which is true by default). If enable_auto_restart=true, it is further determined whether the current restart interval is greater than the preset restart interval. The preset restart interval can be, for example, the automatic restart interval of the automatic restart switch. If the current restart interval is greater than the preset restart interval, the task recovery operation is performed.

[0071] In this embodiment of the invention, cached data is obtained from the database based on the unique identifier of the Flink real-time task. The cached data is the latest checkpoint path stored in the database. Specifically, this embodiment of the invention obtains the latest checkpoint path of the Flink real-time task in real time and stores it in the database table. When task restart and recovery are required, the corresponding checkpoint path can be found from the database based on the unique identifier of the Flink real-time task. At startup, the task is loaded from the latest checkpoint, which can effectively speed up the task recovery efficiency.

[0072] After the task restart is performed, the embodiment of the present invention further confirms the status of the task restart. If the task restart is successful, the task status information of the Flink real-time task is the task execution status. If the task restart fails, the embodiment of the present invention will re-execute step S2 and subsequent steps, and generate alarm information for alarm notification based on the corresponding data of the restart failure.

[0073] In this embodiment of the invention, the method further includes: after generating the alarm information, storing the operational indicator data and the alarm information in a database. The operational indicator data and the alarm information, as historical data, can be used for subsequent data analysis, task optimization, and event review, etc.

[0074] The real-time task precision monitoring and intelligent alarm recovery method described in this embodiment of the invention has the following advantages compared with the prior art:

[0075] 1. Based on the criticality of Flink real-time tasks, personalized alarm rules can be configured for alarm metrics of different tasks, which has better flexibility. In alarm management, by setting multi-level judgment, key alarm information can be effectively filtered out, improving the accuracy of alarms and improving the operational efficiency of alarms.

[0076] 2. In terms of acquiring runtime metrics data, the traditional method of relying on Pushgateway to collect Flink task runtime metrics data is abandoned. Instead, the metrics data provided by Flink NodeManager is provided to downstream processes through program processes. By bypassing the traditional Pushgateway, the runtime metrics managed internally by Flink real-time task jobmanager are exposed in the form of multiple processes, breaking through the performance limitations of a single machine, realizing efficient collection and transmission of monitoring data, improving the real-time performance and accuracy of data collection, and enhancing the scalability and stability of the system in large-scale task monitoring scenarios, providing a solid data foundation for quickly discovering task anomalies.

[0077] 3. By introducing MAX(update_time) difference monitoring between the business source table and the real-time output table, upstream-downstream latency anomalies can be detected within 2 minutes, improving data accuracy verification efficiency by 95% compared to manual verification;

[0078] 4. An alarm notification mechanism based on different levels was introduced to ensure that high-priority tasks can receive sufficient attention and processing resources in a timely manner when problems occur, while avoiding excessive interference from low-priority tasks. This optimized the entire alarm management system and improved operational efficiency.

[0079] 5. It features automatic restart functionality based on alarm priority. When a task fails due to external factors (such as a YARN cluster anomaly), it can quickly identify and automatically restart the task according to preset strategies, greatly reducing business interruption time, decreasing reliance on manual maintenance, and improving the overall stability and availability of the system. Through an automatic restart switch + time interval + latest checkpoint automatic startup, it achieves automatic restart in YARN cluster anomaly scenarios; the average recovery time is reduced from an average of 2 hours to within 5 minutes, and task continuity reaches 99.99%.

[0080] Based on the above embodiments, as a supplement to the above... Figure 1 The present invention provides an embodiment of a real-time task precision monitoring and intelligent alarm recovery device, which implements the method shown. Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices, see reference. Figure 3 As shown, the real-time task precision monitoring and intelligent alarm recovery device includes:

[0081] The task alarm rule configuration module 100 is used to perform alarm rule configuration operations, generate alarm configuration information corresponding to the alarm rules of Flink real-time tasks, and the alarm rules include at least task indicator alarm rules, data quality alarm rules and alarm trigger time rules, and store the configured alarm configuration information in the database.

[0082] The real-time monitoring module 200 is used to expose the running metrics corresponding to the current Flink real-time task in the form of multiple processes, collect the running metrics data of the Flink real-time task, and obtain the real-time output table of the current Flink real-time task.

[0083] The alarm management module 300 is used to determine, within a preset interval, whether the real-time output table is within the data quality alarm rules of the alarm configuration information; to perform threshold judgment on the collected operation indicator data based on the task indicator alarm rules in the alarm configuration information, confirming that the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules; to confirm whether the trigger time of generating alarm information and the trigger interval between two adjacent alarm information are within the range of the alarm trigger time rules; and to confirm whether an alarm notification policy is configured in the alarm configuration information, wherein the alarm notification policy includes the alarm level corresponding to the Flink real-time task and the notification channel corresponding to each alarm level. If a notification policy is configured in the alarm configuration information, the alarm information is notified according to the notification policy; if no notification policy is configured in the alarm configuration information, the alarm information is notified in the default manner.

[0084] The intelligent recovery module 400 is used to monitor the task status of the current Flink real-time task in real time, obtain the task status information of the current Flink real-time task, confirm whether the task status information of the current Flink real-time task indicates task failure, and perform a task recovery operation when it is confirmed that the task status information of the current Flink real-time task indicates task failure.

[0085] In this embodiment of the invention, the intelligent recovery module 400 monitors the task status of the current Flink real-time task in real time, obtains the task status information of the current Flink real-time task, and confirms whether the task status information of the current Flink real-time task indicates task failure, including:

[0086] The task status query results are obtained from the YARN cluster and / or the Flink real-time task execution interface based on the unique identifier of the current Flink real-time task through the xxl-job task scheduling module.

[0087] If any of the task status query results corresponding to the YARN cluster and the Flink real-time task execution interface indicate that the task has failed, then the task status information of the current Flink real-time task is confirmed as a task failure.

[0088] In this embodiment of the invention, the intelligent recovery module 400 performs task recovery operations including:

[0089] Obtain the most recent restart time of the Flink real-time task, and obtain the current restart interval based on the restart time and the current time, and determine whether the current restart interval is greater than the preset restart interval;

[0090] If the current restart interval is not greater than the preset restart interval, the process ends;

[0091] If the current restart interval is greater than the preset restart interval, then a task recovery operation is performed to obtain the cached data of the Flink real-time task in the database and restart the task.

[0092] After the Flink real-time task restarts, the task status information of the current Flink real-time task is obtained again. If the task status information of the Flink real-time task is in the task execution state, the process ends. If the task status information of the Flink real-time task is in the task failure state, step S2 is executed.

[0093] In this embodiment of the invention, the alarm management module 300 is further configured to: after generating the alarm information, store the operating indicator data and the alarm information in the database.

[0094] The real-time task precise monitoring and intelligent alarm recovery device described in this embodiment of the invention can execute the real-time task precise monitoring and intelligent alarm recovery method provided in the above embodiments. The real-time task precise monitoring and intelligent alarm recovery device has the corresponding functional steps and beneficial effects of the real-time task precise monitoring and intelligent alarm recovery method described in the above embodiments. For details, please refer to the embodiments of the real-time task precise monitoring and intelligent alarm recovery method described above. The embodiments of this invention will not be repeated here.

[0095] This invention also provides an electronic device, which may include a processor and a memory, wherein the processor and memory can be connected via a bus or other means. The processor may be a Central Processing Unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the real-time task precise monitoring and intelligent alarm recovery method in this invention embodiment. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby realizing the real-time task precise monitoring and intelligent alarm recovery method in the above method embodiment.

[0096] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. One or more modules are stored in the memory and, when executed by the processor, perform the real-time task precise monitoring and intelligent alarm recovery method as described in the above method embodiments. Specific details of the above electronic device can be understood by referring to the corresponding descriptions and effects in the above method embodiments, and will not be repeated here. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it may include the processes of the embodiments of the above methods. The storage medium may be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above types of memory.

[0097] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0098] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various aspects of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention above. Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and it should be noted that the above embodiments are illustrative of the invention and not restrictive, and that alternative embodiments can be devised by those skilled in the art without departing from its scope.

Claims

1. A method for real-time precise monitoring of tasks and intelligent alarm recovery, characterized in that, The real-time task precision monitoring and intelligent alarm recovery method includes: S1. Perform alarm rule configuration operation to generate alarm configuration information corresponding to the alarm rules of Flink real-time tasks. The alarm rules include at least task indicator alarm rules, data quality alarm rules and alarm trigger time rules, and store the configured alarm configuration information in the database. S2. After exposing the running metrics corresponding to the current Flink real-time task in the form of multiple processes, the running metrics of the Flink real-time task are collected to obtain the running metric data, and the real-time output table of the current Flink real-time task is obtained. S3. Within a preset interval, determine whether the real-time output table is within the data quality alarm rule range of the alarm configuration information. If the real-time output table is within the data quality alarm rule range, then execute step S4. If the real-time output table is not within the data quality alarm rule range, then execute step S7. S4. Based on the task indicator alarm rules in the alarm configuration information, perform threshold judgment on the collected operation indicator data to confirm whether the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules. If the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules, generate alarm information and execute step S5. If the indicator parameters in the operation indicator data do not exceed the threshold range of the task indicator alarm rules, execute step S7. S5. Confirm whether the triggering time of generating alarm information and the triggering interval between two adjacent alarm information are within the range of the alarm triggering time rule. If the triggering time and the triggering interval between two adjacent alarm information are within the range of the alarm triggering time rule, then execute step S6. If the triggering time and the triggering interval between two adjacent alarm information are not within the range of the alarm triggering time rule, then execute step S7. S6. Confirm whether an alarm notification policy is configured in the alarm configuration information. The alarm notification policy includes the alarm level corresponding to the Flink real-time task and the notification channel corresponding to each alarm level. If a notification policy is configured in the alarm configuration information, the alarm information is notified according to the notification policy. If no notification policy is configured in the alarm configuration information, the alarm information is notified in the default way. S7. Monitor the task status of the current Flink real-time task in real time, obtain the task status information of the current Flink real-time task, confirm whether the task status information of the current Flink real-time task indicates task failure, and perform a task recovery operation when it is confirmed that the task status information of the current Flink real-time task indicates task failure. In step S7, the real-time monitoring of the task status of the current Flink real-time task, obtaining the task status information of the current Flink real-time task, and confirming whether the task status information of the current Flink real-time task indicates task failure includes: The task status query results are obtained from the YARN cluster and / or the Flink real-time task execution interface based on the unique identifier of the current Flink real-time task through the xxl-job task scheduling module. If any of the task status query results corresponding to the YARN cluster and the Flink real-time task execution interface indicate that the task has failed, then the task status information of the current Flink real-time task is confirmed as a task failure.

2. The real-time task precise monitoring and intelligent alarm recovery method according to claim 1, characterized in that, The task recovery operation includes: Obtain the most recent restart time of the Flink real-time task, and obtain the current restart interval based on the restart time and the current time, and determine whether the current restart interval is greater than the preset restart interval; If the current restart interval is not greater than the preset restart interval, the process ends; If the current restart interval is greater than the preset restart interval, then a task recovery operation is performed to obtain the cached data of the Flink real-time task in the database and restart the task. After the Flink real-time task restarts, the task status information of the current Flink real-time task is obtained again. If the task status information of the Flink real-time task is in the task execution state, the process ends. If the task status information of the Flink real-time task is in the task failure state, step S2 is executed.

3. The real-time task precise monitoring and intelligent alarm recovery method according to claim 1, characterized in that, The method further includes: after generating the alarm information, storing the operating indicator data and the alarm information in a database.

4. A real-time task precision monitoring and intelligent alarm recovery device, characterized in that, The real-time task precision monitoring and intelligent alarm recovery device includes: The task alarm rule configuration module is used to perform alarm rule configuration operations, generate alarm configuration information corresponding to the alarm rules of Flink real-time tasks, and the alarm rules include at least task indicator alarm rules, data quality alarm rules and alarm trigger time rules, and store the configured alarm configuration information in the database. The real-time monitoring module is used to expose the running metrics corresponding to the current Flink real-time task in the form of multiple processes, collect the running metrics data of the Flink real-time task, and obtain the real-time output table of the current Flink real-time task. The alarm management module is used to determine, within a preset interval, whether the real-time output table is within the data quality alarm rules of the alarm configuration information; to perform threshold judgment on the collected operation indicator data based on the task indicator alarm rules in the alarm configuration information, confirming whether the indicator parameters in the operation indicator data exceed the threshold range of the task indicator alarm rules; to confirm whether the trigger time of generating alarm information and the trigger interval between two adjacent alarm information are within the range of the alarm trigger time rules; and to confirm whether an alarm notification policy is configured in the alarm configuration information, wherein the alarm notification policy includes the alarm level corresponding to the Flink real-time task and the notification channel corresponding to each alarm level. If a notification policy is configured in the alarm configuration information, the alarm information is notified according to the notification policy; if no notification policy is configured in the alarm configuration information, the alarm information is notified in the default manner. The intelligent recovery module is used to monitor the task status of the current Flink real-time task in real time, obtain the task status information of the current Flink real-time task, confirm whether the task status information of the current Flink real-time task indicates task failure, and perform task recovery operation when it is confirmed that the task status information of the current Flink real-time task indicates task failure. The intelligent recovery module monitors the current task status of the Flink real-time task in real time, obtains the task status information of the current Flink real-time task, and confirms whether the task status information of the current Flink real-time task indicates task failure, including: The task status query results are obtained from the YARN cluster and / or the Flink real-time task execution interface based on the unique identifier of the current Flink real-time task through the xxl-job task scheduling module. If any of the task status query results corresponding to the YARN cluster and the Flink real-time task execution interface indicate that the task has failed, then the task status information of the current Flink real-time task is confirmed as a task failure.

5. The real-time task precision monitoring and intelligent alarm recovery device according to claim 4, characterized in that, The intelligent recovery module performs the following recovery operations: Obtain the most recent restart time of the Flink real-time task, and obtain the current restart interval based on the restart time and the current time, and determine whether the current restart interval is greater than the preset restart interval; If the current restart interval is not greater than the preset restart interval, the process ends; If the current restart interval is greater than the preset restart interval, then a task recovery operation is performed to obtain the cached data of the Flink real-time task in the database and restart the task. After the Flink real-time task restarts, the task status information of the current Flink real-time task is obtained again. If the task status information of the Flink real-time task is in the task execution state, the process ends. If the task status information of the Flink real-time task is in the task failure state, step S2 is executed.

6. The real-time task precision monitoring and intelligent alarm recovery device according to claim 4, characterized in that, The alarm management module is also used to: after generating the alarm information, store the operating indicator data and the alarm information in the database.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which are loaded and executed by a processor to perform the operations described in any one of claims 1-3.

8. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements the instructions of the method as described in any one of claims 1-3.

Images