Apache Nifi-based task execution cycle monitoring method and system, and medium

Abstract

The invention discloses a task execution period monitoring method and system based on Apache Nifi, and a medium, mainly relates to the technical field of period monitoring, and is used for solving the problems that an existing scheme is single in monitoring dimension and lacks intelligent prediction capability. Comprising the steps of obtaining an execution information clustering subset where current execution information is located based on the current execution information, an execution information set and a preset clustering algorithm; obtaining an initial early warning time threshold value corresponding to the current task based on all execution information in the execution information clustering subset; feeding back the initial early warning time threshold value to a preset maintenance terminal to obtain a fed-back final early warning time threshold value; and monitoring the currently executed task, starting an early warning notification program when the current executed task exceeds a final early warning time threshold value, acquiring a predicted emergency degree corresponding to the predicted next executed task, and executing the predicted next executed task when the predicted emergency degree is greater than a preset degree threshold value.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of image processing, in particular to an Apache Nifi-based task execution cycle monitoring method and system and a medium. BACKGROUND

[0002] Currently, Apache Nifi task monitoring mainly adopts three types of technical solutions: one is offline analysis based on system logs, which analyzes Nifi generated log files to count task execution time; two is to use monitoring tools such as Prometheus to collect JMX indicators of processor components and set fixed threshold values to trigger alarms; three is to use the native Reporting Task module of Nifi to generate periodic reports. These solutions have obvious shortcomings in e-commerce promotion, financial clearing and other scenarios: log analysis has a delay; JMX monitoring can only obtain basic indicators such as CPU / memory, and cannot be associated with business scenarios; the native reporting function lacks predictive ability. Some enterprises try to extend Nifi source code to add monitoring plugins, but this requires a lot of development resources to maintain and is difficult to adapt to dynamically changing business needs.

[0003] The above-mentioned technology has the following problems: first, the static threshold mechanism cannot distinguish the differentiated requirements of different business scenarios (such as payment order processing and log cleaning) for timeliness, resulting in important task omissions or minor task false alarms; second, there is a lack of mining of the association between tasks, and when upstream data is delayed, the execution risk of downstream critical path tasks cannot be predicted. These problems essentially stem from: 1) single monitoring dimension, without integrating task type, data volume and other context information; 2) lack of intelligent prediction ability. SUMMARY

[0004] The application provides an Apache Nifi-based task execution cycle monitoring method and system and a medium to solve the problem of single monitoring dimension and lack of intelligent prediction ability in existing solutions.

[0005] In a first aspect, the application provides an Apache Nifi-based task execution cycle monitoring method, which comprises: collecting execution information of tasks in a preset time period from Apache Nifi to obtain an execution information set; wherein the execution information at least includes task start time, end time, task type, business scenario, input and output data volume, whether it is overdue, and urgency; inputting the execution information set into a machine learning algorithm to obtain the association between the execution information; detecting a task being executed, reading current execution information; inputting the current execution information into a machine learning algorithm to obtain a predicted next execution task; obtaining an execution information clustering sub-set in which the current execution information is located based on the current execution information, a set of execution information, and a preset clustering algorithm; obtaining an initial early warning time threshold corresponding to the current task based on all execution information in the execution information clustering sub-set; feeding back the initial early warning time threshold to a preset maintenance terminal to obtain a final early warning time threshold fed back; monitoring the task being executed, starting an early warning notification program when the final early warning time threshold is exceeded, and obtaining a predicted emergency level corresponding to the predicted next execution task, and executing the predicted next execution task when the predicted emergency level is greater than a preset degree threshold.

[0006] In an implementation manner of the present application, the execution information of the task is collected from Apache Nifi, and specifically includes: collecting the execution information of the task in a preset time period from distributed log recording; In the collection process, a master-slave backup mechanism is adopted, and an encrypted transmission protocol is introduced for encrypted collection transmission.

[0007] In an implementation manner of the present application, the set of execution information is input into the machine learning algorithm to obtain the association relationship between the execution information, and specifically includes: Based on the start time and end time of the task in the execution information, the FP-Growth algorithm is used to find the execution information with the highest occurrence frequency after each execution information in the set of execution information, and the association relationship between the two is established.

[0008] In an implementation manner of the present application, the task being executed is detected, and the current execution information is read, and specifically includes: reading the start time of the task and the input data volume from the system; obtaining the task type, the task type, the business scenario, and the emergency level through a preset input interface; The end time, the output data volume, and whether the timeout is set to null in the execution information.

[0009] In an implementation manner of the present application, the current execution information is input into the machine learning algorithm to obtain the predicted next execution task, and specifically includes: The current execution information is input into the machine learning algorithm, and the corresponding weight of the end time, the output data volume, and whether the timeout is set to 0 in the machine learning algorithm; and then the predicted next execution task is obtained.

[0010] In an implementation manner of the present application, based on the current execution information, the execution information set and the preset clustering algorithm, an execution information clustering sub-set in which the current execution information is located is obtained, specifically comprising: The current execution information is added to the execution information set, and data corresponding to the end time, the output data volume and whether the timeout in the execution information set is deleted to obtain an input set; The input set is input into the preset clustering algorithm to obtain a clustering sub-set in which the current execution information is located; Data of the end time, the output data volume and whether the timeout is added to execution information corresponding to the clustering sub-set to obtain the execution information clustering sub-set.

[0011] In a second aspect, the present application provides an Apache Nifi-based task execution cycle monitoring system, which comprises: A collection module is configured to collect execution information of tasks in a preset time period from the Apache Nifi to obtain an execution information set; wherein the execution information at least comprises a task start time, an end time, a task type, a business scenario, an input / output data volume, whether the task is timed out and an emergency level; An acquisition module is configured to input the execution information set into a machine learning algorithm to obtain an association relationship between the execution information; detect a currently executed task to read current execution information; input the current execution information into the machine learning algorithm to obtain a predicted next execution task; and based on the current execution information, the execution information set and the preset clustering algorithm, an execution information clustering sub-set in which the current execution information is located is obtained; A warning processing module is configured to obtain an initial warning time threshold corresponding to a current task based on all execution information in the execution information clustering sub-set; feed back the initial warning time threshold to a preset maintenance terminal to obtain a final warning time threshold fed back; monitor the currently executed task, and when the final warning time threshold is exceeded, start a warning notification program, simultaneously obtain a predicted emergency level corresponding to the predicted next execution task, and when the predicted emergency level is greater than a preset degree threshold, execute the predicted next execution task.

[0012] In an implementation manner of the present application, the acquisition module comprises a first acquisition unit, configured to use an FP-Growth algorithm to find execution information with the highest frequency of occurrence after each execution information in the execution information set based on the task start time and the end time in the execution information, and establish an association relationship therebetween.

[0013] In an implementation manner of the present application, the acquisition module comprises a second acquisition unit, configured to add the current execution information to the execution information set, and delete data corresponding to the end time, the output data volume and whether the timeout in the execution information set to obtain an input set; inputting the input set into a preset clustering algorithm to obtain a clustering sub-set in which the current execution information is located; adding the end time, the output data volume, and the data of whether the timeout to the execution information corresponding to the clustering sub-set to obtain the execution information clustering sub-set.

[0014] In a third aspect, the present application provides a non-volatile computer storage medium having computer instructions stored thereon, the computer instructions, when executed, implementing the Apache Nifi-based task execution cycle monitoring method of any one of the above.

[0015] As can be seen from the above technical solutions, the present application has the following advantages: By constructing a new task management system combining multi-dimensional monitoring and intelligent prediction, the two core pain points existing in traditional Apache Nifi monitoring are solved. First, the limitation of single time dimension monitoring is broken through, and a seven-dimensional task feature portrait system is established. By collecting structured feature data such as task type, business scenario, and data throughput, and combining clustering algorithm to generate dynamic early warning threshold, the system can identify 9 implicit abnormal scenarios.

[0016] Predictive management is introduced into the task scheduling field. By analyzing the time sequence relationship of the task chain, two levels of intelligent prediction are realized: task-level prediction and urgency prediction. When it is detected that the current task may be delayed, the system not only triggers an early warning, but also assesses the impact of subsequent tasks in advance. For tasks predicted to be urgent, a resource pre-allocation mechanism is used, which reduces the average waiting time of key tasks. This prediction-execution linkage mechanism directly solves the defects of the traditional "after-the-fact remedy" solution, and improves the system exception response speed. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flow chart of the Apache Nifi-based task execution cycle monitoring method provided by the embodiments of the present application.

[0019] Figure 2 is a schematic diagram of the internal structure of the Apache Nifi-based task execution cycle monitoring system provided by the embodiments of the present application. DETAILED DESCRIPTION

[0020] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0021] Those skilled in the art should understand that the embodiments described below are only preferred embodiments of the present disclosure, and do not represent the only way to implement the present disclosure. The preferred embodiments are only used to explain the technical principles of the present disclosure, and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts still fall within the protection scope of the present disclosure.

[0022] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or other elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0023] The technical solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0024] The embodiments provide an Apache Nifi-based task execution cycle monitoring method, as shown in Figure 1 The method provided by the embodiments of the present application mainly includes the following steps: Step 110, collecting execution information of tasks in a preset time period from Apache Nifi, to obtain an execution information set.

[0025] The execution information at least includes task start time, end time, task type, business scenario, input and output data volume, whether timeout, and emergency degree.

[0026] In some embodiments, the execution information of the tasks is collected from Apache Nifi, and specifically includes: Collecting execution information of tasks in a preset time period from distributed log records; In the collection process, a master-slave backup mechanism is adopted, and an encryption transmission protocol is introduced for encrypted collection and transmission.

[0027] The skilled in the art can understand that, in order to ensure the accuracy and integrity of the data, the system adopts distributed log recording technology to ensure that all relevant data can be accurately recorded even in the case of unstable network.

[0028] The present step can adopt master-slave backup mechanism to ensure seamless switching to standby node when the master node fails, maintaining the continuity of data acquisition process. Introduce encryption transmission protocol (such as TLS) to protect data from illegal access or tampering during acquisition Step 120, input the execution information set into the machine learning algorithm to obtain the association relationship between the execution information; detect the current execution task, read the current execution information; input the current execution information into the machine learning algorithm to obtain the predicted next execution task; based on the current execution information, the execution information set and the preset clustering algorithm, obtain the execution information clustering sub-set where the current execution information is located.

[0029] Among them, the execution information set is input into the machine learning algorithm to obtain the association relationship between the execution information, which can be specifically: Based on the start time and end time of the task in the execution information, the FP-Growth algorithm is used to find the execution information with the highest frequency after each execution information in the execution information set occurs, and the association relationship between them is established.

[0030] Among them, the current execution task is detected, and the current execution information is read, which can be specifically: The start time of the task, the input data volume are read from the system; the task type, the task type, the business scenario, the urgency are obtained through the preset input interface; wherein, the end time, the output data volume, whether the timeout in the execution information are set to null.

[0031] Among them, the current execution information is input into the machine learning algorithm to obtain the predicted next execution task, which can be specifically: The current execution information is input into the machine learning algorithm, and the corresponding weight of the end time, the output data volume, whether the timeout in the machine learning algorithm is set to 0; and then the predicted next execution task is obtained.

[0032] Among them, based on the current execution information, the execution information set and the preset clustering algorithm, the execution information clustering sub-set where the current execution information is located is obtained, which can be specifically: The current execution information is added to the execution information set, and the corresponding data of the end time, the output data volume, whether the timeout in the execution information set is deleted to obtain the input set; the input set is input into the preset clustering algorithm to obtain the clustering sub-set where the current execution information is located; the data of the end time, the output data volume, whether the timeout is added to the corresponding execution information in the clustering sub-set to obtain the execution information clustering sub-set.

[0033] The skilled person in the art can understand that this step improves the prediction accuracy and real-time response capability of task monitoring through multi-stage intelligent processing. First, the FP-Growth algorithm is used to mine task timing association rules (such as 80% probability of triggering task B after task A), which reduces the false positive rate compared to traditional threshold monitoring. The dynamic weight mechanism (setting the weight of incomplete data to zero) ensures that the prediction model focuses on effective features, and the prediction accuracy of the next task is improved to. Through the three-step method of "feature cleaning-clustering-data restoration", the matching efficiency of similar tasks is improved while preserving historical execution result data. This closed-loop process of "correlation analysis-real-time prediction-dynamic clustering" enables the system to identify both task chain rules (macro) and current task features (micro), and compresses the abnormal detection response speed from minutes to seconds. Experiments show that this scheme improves the on-time completion rate of key tasks, and is particularly good at handling sudden load fluctuations in data pipelines.

[0034] Step 130, based on all execution information in the execution information clustered subset, obtaining an initial warning time threshold corresponding to the current task; feeding back the initial warning time threshold to the preset maintenance terminal to obtain the final warning time threshold; monitoring the currently executed task, when exceeding the final warning time threshold, starting the warning notification program, and simultaneously obtaining the predicted emergency degree corresponding to the predicted next execution task, when the predicted emergency degree is greater than the preset degree threshold, executing the predicted next execution task.

[0035] The skilled person in the art can understand that this step realizes the three breakthroughs of the task monitoring system through the synergistic mechanism of dynamic warning and intelligent pre-execution: first, the initial warning threshold generated based on the clustered subset (execution information clustered subset) (such as ETL task average time consumption ± 2σ, i.e. the average time consumption of all execution information in the execution information clustered subset except the current execution information) is calibrated by humans to form the final threshold, which not only retains the data-driven advantage but also incorporates domain knowledge, reducing the false positive rate. Second, the dual-trigger mechanism of "threshold breakthrough + emergency degree prediction" is used, when the task is overdue and the next task emergency degree > 0.7 (preset threshold), the system automatically pre-executes, which compresses the average waiting time of key path tasks. Finally, the system can identify implicit abnormalities (such as gradual performance degradation caused by data skew) by comparing the current execution indicators with the clustered features (such as data volume increase rate / time consumption ratio) in real time, which captures more potential faults than traditional timeout monitoring. This hybrid decision-making mode that combines statistical rules and business rules improves the SLA compliance rate of task scheduling.

[0036] In addition, the application can also use statistical methods and machine learning algorithms to conduct in-depth analysis on the collected task execution data. Through learning from a large amount of historical data, the module can identify task execution patterns and predict potential problems in the future. In addition, the application also supports custom rule configuration, allowing users to set specific analysis logic according to their own needs, further improving the adaptability and flexibility of the system. Combined with time series analysis technology, the application can predict the trend of task execution efficiency in the future, helping enterprises to plan resources in advance. The application can have multiple chart types (such as line charts, bar charts, etc.) built-in to help users intuitively understand the data analysis results.

[0037] In addition, the application can also generate detailed reports on task execution status on a regular basis, which is one of the important means to understand the overall operation status. The report generation module can generate task execution reports containing key indicators on a regular basis or on demand according to user settings. These reports can help managers fully understand the current task execution situation and provide data support for future decision-making. Online editing function is provided to allow users to modify report content according to their own needs, improving the practicality of the report. In addition to showing the performance of a single task, it also supports performance comparison across tasks to find the best practice cases.

[0038] In addition, the application Figure 2 A task execution cycle monitoring system based on Apache Nifi is provided for an embodiment of the application. As shown in Figure 2 The system provided by the embodiment of the application mainly includes: The collection module 210 is configured to collect execution information of tasks in a preset time period from the Apache Nifi, and obtain an execution information set. The execution information at least includes a task start time, an end time, a task type, a business scenario, an input / output data volume, whether to be overtime, and an emergency level.

[0039] The acquisition module 220 is configured to input the execution information set into a machine learning algorithm, acquire an association relationship between the execution information, detect a currently executed task, read a current execution information, input the current execution information into the machine learning algorithm, and obtain a predicted next execution task. The acquisition module 220 is configured to acquire an execution information clustering sub-set to which the current execution information belongs based on the current execution information, the execution information set, and a preset clustering algorithm.

[0040] The acquisition module 220 includes a first acquisition unit configured to use an FP-Growth algorithm to find, based on the task start time and the end time in the execution information, an execution information with the highest occurrence frequency after each execution information in the execution information set occurs, and establish an association relationship between the two.

[0041] The acquisition module 220 comprises a second acquisition unit, configured to add the current execution information into the execution information set, delete the data corresponding to the end time, the output data volume and whether timeout in the execution information set, obtain an input set; input the input set into a preset clustering algorithm, acquire a clustering sub-set in which the current execution information is located; add the data of the end time, the output data volume and whether timeout into the execution information corresponding to the clustering sub-set, and obtain the execution information clustering sub-set.

[0042] The early warning processing module 230 is configured to obtain an initial early warning time threshold corresponding to the current task based on all the execution information in the execution information clustering sub-set; feed back the initial early warning time threshold to a preset maintenance terminal, and obtain a feedback final early warning time threshold; monitor the currently executed task, start an early warning notification program when the final early warning time threshold is exceeded, and simultaneously acquire a predicted next execution task corresponding to a predicted emergency degree; and when the predicted emergency degree is greater than a preset degree threshold, execute the predicted next execution task.

[0043] In addition, the embodiment of the present application further provides a non-volatile computer storage medium, which has executable instructions stored thereon, and the executable instructions are executed to realize the Apache Nifi-based task execution cycle monitoring method.

[0044] The above description of disclosed embodiments enables those skilled in the art to carry out or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An Apache Nifi-based task execution cycle monitoring method, characterized in that, The method comprises: Collecting execution information of tasks in a preset time period from Apache Nifi to obtain an execution information set; wherein the execution information at least includes task start time, end time, task type, business scenario, input and output data volume, whether to timeout, and emergency level; Inputting the execution information set into a machine learning algorithm to obtain the association relationship between the execution information; wherein the execution information and the execution task have a one-to-one relationship; detecting the currently executed task to read the current execution information; inputting the current execution information into the machine learning algorithm to obtain the predicted next execution task; based on the current execution information, the execution information set and a preset clustering algorithm, obtaining the execution information clustering sub-set in which the current execution information is located; Based on all execution information in the execution information clustering sub-set, obtaining the initial early warning time threshold corresponding to the current task; feeding back the initial early warning time threshold to a preset maintenance terminal to obtain the final early warning time threshold of the feedback; monitoring the currently executed task, starting the early warning notification program when the final early warning time threshold is exceeded, and simultaneously obtaining the predicted emergency level corresponding to the predicted next execution task, when the predicted emergency level is greater than a preset degree threshold, executing the predicted next execution task.

2. The Apache Nifi based task execution cycle monitoring method according to claim 1, characterized in that, Collecting execution information of tasks from Apache Nifi, specifically comprising: Collecting execution information of tasks in a preset time period from distributed log records; In the collection process, a master-slave backup mechanism is used, and an encrypted transmission protocol is introduced for encrypted collection transmission.

3. The Apache Nifi based task execution cycle monitoring method according to claim 1, wherein, Inputting the execution information set into a machine learning algorithm to obtain the association relationship between the execution information, specifically comprising: Based on the task start time and end time in the execution information, using the FP-Growth algorithm to find the execution information with the highest occurrence frequency after each execution information in the execution information set, and establishing the association relationship between the two.

4. The Apache Nifi based task execution cycle monitoring method according to claim 1, wherein, Detecting the currently executed task to read the current execution information, specifically comprising: Reading the task start time and input data volume from the system; Obtaining the task type, task type, business scenario, and emergency level through a preset input interface; Wherein the end time, output data volume, and whether to timeout in the execution information are set as null values.

5. The Apache Nifi based task execution cycle monitoring method according to claim 1, wherein, Inputting the current execution information into the machine learning algorithm to obtain the predicted next execution task, specifically comprising: Inputting the current execution information into the machine learning algorithm, and setting the corresponding weights of the end time, output data volume, and whether to timeout in the machine learning algorithm to 0; And then obtaining the predicted next execution task.

6. The Apache Nifi based task execution cycle monitoring method according to claim 1, wherein, Based on the current execution information, the execution information set and a preset clustering algorithm, obtaining the execution information clustering sub-set in which the current execution information is located, specifically comprising: Adding the current execution information to the execution information set, deleting the corresponding data of the end time, output data volume, and whether to timeout in the execution information set to obtain an input set; Inputting the input set into a preset clustering algorithm to obtain the clustering sub-set in which the current execution information is located; Adding the data of the end time, output data volume, and whether to timeout to the corresponding execution information in the clustering sub-set to obtain the execution information clustering sub-set.

7. An Apache Nifi-based task execution cycle monitoring system, characterized in that, The system comprises: The collection module is configured to collect execution information of a task in a preset time period from Apache Nifi to obtain an execution information set, wherein the execution information at least includes a task start time, an end time, a task type, a business scenario, an input / output data volume, whether to be overtime, and an emergency level. The acquisition module is configured to input the execution information set into a machine learning algorithm to obtain an association relationship between the execution information, detect a currently executed task, read current execution information, input the current execution information into the machine learning algorithm to obtain a predicted next execution task, and obtain an execution information clustering sub-set in which the current execution information is located based on the current execution information, the execution information set, and a preset clustering algorithm. The early warning processing module is configured to obtain an initial early warning time threshold corresponding to the current task based on all execution information in the execution information clustering sub-set, feed back the initial early warning time threshold to a preset maintenance terminal to obtain a feedback final early warning time threshold, monitor the currently executed task, start an early warning notification program when the final early warning time threshold is exceeded, simultaneously obtain a predicted emergency level corresponding to the predicted next execution task, and execute the predicted next execution task when the predicted emergency level is greater than a preset degree threshold.

8. The Apache Nifi based task execution cycle monitoring system as claimed in claim 7, wherein, The acquisition module includes a first acquisition unit, configured to use an FP-Growth algorithm to find execution information with the highest occurrence frequency after each execution information in the execution information set based on the task start time and the end time in the execution information, and establish an association relationship therebetween.

9. The Apache Nifi based task execution cycle monitoring system as claimed in claim 7, wherein, The acquisition module includes a second acquisition unit, configured to add the current execution information to the execution information set, delete data corresponding to the end time, the output data volume, and whether to be overtime in the execution information set to obtain an input set, input the input set into a preset clustering algorithm to obtain a clustering sub-set in which the current execution information is located, and add the data of the end time, the output data volume, and whether to be overtime to execution information corresponding to the clustering sub-set to obtain the execution information clustering sub-set. The computer program product has computer instructions stored thereon, and the computer instructions, when executed, implement the Apache Nifi-based task execution cycle monitoring method according to any one of claims 1-6. ​ 10. A non-transitory computer storage medium, comprising, ​

Images