Alarm configuration method and device for data job link

Abstract

The application discloses a kind of data job link alarm configuration method and device, it is related to alarm configuration technical field, wherein the method comprises: with job as node, with the dependent relationship between job as edge, construct directed acyclic graph;The value of end job node is initialized as the timeliness threshold of end job;The average execution duration of the front job corresponding to the edge is determined as the weight of the edge;According to the dependent relationship between job and the data processing system to which job belongs, the job node belonging to different data processing systems and having dependent relationship is determined as key node;From end job node, multi-source breadth-first search is carried out on directed acyclic graph, and the value of each node is updated;According to the recommended timeliness threshold of key node, alarm threshold is configured for the job corresponding to key node.The application can improve the efficiency of job timeliness calculation, improve the accuracy and efficiency of data job link alarm, and reduce the operation and maintenance pressure of job.

Description

Technical Field

[0001] The present invention relates to the technical field of alarm configuration, and in particular to an alarm configuration method and device for a data operation link. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] With the development of financial technology in large commercial banks, the application of big data in bank IT systems has become more and more extensive. This trend has led to an exponential increase in the amount of data that needs to be processed by the bank's back-end, and at the same time, the timeliness requirements for data processing operations have also increased accordingly.

[0004] In development scenarios involving massive data processing jobs, the diverse nature of business and the complexity of data processing operations leads to complex dependencies, manifested in long links and multiple cross-system node dependencies. Consequently, in actual operations and maintenance, significant communication costs are required to determine the dependencies between data processing jobs, resulting in low efficiency in calculating the timeliness of data processing jobs.

[0005] Existing data processing job timeliness monitoring systems use two alarm configuration schemes: one scheme configures alarms for all nodes in the entire data operation chain. If the alarm threshold is not configured properly, when the upstream operation alarms, the downstream operation will also alarm simultaneously, resulting in an alarm storm and many useless alarms, which greatly interferes with the analysis of operation and maintenance personnel; the other scheme only configures alarms for the terminal operation, which cannot detect delays in the upstream operation system in a timely manner, causing the terminal operation system to bear the vast majority of the operation and maintenance pressure, and the operation and maintenance effect is poor. Delays are often discovered after the optimal time for emergency response has been missed. Summary of the Invention

[0006] An embodiment of the present invention provides a method for configuring alarms for a data operation link, which is used to reduce communication costs between terminal operation systems and upstream operation systems, improve the efficiency of operation time calculation, improve the accuracy and efficiency of data operation link alarms, and reduce the operation and maintenance pressure of the operation. The method includes:

[0007] A directed acyclic graph is constructed with jobs as nodes and dependencies between jobs as edges. The direction of the edges is from the predecessor job to the corresponding successor job.

[0008] Initialize the value of the terminal job node to the terminal job's timeliness threshold; determine the average execution time of the predecessor jobs corresponding to the edge as the edge weight; where the terminal job node is a node with an out-degree of 0, and the terminal job's timeliness threshold represents the length of time the terminal job is allowed to be delayed;

[0009] Based on the data processing systems to which the jobs belong and the dependencies between the jobs, job nodes that belong to different data processing systems and have dependencies are identified as key nodes;

[0010] Starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph. The value of each node is updated according to the following rules: the weight of the edge pointing to the current search node is added to the value of the current search node. If the sum is greater than the value of the starting node of the edge, or the starting node of the edge has no value, the value of the starting node of the edge is updated to the sum. The updated value of each node is the recommended timeliness threshold of the corresponding job.

[0011] Configure alarm thresholds for jobs corresponding to key nodes based on the recommended timeliness thresholds for key nodes.

[0012] An embodiment of the present invention further provides an alarm configuration device for a data operation link, which is used to reduce the communication cost between the terminal operation system and the upstream operation system, improve the efficiency of operation time calculation, improve the accuracy and efficiency of data operation link alarms, and reduce the operation and maintenance pressure of the operation. The device includes:

[0013] The link construction module is used to construct a directed acyclic graph with jobs as nodes and dependencies between jobs as edges. The direction of the edges is from the predecessor job to the corresponding successor job.

[0014] The link initialization module is used to initialize the value of the terminal job node to the terminal job's timeliness threshold; the average execution time of the predecessor job corresponding to the edge is determined as the edge weight; the terminal job node is a node with an out-degree of 0, and the terminal job's timeliness threshold represents the length of time the terminal job is allowed to be delayed;

[0015] A key node analysis module is used to determine the job nodes that belong to different data processing systems and have dependencies as key nodes based on the data processing systems to which the jobs belong and the dependencies between the jobs;

[0016] The timeliness threshold analysis module is used to perform a multi-source breadth-first search on the directed acyclic graph starting from the terminal job node, and update the value of each node according to the following rules: the weight of the edge pointing to the current search node is added to the value of the current search node; if the sum is greater than the value of the starting node of the edge, or the starting node of the edge has no value assigned, the value of the starting node of the edge is updated to the sum; the updated value of each node is the recommended timeliness threshold for the corresponding job of each node;

[0017] The alarm configuration module is used to configure alarm thresholds for jobs corresponding to key nodes based on the recommended timeliness thresholds of key nodes.

[0018] An embodiment of the present invention further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned alarm configuration method for the data operation link when executing the computer program.

[0019] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the alarm configuration method for the data operation link.

[0020] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the alarm configuration method for the data operation link.

[0021] Existing data processing job timeliness monitoring systems usually adopt a solution that configures alarms for all nodes in the entire data job chain, or a solution that configures alarms only for terminal jobs. Compared with existing technical solutions, in an embodiment of the present invention, a directed acyclic graph is constructed based on the execution time of the job, the timeliness threshold of the terminal job, and the job dependency relationship. Through the multi-source breadth-first search algorithm of the graph, the recommended timeliness threshold of the job is efficiently solved, the communication cost between the terminal job system and the upstream job system is reduced, and the efficiency of job timeliness calculation is improved. In an embodiment of the present invention, jobs with dependencies between different systems are also identified as key nodes, and all key nodes on the data job chain are automatically identified. Alarms are configured according to the recommended timeliness threshold of the key nodes, thereby improving the accuracy and efficiency of data job chain alarms, avoiding the generation of alarm storms, improving operation and maintenance effects, and reducing operation and maintenance pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0023] Figure 1 Flowchart of the alarm configuration method for a data operation link in an embodiment of the present invention;

[0024] Figure 2 This is a job dependency diagram of a data job link in an embodiment of the present invention;

[0025] Figure 3 A weighted job dependency graph of a data job link in an embodiment of the present invention;

[0026] Figure 4This is a flowchart of the time efficiency analysis of key operations in an embodiment of the present invention;

[0027] Figure 5 Schematic diagram of an alarm configuration device for a data operation link according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0029] First, the relevant technical terms in the embodiments of the present invention are introduced:

[0030] Job: A collection of programs, typically implemented in a programming language based on business logic. These programs are deployed on production servers and automatically executed at fixed times or frequencies through a scheduling management platform. Jobs in big data applications are typically used to load, copy, process, and clean data in a database.

[0031] Job dependency: The relationship between jobs. A subsequent job can only start executing after the predecessor job is completed.

[0032] Data job chain: a directed acyclic graph (DAG) consisting of jobs and job dependencies;

[0033] Topological sorting: A sorting algorithm for directed acyclic graphs that follows the forward and backward dependencies of the directed graph to construct an ordered sequence of nodes, ensuring that the job nodes that are sorted first are completed first.

[0034] Figure 1 FIG. 1 is a flow chart of an alarm configuration method for a data operation link according to an embodiment of the present invention. Figure 1 As shown, the method can be implemented by the following steps:

[0035] Step 101: Construct a directed acyclic graph with jobs as nodes and dependencies between jobs as edges. The edges are oriented from the predecessor job to the corresponding successor job.

[0036] Step 102: Initialize the value of the terminal job node as the terminal job's timeliness threshold; determine the average execution time of the predecessor jobs corresponding to the edge as the edge weight; wherein the terminal job node is a node with an out-degree of 0, and the terminal job's timeliness threshold represents the length of time the terminal job is allowed to be delayed;

[0037] Step 103: Based on the data processing systems to which the jobs belong and the dependencies between the jobs, job nodes that belong to different data processing systems and have dependencies are determined as key nodes.

[0038] Step 104: Starting from the terminal job node, perform a multi-source breadth-first search on the directed acyclic graph and update the value of each node according to the following rules: add the weight of the edge pointing to the current search node to the value of the current search node; if the sum is greater than the value of the starting node of the edge, or the starting node of the edge has no value assigned, then update the value of the starting node of the edge to the sum; the updated value of each node is the recommended timeliness threshold of the corresponding job of each node;

[0039] Step 105: Configure an alarm threshold for the job corresponding to the key node according to the recommended timeliness threshold of the key node.

[0040] Depend on Figure 1 As can be seen from the process, in the embodiment of the present invention, a directed acyclic graph is constructed based on the execution time of the job, the timeliness threshold of the terminal job, and the job dependency relationship. The recommended timeliness threshold of the job is efficiently solved through the multi-source breadth-first search algorithm of the graph, and the jobs with dependencies between different systems are identified as key nodes. Alarms are configured based on the recommended timeliness threshold of the key nodes. Compared with the existing solutions that configure alarms for all nodes in the entire data job chain, or only configure alarms for the terminal job, the embodiment of the present invention uses the multi-source breadth-first search algorithm of the graph to efficiently solve the recommended timeliness threshold of the job, reduce the communication cost between the terminal job system and the upstream job system, improve the efficiency of job timeliness calculation, and can also automatically identify all key nodes on the data job chain. Alarms are configured based on the recommended timeliness threshold of the key nodes, thereby improving the accuracy and efficiency of data job chain alarms, avoiding the generation of alarm storms, improving operation and maintenance effects, and reducing operation and maintenance pressure.

[0041] With the rapid development of banking services, the number of data processing jobs requiring operational management and maintenance in big data development scenarios is increasing. Furthermore, these data processing jobs involve multiple tasks, complex dependencies, and span multiple systems, posing significant challenges to timely monitoring of these jobs. To simplify job dependencies in development scenarios, the inventors leveraged graph theory to abstract data processing jobs into data job chains. This not only clearly demonstrates the dependencies between jobs but also greatly simplifies communication and collaboration between downstream and upstream systems.

[0042] In the embodiment of the present invention, a directed acyclic graph is constructed with jobs as nodes and dependencies between jobs as edges; wherein the direction of the edges is from the predecessor job to the corresponding successor job.

[0043] In an embodiment of the present invention, the value of the terminal job node is initialized as the terminal job's timeliness threshold; the average execution time of the predecessor jobs corresponding to the edge is determined as the edge weight; wherein the terminal job node is a node with an out-degree of 0, and the terminal job's timeliness threshold represents the length of time that the terminal job is allowed to be delayed.

[0044] Figure 2 This is a job dependency diagram of a data job link in an embodiment of the present invention. For example, Figure 2 As shown, there are three end-job nodes (end-job 1-end-job 3), 14 upstream job nodes (job 1-job 14), and four data job systems (system A, system B, system C, and system D). Taking the red end-job nodes in the figure as an example, the deadline threshold for end-job 1 is 32 hours, and the deadline threshold for end-job 2 is 30 hours. The value of the node corresponding to end-job 1 is initialized to the deadline threshold -32, and the value of the node corresponding to end-job 2 is initialized to -30. (The format of the node value can also be: T+N hh:mm:ss, so the value of the node corresponding to end-job 1 is T+1 8:00:00, and the value of the node corresponding to end-job 2 is T+1 6:00:00; where T represents the current date, N represents the number of days allowed for delay, and hh:mm:ss represents the hours: minutes: seconds allowed for delay.)

[0045] In one embodiment, before determining the average execution time of the predecessor job corresponding to the edge as the weight of the edge, the method further includes: collecting the historical execution time of the job; removing outliers in the historical execution time of the job; and calculating the average value of the historical execution time of the job after removing the outliers to obtain the average execution time of the job.

[0046] The inventors have improved the multi-source breadth-first search process, allowing them to simultaneously calculate the values ​​of each node in a directed acyclic graph while performing a multi-source breadth-first search. Leveraging sophisticated graph search algorithms, they have significantly reduced the communication costs between downstream and upstream job systems. To this end, the inventors designed the weights of directed edges in the directed acyclic graph to be the execution time of the corresponding job at the starting point of the directed edge.

[0047] In one embodiment, before determining the average execution time of the predecessor job corresponding to the edge as the weight of the edge, it also includes: starting from the terminal job node, performing a multi-source breadth-first search on the directed acyclic graph to obtain the upstream dependent job set of the terminal job; retaining the nodes in the directed acyclic graph that belong to the upstream dependent job set, and deleting the nodes in the directed acyclic graph that do not belong to the upstream dependent job set.

[0048] Because existing big data development scenarios typically involve multiple end-user jobs, searching the entire data job chain when only analyzing a subset of them wastes computing power. Therefore, in this embodiment, before assigning weights to edges, a multi-source breadth-first search algorithm is used to identify all upstream dependent job sets of the end-user job to be analyzed. Job nodes that are not part of the upstream dependent job set of the end-user job to be analyzed are then deleted, reducing computing power waste.

[0049] In the embodiment of the present invention, job nodes that belong to different data processing systems and have dependency relationships are determined as key nodes based on the data processing systems to which the jobs belong and the dependency relationships between the jobs.

[0050] In existing big data development scenarios, ensuring that all jobs are completed within deadlines is crucial. This includes ensuring data timeliness to meet business needs; rationally allocating computing and storage resources based on job deadlines and resource requirements; and identifying performance bottlenecks and implementing targeted performance optimizations to improve overall development performance. However, data links are often long and involve multiple cross-system node dependencies, resulting in high inter-node communication costs. Therefore, timely locating problematic jobs is a major challenge that urgently needs to be addressed.

[0051] In one embodiment, based on the data processing system to which the job belongs and the dependency relationship between jobs, job nodes that belong to different data processing systems and have dependency relationships are determined as key nodes, including: based on the data processing system to which the job belongs and the dependency relationship between jobs, a job that has a different data processing system from that of a subsequent dependent job is determined as an exit job, and a job that has a different data processing system from that of a preceding dependent job is determined as an entry job; and nodes corresponding to the exit job and the entry job are determined as key nodes.

[0052] Big data development scenarios usually involve multiple data processing systems. Since the communication cost between cross-system operations is much higher than the communication cost between operations within a system, in order to quickly locate the operation node with problems, the inventors identified the operations that interact between different systems as key nodes. Based on the operation time limit of the key node, the data processing system where the problem operation is located can be quickly locked, thereby locating the problem node in time and reducing the operation and maintenance pressure of the operation. Specifically, Figure 2 As shown in the figure, the yellow nodes are key nodes in the upstream job set of terminal job 1 and terminal job 2; among them, job 2, job 4, job 5, job 10, and job 12 are entry jobs, and job 6, job 7, and job 13 are exit jobs.

[0053] Figure 3 This is a weighted job dependency graph of a data job link in an embodiment of the present invention. Figure 3The data operation link in the example is terminal operation 1, terminal operation 2, and their corresponding upstream dependent operation set. In this example, the value of the node corresponding to terminal operation 1 is -31 (T+1 7:00:00), and the value of the node corresponding to terminal operation 2 is -34 (T+1 10:00:00). There are 6 operation nodes in the upstream dependent operation set of the two terminal operation nodes. Figure 3 It can be seen that the average execution time of job 1 is 2 hours, the average execution time of job 2 is 5 hours, the average execution time of job 3 is 2 hours, the average execution time of job 4 is 8 hours, the average execution time of job 5 is 2 hours, and the average execution time of job 6 is 3 hours.

[0054] In an embodiment of the present invention, starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph, and the value of each node is updated according to the following rules: the weight of the edge pointing to the current search node is added to the value of the current search node; if the sum is greater than the value of the starting node of the edge, or the starting node of the edge is not assigned a value, the value of the starting node of the edge is updated to the sum; wherein the updated value of each node is the recommended timeliness threshold of the corresponding job of each node.

[0055] In one embodiment, before performing a multi-source breadth-first search on the directed acyclic graph starting from the terminal job node, it also includes: performing a reverse topological sorting on the directed acyclic graph to obtain the job order; starting from the terminal job node, performing a multi-source breadth-first search on the directed acyclic graph according to the job order.

[0056] For example, first, the job set is sorted in reverse topology of the directed acyclic graph to obtain a job order (the job order obtained here follows the front-to-back dependency relationship of the directed graph to ensure that the job nodes sorted later are completed first); then, according to the obtained job order, a multi-source breadth-first search is performed on the directed acyclic graph; during the multi-source breadth-first search process, the value of each node is updated according to the predetermined rules, and the updated node value is the recommended timeliness threshold of the job corresponding to the node.

[0057] exist Figure 3 In the embodiment shown, a reverse topology sequence is: terminal job 1, terminal job 2, job 1, job 4, job 2, job 3, job 5, job 6. The specific update process of the value of each node is as follows:

[0058] Step 1 (current search node: terminal job 1): -31 + 2 = -29, the value of the node corresponding to job 1 is -29 (T+15:00:00);

[0059] Step 2 (current search node: terminal job 2): -34 + 8 = -26. The value of the node corresponding to job 4 is -26 (T+12:00:00).

[0060] Step 3 (current search node: Job 1): -29 + 5 = -24, the value of the node corresponding to Job 2 is -24 (T+10:00:00); -29 + 2 = -27, the value of the node corresponding to Job 3 is -27 (T+1: 3:00:00);

[0061] Step 4 (current search node: Job 4): -26 + 2 = -24. The value of the node corresponding to Job 5 is -24 (T+024:00:00).

[0062] Step 5 (Current search node: Job 2): Job 2 has no upstream jobs and does not require calculation;

[0063] Step 6 (current search node: Job 3): -27 + 2 = -25 < -24, the value of the node corresponding to Job 5 remains unchanged at -24 (T+024:00:00);

[0064] Step 7 (current search node: Job 5): -24 + 3 = -20, the value of the node corresponding to Job 6 is -20 (T+020:00:00);

[0065] Step 8 (Current search node: Job 6): Job 6 has no upstream jobs and does not require calculation.

[0066] Figure 4 This is a flowchart of the key operation time analysis in the embodiment of the present invention. For example, Figure 4 As shown in the figure, the key task time efficiency analysis can be achieved by following the steps below:

[0067] Step 401, operation threshold configuration, corresponding to step 102, initializing the data link;

[0068] Step 402: Link operation analysis, using a multi-source breadth-first search algorithm to simplify the data link;

[0069] Step 403, key node analysis, corresponding to step 103, determines the key nodes in the data link;

[0070] Step 404 , timeliness threshold analysis, corresponds to step 104 , and obtains the recommended timeliness threshold for each job in the data link.

[0071] In the embodiment of the present invention, an alarm threshold is configured for the job corresponding to the key node according to the recommended timeliness threshold of the key node.

[0072] For example, by performing a timeliness analysis on the data link using the above method, we can ultimately obtain the recommended timeliness thresholds for all upstream dependent jobs of the terminal job. Then, we can filter out the recommended timeliness thresholds for key nodes from the recommended timeliness thresholds for all nodes, and automatically generate alarm configuration information for the data processing jobs corresponding to the key nodes for monitoring.

[0073] The present invention also provides an alarm configuration device for a data operation link, as described in the following embodiments. Since the principles of this device are similar to those of the alarm configuration method for a data operation link, the implementation of this device can refer to the implementation of the alarm configuration method for a data operation link, and the repeated parts will not be repeated.

[0074] Figure 5 FIG. 1 is a schematic diagram of an alarm configuration device for a data operation link according to an embodiment of the present invention. Figure 5 As shown, the device includes:

[0075] The link construction module 501 is used to construct a directed acyclic graph using jobs as nodes and dependencies between jobs as edges, where the direction of the edges is from the predecessor job to the corresponding successor job.

[0076] Link initialization module 502 is used to initialize the value of the terminal job node to the terminal job's timeliness threshold; determine the average execution time of the predecessor jobs corresponding to the edge as the edge weight; the terminal job node is a node with an out-degree of 0, and the terminal job's timeliness threshold represents the length of time the terminal job is allowed to be delayed;

[0077] The key node analysis module 503 is used to determine the job nodes that belong to different data processing systems and have dependencies as key nodes based on the data processing systems to which the jobs belong and the dependencies between the jobs;

[0078] The timeliness threshold analysis module 504 is used to perform a multi-source breadth-first search on the directed acyclic graph starting from the terminal job node, and update the value of each node according to the following rules: the weight of the edge pointing to the current search node is added to the value of the current search node; if the sum of the added values ​​is greater than the value of the starting node of the edge, or the starting node of the edge has no value assigned, the value of the starting node of the edge is updated to the sum value; the updated value of each node is the recommended timeliness threshold of the corresponding job of each node;

[0079] The alarm configuration module 505 is used to configure alarm thresholds for jobs corresponding to key nodes according to the recommended timeliness thresholds of the key nodes.

[0080] In one embodiment, the link initialization module 502 is further configured to, before determining the average execution time of the predecessor job corresponding to the edge as the weight of the edge:

[0081] Collect the historical execution time of the job;

[0082] Remove outliers in the job's historical execution time;

[0083] Calculate the average of the job's historical execution time after removing outliers to obtain the average execution time of the job.

[0084] In one embodiment, the link initialization module 502 is further configured to, before determining the average execution time of the predecessor job corresponding to the edge as the weight of the edge:

[0085] Starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph to obtain the upstream dependent job set of the terminal job;

[0086] The nodes in the directed acyclic graph that belong to the upstream dependent job set are retained, and the nodes in the directed acyclic graph that do not belong to the upstream dependent job set are deleted.

[0087] In one embodiment, the key node analysis module 503 is specifically configured to:

[0088] Based on the data processing system to which the job belongs and the dependency relationship between jobs, a job that has a different data processing system than the subsequent dependent job is determined as an exit job, and a job that has a different data processing system than the preceding dependent job is determined as an entry job;

[0089] The nodes corresponding to the export and import operations are identified as key nodes.

[0090] In one embodiment, the time threshold analysis module 504 is specifically configured to:

[0091] Perform reverse topological sorting on the directed acyclic graph to obtain the job order;

[0092] Starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph according to the job order.

[0093] An embodiment of the present invention further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned alarm configuration method for the data operation link when executing the computer program.

[0094] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the alarm configuration method for the data operation link.

[0095] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the alarm configuration method for the data operation link.

[0096] To summarize, in an embodiment of the present invention, a directed acyclic graph is constructed with jobs as nodes and dependencies between jobs as edges; the value of the terminal job node is initialized to the timeliness threshold of the terminal job; a multi-source breadth-first search is performed with the terminal job node as the starting point to obtain a set of upstream dependent jobs and simplify the directed acyclic graph; after denoising the execution time of the job, the average execution time of the job is calculated; the average execution time of the predecessor job corresponding to the edge is determined as the weight of the edge; based on the data processing system to which the job belongs and the dependency relationship between the jobs, job nodes that belong to different data processing systems and have dependency relationships are determined as key nodes; starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph in the order of inverse topological sorting to update the value of each node. Compared with the technical solutions in the prior art that configure alarms for all nodes in the entire data operation chain, or the technical solutions that configure alarms only for terminal operations, in the embodiments of the present invention, a directed acyclic graph is constructed based on operations and operation dependencies, and graph theory knowledge is used to reduce the communication costs between downstream systems and upstream systems, and improve the efficiency of operation time calculation. In response to the problem of multiple cross-system node dependencies in existing big data development scenarios, data processing operations that interact between systems are identified as key nodes, and all key nodes on the data operation chain are automatically identified. Alarms are configured according to the recommended time thresholds of the key nodes, thereby improving the accuracy and efficiency of data operation chain alarms, avoiding alarm storms, improving operation and maintenance effects, and reducing operation and maintenance pressure.

[0097] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for configuring an alarm for a data operation link, characterized in that: include: A directed acyclic graph is constructed with jobs as nodes and dependencies between jobs as edges. The direction of the edges is from the predecessor job to the corresponding successor job. Initialize the value of the terminal job node to the terminal job's timeliness threshold; determine the average execution time of the predecessor jobs corresponding to the edge as the edge weight; where the terminal job node is a node with an out-degree of 0, and the terminal job's timeliness threshold represents the length of time the terminal job is allowed to be delayed; Based on the data processing systems to which the jobs belong and the dependencies between the jobs, job nodes that belong to different data processing systems and have dependencies are identified as key nodes; Starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph. The value of each node is updated according to the following rules: the weight of the edge pointing to the current search node is added to the value of the current search node. If the sum is greater than the value of the starting node of the edge, or the starting node of the edge has no value, the value of the starting node of the edge is updated to the sum. The updated value of each node is the recommended timeliness threshold of the corresponding job. Configure alarm thresholds for jobs corresponding to key nodes based on the recommended timeliness thresholds for key nodes. Before determining the average execution time of the predecessor job corresponding to the edge as the weight of the edge, the method further includes: starting from the terminal job node, performing a multi-source breadth-first search on the directed acyclic graph to obtain a set of upstream dependent jobs of the terminal job; retaining nodes in the directed acyclic graph that belong to the set of upstream dependent jobs, and deleting nodes in the directed acyclic graph that do not belong to the set of upstream dependent jobs; Based on the data processing system to which the job belongs and the dependency relationship between jobs, job nodes that belong to different data processing systems and have dependency relationships are determined as key nodes, including: based on the data processing system to which the job belongs and the dependency relationship between jobs, jobs that have different data processing systems from those of subsequent dependent jobs are determined as exit jobs, and jobs that have different data processing systems from those of preceding dependent jobs are determined as entry jobs; and nodes corresponding to exit jobs and entry jobs are determined as key nodes.

2. The method according to claim 1, wherein Before determining the average execution time of the predecessor jobs corresponding to an edge as the edge weight, the following is also included: Collect the historical execution time of the job; Remove outliers in the job's historical execution time; Calculate the average of the job's historical execution time after removing outliers to obtain the average execution time of the job.

3. The method according to claim 1, wherein Starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph, including: Perform reverse topological sorting on the directed acyclic graph to obtain the job order; Starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph according to the job order.

4. An alarm configuration device for a data operation link, characterized in that: include: The link construction module is used to construct a directed acyclic graph with jobs as nodes and dependencies between jobs as edges. The direction of the edges is from the predecessor job to the corresponding successor job. The link initialization module is used to initialize the value of the terminal job node to the terminal job's timeliness threshold; the average execution time of the predecessor job corresponding to the edge is determined as the edge weight; the terminal job node is a node with an out-degree of 0, and the terminal job's timeliness threshold represents the length of time the terminal job is allowed to be delayed; A key node analysis module is used to determine the job nodes that belong to different data processing systems and have dependencies as key nodes based on the data processing systems to which the jobs belong and the dependencies between the jobs; The timeliness threshold analysis module is used to perform a multi-source breadth-first search on the directed acyclic graph starting from the terminal job node, and update the value of each node according to the following rules: the weight of the edge pointing to the current search node is added to the value of the current search node; if the sum is greater than the value of the starting node of the edge, or the starting node of the edge has no value assigned, the value of the starting node of the edge is updated to the sum; the updated value of each node is the recommended timeliness threshold for the corresponding job of each node; The alarm configuration module is used to configure alarm thresholds for jobs corresponding to key nodes based on the recommended timeliness thresholds of key nodes; The key node analysis module is specifically used to: determine, based on the data processing system to which the job belongs and the dependency relationship between jobs, jobs that have a different data processing system from that of subsequent dependent jobs as exit jobs, and jobs that have a different data processing system from that of preceding dependent jobs as entry jobs; and determine the nodes corresponding to the exit jobs and entry jobs as key nodes; The time threshold analysis module is specifically used to: perform reverse topological sorting on the directed acyclic graph to obtain the job sequence; starting from the terminal job node, perform a multi-source breadth-first search on the directed acyclic graph according to the job sequence.

5. The device according to claim 4, characterized in that The link initialization module is also used to determine the average execution time of the predecessor job corresponding to the edge as the edge weight: Collect the historical execution time of the job; Remove outliers in the job's historical execution time; Calculate the average of the job's historical execution time after removing outliers to obtain the average execution time of the job.

6. The device according to claim 4, characterized in that The link initialization module is also used to determine the average execution time of the predecessor job corresponding to the edge as the edge weight: Starting from the terminal job node, a multi-source breadth-first search is performed on the directed acyclic graph to obtain the upstream dependent job set of the terminal job; The nodes in the directed acyclic graph that belong to the upstream dependent job set are retained, and the nodes in the directed acyclic graph that do not belong to the upstream dependent job set are deleted.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images