Operation monitoring system and abnormal operation positioning method

Abstract

The invention provides an operation monitoring system and an abnormal operation positioning method, and is applied to the technical field of operation management, and the method comprises the steps: monitoring the working condition of each operation in an operation management and control system in real time, determining the operation as a target operation when a preset alarm condition is satisfied, and generating an alarm signal of the target operation; based on the alarm signal, using an operation link data model to construct an operation link of the target operation; and obtaining current state data of each job, and positioning an abnormal job corresponding to the target job based on each current state data and the job link of the target job. Therefore, operation and maintenance personnel do not need to track the operation links layer by layer, abnormal operation can be quickly positioned, the situation of omission or misjudgment is avoided, and the workload of the operation and maintenance personnel is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of operation management, and in particular to an operation monitoring system and an abnormal operation positioning method. Background Art

[0002] In modern enterprise operations, tracking and analyzing the relationships and processes between various operations in the system is crucial for operation and maintenance management. These operations may involve a variety of business activities such as data processing, service calls, and task scheduling. Their effective management and monitoring are directly related to the operational efficiency and stability of the enterprise. In operation and maintenance emergency scenarios, it is even more necessary to understand the operation status of the operation link in a timely and accurate manner in order to quickly locate and solve problems and ensure the normal operation of the system.

[0003] In traditional job operation and maintenance management, when an abnormality occurs in a job, usually only the static description can be used to trace the upstream root cause of the problem job layer by layer. However, the system scale of large enterprises is huge, often with millions of jobs, and the job chain is more complex. In this case, operation and maintenance personnel often need to spend a lot of time and energy to manually track the job chain level by level to locate the abnormal job. This method is not only time-consuming and labor-intensive, but also prone to omissions or misjudgments, which prolongs the time for troubleshooting and affects the normal operation of the system. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a job monitoring system and an abnormal job locating method. By applying the solution provided by the present application, there is no need for operation and maintenance personnel to track the job links layer by layer. The job links can be constructed using the constructed job link data model, and then the job links can be used to locate abnormal jobs, thereby avoiding omissions or misjudgments. Abnormal jobs can be quickly located, reducing the workload of operation and maintenance personnel.

[0005] To achieve the above objectives, the present application provides the following technical solutions:

[0006] An operation monitoring system, comprising:

[0007] Data collection unit, data processing unit, graph database, query service unit and monitoring service unit;

[0008] The data acquisition unit is used to obtain configuration data of each job from the job management and control system;

[0009] The data processing unit is used to receive the configuration data of each job sent by the data acquisition unit, and extract each node and node relationship data of each job from each configuration data, wherein each node of each job includes a job node and an output node;

[0010] The graph database is used to construct a unified job link data model and store node relationship data of each job. The job link data model is used to construct the job link of the job based on the node relationship data of the job.

[0011] The monitoring service unit is used to monitor the working conditions of each job in the job control system in real time, determine whether there is a job that meets the preset alarm condition, and when there is a job that meets the alarm condition, determine the job as a target job, generate an alarm signal of the target job, and send the alarm signal to the query service unit;

[0012] The query service unit is used to receive the alarm signal, obtain the job link of the target job constructed using the job link data model in the graph database, obtain the current status data of each of the jobs from the data acquisition unit, and locate the abnormal job corresponding to the target job based on the current status data of each of the jobs and the job link of the target job.

[0013] In the above system, optionally, the data processing unit executes a process of extracting each node and node relationship data of each job from each configuration data, comprising:

[0014] Acquire job flow information, job name information and output information of each job from configuration data of each job;

[0015] For each of the jobs, the job name information of the job is applied to construct a job node, and the output information of the job is applied to construct an output node, and the first association information of the job node and the second association information of the output node are generated, and node relationship data is generated based on the job flow information, the first association information and the second association information.

[0016] In the above system, optionally, the query service unit executes a process of acquiring the job link of the target job constructed using the job link data model in the graph database, including:

[0017] For each job node of the job, based on the node relationship data of the job, determine the relationship node of the job node, and establish a relationship edge between the job node and its relationship node according to the job link data model to obtain the link structure of the job;

[0018] Based on the link structure of all jobs, obtain the full amount of job link data;

[0019] The job link of the target job is obtained from the full link data.

[0020] In the above system, optionally, the monitoring service unit performs a process of determining whether there is a job that meets a preset alarm condition, including:

[0021] Obtaining execution status data of each of the jobs;

[0022] Determining the execution status of each of the jobs based on each of the execution status data;

[0023] For each of the jobs, when the execution status of the job matches the alarm status in the alarm condition, it is determined that the job meets the alarm condition.

[0024] In the above system, optionally, the query service unit performs a process of locating an abnormal job corresponding to the target job based on the current status data of each job and the job link of the target job, including:

[0025] Based on the current status data of each of the jobs, determining the current status of each node in the job link of the target job;

[0026] Determine the job node of the target job in the job link of the target job as the target node, and determine each node of the job link of the target job as a to-be-determined node;

[0027] Determine the node to be determined that is closest to the target node and whose current state is characterized as abnormal as an abnormal node;

[0028] The job corresponding to the abnormal node is determined as an abnormal job.

[0029] A method for locating an abnormal operation, comprising:

[0030] Monitor the working status of each job in the job control system in real time to determine whether there is a job that meets the preset alarm conditions. When there is a job that meets the preset alarm conditions, determine the job as the target job and generate an alarm signal for the target job;

[0031] Based on the alarm signal, construct the job link of the target job using a pre-constructed job link data model;

[0032] The current status data of each of the jobs is acquired, and based on the current status data of each of the jobs and the job link of the target job, the abnormal job corresponding to the target job is located.

[0033] In the above method, optionally, the determining whether there is a job that meets a preset alarm condition includes:

[0034] Obtaining execution status data of each of the jobs;

[0035] Determining the execution status of each of the jobs based on each of the execution status data;

[0036] For each of the jobs, when the execution status of the job matches the alarm status in the alarm condition, it is determined that the job meets the alarm condition.

[0037] In the above method, optionally, the step of using a pre-built job link data model to build the job link of the target job includes:

[0038] Acquire configuration data of each of the jobs, and extract each node and node relationship data of each of the jobs from each of the configuration data, wherein each node of each of the jobs includes a job node and an output node;

[0039] For each job node of the job, based on the node relationship data of the job, determine the relationship node of the job node, and establish a relationship edge between the job node and its relationship node according to the job link data model to obtain the link structure of the job;

[0040] Based on the link structure of all jobs, obtain the full amount of job link data;

[0041] The job link of the target job is obtained from the full link data.

[0042] In the above method, optionally, extracting each node and node relationship data of each job from each configuration data includes:

[0043] Acquire job flow information, job name information and output information of each job from configuration data of each job;

[0044] For each of the jobs, the job name information of the job is applied to construct a job node, and the output information of the job is applied to construct an output node, and the first association information of the job node and the second association information of the output node are generated, and node relationship data is generated based on the job flow information, the first association information and the second association information.

[0045] Optionally, the method of locating the abnormal job corresponding to the target job based on the current status data of each job and the job link of the target job includes:

[0046] Based on the current status data of each of the jobs, determining the current status of each node in the job link of the target job;

[0047] Determine the job node of the target job in the job link of the target job as the target node, and determine each node of the job link of the target job as a to-be-determined node;

[0048] Determine the node to be determined that is closest to the target node and whose current state is characterized as abnormal as an abnormal node;

[0049] The job corresponding to the abnormal node is determined as an abnormal job.

[0050] Compared with the prior art, this application has the following advantages:

[0051] The present application provides a job monitoring system and an abnormal job positioning method, including: real-time monitoring of the working conditions of each job in the job control system, when there is a preset alarm condition, the job is determined as a target job, and an alarm signal of the target job is generated; based on the alarm signal, the job link of the target job is constructed using a pre-constructed job link data model; the current status data of each job is obtained, and based on the current status data of each job and the job link of the target job, the abnormal job corresponding to the target job is positioned. Thus, there is no need for operation and maintenance personnel to track the job link layer by layer, and the abnormal job can be quickly located to avoid omissions or misjudgments. In addition, the output results of the job execution are also extracted as nodes in the present application, so the cause of the abnormality can be determined more accurately, and the cause of the system abnormality can also be quickly determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 A schematic diagram of the structure of an operation monitoring system provided in an embodiment of the present application;

[0054] Figure 2 An example diagram of a link structure of a job provided in an embodiment of the present application;

[0055] Figure 3 A flowchart of a method for determining whether a preset alarm condition is met provided in an embodiment of the present application;

[0056] Figure 4 A flow chart of a method for determining an abnormal operation corresponding to an alarm signal provided in an embodiment of the present application;

[0057] Figure 5An example diagram of a job link provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of a job link query process provided in an embodiment of the present application;

[0059] Figure 7 An example diagram of the architecture of the job monitoring system provided in an embodiment of the present application;

[0060] Figure 8 A flowchart of an abnormal operation positioning method provided in an embodiment of the present application;

[0061] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0064] The background technology mentioned that locating abnormal operations is time-consuming and labor-intensive, and is prone to omissions or misjudgments. In addition to the problems in the background technology, since the operation chain is too long and has many branches, the traditional operation management platform cannot provide sufficient support and visualization tools, resulting in difficulty for operation and maintenance personnel to clearly understand the structure and relationship of the entire operation chain, which in turn affects the operation and maintenance personnel's emergency response capabilities for the system.

[0065] In view of the above problems, the present application provides an operation monitoring system and an abnormal operation locating method. Furthermore, the operation monitoring system supports the implementation of the abnormal operation locating method. In other words, the abnormal operation locating method is applied to the operation monitoring system.

[0066] The job monitoring system can be configured using a variety of general-purpose or special-purpose computing devices, such as personal computers, server computers, multi-processor devices, etc.

[0067] Reference Figure 1 , is a schematic diagram of the structure of a job monitoring system provided in an embodiment of the present application, and is specifically described as follows:

[0068] The operation monitoring system includes: a data collection unit 101, a data processing unit 102, a graph database 103, a query service unit 104 and a monitoring service unit 105.

[0069] The data collection unit 101 is used to obtain the configuration data of each job from the job management and control system.

[0070] The job management and control system includes configuration data of multiple jobs and real-time status data of the jobs. Preferably, the data acquisition unit is also used to obtain real-time status data of each job in the job management and control system.

[0071] Preferably, the configuration data can also be called static configuration data, which includes job definition data, job dependency data, job flow data, etc.; wherein, the real-time data of job status includes metadata such as job instance ID, execution batch, actual start time, scheduling node, etc. that uniquely identify the job instance, as well as data reflecting the current running status of the job instance.

[0072] The data processing unit 102 is used to receive the configuration data of each job sent by the data acquisition unit, and extract each node and node relationship data of each job from each configuration data, wherein each node of each job includes a job node and an output node. Preferably, the job flow to which the job belongs can also be extracted as a job flow node.

[0073] The data processing unit is used to receive the configuration data of each job sent by the data acquisition unit, and then perform normalized processing on the received configuration data, so as to extract the job node, output node and node relationship data of each job from each configuration data. Preferably, the node can be called an entity, then the job node can be called a job entity, and the output node can be called an output entity. The entity of this application refers to the object or concept represented by the data. Furthermore, the output node can be divided into a data file node and a custom condition node, wherein the data file node is an abstract node of the data file, and the custom condition node is an abstract node of the custom condition. Both the data file and the custom condition are the results of the job execution output.

[0074] Exemplarily, the process of extracting each node and node relationship data of a job from the configuration data is as follows: obtaining the job flow information, job name information and output information of each job from the configuration data of each job; for each job, applying the job name information of the job to construct a job node, and applying the output information of the job to construct an output node, and generating the first association information of the job node and the second association information of the output node, and generating the node relationship data based on the job flow information, the first association information and the second association information. Preferably, the first association information of the job node includes the information of the node that has a dependency relationship with the job node, and similarly, the second association information of the output node includes the information of the node that has a dependency relationship with the output node.

[0075] Exemplarily, taking job 1 and job 2 as examples, the job node of job 1 is A1, and the output node is B1, the job node of job 2 is A2, and the output node is B2, wherein A2 depends on B1, thus, the first association information of job node A1 includes information associated with output node B1, the second association information of output node B1 includes information associated with job node A1 and information associated with job node A2, the first association information of job node A2 includes information associated with output node B1 and information associated with output node B2, and the second association information of output node B1 includes information associated with job node A2.

[0076] This application also extracts the output results of the job into independent nodes to clarify the type of faulty node and better locate the cause of the fault.

[0077] The node relationship data includes job flow data and the relationship data between each node. The relationship data includes the subordinate relationship between the job and the job flow, the dependency relationship between the job and the data file, and the dependency relationship between the job and the custom conditions. For example, if a job runs and outputs a data file, it is considered that there is a dependency relationship between the two.

[0078] The graph database 103 is used to construct a unified job link data model and store the node relationship data of each job. The job link data model is used to construct the job link of the job based on the node relationship data of the job.

[0079] A unified job chain data model is built based on business information, and node relationship data of each job is stored. Business information is the relevant information of the business supported by the job monitoring system. The job chain data model can be understood as data features such as data structure and data constraints abstracted from business information. The job chain data model is an abstract model.

[0080] Furthermore, the data file node and the custom condition node are connected in series with the upstream and downstream job nodes. The downstream job node can start execution only after receiving the data file or custom condition output by the upstream job after execution.

[0081] Preferably, the graph database uses a graph structure to efficiently organize and store data such as job nodes and dependencies, and also provides graph query capabilities.

[0082] You can use the job link data model in the graph database to build the job link structure, refer to Figure 2 , which is an example diagram of the link structure of a job provided in an embodiment of the present application. As shown in the figure, the relationship edge in the link structure of the job is a directed edge pair, which represents the bidirectional dependency between nodes. The relationship edge includes a directed edge representing the dependency relationship and an undirected edge representing the subordinate relationship; in other words, the relationship edge includes a "dependency relationship" and a "dependent relationship". The "dependency relationship" indicates that a job node depends on the execution results of other nodes and points to the dependent node. The "dependent relationship" is the reverse relationship of the dependency relationship and points to the job node that depends on the node. Furthermore, the "composition relationship" in the figure is used to connect the job flow and the job nodes it contains, indicating that some jobs belong to the same job flow.

[0083] Preferably, through the directed edge pairs of the job link structure, when querying an abnormal job, the complete job link with the job as the central node can be clearly displayed, and the source can be traced layer by layer through the dependency link until the upstream fault node is found. At the same time, the downstream job node can be obtained through the dependency relationship, so as to determine the business impact range of the abnormal job.

[0084] This application uses a graph database to build a job link structure. When a user queries an abnormal job, the job link structure can be displayed to the user in a graph, which can more intuitively explain to the user the business impact of the abnormal job and clearly know the cause of the job abnormality.

[0085] The monitoring service unit 104 is used to monitor the working status of each job in the job management and control system in real time, determine whether there is a job that meets the preset alarm conditions, and when there is a job that meets the alarm conditions, determine the job as the target job, generate an alarm signal for the target job, and send the alarm signal to the query service unit.

[0086] The monitoring service unit can be integrated with the enterprise's internal alarm platform to receive the alarm signal of the job from the alarm platform. When the job meets the alarm condition, the alarm signal of the job will be obtained.

[0087] Reference Figure 3 , is a flow chart of a method for determining whether a preset alarm condition is met provided in an embodiment of the present application, and is specifically described as follows:

[0088] S201. Obtain execution status data of each job.

[0089] Preferably, the execution status data includes information about the current execution status of the job, such as: being executed, or having been executed but with an error, or timed out, etc.

[0090] S202: Determine the execution status of each job based on each execution status data.

[0091] Preferably, the execution status of the job includes but is not limited to a timeout and incomplete status, a completed status, a running status, a running error, and the like.

[0092] S203: For each job, when the execution status of the job matches the alarm status in the alarm condition, determine that the job meets the alarm condition.

[0093] When the execution status of the job does not match the alarm status in the alarm condition, it is determined that the job does not meet the alarm condition.

[0094] Furthermore, the alarm conditions record a variety of alarm states. Exemplarily, the alarm states recorded in the alarm conditions include but are not limited to timeout incomplete state, incomplete state, operation error state, etc. The alarm states are not given examples here.

[0095] For each job, the execution status of the job is matched with each alarm status in the alarm condition. When there is an alarm status consistent with the execution status in the alarm condition, it is determined that the execution status matches the alarm status in the alarm condition. At this time, it is determined that the job meets the alarm condition.

[0096] Preferably, after the alarm signal is generated, the alarm signal is sent to the query service unit, and the alarm signal includes parameters such as the job ID, execution date, and batch number.

[0097] The query service unit 105 is used to receive the alarm signal, obtain the job link of the target job constructed using the job link data model in the graph database, obtain the current status data of each job from the data collection unit, and locate the abnormal job corresponding to the target job based on the current status data of each job and the job link of the target job. The alarm signal contains parameters such as the job name and job ID of the target job.

[0098] After receiving the alarm signal, the query service unit obtains the job link of the target job corresponding to the alarm signal using the job link data model in the graph database. Then, the current status data of each job is obtained from the data collection unit, and based on the current status data of each job and the job link of the target job, the abnormal job corresponding to the target job is located.

[0099] Preferably, the current status data of each job involved in the job chain of the target job can be obtained, and then each current status data is used to locate the abnormal job corresponding to the target job in the job chain of the target job.

[0100] The process of querying the service unit to obtain the job link of the target job constructed using the job link data model in the graph database is as follows:

[0101] For each job node of a job, the relationship node of the job node is determined based on the node relationship data of the job, and the relationship edge between the job node and its relationship node is established according to the job link data model to obtain the link structure of the job; based on the link structure of all jobs, the full job link data is obtained; and the job link of the target job is obtained from the full link data. Preferably, the relationship node of the job node can be one or more, and further, the relationship node of the job node can be the output node of the job to which it belongs, or the output node of other jobs.

[0102] After obtaining the full link data, the job link of the target job is extracted from the full link data. The job link contains the link structure of each job belonging to the link. Subsequently, with the job node of the target job as the center, the upstream and downstream are analyzed layer by layer to determine the abnormal nodes on the job link and further locate the abnormal job.

[0103] In the job monitoring system provided in the embodiment of the present application, the job monitoring system includes a data acquisition unit, a data processing unit, a graph database, a query service unit and a monitoring service unit; the data acquisition unit is used to obtain the configuration data of each job from the job management and control system; the data processing unit is used to extract each node and node relationship data of each job from each configuration data, and each node of each job includes a job node and an output node; the graph database constructs a unified job link data model, and saves the node relationship data of each job, and the job link data model is used to construct the job link of the job based on the node relationship data of the job; the monitoring service unit is used to monitor the working conditions of each job in the job management and control system in real time, and determine whether there is a job that meets the preset alarm condition. When there is a job that meets the alarm condition, the job is determined as a target job, an alarm signal of the target job is generated, and the alarm signal is sent to the query service unit; the query service unit is used to receive the alarm signal, obtain the job link of the target job constructed using the job link data model in the graph database, obtain the current status data of each job from the data acquisition unit, and locate the abnormal job corresponding to the target job based on the current status data of each job and the job link of the target job. The job monitoring system uses the monitoring service unit to monitor the working conditions of each job, and determines the job that meets the alarm conditions as the target job, and then uses the constructed job link data model to build the job link, and then uses the job link to locate the abnormal job. As a result, there is no need for operation and maintenance personnel to track the job link layer by layer, avoid omissions or misjudgments, reduce the workload of operation and maintenance personnel, and quickly locate abnormal jobs. This application also extracts the output results of the job as a node, which is used to build the job link. The constructed link covers various types of nodes. Therefore, when locating abnormal jobs, the cause of the abnormality can be accurately determined.

[0104] Reference Figure 4 , is a flow chart of a method for determining an abnormal operation corresponding to an alarm signal provided in an embodiment of the present application, and the relevant description is as follows:

[0105] S401. Based on the current status data of each job, determine the current status of each node in the job link of the target job.

[0106] The current status data includes information about the current status of each node of the job, such as whether the job has been completed, whether it is waiting to be executed, or whether the result has been output.

[0107] S402: Determine the job node of the target job in the job link of the target job as the target node, and determine each node of the job link of the target job as a to-be-determined node.

[0108] When determining each node in the same relationship link with the target node, it can be determined layer by layer. For example, the target node is taken as the central node, and each node is determined layer by layer according to the "dependency relationship", and each node is determined layer by layer according to the "dependency relationship". Figure 5 Take this as an example to illustrate: Figure 5 An example diagram of the operation link provided for an embodiment of the present application, in which node 5 is used as the central node, and node 2 and node 4 are determined according to the dependency relationship, and then node 1 and node 3 are determined according to the dependency relationship. Thus, each node can be determined layer by layer, and node 6 is determined according to the dependency relationship, and then node 7 is determined according to the dependency relationship. Thus, each node can be determined layer by layer, and each determined node is determined as a node to be determined.

[0109] Preferably, nodes 1-4 can be regarded as upstream nodes of node 5, and nodes 5-6 can be regarded as downstream nodes of node 5.

[0110] S403 . Determine an abnormal node among the nodes to be determined. The abnormal node is a node to be determined that is located upstream or downstream of the target node, is closest to the target node, and has a current status characterized as abnormal.

[0111] The number of abnormal nodes is at least one. Preferably, the abnormal state can be a failure state or a running state. Figure 5 For example, the abnormal node determined is node 1.

[0112] Exemplarily, with the target node as the center, the node to be determined that is located upstream of the target node, closest to the target node and whose current state is characterized as abnormal is determined as an abnormal node; with the target node as the center, the node to be determined that is located downstream of the target node, closest to the target node and whose current state is characterized as abnormal is determined as an abnormal node.

[0113] The distance between the node to be determined and the target node can be determined by the level. The larger the level, the farther from the target node. There may be multiple nodes to be determined at the same level. Figure 5 For example, the nodes to be determined at the first level upstream of node 5 are node 2 and node 4, the nodes to be determined at the second level upstream of node 5 are node 1 and node 3, and so on, which will not be repeated here.

[0114] S404: Determine the job corresponding to the abnormal node as an abnormal job.

[0115] by Figure 5 For example, the job corresponding to the abnormal node is JOB-1001, and the job is determined as an abnormal job.

[0116] In the solution provided in the embodiment of the present application, after receiving the alarm signal, the query service unit uses the job link data model of the graph database to determine the complete job link of the target job based on the request parameters in the alarm signal, the request parameters include but are not limited to job ID, business date, batch number and other parameters; wherein the complete job link includes each node in the same link as the target job; the query service unit obtains the current status data of each node, and then performs link analysis based on the job link of the target job and the current status data of each node, outputs abnormal nodes, and then determines the abnormal job. Preferably, the impact range of the target job can also be determined, the impact range includes abnormal nodes, and includes nodes affected by the target job, and the impact range includes upstream impact range and downstream impact range, so as to determine the abnormal job. Figure 5 For example, the impact range includes nodes 1, 2, 6, and 7.

[0117] Preferably, the impact range can be displayed in the form of a link, so that the impact range can be intuitively presented to the operation and maintenance personnel, so that the operation and maintenance personnel can intuitively see the cause of the abnormality and perform accurate positioning of the abnormal operation.

[0118] Preferably, the job monitoring system can also set up an ES cluster, which receives and stores real-time job status data streams through the data acquisition unit. As a distributed search and analysis engine, the ES cluster stores these real-time data and provides efficient real-time data query capabilities for downstream components such as query service units.

[0119] Reference Figure 6 , is a schematic diagram of the operation link query process provided in the embodiment of the present application, and the specific description is as follows:

[0120] Figure 6 ElasticSearch in the query is the ES cluster mentioned above. Preferably, a timeout threshold is configured for the job in the system. When the execution time of the job exceeds the timeout threshold, the job generates a timeout incomplete alarm (equivalent to the above alarm signal). The operation and maintenance personnel or the automation system initiates a request to the query service unit (equivalent to the above alarm signal), and passes in parameters such as the alarm job ID, execution date and batch number.

[0121] The query service unit calculates and queries the upstream dependent jobs and downstream dependent jobs of the job from the job link data model in the graph database based on the passed parameters and the date / batch number offset rules of the job's configuration data to obtain a complete static job link associated with the alarm job.

[0122] The query service unit obtains the real-time operation status data of each job node in the above-mentioned job chain from the job management and control system. Based on the job chain topology structure and real-time status data, the abnormal job link is quickly located and analyzed, and the upstream dependency link of the job (the job corresponding to the alarm signal) is obtained according to the dependency relationship of the abnormal job, and the execution status of all jobs on the link is checked one by one until the faulty job with a status of "failed" or "running" is located. According to the dependency relationship of the abnormal job, the downstream job associated with the abnormal job is obtained to assist in determining the emergency disposal priority of the abnormal job. At the same time, the upstream and downstream impact range related to the abnormal job is intuitively displayed in a visual form.

[0123] by Figure 5 For example, if job 1003 generates an alarm due to timeout and failure to complete, the present invention can trace back to the source based on the dependency relationship to find the failed job 1001, and find the affected job 1004 based on the dependency relationship, thereby obtaining the abnormal job and determining the impact scope of job 1003.

[0124] Reference Figure 7 , which is an example diagram of the architecture of the job monitoring system provided in an embodiment of the present application. The relevant descriptions of the data acquisition unit, graph database, query service unit, monitoring service unit, and data processing unit in the figure refer to those described above and will not be repeated here. Among them, the relational database module can be integrated in the graph database.

[0125] Figure 7 The operation monitoring system shown is used to support the implementation of the abnormal operation positioning method, referring to Figure 8 , is a flow chart of an abnormal operation positioning method provided by this application, and the specific description is as follows:

[0126] S801. Monitor the working status of each job in the job management and control system in real time to determine whether there is a job that meets the preset alarm conditions. When there is a job that meets the preset alarm conditions, determine the job as a target job and generate an alarm signal for the target job.

[0127] The working status of each job in the monitoring job control system can be determined based on the current status data of each job, and the working status can be regarded as the status of the job.

[0128] S802: Based on the alarm signal, use the pre-built job link data model to build a job link of the target job.

[0129] S803: Acquire current status data of each job, and locate the abnormal job corresponding to the target job based on the current status data of each job and the job link of the target job.

[0130] For details on each step in this process, see Figure 1 The relevant instructions in will not be repeated here.

[0131] In the method provided by the embodiment of the present application, the working conditions of each operation in the operation control system are monitored in real time. When there is a preset alarm condition, the operation is determined as the target operation, and an alarm signal of the target operation is generated; based on the alarm signal, the target node is determined in the operation link model, and the current state of each node in the operation link model is determined; based on the operation link data model, each node in the same relationship link as the target node is determined, and each node in the same relationship link as the target node is determined as a node to be determined; the node to be determined that is closest to the target node and whose current state is characterized as abnormal is determined as an abnormal node, and the operation corresponding to the abnormal node is determined as an abnormal operation. Thus, there is no need for operation and maintenance personnel to track the operation link layer by layer, and abnormal operations can be quickly located to avoid omissions or misjudgments. In addition, the output results of the operation execution are also extracted as nodes in the present application, so that the cause of the abnormality can be determined more accurately, and the cause of the system abnormality can also be quickly determined.

[0132] This application leverages the advantages of graph databases to efficiently manage and query large-scale, complex job chain data, and combines real-time job status monitoring to provide comprehensive job chain visualization and real-time analysis capabilities, thereby improving the emergency response efficiency of large enterprises.

[0133] In the solution provided in this application, a method for bidirectional dependency modeling, querying and analyzing distributed job configuration data based on a graph database is provided. This method utilizes the powerful topological structure data management capabilities of the graph database to uniformly store the scattered job configuration data, and the complex dependencies between jobs are expressed in a graphical manner. The graph query engine built into the graph database can efficiently traverse and analyze the job configuration data, establish a bidirectional dependency model, reconstruct the complete upstream and downstream association links of the job instance, and present it through intuitive visual graphics rendering, so that operation and maintenance personnel can clearly grasp the overall picture of the job process. This innovative method effectively solves the problems of fragmentation of job configuration data and complex link analysis in traditional job management and control platforms. Through innovative technical means such as graph databases, unified management of configuration data, efficient query and link visualization are achieved, which greatly improves the efficiency of operation and maintenance monitoring and emergency response, and is of great value.

[0134] In the solution provided in the embodiment of the present application, a distributed job link monitoring system architecture based on a graph database and real-time status data is provided. The architecture combines static job configuration data with dynamic real-time operation status data, stores and queries job link topology relationships through a graph database, and the real-time monitoring module efficiently stores and queries real-time status data, and integrates the two in a query service unit to achieve comprehensive real-time link monitoring and alarm.

[0135] In the method provided by our company, a unified job configuration data model (i.e., job link data model) is constructed to realize the centralized management of configuration data. By establishing a bidirectional dependency graph model and using the built-in graph traversal and analysis engine of the graph database, the complete job link relationship can be efficiently reconstructed and rendered into an intuitive visual view, greatly simplifying the complexity of link analysis. At the same time, real-time job status data is integrated to realize real-time monitoring, analysis, and visualization of the job link.

[0136] Compared with the traditional method with low query efficiency and insufficient support for complex association relationships, the method of the present invention can query and analyze the real-time execution of the operation link more quickly and accurately, and effectively support the complex association query and real-time monitoring requirements of the mesh topology structure. With the help of the visual link view, operation and maintenance personnel can clearly and intuitively grasp the overall picture of the operation process, timely discover and accurately locate abnormal nodes, and clarify the impact of abnormal operations on downstream, thereby improving the efficiency and quality of emergency response to operations.

[0137] The present application proposes a solution for real-time monitoring and alarming of job links in combination with a graph database. The solution utilizes the efficient query and relationship analysis capabilities of the graph database, provides a simple and intuitive query interface through bidirectional dependency modeling, and introduces real-time data to achieve real-time monitoring and analysis of job links, and provide real-time feedback on the upstream and downstream impact of abnormal jobs. Compared with existing methods, the present invention has higher query efficiency and accuracy, can support multi-level association query and real-time monitoring requirements of complex job links, and significantly improves the response speed of job monitoring and the timeliness of alarms. The present application not only improves the visualization and transparency of job links, but also provides operation and maintenance personnel with efficient decision support tools, which helps to optimize the company's job management and troubleshooting processes.

[0138] The present application can effectively solve the problems of low efficiency and difficulty in tracking and analyzing abnormal operations in the traditional methods based on relational databases or distributed file systems.

[0139] Although the present invention depicts operations in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0140] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0141] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned abnormal operation locating method.

[0142] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Fig. 9 As shown, it specifically includes a memory 901 and one or more instructions 902, wherein the one or more instructions 902 are stored in the memory 901 and are configured to be executed by one or more processors 903 to execute the one or more instructions 902 to perform the above-mentioned abnormal operation locating method.

[0143] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards in the relevant regions.

[0144] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0145] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0146] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0147] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A work monitoring system, characterized in that: include: Data collection unit, data processing unit, graph database, query service unit and monitoring service unit; The data acquisition unit is used to obtain the configuration data of each job from the job management and control system; The data processing unit is used to receive the configuration data of each job sent by the data acquisition unit, and extract each node and node relationship data of each job from each configuration data, wherein each node of each job includes a job node and an output node; The graph database is used to construct a unified job link data model and store node relationship data of each job. The job link data model is used to construct the job link of the job based on the node relationship data of the job. The monitoring service unit is used to monitor the working conditions of each job in the job control system in real time, determine whether there is a job that meets the preset alarm condition, and when there is a job that meets the alarm condition, determine the job as a target job, generate an alarm signal of the target job, and send the alarm signal to the query service unit; The query service unit is used to receive the alarm signal, obtain the job link of the target job constructed using the job link data model in the graph database, obtain the current status data of each of the jobs from the data acquisition unit, and locate the abnormal job corresponding to the target job based on the current status data of each of the jobs and the job link of the target job.

2. The system according to claim 1, characterized in that The data processing unit performs a process of extracting each node and node relationship data of each job from each configuration data, including: Acquire job flow information, job name information and output information of each job from configuration data of each job; For each of the jobs, the job name information of the job is applied to construct a job node, and the output information of the job is applied to construct an output node, and the first association information of the job node and the second association information of the output node are generated, and node relationship data is generated based on the job flow information, the first association information and the second association information.

3. The system according to claim 1, characterized in that The query service unit executes a process of acquiring a job link of the target job constructed using the job link data model in the graph database, including: For each job node of the job, based on the node relationship data of the job, determine the relationship node of the job node, and establish a relationship edge between the job node and its relationship node according to the job link data model to obtain the link structure of the job; Based on the link structure of all jobs, obtain the full amount of job link data; The job link of the target job is obtained from the full link data.

4. The system according to claim 1, characterized in that The monitoring service unit performs a process of determining whether there is a job that meets a preset alarm condition, including: Obtaining execution status data of each of the jobs; Determining the execution status of each of the jobs based on each of the execution status data; For each of the jobs, when the execution status of the job matches the alarm status in the alarm condition, it is determined that the job meets the alarm condition.

5. The system according to claim 1, characterized in that The query service unit performs a process of locating an abnormal job corresponding to the target job based on the current status data of each job and the job link of the target job, including: Based on the current status data of each of the jobs, determining the current status of each node in the job link of the target job; Determine the job node of the target job in the job link of the target job as the target node, and determine each node of the job link of the target job as a to-be-determined node; Determine an abnormal node among the nodes to be determined, wherein the abnormal node is a node to be determined that is located upstream or downstream of the target node, is closest to the target node, and has a current state characterized as abnormal; The job corresponding to the abnormal node is determined as an abnormal job.

6. A method for locating abnormal operations, characterized in that: include: Monitor the working status of each job in the job control system in real time to determine whether there is a job that meets the preset alarm conditions. When there is a job that meets the preset alarm conditions, determine the job as the target job and generate an alarm signal for the target job; Based on the alarm signal, construct the job link of the target job using a pre-constructed job link data model; The current status data of each of the jobs is acquired, and based on the current status data of each of the jobs and the job link of the target job, the abnormal job corresponding to the target job is located.

7. The method according to claim 6, characterized in that The determining whether there is a job that meets a preset alarm condition includes: Obtaining execution status data of each of the jobs; Determining the execution status of each of the jobs based on each of the execution status data; For each of the jobs, when the execution status of the job matches the alarm status in the alarm condition, it is determined that the job meets the alarm condition.

8. The method according to claim 6, characterized in that The using a pre-built job link data model to build the job link of the target job includes: Acquire configuration data of each of the jobs, and extract each node and node relationship data of each of the jobs from each of the configuration data, wherein each node of each of the jobs includes a job node and an output node; For each job node of the job, based on the node relationship data of the job, determine the relationship node of the job node, and establish a relationship edge between the job node and its relationship node according to the job link data model to obtain the link structure of the job; Based on the link structure of all jobs, obtain the full amount of job link data; The job link of the target job is obtained from the full link data.

9. The method according to claim 8, characterized in that The step of extracting each node and node relationship data of each job from each configuration data includes: Acquire job flow information, job name information and output information of each job from configuration data of each job; For each of the jobs, the job name information of the job is applied to construct a job node, and the output information of the job is applied to construct an output node, and the first association information of the job node and the second association information of the output node are generated, and node relationship data is generated based on the job flow information, the first association information and the second association information.

10. The method according to claim 6, characterized in that The locating the abnormal job corresponding to the target job based on the current status data of each job and the job link of the target job includes: Based on the current status data of each of the jobs, determining the current status of each node in the job link of the target job; Determine the job node of the target job in the job link of the target job as the target node, and determine each node of the job link of the target job as a to-be-determined node; Determine an abnormal node among the nodes to be determined, wherein the abnormal node is a node to be determined that is located upstream or downstream of the target node, is closest to the target node, and has a current state characterized as abnormal; The job corresponding to the abnormal node is determined as an abnormal job.

Images