Batch job detection method and device based on directed acyclic graph, and electronic equipment

Abstract

The invention discloses a batch job detection method and device based on directed acyclic graphs and electronic equipment, and relates to the field of financial science and technology, and the method comprises the steps: detecting a corresponding first type directed acyclic graph before the execution of a batch job and a second type directed acyclic graph after the execution of the batch job, nodes in the directed acyclic graphs represent each batch job, connection edges in the directed acyclic graphs represent job dependency relationships, the nodes store job execution information of the corresponding batch jobs, node sets, edge sets and node access degree information between the first type of directed acyclic graphs and the second type of directed acyclic graphs are compared to obtain a comparison result, and the comparison result is sent to the second type of directed acyclic graphs. And outputting execution information of the batch jobs and abnormal job information based on the comparison result. According to the method and the device, the technical problem that the business of the financial institution is abnormal due to the fact that the efficiency is low and the abnormal operation condition is easy to omit in a scene with a large quantity of batch operations and a complex dependency relationship among the operations in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of financial technology or other related fields, and in particular, to a batch job detection method and device based on a directed acyclic graph, and an electronic device. Background Art

[0002] In the information systems of financial institutions, batch jobs are a key component for processing large-scale data and executing complex business logic. In order to ensure the efficient and accurate operation of batch jobs, especially when the job volume is large and the dependencies are complex, monitoring and detecting the execution status of batch jobs has become a challenge. Traditional batch job management methods rely on manual monitoring and post-checking, which is not only inefficient but also easy to miss abnormal situations in scenarios with a large number of jobs and complex dependencies between jobs.

[0003] In the field of automated detection of batch jobs, there are currently a variety of graph-based comparison methods that are used to detect differences before and after job execution to identify possible anomalies, such as methods based on edit distance and information theory. However, although the edit distance-based method is simple to calculate, it may ignore key features when dealing with complex dependency structures; and although the information theory-based method can capture statistical features, it relies on accurate probability distribution and has a high computational complexity.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a batch job detection method and device, and an electronic device based on a directed acyclic graph, so as to at least solve the technical problems in the related technology that, in the scenario where the number of batch jobs is huge and there are complex dependencies between jobs, the efficiency is low, abnormal operation conditions are easily missed, and abnormalities occur in the business of financial institutions.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a batch job detection method based on a directed acyclic graph is provided, including: detecting a first type of directed acyclic graph corresponding to before the batch job is executed and a second type of directed acyclic graph after the batch job is executed, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependencies, and the nodes store the job execution information of the corresponding batch job; comparing the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result; based on the comparison result, outputting the execution information and abnormal job information of the batch job.

[0007] Optionally, the step of detecting the first type of directed acyclic graph corresponding to the batch job before the batch job is executed and the second type of directed acyclic graph after the batch job is executed includes: before the batch job is executed, detecting the basic job information of each job in the batch job, and constructing a node based on the basic job information, wherein the basic job information includes: job name, group name of the group to which the job belongs, job type, job trigger time, and job function description information; before the job is executed, detecting the dependency relationship between the jobs in the batch job, taking the dependent node in the dependency relationship as the end point of the edge, taking the dependent node in the dependency relationship as the starting point of the edge, and establishing directed connection edges between the nodes; constructing the first type of directed acyclic graph based on the nodes and the directed connection edges; during the execution of the batch job, detecting the running status and job execution result of each job in the batch job, and recording the running status and the job execution result in the auxiliary information of the node; after the execution of the batch job is completed, constructing the second type of directed acyclic graph based on the auxiliary information of the node and the new directed connection edges established during the batch job operation.

[0008] Optionally, the step of comparing the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result includes: comparing whether the node set corresponding to the first type of directed acyclic graph and the node set corresponding to the second type of directed acyclic graph completely match; if the two node sets do not completely match, confirming that the batch job execution is abnormal and outputting the abnormal comparison result of the node; if the two node sets completely match, outputting the normal comparison result of the node.

[0009] Optionally, the step of comparing the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result includes: comparing whether the edge set corresponding to the first type of directed acyclic graph and the edge set corresponding to the second type of directed acyclic graph completely match; in the case where it is detected that any directed connection edge only exists in one of the edge sets, or the weights of the directed connection edges of the two types of directed acyclic graphs are different, confirming that the batch job execution is abnormal, and outputting the abnormal comparison result of the connection edge, wherein the weight of the connection edge is determined based on the job importance level and the job execution result during the job execution process; in the case where the directed connection edges in the two edge sets completely match, outputting the normal comparison result of the connection edge.

[0010] Optionally, the step of comparing the node set, edge set and node in-degree and out-degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result includes: comparing whether the in-degree and out-degree of each node in the first type of directed acyclic graph and the second type of directed acyclic graph are equal; if there is any node with different in-degree or out-degree in the two types of directed acyclic graphs, confirming that the batch job execution is abnormal, and outputting the abnormal comparison result of the node in-degree and out-degree; if the in-degree or out-degree in the two types of directed acyclic graphs are the same, outputting the normal comparison result of the node in-degree and out-degree.

[0011] Optionally, the batch job detection method based on directed acyclic graphs also includes: executing a topological sorting algorithm on the first type of directed acyclic graph and the second type of directed acyclic graph, respectively, to obtain a first topological sorting result and a second topological sorting result; when the first topological sorting result and the second topological sorting result are different, confirming that the batch job execution is abnormal, and outputting an abnormal comparison result of the graph topological sorting; when the first topological sorting result and the second topological sorting result are the same, outputting a normal comparison result of the graph topological sorting.

[0012] Optionally, the topological sorting algorithm adopts one of the following: a hash algorithm, a depth-first search algorithm or a breadth-first search algorithm, and detects whether the two types of directed acyclic graphs have consistent structures during the topological sorting process, and determines the execution status of the batch job based on the detection result.

[0013] According to another aspect of an embodiment of the present invention, a batch job detection device based on a directed acyclic graph is also provided, including: a graph detection unit, used to detect a first type of directed acyclic graph corresponding to the batch job before execution and a second type of directed acyclic graph after execution of the batch job, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependencies, and the nodes store the job execution information of the corresponding batch job; a graph comparison unit, used to compare the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result; a job execution result output unit, used to output the execution information and abnormal job information of the batch job based on the comparison result.

[0014] Optionally, the graph detection unit includes: a first job detection module, which is used to detect the basic job information of each job in the batch job before the batch job is executed, and construct a node based on the basic job information, wherein the basic job information includes: job name, group name of the group to which the job belongs, job type, job trigger time, and job function description information; a second job detection module, which is used to detect the dependency relationship between the jobs in the batch job before the job is executed, and use the dependent node in the dependency relationship as the end point of the edge and the dependent node in the dependency relationship as the starting point of the edge to establish a directed connection edge between the nodes; a first graph construction module, which is used to construct the first type of directed acyclic graph based on the nodes and the directed connection edges; a third job detection module, which is used to detect the running status and job execution result of each job in the batch job during the execution of the batch job, and record the running status and the job execution result in the auxiliary information of the node; a second graph construction module, which is used to construct the second type of directed acyclic graph based on the auxiliary information of the node and the new directed connection edges established during the batch job execution after the batch job is executed.

[0015] Optionally, the graph comparison unit includes: a node comparison module, used to compare whether the node set corresponding to the first type of directed acyclic graph and the node set corresponding to the second type of directed acyclic graph are completely matched; a first result output module, used to confirm that the batch job execution is abnormal and output the abnormal comparison result of the node when the two node sets are not completely matched; and a second result output module, used to output the normal comparison result of the node when the two node sets are completely matched.

[0016] Optionally, the graph comparison unit also includes: an edge comparison module, used to compare whether the edge set corresponding to the first type of directed acyclic graph is completely matched with the edge set corresponding to the second type of directed acyclic graph; a third result output module, used to confirm the abnormality of batch job execution when it is detected that any directed connection edge only exists in one of the edge sets, or when the weights of the directed connection edges of the two types of directed acyclic graphs are different, and output the abnormal comparison result of the connection edge, wherein the weight of the connection edge is determined based on the job importance level and the job execution result during the job execution process; a fourth result output module, used to output the normal comparison result of the connection edge when the directed connection edges in the two edge sets are completely matched.

[0017] Optionally, the graph comparison unit also includes: an in-degree comparison module, used to compare whether the in-degree and out-degree of each node in the first type of directed acyclic graph and the second type of directed acyclic graph are equal; a fifth result output module, used to confirm that the batch job execution is abnormal when there is a node with different in-degree or out-degree in the two types of directed acyclic graphs, and output the abnormal comparison result of the node in-degree; a sixth result output module, used to output the normal comparison result of the node in-degree when the in-degree or out-degree in the two types of directed acyclic graphs are the same.

[0018] Optionally, the batch job detection device based on directed acyclic graphs also includes: a topological sorting unit, used to execute a topological sorting algorithm on the first type of directed acyclic graph and the second type of directed acyclic graph, respectively, to obtain a first topological sorting result and a second topological sorting result; a seventh result output module, used to confirm that the batch job execution is abnormal when the first topological sorting result and the second topological sorting result are different, and output an abnormal comparison result of the graph topological sorting; an eighth result output module, used to output a normal comparison result of the graph topological sorting when the first topological sorting result and the second topological sorting result are the same.

[0019] Optionally, the topological sorting algorithm adopts one of the following: a hash algorithm, a depth-first search algorithm or a breadth-first search algorithm, and detects whether the two types of directed acyclic graphs have consistent structures during the topological sorting process, and determines the execution status of the batch job based on the detection result.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned directed acyclic graph-based batch job detection methods.

[0021] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned directed acyclic graph-based batch job detection methods.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned directed acyclic graph-based batch job detection methods are implemented.

[0023] In the present disclosure, a first type of directed acyclic graph corresponding to a batch job before execution and a second type of directed acyclic graph after execution of the batch job are detected, and the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph are compared to obtain a comparison result. Based on the comparison result, the execution information of the batch job and the abnormal job information are output, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependency, and the nodes store the job execution information of the corresponding batch job.

[0024] Based on the above-mentioned public content, the directed acyclic graph structure before and after the execution of batch jobs can be comprehensively compared, including the node set, edge set, and node in-and-out degree. This enables timely discovery and location of abnormal situations in the execution of batch jobs, thereby improving the efficiency of exception handling. At the same time, it realizes the automation of batch job execution status detection, improves the accuracy and reliability of job monitoring, and thereby solves the technical problems in the related technologies of low efficiency and easy omission of abnormal operation situations, which lead to abnormal business of financial institutions in scenarios where the number of batch jobs is large and there are complex dependencies between jobs. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a batch job detection method based on a directed acyclic graph is shown;

[0027] Figure 2 is a flow chart of an optional directed acyclic graph-based batch job detection method according to an embodiment of the present invention;

[0028] Figure 3 is a schematic diagram of an optional directed acyclic graph-based batch job detection device according to an embodiment of the present invention;

[0029] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] To facilitate those skilled in the art to understand the present invention, some terms or nouns involved in the embodiments of the present invention are explained below:

[0033] Directed Acyclic Graph, referred to as DAG, nodes are connected by directed edges to form a series of paths. In the present invention, DAG is used to represent the dependency relationship between batch jobs, where: a node represents a specific batch job (which can be a data processing task, a report generation task, an account settlement task, etc.), and a directed connection edge represents the dependency relationship between two jobs, that is, one job must be executed after the other job is completed, which reflects the execution order and conditional constraints of the jobs. The hierarchical layout of DAG helps to reduce the intersection between edges, making the visualization of the dependency relationship clearer, which is convenient for developers and system administrators to understand the execution process and logic of the job, especially in complex batch job management.

[0034] Hash algorithm, a data processing method, can convert an input of arbitrary length (called a pre-map or message) into an output of fixed length (called a hash or message digest). In the present invention, the hash algorithm can be used to quickly compare the structure or characteristics of two DAGs, such as node sets, edge sets, and topological sorting results. By calculating the hash value of each graph and then comparing these values, it is possible to quickly determine whether the two graphs are the same or in what aspects there are differences. This helps to improve detection efficiency and reduce unnecessary complex calculations.

[0035] Depth-First Search, or DFS for short, is an algorithm for traversing or searching a tree or graph. During the search, DFS explores the branch paths of the graph as deeply as possible, that is, it goes deep along a path until it can no longer go further, then backtracks to the previous branch point and continues to explore other branch paths. In the scenario of batch job detection in the present invention, DFS can be used to verify whether the job execution follows the expected dependencies, that is, whether the jobs are executed in the correct order, and whether there are any unnecessary dependencies or circular dependencies.

[0036] Breadth-First Search, or BFS for short, is an algorithm for traversing or searching a tree or graph. Unlike depth-first search, BFS starts from the root node and traverses all nodes layer by layer, that is, first visits all nodes directly connected to the root node, then visits all nodes directly connected to the first-layer nodes, and so on. In the batch job detection scenario of the present invention, BFS can be used to check whether the hierarchical structure of the DAG remains unchanged, and to verify whether the dependencies between jobs are consistent with the preset structure, which is crucial to ensure the correct execution of jobs and timely detection of problems.

[0037] It should be noted that the batch job detection method based on directed acyclic graph and the device thereof in the present disclosure can be used in the field of financial technology. When the execution results of batch jobs and job dependencies are quickly detected based on the directed acyclic graph, it can also be used in any field other than the field of financial technology. When the execution results of batch jobs and job dependencies are quickly detected based on the directed acyclic graph, the present disclosure does not limit the application field of the batch job detection method based on directed acyclic graph and the device thereof.

[0038] It should be noted that the information (including but not limited to operation information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0039] It should be noted that in the present disclosure, when collecting and analyzing customer information, corresponding operation entrances are provided for users to choose to agree or reject the automated decision results; if the user chooses to reject, the expert decision-making process is entered.

[0040] The following embodiments of the present invention can be applied to various systems / applications / devices for batch job detection based on directed acyclic graphs. Compared with the existing batch job detection strategies, when the number of batch jobs is large and the dependencies are complex, it is impossible to accurately detect abnormal situations in the job execution process, and abnormal jobs are prone to omission. The present invention provides a solution specifically for batch job scenarios of financial institutions, which implements batch job detection based on directed acyclic graphs and can comprehensively compare the directed acyclic graph structure before and after the batch job execution, including node sets, edge sets, node in-and-out degrees, and topological sorting results, so as to more accurately and efficiently detect abnormal execution of batch jobs, locate erroneous jobs, and improve the automation level and accuracy of job monitoring and management.

[0041] At the same time, the present invention can construct a first type of directed acyclic graph based on the basic information of the job and the dependency relationship between the jobs before the batch job is executed, and after the batch job is executed, it can construct a second type of directed acyclic graph based on the execution status information, execution results and possible updated dependencies of the job, and then compare the node set, edge set, node in-and-out degree and topological sorting to ensure that there is no omission or addition of the job, and confirm that the dependency relationship between the jobs has not changed, including checking the existence and weight of the edge, ensuring that the execution order and dependency relationship of the job are correct, verifying whether the linear order of the job execution is consistent with expectations, outputting the execution status information of the batch job and the information of the abnormal job, and assisting developers to quickly locate the problem. The need for manual intervention is reduced, the automation of batch job execution status detection is realized, the accuracy and reliability of job monitoring are improved, and it can be applied to the processing of large-scale batch jobs. Even if the number of jobs and the complexity of dependency relationships increase, efficient detection capabilities can be maintained.

[0042] The present invention is described in detail below in conjunction with various embodiments.

[0043] Embodiment 1

[0044] According to an embodiment of the present invention, an embodiment of a batch job detection method based on a directed acyclic graph is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] The batch job detection method embodiment based on a directed acyclic graph provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a batch job detection method based on a directed acyclic graph. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 , 102n are used to illustrate) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU (Microcontroller Unit) or a programmable logic device FPGA (Field Programmable Gate Array)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0046] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0047] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the batch job detection method based on a directed acyclic graph in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the batch job detection method based on a directed acyclic graph is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0050] It should be noted that this embodiment can be applied to the IT system of a financial institution. A large number of batch jobs are an indispensable part of daily operations, involving multiple business links from data processing, report generation, risk assessment to account settlement. Before the batch job is executed, a directed acyclic graph (DAG) is constructed according to the business logic and dependency relationship of the job. The nodes in the graph represent each job, and the connecting edges represent the dependency relationship between the jobs. When the batch job is executed, the technical solution of the present invention will generate a new DAG, which not only reflects the dependency relationship of the original job, but also contains the status information after the job is executed, such as execution time, execution result, job status (success / failure), etc. The embodiment of the present invention can quickly compare the differences between the job DAG before execution and the job DAG after execution, including the node set, edge set, node in-and-out degree and topological sorting result, which can help confirm whether the job has been executed correctly, and can also locate which jobs have abnormalities during execution, such as job failure, wrong execution order or dependency change.

[0051] In addition, if during the execution of a batch job, the dependencies of the jobs change due to some reasons (such as system upgrades, job optimization), the technical solution provided by the embodiment of the present invention can detect these changes and report them to the system administrator. This helps to adjust the job scheduling strategy in a timely manner to avoid potential execution conflicts or data inconsistencies. By updating the DAG regularly or in real time and using the comparison method of the present invention, the execution efficiency and resource consumption of batch jobs can be continuously monitored. System administrators can optimize the execution order of jobs and reasonably allocate system resources based on the analysis results of the DAG, thereby improving the overall job processing capability and system response speed.

[0052] Under the above operating environment, this application provides Figure 2 The batch job detection method based on directed acyclic graph is shown. Figure 2 is a flow chart of an optional directed acyclic graph-based batch job detection method according to an embodiment of the present invention, such as Figure 2 As shown, the method comprises the following steps:

[0053] Step S201, detect the first type of directed acyclic graph corresponding to the batch job before execution and the second type of directed acyclic graph after the batch job is executed, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependency, and the nodes store the job execution information of the corresponding batch job.

[0054] Optionally, step S201 includes: before the batch job is executed, detecting the basic job information of each job in the batch job, and constructing a node based on the basic job information, wherein the basic job information includes: job name, group name of the group to which the job belongs, job type, job trigger time, and job function description information; before the job is executed, detecting the dependency relationship between the jobs in the batch job, taking the dependent node in the dependency relationship as the end point of the edge, taking the dependent node in the dependency relationship as the starting point of the edge, and establishing directed connection edges between the nodes; constructing a first type of directed acyclic graph based on the nodes and the directed connection edges; during the execution of the batch job, detecting the running status and job execution results of each job in the batch job, and recording the running status and job execution results in the auxiliary information of the node; after the batch job is executed, constructing a second type of directed acyclic graph based on the auxiliary information of the node and the new directed connection edges established during the batch job operation.

[0055] Before executing a batch job, the system needs to collect basic information about each job. The job name is used to uniquely identify each job, which is convenient for subsequent query and tracking. The group name of the job group is used to determine the context and priority of the job. Jobs are usually grouped according to functions or business areas. The group name helps to understand the context and priority of the job. The job type refers to whether the job belongs to data processing, report generation, risk assessment or other types. The job type information helps to classify nodes in the DAG, which is convenient for subsequent analysis and optimization. The job trigger time is used to determine the time point when the job is expected to start execution, and is used to schedule and monitor whether the job starts on time. The job function description information is used to describe the specific function and goal of the job, which is crucial for understanding the business logic of the job and verifying the execution results. This basic information is used to create nodes in the DAG. Each node represents a specific batch job. In addition, the system also needs to detect the dependency relationship between jobs, that is, the execution of one job depends on the completion of one or more jobs. The dependency relationship is represented by establishing directed edges in the DAG. The direction of the edge points from the dependent job to the dependent job, which ensures that the execution order of the jobs conforms to the business logic.

[0056] During the execution of batch jobs, the system will continuously monitor the running status and execution results of each job. The running status can be "running", "paused", "abnormal interruption", "waiting for resources", "success", "failure", etc. The execution results include the output data, error information, execution time, etc. of the job. This information is used as auxiliary information of the node to update the node status in the DAG to reflect the actual execution status of the job.

[0057] When all batch jobs are executed, the system rebuilds a directed acyclic graph (i.e., the second type of directed acyclic graph) DAG based on the collected job execution information and possible new or modified dependencies. At this time, the DAG not only contains the initial nodes and edges, but also reflects the actual status of job execution. For example, some jobs may fail, resulting in the inability to continue to execute dependencies, or some jobs take too long to execute, affecting the scheduling of subsequent jobs. Therefore, the second type of directed acyclic graph can provide a more realistic and detailed view of job execution, which is convenient for subsequent comparison and anomaly detection.

[0058] Before batch jobs are executed, the system reads basic information about each job from the database or configuration file, including the job name, group name, type, trigger time, and function description. This information is not only used to create nodes in the DAG, but is also stored inside the node as a node attribute for subsequent query and analysis.

[0059] It should be noted that when establishing directed connection edges based on dependencies, the system can analyze the dependencies between jobs, which may come from business rules, system configurations, or historical execution data. Each dependency is represented as a directed edge, from the node of the dependent job to the node of the dependent job, ensuring that the DAG can accurately reflect the execution order and conditional constraints of the jobs.

[0060] It should also be noted that when the DAG is updated according to the job execution information to obtain the second type of directed acyclic graph, the system collects the running status and execution results of the job in real time during the job execution. Once the job is executed, its status and result information will be recorded in the auxiliary information of the node. If new dependencies are added during the job execution (for example, a new data verification job is added due to data quality issues), or some dependencies are removed due to job optimization, these changes will be reflected in the second type of directed acyclic graph to ensure the dynamic and accuracy of the DAG.

[0061] Through the above steps, the embodiment of the present invention can build a DAG model that fully reflects the execution of batch jobs, providing a solid foundation for subsequent comparison and anomaly detection. This not only helps to timely discover and locate problems in job execution, but also provides data support for job scheduling and optimization strategies, ensuring the efficient and stable operation of the IT system of financial institutions.

[0062] Step S202 , comparing the node set, edge set, and node in-degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result.

[0063] It should be noted that in the process of comparing DAG graphs, the node sets can be compared first to ensure that there are no omissions or additions to the job. Optionally, the step of comparing the node set, edge set, and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain the comparison result includes: comparing whether the node set corresponding to the first type of directed acyclic graph and the node set corresponding to the second type of directed acyclic graph completely match; if the two node sets do not completely match, confirming that the batch job execution is abnormal, and outputting the abnormal comparison result of the node; if the two node sets completely match, outputting the normal comparison result of the node.

[0064] Compare whether the node sets in the first type of directed acyclic graph (DAG before execution) and the second type of directed acyclic graph (DAG after execution) are completely matched. This is to confirm whether all scheduled jobs are executed and whether any unplanned jobs are triggered. If the two node sets do not completely match, that is, there are unexpected node additions or expected node deletions, this indicates that there are abnormalities in the batch job execution process, such as job skipping, job addition, etc. In this case, the system will confirm the batch job execution abnormality and output the abnormal comparison results of the nodes, identifying which jobs were not executed correctly or additional jobs were executed. If the two node sets completely match, that is, all jobs are executed as expected, the system will output the normal comparison results of the nodes.

[0065] Further compare the edge sets to confirm that the dependencies between jobs have not changed, including checking the existence and weight of the edges to ensure that the execution order and dependencies of the jobs are correct. Optionally, compare the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain the comparison result, including: comparing the edge set corresponding to the first type of directed acyclic graph and the edge set corresponding to the second type of directed acyclic graph to see if they completely match; in the case where it is detected that any directed connection edge exists only in one of the edge sets, or the weights of the directed connection edges of the two types of directed acyclic graphs are different, confirm that the batch job execution is abnormal, and output the abnormal comparison result of the connection edge, wherein the weight of the connection edge is determined based on the job importance level and the job execution result during the job execution; in the case where the directed connection edges in the two edge sets completely match, output the normal comparison result of the connection edge.

[0066] The comparison of edge sets is used to verify whether the dependencies between jobs are correctly maintained. It compares whether the edge sets in the first type of directed acyclic graph and the second type of directed acyclic graph are completely matched, that is, to check whether unexpected dependencies are established or expected dependencies are not followed. If any edge is found to exist only in one of the edge sets, or the weights of the edges in the two types of DAGs (such as the values ​​calculated based on the job importance level and the execution results) are different, this indicates that the dependencies have changed during the job execution process. There may be errors in the job order, changes in dependencies, or job execution results that affect subsequent dependencies. At this time, the system confirms that the batch job execution is abnormal and outputs the abnormal comparison results of the connection edges, detailing which dependencies are not executed correctly. If the directed connection edges in the two edge sets are completely matched, the system will output the normal comparison results of the connection edges, indicating that the dependencies are correctly followed.

[0067] Further compare the node in-degrees and out-degrees, optionally, compare the node set, edge set and node in-degree and out-degree information between the first type of directed acyclic graph and the second type of directed acyclic graph, and obtain the comparison result, including: comparing whether the in-degree and out-degree of each node in the first type of directed acyclic graph and the second type of directed acyclic graph are equal; in the case that the in-degree or out-degree of any node in the two types of directed acyclic graphs is different, confirm that the batch job execution is abnormal, and output the abnormal comparison result of the node in-degree; in the case that the in-degree or out-degree in the two types of directed acyclic graphs are the same, output the normal comparison result of the node in-degree.

[0068] The in-degree of a node means the number of edges pointing to the node (in-degree) and the number of edges departing from the node (out-degree). By comparing the in-degree of each node in the two types of DAGs, it is possible to check whether the job execution follows the predetermined dependency and execution order. If there is any node with different in-degree or out-degree in the two types of DAGs, this may mean that the job execution order is wrong, the dependency has changed, or the job has failed, resulting in the failure of subsequent jobs to execute. At this time, the system confirms that the batch job execution is abnormal, and outputs the abnormal comparison results of the node in-degree, identifies the nodes with changed out-degree or in-degree, and helps locate the problem. If the in-degree of all nodes in the two types of DAGs is the same, the system will output the normal comparison results of the node in-degree, indicating that all jobs are completed according to the expected dependencies.

[0069] Optionally, the batch job detection method based on directed acyclic graphs also includes: executing a topological sorting algorithm on the first type of directed acyclic graph and the second type of directed acyclic graph, respectively, to obtain a first topological sorting result and a second topological sorting result; when the first topological sorting result and the second topological sorting result are different, confirming that the batch job execution is abnormal, and outputting an abnormal comparison result of the graph topological sorting; when the first topological sorting result and the second topological sorting result are the same, outputting a normal comparison result of the graph topological sorting.

[0070] Topological sorting is a method of sorting nodes in a directed acyclic graph so that for any pair of nodes, as long as there is a path from the first node to the second node, the first node always appears before the second node in the sort. By executing a topological sorting algorithm (such as depth-first search or breadth-first search) on the two types of DAGs respectively, the system obtains a first topological sorting result and a second topological sorting result. Comparing these two results can check whether the job execution follows the correct order. If the first topological sorting result and the second topological sorting result are different, this usually means that the job execution order does not match the predetermined order. The system will confirm that the batch job execution is abnormal and output the abnormal comparison result of the graph topological sorting to help identify jobs with incorrect execution order. If the two topological sorting results are the same, the system will output a normal comparison result of the graph topological sorting, indicating that the job execution order is consistent with the predetermined order and there is no abnormality.

[0071] Optionally, the topological sorting algorithm adopts one of the following: a hash algorithm, a depth-first search algorithm or a breadth-first search algorithm. During the topological sorting process, it is detected whether the two types of directed acyclic graphs have consistent structures, and the execution status of the batch job is determined based on the detection result.

[0072] Specifically, in the batch job detection process, a topological sorting algorithm is used to verify whether the job execution order is consistent with the predetermined dependency relationship between jobs. Among them, this embodiment selects three algorithms: hash algorithm, depth-first search algorithm (DFS) and breadth-first search algorithm (BFS). Depth-first search is a graph traversal algorithm. In the scenario of batch jobs, DFS can be used as part of topological sorting to sequentially access each job and check whether the dependency relationship between jobs is correctly followed. The application of DFS in topological sorting can not only help determine the execution order of jobs, but also check the out-degree and in-degree of each node when accessing it, and immediately find possible circular dependencies or dependency anomalies. The use of breadth-first search in the topological sorting process, in addition to being able to generate the correct job execution order, can also compare the status of nodes in the DAG before and after execution when traversing each level, and immediately find missing or additional jobs, as well as abnormal changes in job status. This feature of BFS makes it an effective tool for detecting the consistency of DAG structure, ensuring that batch jobs are executed in a predetermined order and dependency relationship.

[0073] In the scenario of batch jobs, the main goal of the topological sorting algorithm (using DFS or BFS) is to generate a linear sequence that reflects the job dependencies and execution order. During the topological sorting process, the system can detect the structural consistency of the two types of DAGs in real time. The specific methods are as follows: Node status comparison: During the sorting process, check the status information of each node before and after execution, such as the running results, execution time, status code, etc. of the job to determine whether the job is completed in the predetermined order. Edge consistency verification: During the DFS or BFS traversal process, check the existence and weight of the edges to verify whether the job dependencies are correctly followed and whether all expected dependency paths are correctly executed. Cyclic dependency detection: The core of topological sorting is to ensure that there are no cyclic dependencies in the graph. If a cyclic path is detected by DFS or BFS during the sorting process, it indicates that there is a serious error in the execution of the batch job and immediate intervention is required.

[0074] Based on the above detection, the system can evaluate the execution status of the batch job: Normal state: If the topological sorting results of the two types of DAGs are exactly the same, it indicates that the batch job is executed normally and all jobs are completed according to the predetermined dependencies and order. Abnormal state: If structural inconsistency is detected, such as node status mismatch, edge information abnormality, or circular dependency, the system will confirm that the batch job execution is abnormal and output detailed abnormal job information, including the type of abnormality, the name and location of the abnormal job, and possible cause analysis, to help quickly locate and solve the problem.

[0075] Through comprehensive comparison of the above steps, the embodiment of the present invention can comprehensively evaluate the normality of batch job execution, not only detecting the information of nodes and edges, but also going deep into the details of job execution order and dependencies, ensuring the accuracy and reliability of the detection results, which is crucial to maintaining the execution efficiency and data consistency of batch jobs of financial institutions, and helps to promptly discover and correct errors in execution, avoiding business process interruptions or data processing anomalies.

[0076] Step S203: outputting batch job execution information and abnormal job information based on the comparison result.

[0077] In the process of batch job detection, the job execution status and exceptions are identified by comparing the first type of directed acyclic graph (graph before execution) with the second type of directed acyclic graph (graph after execution). The comparison results include the difference information of the two graphs in terms of node set, edge set and node in-degree and out-degree. Based on the comparison results, the system can generate detailed batch job execution information and abnormal job information. The execution information provides the overall situation of job execution, including but not limited to: Execution overview: List all jobs executed as planned, confirm the matching of node sets in the DAG before and after execution, and indicate the integrity of the job. Dependency verification: Confirm the matching of edge sets in the DAG before and after execution, and show whether the dependencies between jobs are correctly followed. Node in-degree comparison: Show whether the in-degree of each job is consistent before and after execution, and confirm whether the job execution order and dependency are as expected, which helps system administrators and developers understand the execution efficiency and resource usage of batch jobs and evaluate the effectiveness of job scheduling strategies.

[0078] When the comparison results reveal differences between the DAGs before and after execution, the system will output abnormal job information to quickly locate and fix the problem. For example, abnormal information may include missing jobs: listing nodes that could not be found in the DAG after execution, which means that the corresponding job may not be triggered or failed to execute. Exception information may also include additional jobs: indicating newly added nodes in the DAG after execution, which may be due to the addition of unplanned dependencies between jobs or repeated execution of jobs; abnormal information may also include dependency exceptions: identifying unmatched edges in the DAG before and after execution, indicating that the dependency between jobs has changed, such as missing dependencies or new dependencies; abnormal information may also include execution order errors: indicating inconsistent jobs in the topological sorting results, indicating that the job execution order does not match the predetermined order; abnormal information may also include node in-degree changes: listing jobs with inconsistent in-degrees, which may indicate errors in execution order or dependencies, or changes in job status. The output of abnormal job information not only includes basic information such as the job name, type, and group to which it belongs, but may also include detailed exception type, time point when the exception occurred, execution status of the abnormal job (such as failure, timeout, not started), and possible cause analysis, providing data support for subsequent troubleshooting and system optimization.

[0079] Assume that a financial institution needs to execute a set of batch jobs for account balance update and transaction review every night. Before the job is executed, the DAG shows that all jobs are arranged in order, without cross-dependencies, and the status of each job is marked as "to be executed". After the job is executed, the job status will be updated to "success" or "failure" in the generated DAG, and any newly added dependencies (such as temporary data verification jobs that need to be added due to data quality issues) will also be recorded. Developers use the technical solution of this embodiment to compare the DAG before and after execution and find that the execution order of the "account balance update" job has changed from the "transaction review" job. Originally, the "account balance update" should be executed before the "transaction review" job, but in the latest DAG, the order is reversed. This abnormal situation is immediately detected and reported by the system, and developers can quickly locate the problem, adjust the job order, and avoid data processing errors. Through the application of this embodiment, financial institutions can effectively monitor and manage complex batch job processes, ensure business continuity and data accuracy, and improve the overall operating efficiency of the system.

[0080] Through the above steps, the first type of directed acyclic graph corresponding to the batch job before execution and the second type of directed acyclic graph after the batch job is executed can be detected, and the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph can be compared to obtain a comparison result. Based on the comparison result, the execution information of the batch job and the abnormal job information are output, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependency, and the nodes store the job execution information of the corresponding batch job. In this embodiment, the directed acyclic graph structure before and after the batch job execution can be comprehensively compared, including the node set, edge set, and node in-and-out degree, so that the abnormal situation in the batch job execution can be discovered and located in time, and the efficiency of abnormal handling is improved. At the same time, the automation of batch job execution status detection is realized, and the accuracy and reliability of job monitoring are improved, thereby solving the technical problems in the related technology that the number of batch jobs is large and there are complex dependencies between jobs, the efficiency is low, and abnormal operation situations are easily missed, resulting in abnormal business of financial institutions.

[0081] In order to ensure the accuracy and readability of the output execution information and abnormal job information, the system may provide real-time feedback. Once an abnormality is found, the abnormal information is output immediately without waiting for all jobs to be executed, which helps to respond and handle problems quickly. The system can also implement abnormality classification. According to the severity and urgency of the abnormality, abnormal jobs are classified into different levels. For example, job failures are marked as the highest priority, while slight deviations in the order of jobs may be marked as lower priorities, which is convenient for administrators to prioritize critical abnormalities. The system can also output automatic repair suggestions: Based on the type of abnormality and job characteristics, the system can provide automatic repair suggestions, such as rescheduling jobs, adjusting dependencies, optimizing job execution logic, etc. The system can also output visual reports: The output information can be presented in a graphical way, such as using color coding or icons to indicate job status and abnormality types, so as to intuitively understand the overall execution of batch jobs.

[0082] Based on the output of the comparison results, it not only provides real-time feedback on the execution of batch jobs, but also goes deep into the specific diagnosis of abnormal jobs, providing comprehensive and accurate job management information for IT system administrators and developers of financial institutions. In this way, financial institutions can effectively monitor the execution of batch jobs, promptly discover and solve problems in the execution process, and ensure business continuity and data accuracy.

[0083] The following is a detailed description in conjunction with another embodiment.

[0084] Embodiment 2

[0085] A batch job detection device based on a directed acyclic graph provided in this embodiment includes multiple implementation units, each of which corresponds to each implementation step in the above-mentioned embodiment 1. Its specific implementation methods and beneficial effects can refer to the above-mentioned method embodiments and will not be repeated here.

[0086] Figure 3 is a schematic diagram of an optional directed acyclic graph-based batch job detection device according to an embodiment of the present invention, such as Figure 3 As shown, the batch job detection device based on directed acyclic graph may include: a graph detection unit 31, a graph comparison unit 32, and a job execution result output unit 33.

[0087] Among them, the graph detection unit 31 is used to detect the first type of directed acyclic graph corresponding to the batch job before execution and the second type of directed acyclic graph after the batch job is executed, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependency, and the nodes store the job execution information of the corresponding batch job.

[0088] The graph comparison unit 32 is used to compare the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result.

[0089] The job execution result output unit 33 is used to output the batch job execution information and abnormal job information based on the comparison result.

[0090] The above-mentioned batch job detection device based on directed acyclic graph can detect the first type of directed acyclic graph corresponding to the batch job before the execution and the second type of directed acyclic graph after the batch job is executed through the graph detection unit 31, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependency, the nodes store the job execution information of the corresponding batch job, and the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph are compared through the graph comparison unit 32 to obtain a comparison result, and the job execution result output unit 33 outputs the execution information of the batch job and the abnormal job information based on the comparison result. In this embodiment, the directed acyclic graph structure before and after the batch job execution can be comprehensively compared, including the node set, edge set, and node in-and-out degree, so that abnormal situations in the batch job execution can be discovered and located in a timely manner, thereby improving the efficiency of exception handling. At the same time, the automation of batch job execution status detection is realized, and the accuracy and reliability of job monitoring are improved, thereby solving the technical problems in the related technology that, in scenarios where the number of batch jobs is large and there are complex dependencies between jobs, the efficiency is low, abnormal operation situations are easily missed, and abnormal business of financial institutions occurs.

[0091] Optionally, the graph detection unit includes: a first job detection module, which is used to detect the basic job information of each job in the batch job before the batch job is executed, and construct nodes based on the basic job information, wherein the basic job information includes: job name, group name of the group to which the job belongs, job type, job trigger time, and job function description information; a second job detection module, which is used to detect the dependency relationship between each job in the batch job before the job is executed, and use the dependent node in the dependency relationship as the end point of the edge and the dependent node in the dependency relationship as the starting point of the edge to establish directed connection edges between the nodes; a first graph construction module, which is used to construct a first type of directed acyclic graph based on the nodes and the directed connection edges; a third job detection module, which is used to detect the running status and job execution results of each job in the batch job during the execution of the batch job, and record the running status and job execution results in the auxiliary information of the node; a second graph construction module, which is used to construct a second type of directed acyclic graph based on the auxiliary information of the node and the new directed connection edges established during the batch job execution after the batch job is executed.

[0092] Optionally, the graph comparison unit includes: a node comparison module, used to compare whether the node set corresponding to the first type of directed acyclic graph is completely matched with the node set corresponding to the second type of directed acyclic graph; a first result output module, used to confirm that the batch job execution is abnormal and output the abnormal comparison result of the node when the two node sets are not completely matched; and a second result output module, used to output the normal comparison result of the node when the two node sets are completely matched.

[0093] Optionally, the graph comparison unit also includes: an edge comparison module, used to compare whether the edge set corresponding to the first type of directed acyclic graph is completely matched with the edge set corresponding to the second type of directed acyclic graph; a third result output module, used to confirm the abnormality of batch job execution when it is detected that any directed connection edge only exists in one of the edge sets, or when the weights of the directed connection edges of the two types of directed acyclic graphs are different, and output the abnormal comparison result of the connection edge, wherein the weight of the connection edge is determined based on the job importance level and the job execution result during the job execution process; a fourth result output module, used to output the normal comparison result of the connection edge when the directed connection edges in the two edge sets are completely matched.

[0094] Optionally, the graph comparison unit also includes: an in-degree comparison module, which is used to compare whether the in-degree and out-degree of each node in the first type of directed acyclic graph and the second type of directed acyclic graph are equal; a fifth result output module, which is used to confirm that the batch job execution is abnormal when there is any node with different in-degree or out-degree in the two types of directed acyclic graphs, and output the abnormal comparison result of the node in-degree and out-degree; a sixth result output module, which is used to output the normal comparison result of the node in-degree and out-degree when the in-degree or out-degree in the two types of directed acyclic graphs are the same.

[0095] Optionally, the batch job detection device based on directed acyclic graphs also includes: a topological sorting unit, used to execute a topological sorting algorithm on the first type of directed acyclic graph and the second type of directed acyclic graph, respectively, to obtain a first topological sorting result and a second topological sorting result; a seventh result output module, used to confirm that the batch job execution is abnormal when the first topological sorting result and the second topological sorting result are different, and output an abnormal comparison result of the graph topological sorting; an eighth result output module, used to output a normal comparison result of the graph topological sorting when the first topological sorting result and the second topological sorting result are the same.

[0096] Optionally, the topological sorting algorithm adopts one of the following: a hash algorithm, a depth-first search algorithm or a breadth-first search algorithm. During the topological sorting process, it is detected whether the two types of directed acyclic graphs have consistent structures, and the execution status of the batch job is determined based on the detection result.

[0097] The above-mentioned batch job detection device based on directed acyclic graph can also include a processor and a memory. The above-mentioned graph detection unit 31, graph comparison unit 32, job execution result output unit 33, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0098] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the abnormality detection of batch jobs based on the directed acyclic graph can be realized by adjusting the kernel parameters.

[0099] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM), and the memory includes at least one storage chip.

[0100] Embodiment 3

[0101] An embodiment of the present application may provide an electronic device, Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (only one is shown) processor 402, memory 404, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0102] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the batch job detection method and device based on directed acyclic graph in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the batch job detection method based on directed acyclic graph is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0103] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: detect the first type of directed acyclic graph corresponding to the batch job before the execution and the second type of directed acyclic graph after the batch job is executed, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependency, and the nodes store the job execution information of the corresponding batch job; compare the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result; based on the comparison result, output the execution information of the batch job and the abnormal job information.

[0104] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: before the batch job is executed, detect the basic job information of each job in the batch job, and build a node based on the basic job information, wherein the basic job information includes: job name, group name of the group to which the job belongs, job type, job trigger time, and job function description information; before the job is executed, detect the dependency relationship between the jobs in the batch job, use the dependent node in the dependency relationship as the end point of the edge, and use the dependent node in the dependency relationship as the starting point of the edge to establish directed connection edges between the nodes; construct a first type of directed acyclic graph based on the nodes and the directed connection edges; during the execution of the batch job, detect the running status and job execution results of each job in the batch job, and record the running status and job execution results in the auxiliary information of the node; after the batch job is executed, construct a second type of directed acyclic graph based on the auxiliary information of the node and the new directed connection edges established during the batch job operation.

[0105] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: compare whether the node set corresponding to the first type of directed acyclic graph is completely matched with the node set corresponding to the second type of directed acyclic graph; if the two node sets do not completely match, confirm that the batch job execution is abnormal and output the abnormal comparison result of the node; if the two node sets completely match, output the normal comparison result of the node.

[0106] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: compare whether the edge set corresponding to the first type of directed acyclic graph is completely matched with the edge set corresponding to the second type of directed acyclic graph; when it is detected that any directed connection edge only exists in one of the edge sets, or when the weights of the directed connection edges of the two types of directed acyclic graphs are different, confirm that the batch job execution is abnormal, and output the abnormal comparison result of the connection edge, wherein the weight of the connection edge is determined based on the job importance level and the job execution result during the job execution process; when the directed connection edges in the two edge sets are completely matched, output the normal comparison result of the connection edge.

[0107] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: compare whether the in-degree and out-degree of each node in the first type of directed acyclic graph and the second type of directed acyclic graph are equal; if there is any node with different in-degree or out-degree in the two types of directed acyclic graphs, confirm that the batch job execution is abnormal and output the abnormal comparison result of the node in-degree and out-degree; if the in-degree or out-degree in the two types of directed acyclic graphs are the same, output the normal comparison result of the node in-degree and out-degree.

[0108] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: execute the topological sorting algorithm on the first type of directed acyclic graph and the second type of directed acyclic graph respectively to obtain the first topological sorting result and the second topological sorting result; when the first topological sorting result and the second topological sorting result are different, confirm that the batch job execution is abnormal, and output the abnormal comparison result of the graph topological sorting; when the first topological sorting result and the second topological sorting result are the same, output the normal comparison result of the graph topological sorting.

[0109] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: the topological sorting algorithm adopts one of the following: hash algorithm, depth-first search algorithm or breadth-first search algorithm, and detects whether the two types of directed acyclic graphs are structurally consistent during the topological sorting process, and determines the execution status of the batch job based on the detection results.

[0110] According to the embodiment of the present application, a first type of directed acyclic graph is constructed based on the basic information of the job and the dependency relationship between the jobs. After the batch job is executed, a second type of directed acyclic graph is constructed based on the execution status information, execution results and possibly updated dependency relationships of the job. Then, the node set, edge set, node in-and-out degree and topological sorting are compared to ensure that no job is omitted or newly added, and to confirm that the dependency relationship between jobs has not changed, including checking the existence and weight of the edge, ensuring that the execution order and dependency relationship of the job are correct, verifying whether the linear order of job execution is consistent with expectations, outputting the execution status information of the batch job and the information of the abnormal job, and assisting developers in quickly locating problems. The need for manual intervention is reduced, the automation of batch job execution status detection is realized, the accuracy and reliability of job monitoring are improved, and it can be applied to the processing of large-scale batch jobs. Even if the number of jobs and the complexity of dependency relationships increase, efficient detection capabilities can be maintained.

[0111] It can be understood by those skilled in the art that Figure 4The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, etc. Figure 4 The structure of the electronic device is not limited. Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 4 Different configurations shown.

[0112] A person skilled in the art can understand that all or part of the steps in the various directed acyclic graph-based batch job detection methods in the above-mentioned embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0113] Embodiment 4

[0114] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the directed acyclic graph-based batch job detection method provided in the first embodiment.

[0115] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the directed acyclic graph-based batch job detection methods in the above-mentioned embodiment 1.

[0116] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0117] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the directed acyclic graph-based batch job detection method described in each embodiment of the present application.

[0118] The present application also provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the batch job detection method based on a directed acyclic graph described in each embodiment of the present application are implemented.

[0119] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0120] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0123] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0125] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A batch job detection method based on directed acyclic graph, characterized in that: include: Detecting a first type of directed acyclic graph corresponding to the batch job before execution and a second type of directed acyclic graph after the batch job is executed, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent the job dependency, and the nodes store the job execution information of the corresponding batch job; Comparing the node set, edge set, and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result; Based on the comparison result, the execution information of the batch job and the abnormal job information are output.

2. The batch job detection method according to claim 1, characterized in that: The step of detecting the first directed acyclic graph corresponding to the batch job before the batch job is executed and the second directed acyclic graph after the batch job is executed includes: Before the batch job is executed, the basic job information of each job in the batch job is detected, and a node is constructed based on the basic job information, wherein the basic job information includes: job name, group name of the group to which the job belongs, job type, job trigger time, and job function description information; Before the job is executed, the dependency relationship between the jobs in the batch job is detected, the dependent node in the dependency relationship is used as the end point of the edge, the dependent node in the dependency relationship is used as the starting point of the edge, and a directed connection edge is established between the nodes; Constructing the first type of directed acyclic graph according to the nodes and the directed connecting edges; During the batch job execution process, detecting the running status and job execution result of each job in the batch job, and recording the running status and the job execution result in the auxiliary information of the node; After the batch job is executed, the second type of directed acyclic graph is constructed based on the auxiliary information of the node and the new directed connection edges established during the batch job running process.

3. The batch job detection method according to claim 1, characterized in that: The step of comparing the node set, edge set, and node in-degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result includes: Comparing whether the node set corresponding to the first type of directed acyclic graph and the node set corresponding to the second type of directed acyclic graph completely match each other; When two sets of nodes do not completely match, confirm that the batch job is abnormal and output the abnormal comparison results of the nodes; In the case where the two node sets match exactly, the normal comparison result of the nodes is output.

4. The batch job detection method according to claim 1, characterized in that: The step of comparing the node set, edge set, and node in-degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result includes: Comparing whether the edge set corresponding to the first type of directed acyclic graph and the edge set corresponding to the second type of directed acyclic graph completely match each other; When it is detected that any directed connection edge exists only in one of the edge sets, or when the weights of the directed connection edges of the two types of directed acyclic graphs are different, confirm that the batch job execution is abnormal, and output the abnormal comparison result of the connection edge, wherein the weight of the connection edge is determined based on the job importance level and the job execution result during the job execution process; When the directed connecting edges in the two edge sets completely match, the normal comparison results of the connecting edges are output.

5. The batch job detection method according to claim 1, characterized in that: The step of comparing the node set, edge set, and node in-degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result includes: Compare whether the in-degree and out-degree of each node in the first type of directed acyclic graph and the second type of directed acyclic graph are equal; If the in-degree or out-degree of any node in the two types of directed acyclic graphs is different, confirm that the batch job execution is abnormal, and output the abnormal comparison result of the node in-degree and out-degree; When the in-degree or out-degree in two types of directed acyclic graphs are the same, the normal comparison results of the in-degree and out-degree of the nodes are output.

6. The batch job detection method according to claim 1, characterized in that: Also includes: Execute a topological sorting algorithm on the first type of directed acyclic graph and the second type of directed acyclic graph respectively to obtain a first topological sorting result and a second topological sorting result; When the first topological sorting result and the second topological sorting result are different, confirming that the batch job execution is abnormal, and outputting the abnormal comparison result of the graph topological sorting; When the first topological sorting result and the second topological sorting result are the same, a normal comparison result of the graph topological sorting is output.

7. The batch job detection method according to claim 6, characterized in that: The topological sorting algorithm adopts one of the following: a hash algorithm, a depth-first search algorithm or a breadth-first search algorithm. During the topological sorting process, it is detected whether the two types of directed acyclic graphs have consistent structures, and the execution status of the batch job is determined based on the detection result.

8. A batch job detection device based on directed acyclic graph, characterized in that: include: A graph detection unit, used to detect a first type of directed acyclic graph corresponding to a batch job before execution and a second type of directed acyclic graph after execution of the batch job, wherein the nodes in the directed acyclic graph represent each batch job, the connecting edges in the directed acyclic graph represent job dependencies, and the nodes store job execution information of the corresponding batch job; A graph comparison unit, used to compare the node set, edge set and node in-and-out degree information between the first type of directed acyclic graph and the second type of directed acyclic graph to obtain a comparison result; The job execution result output unit is used to output the execution information of the batch job and the abnormal job information based on the comparison result.

9. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the batch job detection method based on a directed acyclic graph as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the directed acyclic graph-based batch job detection method described in any one of claims 1 to 7 are implemented.

Images