Dag loop detection method, apparatus, device, and storage medium

Abstract

The application discloses a directed acyclic graph loop detection method and device, equipment and a storage medium, and the method comprises the following steps: constructing a plurality of directed acyclic graphs according to the dependency relationship between batch processing tasks in each batch processing service; for a new task to be added, determining target batch processing tasks in each batch processing service that have a dependency relationship with the new task; based on a high-performance asynchronous processing framework, traversing each directed acyclic graph through a depth-first traversal algorithm, and judging whether a first node corresponding to the new task and a second node corresponding to the target batch processing task exist in each directed acyclic graph; if the first node and the second node do not exist in each directed acyclic graph, the addition of the new task will not form a loop. According to the application, a plurality of directed acyclic graphs are constructed according to each batch processing service, and then each directed acyclic graph is traversed through a depth-first traversal algorithm based on a high-performance asynchronous processing framework, so that the complexity of directed acyclic graph configuration is effectively reduced, and the efficiency of loop detection is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for detecting loops in directed acyclic graphs. Background Technology

[0002] In existing software applications, there are always various batch processing tasks, which often require collaborative operation. Since these batch tasks execute in a specific order, there is a need for dependency management. While large-scale software that purchases external software to address these dependencies is usually quite mature, it is too expensive. Therefore, dependency management is transformed into a directed acyclic graph (DAG) approach to build a dependency management system.

[0003] However, using directed acyclic graphs (DAGs) inevitably triggers cycle detection when adding dependent nodes. If dependencies were initially designed for batch processing, a single dependent node might exist in multiple DAG dependencies. Therefore, striking a balance between DAG design and cycle detection presents the following problems: First, cycle detection is performed after merging. This can lead to situations where, after merging, a cycle is found in the merged DAG, rendering the merge invalid and requiring a rollback. Second, as DAGs become more complex, the merging time increases, resulting in longer waiting times.

[0004] Therefore, there is an urgent need for a directed acyclic graph (DAG) loop detection method that can effectively reduce the complexity of DAG configuration and improve loop detection efficiency. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for loop detection in directed acyclic graphs, aiming to solve the technical problem of how to reduce the complexity of directed acyclic graph configuration and improve loop detection efficiency.

[0006] To achieve the above objectives, the present invention provides a method for detecting loops in a directed acyclic graph, the method comprising the following steps:

[0007] Construct multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business;

[0008] For any new task to be added, identify the target batch processing tasks in each batch processing business that have a dependency relationship with the new task;

[0009] Based on a high-performance asynchronous processing framework, a depth-first traversal algorithm is used to traverse each of the directed acyclic graphs to determine whether there is a first node corresponding to the new task and a second node corresponding to the target batch processing task in each of the directed acyclic graphs.

[0010] If neither the first node nor the second node exists in any of the directed acyclic graphs, then the addition of the new task will not form a cycle.

[0011] Optionally, after determining whether the first node corresponding to the new task and the second node corresponding to the target batch processing task exist in each of the directed acyclic graphs, the method further includes:

[0012] If the first node and the second node exist in the directed acyclic graph, then the directed acyclic graph containing the first node and the second node is taken as the target directed acyclic graph.

[0013] If the number of target directed acyclic graphs is greater than 1, then all target directed acyclic graphs are merged to obtain a merged result;

[0014] The merging result is converted into the corresponding adjacency matrix, and loop detection is performed in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result;

[0015] If the loop detection result indicates the existence of a loop, then the addition of the new task will form a loop.

[0016] Optionally, after the step of using the directed acyclic graph containing the first node and the second node as the target directed acyclic graph, the method further includes:

[0017] If the number of the target directed acyclic graph is equal to 1, then the target directed acyclic graph is converted into the corresponding adjacency matrix;

[0018] The step of performing loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result.

[0019] Optionally, after the step of performing loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result, the method further includes:

[0020] If the loop detection result indicates that no loop exists, then the addition of the new task will not create a loop.

[0021] Optionally, after the step of ensuring that the addition of the new task does not form a loop, the method further includes:

[0022] In a directed acyclic graph containing the second node, a new directed acyclic graph is constructed based on the dependency relationship between the first node and the second node;

[0023] Determine whether the number of nodes in the new directed acyclic graph exceeds a preset threshold;

[0024] If the number of nodes in the new directed acyclic graph exceeds a preset threshold, the new directed acyclic graph is split to obtain the splitting result corresponding to the new directed acyclic graph.

[0025] Optionally, the step of constructing multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business includes:

[0026] Each batch processing service is introduced, and each batch processing service includes at least one batch processing task;

[0027] Using the unique identifier of each batch processing task as a node, multiple directed acyclic graphs are constructed based on the dependencies between batch processing tasks in each batch processing business.

[0028] Accordingly, after the step of constructing multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business, the method further includes:

[0029] Sequentially determine whether the number of nodes in each of the directed acyclic graphs exceeds the preset threshold;

[0030] If the number of nodes in the directed acyclic graph exceeds the preset threshold, the directed acyclic graph is split to obtain the splitting result corresponding to the directed acyclic graph.

[0031] Optionally, the high-performance asynchronous processing framework is the Disruptor high-performance asynchronous processing framework. The step of traversing each of the directed acyclic graphs using a depth-first search algorithm based on the high-performance asynchronous processing framework, and determining whether there exists a first node corresponding to the new task and a second node corresponding to the target batch processing task in each of the directed acyclic graphs, includes:

[0032] Using the event notification mechanism of the Disruptor high-performance asynchronous processing framework, the addressing instructions of the first node corresponding to the new task and the second node corresponding to the target batch processing task are broadcast.

[0033] Upon receiving the addressing instruction, a depth-first traversal algorithm is used to traverse each of the directed acyclic graphs to determine whether the first node and the second node exist in each of the directed acyclic graphs.

[0034] Furthermore, to achieve the above objectives, the present invention also proposes a directed acyclic graph loop detection device, the device comprising:

[0035] The structure building module is used to construct multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business.

[0036] The task determination module is used to determine, for the new task to be added, the target batch processing tasks in each batch processing business that have a dependency relationship with the new task;

[0037] The loop detection module is used to traverse each of the directed acyclic graphs based on a high-performance asynchronous processing framework using a depth-first traversal algorithm, and to determine whether there is a first node corresponding to the new task and a second node corresponding to the target batch processing task in each of the directed acyclic graphs.

[0038] The result output module is configured to ensure that the addition of the new task will not form a loop if the first node and the second node do not exist in each of the directed acyclic graphs.

[0039] Furthermore, to achieve the above objectives, the present invention also proposes a directed acyclic graph loop detection device, the device comprising: a memory, a processor, and a directed acyclic graph loop detection program stored in the memory and executable on the processor, the directed acyclic graph loop detection program being configured to implement the steps of the directed acyclic graph loop detection method as described above.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a directed acyclic graph loop detection program, which, when executed by a processor, implements the steps of the directed acyclic graph loop detection method as described above.

[0041] This invention constructs multiple directed acyclic graphs (DAGs) based on the dependencies between batch processing tasks in each batch processing business. For a new task to be added, a target batch processing task with a dependency relationship to the new task is identified in each batch processing business. Based on a high-performance asynchronous processing framework, a depth-first traversal algorithm is used to traverse each DAG to determine whether a first node corresponding to the new task and a second node corresponding to the target batch processing task exist in each DAG. If neither the first node nor the second node exists in any DAG, the addition of the new task will not form a loop. Because this invention constructs multiple DAGs based on the dependencies between batch processing tasks in each batch processing business, and then uses a depth-first traversal algorithm based on a high-performance asynchronous processing framework to traverse each DAG and determine whether a first node corresponding to the new task and a second node corresponding to the target batch processing task exist in each DAG, loop detection is achieved. Compared to existing technologies that merge multiple DAGs before performing loop detection, this invention not only effectively reduces the complexity of DAG configuration but also improves the efficiency of loop detection. Attached Figure Description

[0042] Figure 1This is a schematic diagram of the structure of the directed acyclic graph loop detection device for the hardware operating environment involved in the embodiments of the present invention;

[0043] Figure 2 This is a flowchart illustrating the first embodiment of the directed acyclic graph loop detection method of the present invention;

[0044] Figure 3 This is a schematic diagram of the first stage of the loop detection method for directed acyclic graphs of the present invention.

[0045] Figure 4 This is a flowchart illustrating the second embodiment of the directed acyclic graph loop detection method of the present invention;

[0046] Figure 5 This is a schematic diagram of the second stage of the loop detection method in the directed acyclic graph of the present invention.

[0047] Figure 6 This is a flowchart illustrating the third embodiment of the directed acyclic graph loop detection method of the present invention;

[0048] Figure 7 This is a structural block diagram of the first embodiment of the directed acyclic graph loop detection device of the present invention.

[0049] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0051] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a directed acyclic graph loop detection device for the hardware operating environment involved in the embodiments of the present invention.

[0052] like Figure 1As shown, the directed acyclic graph loop detection device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the directed acyclic graph loop detection device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0054] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a directed acyclic graph loop detection program.

[0055] exist Figure 1 In the directed acyclic graph loop detection device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the directed acyclic graph loop detection device of the present invention can be set in the directed acyclic graph loop detection device, and the directed acyclic graph loop detection device calls the directed acyclic graph loop detection program stored in the memory 1005 through the processor 1001 and executes the directed acyclic graph loop detection method provided in the embodiment of the present invention.

[0056] This invention provides a method for detecting loops in a directed acyclic graph, referring to... Figure 2 , Figure 2 This is a schematic flowchart of the first embodiment of the directed acyclic graph loop detection method of the present invention.

[0057] In this embodiment, the directed acyclic graph loop detection method includes the following steps:

[0058] Step S10: Construct multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business.

[0059] It should be noted that the executing entity in this embodiment can be a computing service device with data processing and program execution functions, such as a server, tablet computer, or personal computer, or an electronic device capable of performing the same or similar functions, such as the one described above. Figure 1 The example shown is a directed acyclic graph loop detection device. The following uses a directed acyclic graph loop detection device as an example to illustrate this embodiment and the embodiments described below.

[0060] Understandably, batch processing involves combining a series of batch tasks in a specific order for batch execution. The dependencies between batch tasks within a batch processing workflow can be defined by the order in which they are executed.

[0061] It should be understood that in mathematics, graph theory, and computer science, a directed acyclic graph refers to a directed graph without loops.

[0062] In a specific implementation, batch processing services can be introduced, each including at least one batch processing task; the unique identifier of the batch processing task is used as a node, and multiple directed acyclic graphs are constructed based on the dependencies between batch processing tasks in each batch processing service.

[0063] Step S20: For the new task to be added, identify the target batch processing task in each batch processing business that has a dependency relationship with the new task.

[0064] It should be explained that the aforementioned new task can be a new batch processing task that needs to be added to a certain batch processing business and has a dependency relationship with its batch processing task.

[0065] It should be noted that when using directed acyclic graphs (DAGs) for dependency management of batch processing tasks, DAG loop detection will inevitably be triggered when adding dependent nodes. However, if the dependency relationship was initially designed based on batch processing business units, a certain dependent node may exist in multiple DAGs. That is, the node corresponding to the target batch processing task that has a dependency relationship with the new task may exist in multiple DAGs.

[0066] Step S30: Based on the high-performance asynchronous processing framework, the directed acyclic graphs are traversed using a depth-first traversal algorithm to determine whether the first node corresponding to the new task and the second node corresponding to the target batch processing task exist in each directed acyclic graph.

[0067] It should be explained that the high-performance asynchronous processing framework mentioned above is the Disruptor high-performance asynchronous processing framework, which is a high-performance asynchronous processing framework implemented based on the ringbuffer data structure.

[0068] It should be understood that the current balance between designing directed acyclic graphs (DAGs) and loop detection presents the following problems: First, loop detection is performed after merging, which may result in the merged DAG containing loops, rendering the merge invalid and requiring a rollback. Second, as DAGs become more complex, the merging time increases, leading to longer waiting times.

[0069] Furthermore, this embodiment eliminates the need to merge multiple directed acyclic graphs, and the maintenance of directed acyclic graphs is based on batch processing tasks, allowing the same nodes to exist between different directed acyclic graphs.

[0070] In the specific implementation, the event notification mechanism of the Disruptor high-performance asynchronous processing framework is used to broadcast the addressing instructions of the first node corresponding to the new task and the second node corresponding to the target batch processing task. After receiving the addressing instructions, the directed acyclic graph is traversed by a depth-first traversal algorithm to determine whether the first node and the second node exist in each directed acyclic graph.

[0071] For example, refer to Figure 3 , Figure 3 This is a schematic diagram of the first stage of the loop detection method for directed acyclic graphs (DGAs) of the present invention. Assuming four DGAs (DGA1, DGA2, DGA3, and DGA4) are constructed, when a new task is added, to improve the efficiency of loop detection, the event notification mechanism of the Disruptor high-performance asynchronous processing framework (i.e., the DGA event core) can be used to broadcast the addressing instructions for the first node corresponding to the new task and the second node corresponding to the target batch processing task to each DGA (i.e., DGA1, DGA2, DGA3, and DGA4). Then, after receiving the addressing instructions, each DGA traverses the DGAs using a depth-first search algorithm (DFS algorithm) to determine whether the first node and the second node exist in each DGA (i.e., whether the DFS addressing is successful). The global addressing process ends after the addressing of DGA1, DGA2, DGA3, and DGA4 is completed.

[0072] For example, leveraging the event notification mechanism of the Disruptor high-performance asynchronous processing framework, node change events can be broadcast, such as the dependency relationship A->B. This involves broadcasting the addressing of node A and the addressing of node B. Each message handler bound to a DAG, upon sensing the addressing event, uses the Depth-First Search (DFS) algorithm to determine whether nodes A and B exist in the current DAG. If they do not exist in the current DAG, the addressing process ends. The addressing event concludes once all DAGs have been processed.

[0073] Step S40: If neither the first node nor the second node exists in any of the directed acyclic graphs, then the addition of the new task will not form a loop.

[0074] In a specific implementation, after the first stage of the loop detection method for directed acyclic graphs is completed, if neither the first node nor the second node exists in any of the directed acyclic graphs, the addition of the new task will not form a loop. Then, in the directed acyclic graphs where the second node exists, a new directed acyclic graph is constructed based on the dependency relationship between the first node and the second node.

[0075] This embodiment constructs multiple directed acyclic graphs (DAGs) based on the dependencies between batch processing tasks in each batch processing business. For a new task to be added, a target batch processing task with a dependency relationship to the new task is identified in each batch processing business. Based on a high-performance asynchronous processing framework, a depth-first traversal algorithm is used to traverse each DAG to determine whether a first node corresponding to the new task and a second node corresponding to the target batch processing task exist in each DAG. If neither the first node nor the second node exists in any DAG, the addition of the new task will not form a loop. Since this embodiment constructs multiple DAGs based on the dependencies between batch processing tasks in each batch processing business, and then uses a depth-first traversal algorithm based on a high-performance asynchronous processing framework to traverse each DAG and determine whether a first node corresponding to the new task and a second node corresponding to the target batch processing task exist in each DAG, loop detection is achieved. Compared to existing technologies that merge multiple DAGs before performing loop detection, this embodiment not only effectively reduces the complexity of DAG configuration but also improves the efficiency of loop detection.

[0076] refer to Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the directed acyclic graph loop detection method of the present invention.

[0077] Based on the first embodiment described above, in this embodiment, after step S30, the method further includes:

[0078] Step S401: If the first node and the second node exist in the directed acyclic graph, then the directed acyclic graph containing the first node and the second node is taken as the target directed acyclic graph.

[0079] It should be explained that after the first stage of loop detection in the directed acyclic graph loop detection method is completed, if the first node and the second node exist in the directed acyclic graph, that is, if an addressing hit event exists, then the directed acyclic graph containing the first node and the second node is taken as the target directed acyclic graph, and then the number of target directed acyclic graphs is determined.

[0080] Step S402: If the number of target directed acyclic graphs is greater than 1, then the target directed acyclic graphs are merged to obtain a merged result.

[0081] Step S403: Convert the merging result into the corresponding adjacency matrix, and perform loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result.

[0082] Understandably, the above loop detection results include both the absence of loops and the presence of loops.

[0083] Step S404: If the loop detection result indicates the existence of a loop, then the addition of the new task will form a loop.

[0084] It should be understood that if the loop detection result is that no loop exists, the addition of the new task will not form a loop.

[0085] It should be noted that if the number of the target directed acyclic graph is equal to 1, then the target directed acyclic graph is converted into the corresponding adjacency matrix; the step of performing loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result is executed; if the loop detection result indicates that a loop exists, then the addition of the new task will form a loop, and if the loop detection result indicates that no loop exists, then the addition of the new task will not form a loop.

[0086] For example, refer to Figure 5 , Figure 5This diagram illustrates the second stage of the loop detection method for directed acyclic graphs (DAGs) according to the present invention. After the first stage of loop detection is completed (i.e., after DGA1, DGA2, DGA3, DGA4, and DGA5 addressing), if both the first node and the second node exist in DGA1 and DGA5, then the DGAs are merged (i.e., DGA1 and DGA5 are merged in memory) to obtain a merged result. The merged result is then converted into an adjacency matrix. Loop detection is then performed on the adjacency matrix using a depth-first search algorithm (DFS) to obtain the loop detection result. If the result is positive, then no loop exists; otherwise, a loop exists.

[0087] This embodiment constructs multiple directed acyclic graphs (DAGs) based on the dependencies between batch processing tasks in each batch processing business. For a new task to be added, target batch processing tasks that have dependencies on the new task are identified in each batch processing business. Based on a high-performance asynchronous processing framework, a depth-first traversal algorithm is used to traverse each DAG to determine whether a first node corresponding to the new task and a second node corresponding to the target batch processing task exist in each DAG. If the first node and the second node exist in the DAG, the DAG containing the first node and the second node is taken as the target DAG. If the number of target DAGs is greater than 1, the target DAGs are merged to obtain a merge result. The merge result is converted into a corresponding adjacency matrix, and loop detection is performed in the adjacency matrix based on the depth-first traversal algorithm to obtain a loop detection result. If the loop detection result indicates the existence of a loop, the addition of the new task will form a loop. This embodiment manages the directed acyclic graph (DAG) within a single batch processing business unit, avoiding the problem that adding new dependencies to a single DAG configuration can lead to continuously increasing complexity. At the same time, it utilizes the event notification mechanism of the Disruptor high-performance asynchronous processing framework to reduce the complexity of DAG configuration and decrease the cost of DAG loop detection.

[0088] refer to Figure 6 , Figure 6 This is a flowchart illustrating the third embodiment of the directed acyclic graph loop detection method of the present invention.

[0089] Based on the above embodiments, in this embodiment, after step S40, the method further includes:

[0090] Step S50: In the directed acyclic graph containing the second node, construct a new directed acyclic graph based on the dependency relationship between the first node and the second node.

[0091] Understandably, if neither the first node nor the second node exists in any of the directed acyclic graphs, the addition of the new task will not form a loop. Then, in the directed acyclic graph where the second node exists, a new directed acyclic graph is constructed based on the dependency relationship between the first node and the second node.

[0092] It should be noted that, in order to prevent the complexity of the new directed acyclic graph from becoming too large, the new directed acyclic graph can be split according to the number of nodes in the new directed acyclic graph. In this way, the complexity of configuring dependencies can be controlled at a reasonable level.

[0093] Step S60: Determine whether the number of nodes in the new directed acyclic graph exceeds a preset threshold.

[0094] It should be noted that the aforementioned preset threshold can be user-defined or set based on batch processing business analysis using a specific algorithm; this embodiment does not impose any restrictions on this. For example, the aforementioned preset threshold can be 6, 8, or 10, etc.

[0095] Step S70: If the number of nodes in the new directed acyclic graph exceeds a preset threshold, the new directed acyclic graph is split to obtain the splitting result corresponding to the new directed acyclic graph.

[0096] Accordingly, in order to prevent the number of nodes in multiple directed acyclic graphs constructed based on the dependencies between batch processing tasks in each batch processing business from exceeding a preset threshold, after step S10, the method further includes: sequentially determining whether the number of nodes in each directed acyclic graph exceeds the preset threshold; if the number of nodes in the directed acyclic graph exceeds the preset threshold, then splitting the directed acyclic graph to obtain the splitting result corresponding to the directed acyclic graph.

[0097] In this embodiment, a new directed acyclic graph (DAG) is constructed based on the dependency relationship between the first and second nodes in a DAG containing a second node. It then determines whether the number of nodes in the new DAG exceeds a preset threshold. If the number of nodes exceeds the preset threshold, the new DAG is split to obtain the corresponding splitting result. Compared to existing technologies, this embodiment allows for further splitting of the DAG after its complexity reaches a relatively high level. This ensures that the complexity of configuring dependencies in the DAG is always controlled at a reasonable level, reducing not only the complexity of DAG configuration but also the cost of loop detection in DAGs.

[0098] Furthermore, this embodiment of the invention also proposes a storage medium storing a directed acyclic graph loop detection program, which, when executed by a processor, implements the steps of the directed acyclic graph loop detection method described above.

[0099] Reference Figure 7 , Figure 7 This is a structural block diagram of the first embodiment of the directed acyclic graph loop detection device of the present invention.

[0100] like Figure 7 As shown, the directed acyclic graph loop detection device proposed in this embodiment of the invention includes: a structure construction module 701, a task determination module 702, a loop detection module 703, and a result output module 704.

[0101] The structure building module 701 is used to construct multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business.

[0102] The task determination module 702 is used to determine, for the new task to be added, the target batch processing task in each batch processing business that has a dependency relationship with the new task.

[0103] The loop detection module 703 is used to traverse each of the directed acyclic graphs using a depth-first traversal algorithm based on a high-performance asynchronous processing framework, and to determine whether there is a first node corresponding to the new task and a second node corresponding to the target batch processing task in each of the directed acyclic graphs.

[0104] The result output module 704 is configured to ensure that the addition of the new task will not form a loop if the first node and the second node do not exist in each of the directed acyclic graphs.

[0105] The loop detection module 703 is also used to broadcast addressing instructions for the first node corresponding to the new task and the second node corresponding to the target batch processing task using the event notification mechanism of the Disruptor high-performance asynchronous processing framework; after receiving the addressing instructions, it traverses each of the directed acyclic graphs using a depth-first traversal algorithm to determine whether the first node and the second node exist in each of the directed acyclic graphs.

[0106] This embodiment constructs multiple directed acyclic graphs (DAGs) based on the dependencies between batch processing tasks in each batch processing business. For a new task to be added, a target batch processing task with a dependency relationship to the new task is identified in each batch processing business. Based on a high-performance asynchronous processing framework, a depth-first traversal algorithm is used to traverse each DAG to determine whether a first node corresponding to the new task and a second node corresponding to the target batch processing task exist in each DAG. If neither the first node nor the second node exists in any DAG, the addition of the new task will not form a loop. Since this embodiment constructs multiple DAGs based on the dependencies between batch processing tasks in each batch processing business, and then uses a depth-first traversal algorithm based on a high-performance asynchronous processing framework to traverse each DAG and determine whether a first node corresponding to the new task and a second node corresponding to the target batch processing task exist in each DAG, loop detection is achieved. Compared to existing technologies that merge multiple DAGs before performing loop detection, this embodiment not only effectively reduces the complexity of DAG configuration but also improves the efficiency of loop detection.

[0107] Based on the first embodiment of the directed acyclic graph loop detection device of the present invention, a second embodiment of the directed acyclic graph loop detection device of the present invention is proposed.

[0108] In this embodiment, the loop detection module 703 is further configured to: if the first node and the second node exist in the directed acyclic graph, then take the directed acyclic graph containing the first node and the second node as the target directed acyclic graph; if the number of target directed acyclic graphs is greater than 1, then merge the target directed acyclic graphs to obtain a merge result; convert the merge result into a corresponding adjacency matrix, and perform loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain a loop detection result; if the loop detection result indicates the existence of a loop, then the addition of the new task will form a loop.

[0109] The loop detection module 703 is further configured to convert the target directed acyclic graph into a corresponding adjacency matrix if the number of the target directed acyclic graph is equal to 1; and to perform the loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result.

[0110] The result output module 704 is further configured to ensure that the addition of the new task will not form a loop if the loop detection result indicates that no loop exists.

[0111] The result output module 704 is further configured to construct a new directed acyclic graph based on the dependency relationship between the first node and the second node in a directed acyclic graph containing the second node; determine whether the number of nodes in the new directed acyclic graph exceeds a preset threshold; if the number of nodes in the new directed acyclic graph exceeds the preset threshold, then split the new directed acyclic graph to obtain the splitting result corresponding to the new directed acyclic graph.

[0112] Other embodiments or specific implementations of the directed acyclic graph loop detection device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0113] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0114] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0116] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for detecting loops in a directed acyclic graph, characterized in that, The method includes the following steps: Construct multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business; For any new task to be added, identify the target batch processing tasks in each batch processing business that have a dependency relationship with the new task; Based on a high-performance asynchronous processing framework, the directed acyclic graphs are traversed using a depth-first traversal algorithm to determine whether the first node corresponding to the new task and the second node corresponding to the target batch processing task exist in each directed acyclic graph. If neither the first node nor the second node exists in any of the directed acyclic graphs, then the addition of the new task will not form a loop; After the step of determining whether the first node corresponding to the new task and the second node corresponding to the target batch processing task exist in each of the directed acyclic graphs, the method further includes: If the first node and the second node exist in the directed acyclic graph, then the directed acyclic graph containing the first node and the second node is taken as the target directed acyclic graph. If the number of target directed acyclic graphs is greater than 1, then all target directed acyclic graphs are merged to obtain a merged result; The merging result is converted into a corresponding adjacency matrix, and loop detection is performed in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result; If the loop detection result indicates the existence of a loop, then the addition of the new task will form a loop.

2. The method for detecting loops in a directed acyclic graph as described in claim 1, characterized in that, After the step of taking the directed acyclic graph containing the first node and the second node as the target directed acyclic graph, the method further includes: If the number of the target directed acyclic graph is equal to 1, then the target directed acyclic graph is converted into the corresponding adjacency matrix; The step of performing loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result.

3. The method for detecting loops in a directed acyclic graph as described in any one of claims 1-2, characterized in that, After the step of performing loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain the loop detection result, the method further includes: If the loop detection result indicates that no loop exists, then the addition of the new task will not create a loop.

4. The method for detecting loops in a directed acyclic graph as described in claim 3, characterized in that, Following the step of ensuring that the addition of the new task will not create a loop, the method further includes: In a directed acyclic graph containing the second node, a new directed acyclic graph is constructed based on the dependency relationship between the first node and the second node; Determine whether the number of nodes in the new directed acyclic graph exceeds a preset threshold; If the number of nodes in the new directed acyclic graph exceeds a preset threshold, the new directed acyclic graph is split to obtain the splitting result corresponding to the new directed acyclic graph.

5. The method for detecting loops in a directed acyclic graph as described in claim 4, characterized in that, The step of constructing multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business includes: Each batch processing service is introduced, and each batch processing service includes at least one batch processing task; Using the unique identifier of each batch processing task as a node, multiple directed acyclic graphs are constructed based on the dependencies between batch processing tasks in each batch processing business. Accordingly, after the step of constructing multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business, the method further includes: Sequentially determine whether the number of nodes in each of the directed acyclic graphs exceeds the preset threshold; If the number of nodes in the directed acyclic graph exceeds the preset threshold, the directed acyclic graph is split to obtain the splitting result corresponding to the directed acyclic graph.

6. The method for detecting loops in a directed acyclic graph as described in claim 1, characterized in that, The high-performance asynchronous processing framework is the Disruptor high-performance asynchronous processing framework. The step of traversing each directed acyclic graph using a depth-first search algorithm based on this framework, and determining whether each directed acyclic graph contains a first node corresponding to the new task and a second node corresponding to the target batch processing task, includes: Using the event notification mechanism of the Disruptor high-performance asynchronous processing framework, the addressing instructions of the first node corresponding to the new task and the second node corresponding to the target batch processing task are broadcast. Upon receiving the addressing instruction, the directed acyclic graphs are traversed using a depth-first traversal algorithm to determine whether the first node and the second node exist in each directed acyclic graph.

7. A directed acyclic graph loop detection device, characterized in that, The device includes: The structure building module is used to construct multiple directed acyclic graphs based on the dependencies between batch processing tasks in each batch processing business. The task determination module is used to determine, for the new task to be added, the target batch processing tasks in each batch processing business that have a dependency relationship with the new task; The loop detection module is used to traverse each of the directed acyclic graphs based on a high-performance asynchronous processing framework using a depth-first traversal algorithm, and to determine whether there is a first node corresponding to the new task and a second node corresponding to the target batch processing task in each of the directed acyclic graphs. The result output module is configured to ensure that the addition of the new task will not form a loop if the first node and the second node do not exist in each of the directed acyclic graphs. The loop detection module is further configured to: if the first node and the second node exist in the directed acyclic graph, then take the directed acyclic graph containing the first node and the second node as the target directed acyclic graph; if the number of target directed acyclic graphs is greater than 1, then merge the target directed acyclic graphs to obtain a merge result; convert the merge result into a corresponding adjacency matrix, and perform loop detection in the adjacency matrix based on the depth-first traversal algorithm to obtain a loop detection result; if the loop detection result indicates the existence of a loop, then the addition of the new task will form a loop.

8. A directed acyclic graph loop detection device, characterized in that, The device includes: a memory, a processor, and a directed acyclic graph loop detection program stored in the memory and executable on the processor, the directed acyclic graph loop detection program being configured to implement the steps of the directed acyclic graph loop detection method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a directed acyclic graph loop detection program, which, when executed by a processor, implements the steps of the directed acyclic graph loop detection method as described in any one of claims 1 to 6.

Images