Clustering Scheduling Method and Device Based on Replication and Redundancy Removal

Abstract

The present application relates to a clustering scheduling method, device, storage medium, and computer program product based on replication and redundancy removal, including: obtaining a first processing sub-queue for processing target tasks, where the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target tasks; decomposing the first processing sub-queue into a second processing sub-queue based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target tasks; obtaining the estimated completion time of each sub-task node in the second processing sub-queue; sending a first notification message, where the first notification message includes the estimated completion time of each sub-task node, and the first notification message is used to instruct the processor executing the tasks to delete the first target sub-task nodes, and the first target sub-task nodes are the sub-task nodes on the processor executing the tasks whose estimated completion time is later than the estimated completion time in the first notification message. Using this method can improve the scheduling efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular, to a clustering scheduling method, device, computer device, computer-readable storage medium, and computer program product based on replication and redundancy removal. Background Art

[0002] With the slowdown of the computer development speed and the increasing demand for computing power, task parallelization and distributed computing have become more and more popular. When processing tasks based on computing resources, a scheduling method for reasonably and effectively allocating computing resources is required to be able to complete tasks as soon as possible and maximize resource utilization.

[0003] Generally, a scheduling method based on task replication can be used to achieve effective scheduling of resources. However, the above method will result in low scheduling efficiency of tasks. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a clustering scheduling method, device, computer device, computer-readable storage medium, and computer program product based on replication and redundancy removal that can improve scheduling efficiency.

[0005] In a first aspect, the present application provides a clustering scheduling method based on replication and redundancy removal, including: obtaining a first processing sub-queue for processing a target task, where the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, decomposing the first processing sub-queue into a second processing sub-queue, where the second processing sub-queue includes multiple sub-task nodes corresponding to each of the task nodes; obtaining the estimated completion time of each of the sub-task nodes in the second processing sub-queue, and the estimated completion time of any one sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node of the sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node; sending a first notification message, where the first notification message includes the estimated completion time of each of the sub-task nodes, and the first notification message is used to instruct a processor executing a task to delete a first target sub-task node, where the first target sub-task node is a sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message.

[0006] In one of the embodiments, the obtaining a first processing sub-queue for processing a target task includes: obtaining the reduced directed acyclic graph of the target task; dividing the task nodes in the reduced directed acyclic graph to obtain two or more queues; and determining one of the two or more queues as the first processing sub-queue.

[0007] In one embodiment, partitioning the task nodes in the reduced directed acyclic graph to obtain more than two queues includes: if there are multiple leaf task nodes in the reduced directed acyclic graph, respectively allocating each of the leaf task nodes to a first queue, where each of the leaf task nodes is a task node with an out-degree of 0; determining the critical predecessor task nodes corresponding to each of the leaf task nodes, and multiple predecessor task nodes of the corresponding critical predecessor task nodes, and allocating the corresponding critical predecessor task nodes and the multiple predecessor task nodes of the corresponding critical predecessor task nodes to the corresponding first queue; if there are no leaf task nodes in the reduced directed acyclic graph, obtaining multiple task nodes in the reduced directed acyclic graph that have not been allocated to a queue; allocating each of the task nodes that have not been allocated to a processing queue to a second queue; the more than two queues include the first queue and the second queue.

[0008] In one embodiment, determining the critical predecessor task nodes of the leaf task nodes includes: calculating the earliest completion time of multiple predecessor task nodes of the leaf task node; the earliest completion time is the sum of the earliest start time of each of the predecessor task nodes and the estimated processing time of each of the predecessor task nodes, and the earliest start time is the latest completion time of multiple predecessor task nodes of each of the predecessor task nodes; determining the predecessor task node with the largest earliest completion time among each of the predecessor task nodes as the critical predecessor task node.

[0009] In one embodiment, obtaining a first processing sub-queue for a processing target task includes: receiving the first processing sub-queue sent by a target processor; wherein, the first processing sub-queue is one of the more than two queues obtained by the target processor by obtaining the reduced directed acyclic graph of the target task and partitioning the task nodes in the reduced directed acyclic graph.

[0010] In one embodiment, the method further includes: receiving second notification information, where the second notification information includes the estimated completion time of each sub-task node in the processing sub-queue executed by a processor executing a task; determining whether there is a second target sub-task node, where the second target sub-task node is a sub-task node in the second processing sub-queue whose estimated completion time is later than the estimated completion time in the second notification information; if there is the second target sub-task node, deleting the second target sub-task node in the second processing sub-queue to obtain an updated second processing sub-queue.

[0011] In one embodiment, the method further includes: obtaining the estimated completion time of each of the subtask nodes in the updated second processing subqueue; sending a third notification message, where the third notification message includes the estimated completion time of each of the subtask nodes in the updated second processing subqueue, and the third notification message is used to instruct the processor executing the task to delete the third target subtask node, where the third target subtask node is a subtask node whose estimated completion time on the processor executing the task is later than the estimated completion time in the third notification message.

[0012] In a second aspect, the present application provides a clustering scheduling device based on replication and redundancy removal, including: a queue acquisition module, configured to acquire a first processing subqueue for processing a target task, where the first processing subqueue includes multiple task nodes of a reduced directed acyclic graph of the target task; a queue decomposition module, configured to decompose the first processing subqueue into a second processing subqueue based on the correspondence between the task nodes of the reduced directed acyclic graph and the subtask nodes of the partitioned directed acyclic graph of the target task, where the second processing subqueue includes multiple subtask nodes corresponding to each of the task nodes; a time acquisition module, configured to obtain the estimated completion time of each of the subtask nodes in the second processing subqueue, and the estimated completion time of any one subtask node is the sum of the estimated completion time of the previous adjacent subtask node of the subtask node in the second processing subqueue and the estimated processing time of the subtask node; a processing module, configured to send a first notification message, where the first notification message includes the estimated completion time of each of the subtask nodes, and the first notification message is used to instruct the processor executing the task to delete the first target subtask node, where the first target subtask node is a subtask node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message.

[0013] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: obtaining a first processing sub-queue for processing a target task, where the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, decomposing the first processing sub-queue into a second processing sub-queue, where the second processing sub-queue includes multiple sub-task nodes corresponding to each of the task nodes; obtaining the estimated completion time of each of the sub-task nodes in the second processing sub-queue, and the estimated completion time of any one sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node of the sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node; sending a first notification message, where the first notification message includes the estimated completion time of each of the sub-task nodes, and the first notification message is used to instruct the processor executing the task to delete a first target sub-task node, where the first target sub-task node is a sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message.

[0014] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the following steps are implemented: obtaining a first processing sub-queue for processing a target task, where the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, decomposing the first processing sub-queue into a second processing sub-queue, where the second processing sub-queue includes multiple sub-task nodes corresponding to each of the task nodes; obtaining the estimated completion time of each of the sub-task nodes in the second processing sub-queue, and the estimated completion time of any one sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node of the sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node; sending a first notification message, where the first notification message includes the estimated completion time of each of the sub-task nodes, and the first notification message is used to instruct the processor executing the task to delete a first target sub-task node, where the first target sub-task node is a sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message.

[0015] Fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented: obtaining a first processing sub-queue for processing a target task, where the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, decomposing the first processing sub-queue into a second processing sub-queue, where the second processing sub-queue includes multiple sub-task nodes corresponding to each of the task nodes; obtaining the estimated completion time of each of the sub-task nodes in the second processing sub-queue, and the estimated completion time of any one sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node of the sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node; sending a first notification message, where the first notification message includes the estimated completion time of each of the sub-task nodes, and the first notification message is used to instruct the processor executing the task to delete a first target sub-task node, and the first target sub-task node is a sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message.

[0016] The above clustering scheduling method, device, computer device, storage medium and computer program product based on replication and redundancy removal, by obtaining a first processing sub-queue for processing a target task, since the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task, based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, the first processing sub-queue can be decomposed into a second processing sub-queue, and the second processing sub-queue includes multiple sub-task nodes corresponding to each task node. Furthermore, by obtaining the estimated completion time of each sub-task node in the second processing sub-queue, a first notification message can be sent. The first notification message includes the estimated completion time of each sub-task node, and the first notification message is used to instruct the processor executing the task to delete a first target sub-task node. The first target sub-task node is a sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message. In this way, when processing the corresponding sub-task nodes based on the task processor and the processor executing the task, it is possible to avoid processing the same sub-task node on different processors, thereby improving the scheduling efficiency.

[0017] Sixth aspect, the present application provides a clustering scheduling method based on replication and redundancy removal, including: sending a first processing sub-queue for processing a target task to a task processor; the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; receiving a first notification message; the first notification message includes the estimated completion time of each sub-task node in a second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor decomposing the first processing sub-queue based on the corresponding relationship between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task; sending the first notification message to a target task processor, where the target task processor includes at least a task processor having at least one same sub-task node as the second processing sub-queue, and the first notification message is used to instruct the target task processor to delete a first target sub-task node, and the first target sub-task node is a sub-task node whose estimated completion time on the target task processor is later than the estimated completion time in the first notification message.

[0018] Seventh aspect, the present application provides a clustering scheduling device based on replication and redundancy removal, including: a queue sending module, configured to send a first processing sub-queue for processing a target task to a task processor; the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; an information receiving module, configured to receive a first notification message; the first notification message includes the estimated completion time of each sub-task node in a second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor decomposing the first processing sub-queue based on the corresponding relationship between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task; a message sending module, configured to send the first notification message to a target task processor, where the target task processor includes at least a task processor having at least one same sub-task node as the second processing sub-queue, and the first notification message is used to instruct the target task processor to delete a first target sub-task node, and the first target sub-task node is a sub-task node whose estimated completion time on the target task processor is later than the estimated completion time in the first notification message.

[0019] In an eighth aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: sending a first processing sub-queue for processing a target task to a task processor; the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; receiving first notification information; the first notification information includes the estimated completion time of each sub-task node in a second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor decomposing the first processing sub-queue based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task; sending the first notification information to a target task processor, where the target task processor includes at least a task processor having at least one same sub-task node as the second processing sub-queue, and the first notification information is used to instruct the target task processor to delete a first target sub-task node, and the first target sub-task node is a sub-task node whose estimated completion time on the target task processor is later than the estimated completion time in the first notification information.

[0020] In a ninth aspect, the present application further provides a computer-readable storage medium. On the computer-readable storage medium, there is stored a computer program, and when the computer program is executed by a processor, the following steps are implemented: sending a first processing sub-queue for processing a target task to a task processor; the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; receiving first notification information; the first notification information includes the estimated completion time of each sub-task node in a second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor decomposing the first processing sub-queue based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task; sending the first notification information to a target task processor, where the target task processor includes at least a task processor having at least one same sub-task node as the second processing sub-queue, and the first notification information is used to instruct the target task processor to delete a first target sub-task node, and the first target sub-task node is a sub-task node whose estimated completion time on the target task processor is later than the estimated completion time in the first notification information.

[0021] Tenth aspect, the present application also provides a computer program product. The computer program product includes a computer program which, when executed by a processor, implements the following steps: sending a first processing sub-queue for processing a target task to a task processor; the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; receiving first notification information; the first notification information includes the estimated completion time of each sub-task node in a second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor decomposing the first processing sub-queue based on the corresponding relationship between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task; sending the first notification information to a target task processor, the target task processor includes at least a task processor having at least one same sub-task node as the second processing sub-queue, and the first notification information is used to instruct the target task processor to delete a first target sub-task node, and the first target sub-task node is a sub-task node whose estimated completion time on the target task processor is later than the estimated completion time in the first notification information.

[0022] In the above clustering scheduling method, device, computer device, storage medium and computer program product based on replication and redundancy removal, the target processor sends a first processing sub-queue for processing a target task to the task processor, so that the task processor decomposes the first processing sub-queue based on the corresponding relationship between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task to obtain a second processing queue. Then, the task processor can obtain the estimated completion time of each sub-task node in the second processing sub-queue and send a first notification message to the target processor. Since the first notification information includes the estimated completion time of each sub-task node in the second processing sub-queue of the task processor, in this way, the target processor receives the estimated completion time of each sub-task node in the queue sent by the executing task processor. By comparing with the estimated completion time of each sub-task node in the second processing sub-queue in the first notification message, the target processor can determine the target task processor where the first target sub-task node whose estimated completion time is later than the estimated completion time in the first notification information is located, and send the first notification information to the target task processor, so that the target task processor can delete the first target sub-task node, so that the first target sub-task node only exists in the task processor, which can avoid the existence of the same first target sub-task node on different processors, thereby improving the scheduling efficiency. Description of the Drawings

[0023] Figure 1 It is an application environment diagram of the clustering scheduling method based on replication and redundancy removal in an embodiment;

[0024] Figure 2Schematic flowchart of a clustering scheduling method based on replication and redundancy removal in an embodiment;

[0025] Figure 3 Schematic diagram of a reduced directed acyclic graph of a target task in an embodiment;

[0026] Figure 4 Schematic diagram of a queue for processing a target task in an embodiment;

[0027] Figure 5 Schematic diagram of a partitioned directed acyclic graph of a target task in an embodiment;

[0028] Figure 6 Schematic diagram of a queue for processing a target task in an embodiment;

[0029] Figure 7 Schematic flowchart of a process for obtaining a first processing sub-queue for processing a target task in an embodiment;

[0030] Figure 8 Schematic flowchart of a process for dividing task nodes in a reduced directed acyclic graph to obtain two or more queues in an embodiment;

[0031] Figure 9 Schematic flowchart of a process for determining key predecessor task nodes of leaf task nodes in an embodiment;

[0032] Figure 10 Schematic diagram of assigning queues to each task node in a reduced directed acyclic graph in an embodiment;

[0033] Figure 11 Schematic flowchart of a clustering scheduling method based on replication and redundancy removal in an embodiment;

[0034] Figure 12 Schematic flowchart of a clustering scheduling method based on replication and redundancy removal in an embodiment;

[0035] Figure 13 Schematic diagram of sub-task nodes for processing a target task in an embodiment;

[0036] Figure 14 Schematic flowchart of a clustering scheduling method based on replication and redundancy removal in an embodiment;

[0037] Figure 15 Schematic diagram of a clustering scheduling method based on replication and redundancy removal in an embodiment;

[0038] Figure 16 Structural block diagram of a clustering scheduling device based on replication and redundancy removal in an embodiment;

[0039] Figure 17It is a structural block diagram of a clustering scheduling device based on replication and redundancy removal in an embodiment;

[0040] Figure 18 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] With the increasing demand for computing power, task parallelization and distributed computing are becoming more and more popular. When processing tasks in deep learning, graphics analysis, and edge computing, various computing resources are required. Therefore, a scheduling method for reasonably and effectively allocating computing resources is needed to be able to complete tasks as soon as possible and maximize resource utilization. Among them, computing resources can refer to a central processing unit (CPU), network bandwidth, memory, etc.

[0043] Generally, tasks to be processed are usually modeled as large directed acyclic graphs (DAGs). Among them, the nodes in the DAG represent tasks, and the connections between the nodes represent the dependencies between tasks. When the number of nodes in the DAG ranges in the tens of thousands, the tasks to be processed can be called large tasks, and the scheduling of the computing resources required for processing large tasks can be called large task scheduling.

[0044] Large task scheduling is an important and challenging research field in computer science. The task scheduling system needs to balance the execution time of tasks and the available computing resources so that applications for processing large tasks can be executed in the shortest time and maximize resource utilization; due to the different characteristics of large tasks, the requirements for computer resources when executing large tasks are very different. For example, some large tasks require a large amount of CPU for computing, while some large tasks have higher requirements for input / output (I / O) operations; at the same time, there are many criteria to evaluate the effectiveness of task scheduling based on fairness, low latency, and high throughput; moreover, if the task nodes appear in chronological order, this is a dynamic scheduling problem, and if the tasks are specified and all information related to the tasks is known before task scheduling, this is a static scheduling problem.

[0045] Generally, task scheduling methods can be divided into three categories, namely list-based task scheduling methods, cluster-based task scheduling methods, and replication-based task scheduling methods. Among them, list-based task scheduling methods are simple and easy to implement. However, this will lead to waste of computing resources; replication-based task scheduling methods perform well when scheduling small-scale tasks with 1000 nodes or fewer in DAD. However, once the scale of the graph becomes larger, for example, tens of thousands of nodes, it will require a long scheduling time, resulting in complex processing and low scheduling efficiency; the decisions of cluster-based task scheduling methods are local and do not consider the global structure of the DAG. Moreover, if there are a sufficient number of processors, cluster-based task scheduling methods tend to lead to waste of computing resources.

[0046] In view of this, the present application provides a clustering scheduling method based on replication and redundancy removal. This method can be applied to an application environment composed of task processors. Among them, the task processor can be a processor cluster composed of multiple processors, and one of the multiple processors can be the target processor; alternatively, this method can be applied to an application environment composed of a task processor and a target processor, where the task processor can be a processor cluster composed of multiple processors, and the task processor and the target processor are independent processors respectively.

[0047] For the sake of convenience of description, hereinafter, an example where the task processor and the target processor are independent processors respectively will be used for exemplary illustration. Specifically, the clustering scheduling method based on replication and redundancy removal provided by the present application can be applied to, for example Figure 1In the application environment shown, the task processor 102 communicates with the target processor 104 via a network. Specifically, the task processor 102 can obtain a first processing sub-queue for processing the target task from the target processor 104. The first processing sub-queue includes multiple task nodes of the reduced directed acyclic graph of the target task. Thus, based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, the task processor 102 can decompose the first processing sub-queue into a second processing sub-queue, which includes multiple sub-task nodes corresponding to each task node. The task processor 102 obtains the estimated completion time of each sub-task node in the second processing sub-queue. The estimated completion time of any one sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node. Furthermore, after obtaining the estimated time, the task processor 102 can send a first notification message, which includes the estimated completion time of each sub-task node. The first notification message is used to instruct the processor executing the task to delete the first target sub-task node, where the first target sub-task node is the sub-task node on the processor executing the task whose estimated completion time is later than the estimated completion time in the first notification message.

[0048] In one embodiment, as Figure 2 shown, a clustering scheduling method based on replication and redundancy removal is provided. Taking the task processor 102 in Figure 1 as an example, the method includes the following steps:

[0049] S202, obtain a first processing sub-queue for processing the target task, where the first processing sub-queue includes multiple task nodes of the reduced directed acyclic graph of the target task.

[0050] In this embodiment, the reduced directed acyclic graph of the target task is obtained by clustering the task nodes in the partitioned directed acyclic graph of the target task. The task nodes in the partitioned directed acyclic graph of the target task are obtained by partitioning the sub-task nodes in the original directed acyclic graph of the target task. In this partitioning process, the number of partitions can be controlled according to the number of processors executing the task.

[0051] Among them, when obtaining the partitioned directed acyclic graph based on the original directed acyclic graph of the target task, it is necessary to ensure that there are no loops in the original directed acyclic graph of the target task, so that there will be no duplicate task nodes among the sub-task nodes of the reduced directed acyclic graph.

[0052] Exemplarily, as Figure 3As shown, a schematic diagram of a reduced directed acyclic graph for a target task is provided. Different shapes represent different task nodes. Specifically, the reduced directed acyclic graph includes task nodes 1 to 5.

[0053] Based on Figure 3 the schematic diagram shown above, as Figure 4 shown, a schematic diagram of a queue for processing the target task is provided. Among them, the queue for processing the target task includes queue 1 and queue 2. Queue 1 includes task nodes 1 and 4, and queue 2 includes task nodes 1, 2, 3, and 5. The task processor can determine one of queue 1 or queue 2 as the first processing sub-queue.

[0054] S204, based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, decompose the first processing sub-queue into a second processing sub-queue, and the second processing sub-queue includes multiple sub-task nodes corresponding to each task node.

[0055] In this embodiment, the reduced directed acyclic graph of the target task is obtained by clustering the task nodes in the partitioned directed acyclic graph of the target task. Therefore, the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task can be determined. In this way, based on this correspondence, the first processing sub-queue can be decomposed into a second processing sub-queue, and the number of task nodes in the first processing sub-queue is less than the number of task nodes in the second processing queue.

[0056] Exemplarily, as Figure 5 shown, a schematic diagram of a partitioned directed acyclic graph of a target task is provided. Different shapes represent different sub-task nodes. Specifically, the partitioned directed acyclic graph includes sub-task nodes 1 to 15. After Figure 5 clustering the partitioned directed acyclic graph shown above, a reduced directed acyclic graph as shown in Figure 3 can be obtained; among them, the arrows between the sub-task nodes represent the dependency relationships between the sub-task nodes, and the numbers on the arrows between the sub-task nodes represent the communication delays, that is, the time when the task result corresponding to a certain sub-task node is sent to the sub-task node pointed to by the arrow.

[0057] Based on Figures 3 to 5 the content shown above, as Figure 6 shown, a schematic diagram of a queue for processing the target task is provided. Among them, by combining Figure 3 the reduced directed acyclic graph shown with Figure 5 the corresponding relationship of the partitioned directed acyclic graph shown, the Figure 4 sub-task nodes corresponding to the task nodes in can be obtained.

[0058] If the first processing sub - queue is Figure 4 the queue 1 in Figure 3 the abbreviated directed acyclic graph shown in Figure 5 and the corresponding relationship with the partitioned directed acyclic graph shown in Figure 3 the task node 1 in Figure 5 corresponds to the sub - task nodes 1, 3, 4, and 5 in Figure 3 the task node 4 in Figure 5 corresponds to the sub - task nodes 2, 8, and 12 in Figure 4 then the second processing queue obtained by decomposing the queue 1 in

[0059] Figure 4 If the first processing sub - queue is Figure 4 the queue 2 in Figure 3 the abbreviated directed acyclic graph shown in Figure 5 and the corresponding relationship with the partitioned directed acyclic graph shown in Figure 3 the task node 1 in Figure 5 corresponds to the sub - task nodes 1, 3, 4, and 5 in Figure 3 the task node 2 in Figure 5 corresponds to the sub - task node 7 in Figure 3 the task node 3 in Figure 5 corresponds to the sub - task nodes 7 and 11 in Figure 3 the task node 5 in Figure 5 corresponds to the sub - task nodes 9, 10, 13, 14, and 15 in

[0060] Among them, the execution order of the sub - tasks corresponding to the sub - task nodes in the second processing sub - queue can be set according to the actual application scenario, and this embodiment does not make a limitation.

[0061] S206. Obtain the estimated completion time of each sub - task node in the second processing sub - queue. The estimated completion time of any sub - task node is the sum of the estimated completion time of the previous adjacent sub - task node of the sub - task node in the second processing sub - queue and the estimated processing time of the sub - task node.

[0062] In this embodiment, since there are dependencies among the sub-task nodes in the partitioned directed acyclic graph of the target task, that is, after each sub-task node completes the corresponding task calculation, it is necessary to send the corresponding task result to the sub-task node with which it has a dependency. Therefore, when calculating the estimated completion time of each sub-task node in the second processing sub-queue, it is necessary to combine the estimated completion times of the previous adjacent sub-nodes of each sub-task node. In this way, by combining the estimated completion times of the previous adjacent sub-nodes of each sub-task node and the estimated processing time of the sub-task node, the estimated completion time of the sub-task node can be obtained.

[0063] Combined with Figure 5 , sub-task nodes 1 to 15 respectively correspond to v1 to v15. The node weight of the sub-task node is the estimated processing time of the sub-task node. The node weight of the sub-task can also be understood as the execution time of the sub-task corresponding to the sub-task node on the processor. Moreover, the node weight of the sub-task node will not change due to the cluster. Specifically, as shown in Table 1.

[0064] Table 1

[0065]

[0066] If the sub-task nodes executed in the second processing sub-queue are sub-task node 1, sub-task node 2, sub-task node 3, sub-task node 8, sub-task node 5, sub-task node 4, and sub-task node 12, taking sub-task node 3 as an example, the previous adjacent sub-task node of sub-task node 3 is 2. According to Table 1, the estimated completion time of sub-task node 3 is 88, and the estimated completion time of sub-task node 2 is 129. Then the estimated completion time of sub-task node 3 is 217; among them, the communication delay between the task nodes in the same processing sub-queue can be ignored.

[0067] S208, send the first notification message. The first notification message includes the estimated completion time of each sub-task node. The first notification message is used to instruct the processor executing the task to delete the first target sub-task node. The first target sub-task node is the sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message.

[0068] It can be understood that by deleting the first target sub-task node, on the one hand, the global completion time of processing the target task can be reduced, and on the other hand, it can be ensured that there will be no duplicate first target sub-task nodes on different processors, thereby improving the scheduling efficiency.

[0069] It should be noted that the task processor may directly send the first notification message to the processor executing the task, or the task processor may also send the first notification message to the target processor. When the target processor receives the estimated completion time of the subtask node in the processing queue from the processor executing the task, by comparing it with the estimated completion time of the subtask node in the first notification message, after determining that the subtask node later than the estimated completion time in the first notification message is the first target subtask node, it can directly send a message for instructing to delete the first target subtask node to the processor executing the task, or the task processor may also send the first notification message to the target processor, and the target processor sends the first notification message to the processor executing the task. In this way, the processor executing the task can delete the first target subtask node based on the first notification message.

[0070] In summary, in Figure 2 the embodiment shown, by obtaining the first processing sub-queue for processing the target task, since the first processing sub-queue includes multiple task nodes of the reduced directed acyclic graph of the target task, based on the corresponding relationship between the task nodes of the reduced directed acyclic graph and the subtask nodes of the partitioned directed acyclic graph of the target task, the first processing sub-queue can be decomposed into a second processing sub-queue. The second processing sub-queue includes multiple subtask nodes corresponding to each task node. Furthermore, by obtaining the estimated completion time of each subtask node in the second processing sub-queue, a first notification message can be sent. The first notification message includes the estimated completion time of each subtask node, and the first notification message is used to instruct the processor executing the task to delete the first target subtask node. The first target subtask node is the subtask node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message. In this way, when processing the corresponding subtask nodes based on the task processor and the processor executing the task, it is possible to avoid processing the same subtask node on different processors, thereby improving the scheduling efficiency.

[0071] In Figure 2 the embodiment shown, as Figure 7 shown, a flowchart of obtaining the first processing sub-queue for processing the target task is provided. Taking the method applied to Figure 1 the task processor 102 therein as an example for illustration, it includes the following steps:

[0072] S702, obtain the reduced directed acyclic graph of the target task.

[0073] In this embodiment, the target task is modeled as an original directed acyclic graph. After partitioning the subtask nodes of the original directed acyclic graph to obtain a partitioned directed acyclic graph, a cluster-based task scheduling method is used to cluster the task nodes in the partitioned directed acyclic graph, so as to obtain a reduced directed acyclic graph of the target task. Moreover, the cluster-based task scheduling method can reduce the complexity of task processing. Among them, the thumbnail of the target task can be as shown in Figure 3 the schematic diagram shown.

[0074] S704. Divide the task nodes in the reduced directed acyclic graph to obtain two or more queues.

[0075] In this embodiment, by dividing the task nodes in the reduced directed acyclic graph, two or more queues containing multiple subtasks corresponding to the target task can be allocated to different processors for executing tasks. On the one hand, the computing pressure on the processor can be reduced. On the other hand, the task processing efficiency can be improved.

[0076] S706. Determine one of the two or more queues as the first processing sub-queue.

[0077] In summary, in the embodiment shown in Figure 7 by obtaining the reduced directed acyclic graph of the target task, the task nodes in the reduced directed acyclic graph can be divided to obtain two or more queues, and one of the two or more queues is determined as the first processing sub-queue. Then, the task processor performs subsequent processes based on the first processing sub-queue to improve the scheduling efficiency.

[0078] In the embodiment shown in Figure 7 as shown in Figure 8 a schematic flowchart of dividing the task nodes in the reduced directed acyclic graph to obtain two or more queues is provided. Taking the method applied to the task processor 102 in Figure 1 as an example, the steps are as follows:

[0079] S802. If there are multiple leaf task nodes in the reduced directed acyclic graph, allocate each leaf task node to the first queue respectively. Each leaf task node is a task node with an out-degree of 0.

[0080] S804. Determine the critical predecessor task nodes corresponding to each leaf task node, and the multiple predecessor task nodes of the corresponding critical predecessor task nodes, and allocate the corresponding critical predecessor task nodes and the multiple predecessor task nodes of the corresponding critical predecessor task nodes to the corresponding first queue.

[0081] In this embodiment, one of the task nodes pointing to the leaf task node is the critical predecessor task node of the leaf task node, and all the task nodes pointing to the critical predecessor task node are the multiple predecessor task nodes of the critical predecessor task node.

[0082] For example, in combination with Figure 3 , the number of leaf task nodes in the reduced directed acyclic graph is 1. The leaf task node in the reduced directed acyclic graph is task node 5. The task nodes pointing to task node 5 include task node 3 and task node 4. One of task node 3 and task node 4 is the critical predecessor task node of task node 5; if the critical predecessor task node of task node 5 is task node 3, all the task nodes pointing to task node 3 are task node 1 and task node 2. Therefore, the predecessor task nodes of task node 3 are task node 1 and task node 2, and then task node 1, task node 2, task node 3, and task node 5 are allocated in the first queue.

[0083] S806, if there is no leaf task node in the reduced directed acyclic graph, obtain multiple task nodes in the reduced directed acyclic graph that are not allocated to a queue.

[0084] S808, allocate each task node that is not allocated to a processing queue in the second queue. Two or more queues include the corresponding first queue and the second queue.

[0085] In this embodiment, based on the leaf task nodes in the reduced directed acyclic graph, multiple task nodes in the reduced directed acyclic graph can be allocated to the first queue.

[0086] For example, in combination with Figure 3 , the leaf task node is task node 5, and task node 1, task node 2, task node 3, and task node 5 are allocated in the first queue. Multiple task nodes that are not allocated to a queue can also be allocated in the second queue. For example, in combination with Figure 3 , the multiple task nodes in the reduced directed acyclic graph that are not allocated to a queue are task node 1 and task node 4, and task node 1 and task node 4 are allocated in the second queue. In this way, the task processor can determine one of the first queue and the second queue as the first processing sub-queue.

[0087] In Figure 8 the shown embodiment, as Figure 9 shown, a flow diagram for determining the critical predecessor task node of the leaf task node is provided. Taking the method applied to the task processor 102 in Figure 1 as an example, it includes the following steps:

[0088] S902, calculate the earliest completion time of multiple predecessor task nodes of the leaf task node.

[0089] In this embodiment, the earliest completion time is the sum of the earliest start times of each predecessor task node and the estimated processing times of each predecessor task node, and the earliest start time is the time with the latest estimated completion time among the multiple predecessor task nodes of each predecessor task node.

[0090] For example, in combination with Figure 3 , task nodes 1 to 5 respectively correspond to node weights, and the node weight of a task node is the estimated processing time of the task node. Specifically, as shown in Table 2.

[0091] Table 2

[0092] Task Node 1 2 3 4 5 Node Weight 233 80 144 204 248

[0093] Specifically, the multiple predecessor task nodes of the leaf task node 5 are respectively task node 3 and task node 4, and the multiple predecessor task nodes of task node 3 are respectively task node 1 and task node 2; the estimated completion time of task node 1 is 233, and the estimated completion time of task node 2 is the sum of the estimated processing time of task node 1 and the estimated processing time of task node 2, that is, the estimated completion time of task node 2 is 313. Since the estimated completion time of task node 2 is greater than the estimated completion time of task node 1, the earliest start time of task node 3 is the estimated completion time of task node 2, that is, the earliest start time of task node 3 is 313, and the earliest completion time of task node 3 is 457; the predecessor task node of task node 4 is only task node 1, so the estimate of task node 4 is the estimated processing time of task node 1, that is, the earliest start time of task node 4 is 233, and the earliest completion time of task node 4 is 437.

[0094] S904, determine the predecessor task node with the maximum earliest completion time among each predecessor task node as the critical predecessor task node.

[0095] For example, in combination with Figure 3 , the multiple predecessor task nodes of the leaf task node 5 are respectively task node 3 and task node 4, the earliest completion time of task node 3 is 457, the earliest completion time of task node 4 is 437, and the earliest completion time of task node 3 is greater than the earliest completion time of task node 4, so task node 3 is the critical predecessor task node of the leaf task node 5.

[0096] In summary, in the Figure 9 shown embodiment, by calculating the earliest completion times of the multiple predecessor task nodes of the leaf task node, it is possible to determine the predecessor task node with the maximum earliest completion time among each predecessor task node as the critical predecessor task node. In this way, based on the critical predecessor task node of the leaf task node, processing queues can be allocated to each task node in the reduced directed acyclic graph.

[0097] Based on Figure 8 and Figure 9 the embodiments shown, as Figure 10 shown, a schematic diagram of allocating queues to each task node in a reduced directed acyclic graph is provided. Specifically, by obtaining the leaf task nodes in the reduced directed acyclic graph, when it is determined that there are leaf task nodes, the leaf task nodes can be allocated in the first queue. After determining the critical predecessor task nodes of the leaf task nodes, the critical predecessor task nodes are also allocated in the first queue. At the same time, continue to obtain the leaf task nodes in the reduced directed acyclic graph. If there are leaf task nodes, perform the same allocation process as the previous leaf task node. If there are no leaf task nodes, obtain the task nodes in the unallocated queue and allocate the unallocated queue in the second queue.

[0098] Among them, if the target task is a large task, the number of leaf task nodes in the reduced directed acyclic graph of the target task can be multiple. Different first queues corresponding to different leaf task nodes can be specifically set according to the actual target task, and this embodiment does not make a limitation.

[0099] Based on Figure 2 the embodiments shown, in one embodiment, obtaining the first processing sub-queue for processing the target task includes:

[0100] Receiving the first processing sub-queue sent by the target processor; wherein, the first processing sub-queue is one of the two or more queues obtained by the target processor by obtaining the reduced directed acyclic graph of the target task and partitioning the task nodes in the reduced directed acyclic graph.

[0101] Among them, the implementation manner for the target processor to obtain two or more queues can refer to the implementation manner for the task processor to obtain two or more queues, which will not be elaborated here.

[0102] Based on Figure 2 the embodiments shown, in one embodiment, as Figure 11 shown, a schematic diagram of a clustering scheduling method based on replication and redundancy removal is provided. Taking this method applied to Figure 1 the task processor 102 therein as an example for illustration, it includes the following steps:

[0103] S1102, receiving the second notification information, where the second notification information includes the estimated completion time of each sub-task node in the processing sub-queue executed by the processor executing the task.

[0104] S1104, Determine whether there is a second target sub - task node. The second target sub - task node is a sub - task node in the second processing sub - queue whose estimated completion time is later than the estimated completion time in the second notification message.

[0105] S1106, If there is a second target sub - task node, delete the second target sub - task node in the second processing sub - queue to obtain an updated second processing sub - queue.

[0106] It should be noted that the task processor can receive the second notification message sent by the target processor. The target processor receives the second notification message sent by the processor executing the task. Or, the task processor can directly receive the second notification information sent by the processor executing the task. This embodiment does not make specific limitations.

[0107] In summary, in Figure 11 the illustrated embodiment, the task processor can determine whether there is a second target sub - task node by receiving the second notification message sent by the processor executing the task. If there is a second target sub - task node, delete the second target sub - task node in the second processing sub - queue to obtain an updated second processing sub - queue. In this way, the task processor deletes the second target sub - task node, so that the second target sub - task node only exists in the processor executing the task, which can avoid processing the same sub - task node on different processors, thereby improving the scheduling efficiency.

[0108] Based on the embodiment shown in Figure 11 in one of the embodiments, as Figure 12 shown, a flowchart of a clustering scheduling method based on replication and redundancy removal is provided. Taking this method applied to Figure 1 the task processor 102 as an example, it includes the following steps:

[0109] S1202, Obtain the estimated completion time of each sub - task node in the updated second processing sub - queue.

[0110] S1204, Send a third notification message. The third notification message includes the estimated completion time of each sub - task node in the updated second processing sub - queue. The third notification message is used to instruct the processor executing the task to delete the third target sub - task node. The third target sub - task node is a sub - task node on the processor executing the task whose estimated completion time is later than the estimated completion time in the third notification message.

[0111] It should be noted that the task processor may directly send a third notification message to the processor that executes the task, or the task processor may also send a third notification message to the target processor. When the target processor receives the estimated completion time of the subtask node in the processing queue from the processor that executes the task, by comparing it with the estimated completion time of the subtask node in the third notification message, after determining that the subtask node whose estimated completion time is later than that in the third notification message is the second target subtask node, it can directly send a message for instructing to delete the second target subtask node to the processor that executes the task, or the task processor may also send a third notification message to the target processor, and the target processor sends the third notification message to the processor that executes the task. In this way, the processor that executes the task can delete the second target subtask node based on the third notification message.

[0112] In summary, in Figure 12 the illustrated embodiment, after the task processor obtains the estimated completion time of each subtask node in the updated second processing subqueue, it can send a third notification message. Since the estimated completion time of each subtask node in the updated second processing subqueue and the third notification message are used to instruct the processor that executes the task to delete the third target subtask node, and the third target subtask node is the subtask node whose estimated completion time on the processor that executes the task is later than the estimated completion time in the third notification message. In this way, the processor that executes the task can delete the subtask node whose estimated completion time is later than the estimated completion time in the third notification message, so that the second target subtask node only exists in the task processor, and it is possible to avoid having the same first target subtask node on different processors, thereby improving the scheduling efficiency.

[0113] Combined with Figures 2 to 12 the content shown, as Figure 13 shown, a schematic diagram of processing the subtask nodes of the target task is provided. Among them, queue 1 is the queue on the task processor, and queue 2 is the processor that executes the task. It can be seen that there are no duplicate subtask nodes on the task processor and the processor that executes the task. Therefore, the scheduling efficiency can be improved.

[0114] In one of the embodiments, as Figure 14 shown, a flowchart of a clustering scheduling method based on replication and redundancy elimination is provided. Taking the method applied to Figure 1 the target processor 104 therein as an example for illustration, it includes the following steps:

[0115] S1402, send the first processing subqueue for processing the target task to the task processor; the first processing subqueue includes multiple task nodes of the abbreviated directed acyclic graph of the target task.

[0116] S1404, receive the first notification message; the first notification message includes the estimated completion time of each sub-task node in the second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor decomposing the first processing sub-queue based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task.

[0117] S1406, send the first notification message to the target task processor, where the target task processor includes at least one task processor with at least one same sub-task node as the second processing sub-queue, and the first notification message is used to instruct the target task processor to delete the first target sub-task node, where the first target sub-task node is the sub-task node on the target task processor whose estimated completion time is later than the estimated completion time in the first notification message.

[0118] In this embodiment, the specific content of S1402 - S1406 can refer to the foregoing content for adaptation and description, and will not be elaborated again here; among them, combined with Figure 1 , the target task processor can be Figure 1 one of the task processors in

[0119] It can be understood that in this embodiment, if the target processor is one of the task processors, after the target processor determines the processing sub-queue for processing the target task, the target processor can send the first processing sub-queue for processing the target task to other processors in the task processor. At the same time, the target processor can also execute other queues for processing the target task. The specific content of the queue executed by the target processor is not limited in this embodiment.

[0120] In summary, in Figure 14In the illustrated embodiment, the target processor sends a first processing sub-queue for processing a target task to the task processor, enabling the task processor to summarize the correspondence between the task nodes of the directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task. After decomposing the first processing sub-queue to obtain a second processing queue, the task processor can obtain the estimated completion times of the sub-task nodes in the second processing sub-queue and send a first notification message to the target processor. Since the first notification information includes the estimated completion times of the sub-task nodes in the second processing sub-queue of the task processor, in this way, the target processor receives the estimated completion times of the sub-task nodes in the queue sent by the executing task processor. By comparing with the estimated completion times of the sub-task nodes in the second processing sub-queue in the first notification message, the target processor can determine the target task processor where the first target sub-task node that is later than the estimated completion time in the first notification information is located and send the first notification information to the target task processor, enabling the target task processor to delete the first target sub-task node, so that the first target sub-task node only exists in the task processor, which can avoid the existence of the same first target sub-task node on different processors, thereby improving the scheduling efficiency.

[0121] Based on Figures 2 to 14 the content shown above, as Figure 15 shown, a schematic diagram of a clustering scheduling method based on replication and redundancy elimination is provided. Among them, after partitioning the sub-task nodes in the original directed acyclic graph of the target task, a partitioned directed acyclic graph can be obtained. After clustering the partitioned directed acyclic graph, a reduced directed acyclic graph can be obtained. Furthermore, based on the leaf task nodes in the reduced directed acyclic graph, queues are assigned to the task nodes in the reduced directed acyclic graph. And based on the correspondence between the task nodes in the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph, the task nodes in the assigned queues can be decomposed, so that the decomposed queues include sub-task nodes. Then, based on the estimated completion times of the sub-task nodes in the decomposed queues, the duplicate sub-task nodes on one of the executing task processors are deleted. The specific content can refer to the foregoing description and will not be elaborated here.

[0122] Combined with the above content, it should be noted that in the clustering scheduling method based on replication and redundancy removal provided in this application, a task scheduling method based on clusters and a task scheduling method based on replication are combined. That is, by partitioning the subtask nodes in the original directed acyclic graph of the target task, the complexity of task processing can be reduced, but it will lead to a loss of scheduling efficiency. Moreover, generally, when dealing with large-scale task scheduling, the number of tasks is much larger than the number of available processors. Therefore, in this application, by clustering the subtask nodes in the partitioned directed acyclic graph, and based on the correspondence between the task nodes in the reduced directed acyclic graph and the subtask nodes in the partitioned directed acyclic graph of the target task, mapping the task nodes in the reduced directed acyclic graph back to the subtask nodes in the partitioned directed acyclic graph, and by deleting duplicate subtask nodes in other queues based on the estimated processing time of each subtask node, the efficiency of task scheduling can be improved on a limited number of processors, achieving an ideal scheduling effect and avoiding waste of computing resources.

[0123] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0124] Based on the same inventive concept, an embodiment of this application also provides a clustering scheduling device based on replication and redundancy removal for implementing the above-mentioned clustering scheduling method based on replication and redundancy removal. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the clustering scheduling device based on replication and redundancy removal provided below can refer to the limitations on the clustering scheduling method based on replication and redundancy removal in the above text, and will not be repeated here.

[0125] In one embodiment, as Figure 16 shown, a clustering scheduling device based on replication and redundancy removal is provided, including: a queue acquisition module 1602, a queue decomposition module 1604, a time acquisition module 1606, and a processing module 1608, where:

[0126] A queue acquisition module 1602 is configured to acquire a first processing sub-queue for processing a target task. The first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task. A queue decomposition module 1604 is configured to decompose the first processing sub-queue into a second processing sub-queue based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task. The second processing sub-queue includes multiple sub-task nodes corresponding to each task node. A time acquisition module 1606 is configured to acquire the estimated completion time of each sub-task node in the second processing sub-queue. The estimated completion time of any one sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node. A processing module 1608 is configured to send a first notification message. The first notification message includes the estimated completion time of each sub-task node. The first notification message is used to instruct the processor executing the task to delete a first target sub-task node. The first target sub-task node is a sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message.

[0127] In one embodiment, the queue acquisition module 1602 is further configured to acquire the reduced directed acyclic graph of the target task; divide the task nodes in the reduced directed acyclic graph to obtain more than two queues; and determine one of the more than two queues as the first processing sub-queue.

[0128] In one embodiment, the queue acquisition module 1602 is further configured to, if there are multiple leaf task nodes in the reduced directed acyclic graph, respectively allocate each leaf task node in the first queue. Each leaf task node is a task node with an out-degree of 0; determine the critical predecessor task nodes corresponding to each leaf task node, and multiple predecessor task nodes of the corresponding critical predecessor task nodes, and allocate the corresponding critical predecessor task nodes and the multiple predecessor task nodes of the corresponding critical predecessor task nodes in the corresponding first queue; if there are no leaf task nodes in the reduced directed acyclic graph, acquire multiple task nodes in the reduced directed acyclic graph that have not been allocated queues; allocate each task node without an allocated processing queue in the second queue; and the more than two queues include the corresponding first queue and the second queue.

[0129] In one embodiment, the queue acquisition module 1602 is further configured to calculate the earliest completion time of multiple predecessor task nodes of a leaf task node; the earliest completion time is the sum of the earliest start time of each predecessor task node and the estimated processing time of each predecessor task node, and the earliest start time is the latest completion time of multiple predecessor task nodes of each predecessor task node; and determine the predecessor task node with the maximum earliest completion time among each predecessor task node as the critical predecessor task node.

[0130] In one embodiment, the queue acquisition module 1602 is further configured to receive a first processing sub-queue sent by a target processor; wherein, the first processing sub-queue is one of more than two queues obtained by the target processor acquiring a reduced directed acyclic graph of a target task and partitioning task nodes in the reduced directed acyclic graph.

[0131] In one embodiment, the processing module 1608 is further configured to receive second notification information, where the second notification information includes the estimated completion time of each sub-task node in the processing sub-queue executed by the processor executing the task; determine whether there is a second target sub-task node, where the second target sub-task node is a sub-task node in the second processing sub-queue whose estimated completion time is later than the estimated completion time in the second notification information; if there is a second target sub-task node, delete the second target sub-task node in the second processing sub-queue to obtain an updated second processing sub-queue.

[0132] In one embodiment, the processing module 1608 is further configured to obtain the estimated completion time of each sub-task node in the updated second processing sub-queue; send third notification information, where the third notification information includes the estimated completion time of each sub-task node in the updated second processing sub-queue, and the third notification information is used to instruct the processor executing the task to delete a third target sub-task node, where the third target sub-task node is a sub-task node on the processor executing the task whose estimated completion time is later than the estimated completion time in the third notification information.

[0133] In one embodiment, as Figure 17 shown, a clustering scheduling device based on replication and redundancy removal is provided, including: a queue sending module 1702, an information receiving module 1704, and a message sending module 1706, where:

[0134] The queue sending module 1702 is configured to send a first processing sub-queue for processing a target task to a task processor; the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task.

[0135] The information receiving module 1704 is configured to receive first notification information; the first notification information includes the estimated completion time of each sub-task node in a second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor decomposing the first processing sub-queue based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task.

[0136] A message sending module 1706 is configured to send a first notification message to a target task processor. The target task processor includes at least a task processor having at least one same sub-task node as that in a second processing sub-queue. The first notification message is used to instruct the target task processor to delete a first target sub-task node, where the first target sub-task node is a sub-task node on the target task processor whose estimated completion time is later than the estimated completion time in the first notification message.

[0137] Each module in the above clustering scheduling device based on replication and redundancy removal can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in a processor in a computer device in a hardware form or be independent of the processor, or can be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to each of the above modules.

[0138] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 18 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Wherein, the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store notification messages and processing sub-queues. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a clustering scheduling method based on replication and redundancy removal.

[0139] Those skilled in the art can understand that Figure 18 the structure shown in

[0140] is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0141] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0142] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps in the above-described method embodiments.

[0143] Those of ordinary skill in the art can understand that all or part of the processes in the above-described method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-described method embodiments. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0144] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0145] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A clustering scheduling method based on replication and redundancy removal, characterized in that, Including: Obtain a first processing sub-queue for processing a target task, where the first processing sub-queue includes multiple task nodes of a reduced directed acyclic graph of the target task; The reduced directed acyclic graph of the target task is obtained by modeling the target task as an original directed acyclic graph, partitioning sub-task nodes of the original directed acyclic graph to obtain a partitioned directed acyclic graph, and then clustering the task nodes in the partitioned directed acyclic graph using a cluster-based task scheduling method; Based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, decompose the first processing sub-queue into a second processing sub-queue, where the second processing sub-queue includes multiple sub-task nodes corresponding to each of the task nodes; Obtain the estimated completion time of each of the sub-task nodes in the second processing sub-queue. The estimated completion time of any one sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node of the sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node; Send a first notification message, where the first notification message includes the estimated completion time of each of the sub-task nodes. The first notification message is used to instruct a processor executing a task to delete a first target sub-task node, and the first target sub-task node is a sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the first notification message.

2. The method according to claim 1, wherein The obtaining the first processing sub-queue for processing a target task includes: Obtain the reduced directed acyclic graph of the target task; Partition the task nodes in the reduced directed acyclic graph to obtain two or more queues; Determine one of the two or more queues as the first processing sub-queue.

3. The method according to claim 2, wherein The partitioning the task nodes in the reduced directed acyclic graph to obtain two or more queues includes: If there are multiple leaf task nodes in the reduced directed acyclic graph, allocate each of the leaf task nodes in a first queue, and each of the leaf task nodes is a task node with an out-degree of 0; Determine the critical predecessor task nodes corresponding to each of the leaf task nodes, and multiple predecessor task nodes of the corresponding critical predecessor task nodes, and allocate the corresponding critical predecessor task nodes and the multiple predecessor task nodes of the corresponding critical predecessor task nodes in the corresponding first queue; If there are no leaf task nodes in the reduced directed acyclic graph, obtain multiple task nodes in the reduced directed acyclic graph that have not been allocated a queue; Allocate each of the task nodes that have not been allocated a processing queue in a second queue; the two or more queues include the first queue and the second queue.

4. The method according to claim 3, wherein Determining the critical predecessor task nodes of the leaf task nodes includes: Calculate the earliest completion times of multiple predecessor task nodes of the leaf task node; the earliest completion time is the sum of the earliest start times of the predecessor task nodes and the estimated processing times of the predecessor task nodes, and the earliest start time is the latest completion time of multiple predecessor task nodes of each predecessor task node; Determine the predecessor task node with the maximum earliest completion time among the predecessor task nodes as the critical predecessor task node of the leaf task node.

5. The method according to claim 1, wherein The obtaining of the first processing sub-queue for processing the target task includes: Receiving the first processing sub-queue sent by the target processor; wherein, the first processing sub-queue is one of more than two queues obtained by the target processor by obtaining the reduced directed acyclic graph of the target task and partitioning the task nodes in the reduced directed acyclic graph.

6. The method according to claim 1, characterized in that, The method further includes: Receiving second notification information, where the second notification information includes the estimated completion times of each sub-task node in the processing sub-queue executed by the processor executing the task; Determine whether there is a second target sub-task node, where the second target sub-task node is a sub-task node in the second processing sub-queue whose estimated completion time is later than the estimated completion time in the second notification information; If there is the second target sub-task node, delete the second target sub-task node in the second processing sub-queue to obtain an updated second processing sub-queue.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the estimated completion times of each sub-task node in the updated second processing sub-queue; Send third notification information, where the third notification information includes the estimated completion times of each sub-task node in the updated second processing sub-queue, and the third notification information is used to instruct the processor executing the task to delete the third target sub-task node, where the third target sub-task node is a sub-task node whose estimated completion time on the processor executing the task is later than the estimated completion time in the third notification information.

8. A clustering scheduling method based on replication and redundancy removal, characterized in that, Includes: Sending a first processing sub-queue for processing the target task to the task processor; the first processing sub-queue includes multiple task nodes of the reduced directed acyclic graph of the target task; The reduced directed acyclic graph of the target task is obtained by modeling the target task as an original directed acyclic graph, partitioning the sub-task nodes of the original directed acyclic graph to obtain a partitioned directed acyclic graph, and then clustering the task nodes in the partitioned directed acyclic graph using a cluster-based task scheduling method; Receiving first notification information; the first notification information includes the estimated completion times of each sub-task node in the second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor decomposing the first processing sub-queue based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task; the estimated completion time of any sub-task node is the sum of the estimated completion time of the pre-adjacent sub-task node of the sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node; Send the first notification message to the target task processor, where the target task processor includes at least a task processor having at least one same sub-task node as the second processing sub-queue. The first notification message is used to instruct the target task processor to delete the first target sub-task node, and the first target sub-task node is a sub-task node on the target task processor whose estimated completion time is later than the estimated completion time in the first notification message.

9. A clustering scheduling device based on replication and redundancy removal, characterized in that Comprising: A queue acquisition module, configured to acquire a first processing sub-queue for processing a target task, where the first processing sub-queue includes a plurality of task nodes of a reduced directed acyclic graph of the target task; the reduced directed acyclic graph of the target task is obtained by modeling the target task as an original directed acyclic graph, partitioning the sub-task nodes of the original directed acyclic graph to obtain a partitioned directed acyclic graph, and then clustering the task nodes in the partitioned directed acyclic graph by using a cluster-based task scheduling method; A queue decomposition module, configured to decompose the first processing sub-queue into a second processing sub-queue based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task, where the second processing sub-queue includes a plurality of sub-task nodes corresponding to each of the task nodes; A time acquisition module, configured to acquire the estimated completion time of each of the sub-task nodes in the second processing sub-queue, and the estimated completion time of any one sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node of the sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node; A processing module, configured to send a first notification message, where the first notification message includes the estimated completion time of each of the sub-task nodes, and the first notification message is used to instruct the processor executing the task to delete the first target sub-task node, and the first target sub-task node is a sub-task node on the processor executing the task whose estimated completion time is later than the estimated completion time in the first notification message.

10. A clustering scheduling device based on replication and redundancy removal, characterized in that, Comprising: A queue sending module, configured to send a first processing sub-queue for processing a target task to a task processor; the first processing sub-queue includes a plurality of task nodes of a reduced directed acyclic graph of the target task; the reduced directed acyclic graph of the target task is obtained by modeling the target task as an original directed acyclic graph, partitioning the sub-task nodes of the original directed acyclic graph to obtain a partitioned directed acyclic graph, and then clustering the task nodes in the partitioned directed acyclic graph by using a cluster-based task scheduling method; An information receiving module, configured to receive a first notification message; the first notification message includes the estimated completion time of each sub-task node in the second processing sub-queue of the task processor, and the second processing sub-queue is obtained by the task processor based on the correspondence between the task nodes of the reduced directed acyclic graph and the sub-task nodes of the partitioned directed acyclic graph of the target task; the estimated completion time of any sub-task node is the sum of the estimated completion time of the previous adjacent sub-task node of the sub-task node in the second processing sub-queue and the estimated processing time of the sub-task node. A message sending module, configured to send the first notification message to a target task processor, where the target task processor includes at least a task processor having at least one same sub-task node as the second processing sub-queue, and the first notification message is used to instruct the target task processor to delete a first target sub-task node, where the first target sub-task node is a sub-task node on the target task processor whose estimated completion time is later than the estimated completion time in the first notification message.

Images