A workflow parallel scheduling method and system based on a dynamic resource selection strategy

Abstract

The application belongs to the field of workflow intelligent scheduling, and more particularly to a workflow parallel scheduling method and system based on a dynamic resource selection strategy. The timing relationship and task load information between tasks are accurately represented by constructing a DAG task flow graph, and the nodes of the DAG task flow graph are aggregated by a graph convolutional neural network to obtain a DAG task flow subgraph that can reflect the characteristics of each task of each independent task path. The target comprehensive feature vector that can fuse the individual task features, independent task path features and global features of the DAG task flow graph is obtained by performing DAG feature aggregation and global feature aggregation operations on each DAG task flow subgraph, and is sent to a deep reinforcement learning network in the form of an embedded vector together with system environment information, so that the dynamic adaptability of the generated workflow scheduling scheme to the computing resources and task load and the comprehensive resource utilization rate can be effectively improved.

Description

Technical Field

[0001] This application belongs to the field of intelligent workflow scheduling, and more specifically, relates to a workflow parallel scheduling method and system based on a dynamic resource selection strategy. Background Technology

[0002] With the rapid development of cloud computing, edge computing, and high-performance computing, modern computing systems exhibit highly heterogeneous characteristics, integrating various types of computing components (computing resources) such as CPUs, GPUs, and FPGAs. In these complex heterogeneous computing systems, how to efficiently schedule workflow tasks with complex dependencies in parallel to shorten workflow execution time and improve computing resource utilization has become a key research focus in operating system workflow task scheduling methods.

[0003] Currently, existing workflow scheduling methods are mainly divided into two categories: target exploration algorithms based on preset rules (such as the African vulture algorithm) and optimization algorithms based on static clustering (DAG clustering based on centroids). However, in real-world complex work environments, the operating status of computing resources (such as CPU utilization, memory usage, and disk I / O resource utilization) is dynamically changing. Target exploration algorithms based on preset rules cannot promptly obtain the specific changes in the operating status of computing resources caused by fluctuations in the work environment, making it difficult to timely schedule the allocation of computing resources for each task in the workflow.

[0004] Furthermore, workflow scheduling methods based on static clustering optimization algorithms typically assume that task execution times and dependencies are known and fixed. The allocation of computing resources is calculated based on predefined node priorities, making it difficult to dynamically capture changes in the workflow graph's topology. This can lead to situations where some nodes are overloaded while others remain idle during resource allocation, resulting in reduced overall operating system computing resource utilization. Therefore, existing workflow scheduling methods suffer from poor dynamic adaptability to computing resources and task loads, and low overall resource utilization, making it difficult to meet the requirements for efficient dynamic parallel scheduling of workflow tasks on heterogeneous computing systems. Summary of the Invention

[0005] To address the aforementioned deficiencies in existing technologies, this application provides a workflow parallel scheduling method and system based on a dynamic resource selection strategy, aiming to solve the problems of reduced dynamic adaptability to computing resources and task load, and reduced overall resource utilization caused by changes in the working environment under complex working conditions.

[0006] Firstly, this application provides a workflow parallel scheduling method based on a dynamic resource selection strategy, including: S1. Based on the temporal dependencies between tasks in the workflow, construct a DAG task flow graph. Each node of the DAG task flow graph can represent a task and the task load information of the task, and each edge represents the temporal relationship between two tasks. S2. Based on the graph convolutional neural network, node features are aggregated for each node to obtain multiple DAG task flow subgraphs. DAG feature aggregation and global feature aggregation operations are performed on each DAG task flow subgraph in sequence. The results of each DAG task flow subgraph, DAG feature aggregation and global feature aggregation operations are converted into target comprehensive feature vectors. S3. Obtain the system environment information of the target operating system, and input the environment information and the target comprehensive feature vector into the trained deep reinforcement learning network. The system environment information includes the real-time operating parameters of each computing resource. S4. Constrain the decision-making process of the deep reinforcement learning network through the objective constraint algorithm to obtain the objective workflow scheduling scheme. The objective workflow scheduling scheme includes the execution path, execution time and scheduling results of computing resource allocation for each task.

[0007] In this process, a graph convolutional neural network is used to aggregate node features for each node. This is because the DAG task flow graph is a directed acyclic graph, and the graph convolutional neural network can directly extract features from the task load information and the temporal relationship between tasks. At the same time, it can adapt to the dynamically changing workflow structure. That is, when tasks are added, deleted, or the temporal relationship is adjusted in the workflow, the node features, local features of each independent path, and global features of the acquired DAG task flow graph can be updated in real time.

[0008] Furthermore, based on graph convolutional neural networks, node features are aggregated to obtain multiple DAG task flow subgraphs, including: S11. Based on the topological structure between nodes of the DAG task flow graph, the task load information corresponding to the child node without a successor node is transmitted to the adjacent parent node in the form of an embedding vector through the graph neural network, and the feature is fused with the task load information of the parent node to obtain the embedding vector embedded in the parent node. S12. After fusing the node information features, the parent node is used as the child node. Repeat step S11 until each node in the DAG task flow graph has a corresponding embedding vector. Then, the branches corresponding to each node in the DAG task flow graph that do not have temporal relationships are used as DAG task flow subgraphs.

[0009] Furthermore, DAG feature aggregation operations are performed sequentially on each DAG task flow subgraph, including: Add a local virtual parent node to each DAG task flow subgraph. The local virtual parent node is the common parent node of all nodes in the DAG task flow subgraph. The task load information of the parent node of each DAG task flow subgraph is passed to the corresponding local virtual parent node of the DAG task flow subgraph in the form of an embedding vector. The embedding vectors passed to the local virtual parent node are then fused to obtain the embedding vector embedded in the local virtual parent node.

[0010] Furthermore, DAG feature aggregation and global feature aggregation operations are sequentially performed on each DAG task flow subgraph. The results of each DAG task flow subgraph, DAG feature aggregation, and global feature aggregation operations are then converted into a target comprehensive feature vector, including: Add a common global virtual parent node to all local virtual parent nodes, and fuse the embedding vectors of each local virtual parent node to obtain the embedding vector of the global virtual parent node. The embedding vectors of all nodes, local virtual parent nodes, and global virtual parent nodes in each DAG task flow graph are converted into target comprehensive feature vectors.

[0011] Furthermore, the deep reinforcement learning network is an Actor-Critic policy network. The Actor network of the deep reinforcement learning network is used to generate workflow scheduling schemes. The Actor network includes multiple sub-decision networks, and the types of sub-decision networks include task scheduling sub-networks, CPU resource scheduling sub-networks, memory resource scheduling sub-networks, and network bandwidth resource scheduling sub-networks.

[0012] Furthermore, the backbone networks of both the Actor and Critic networks in deep reinforcement learning networks consist of Transformer layers and MoE layers, while the sub-decision networks of the Actor network are MLP networks.

[0013] Furthermore, the decision-making process of the deep reinforcement learning network is constrained through an objective constraint algorithm, including: The process of generating workflow scheduling schemes by constraining deep reinforcement learning networks based on the EFT algorithm yields a first workflow scheduling scheme with the objective of minimizing the independent execution time of each task. The process of generating workflow scheduling schemes based on the CPEFT algorithm constrains the deep reinforcement learning network, resulting in a second workflow scheduling scheme with the objective of minimizing the execution time of tasks corresponding to key nodes in the DAG task flow graph, where each key node has at least two child nodes. The workflow scheduling scheme with the shortest total task execution time between the first and second workflow scheduling schemes will be selected as the target workflow scheduling scheme.

[0014] Secondly, this application also provides a workflow parallel scheduling system based on a dynamic resource selection strategy for executing any of the methods in the first aspect, including: The task flow graph construction module is used to construct a DAG task flow graph based on the temporal dependencies between tasks in the workflow. The feature aggregation module is used to aggregate node features based on graph convolutional neural networks to obtain multiple DAG task flow subgraphs, and to perform DAG feature aggregation and global feature aggregation operations on each DAG task flow subgraph in sequence. The feature vector acquisition module is used to convert the results of each DAG task flow subgraph, DAG feature aggregation, and global feature aggregation operations into a target comprehensive feature vector. The feature vector concatenation module is used to obtain the environmental feature vector corresponding to the system environment information of the target operating system, and to concatenate the environmental feature vector with the target comprehensive feature vector. The scheduling scheme acquisition module is used to input the feature concatenation result into the trained deep reinforcement learning network, and to constrain the decision-making process of the deep reinforcement learning network through the target constraint algorithm to obtain the target workflow scheduling scheme.

[0015] Thirdly, this application also provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0017] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art: 1. This application accurately represents the temporal relationships and task load information between tasks by constructing a DAG task flow graph. A graph convolutional neural network is used to aggregate nodes in the DAG task flow graph, resulting in a DAG task flow subgraph that reflects the characteristics of each task along its independent path. By performing DAG feature aggregation and global feature aggregation operations on each DAG task flow subgraph, a target comprehensive feature vector is obtained that integrates individual task features, independent task path features, and global features of the DAG task flow graph. This vector, along with system environment information, is sent to a deep reinforcement learning network in the form of an embedded vector. This allows the deep reinforcement learning network to dynamically and accurately acquire the real-time features of the DAG task flow graph and the real-time operating parameters of each system's computing resources, thereby effectively improving the dynamic adaptability of the generated workflow scheduling scheme in terms of computing resources and task load, as well as the overall resource utilization rate.

[0018] 2. By introducing EFT and CPEFT algorithms respectively to constrain the process of deep reinforcement learning network generating workflow scheduling schemes, the first and second workflow scheduling schemes are obtained. Then, the scheme with the shortest total task execution time can be selected as the target workflow scheduling scheme. Thus, by combining EFT and CPEFT algorithms, the scheduling efficiency of the generated workflow scheduling scheme and the stability of each computing resource in processing tasks can be effectively improved. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a workflow parallel scheduling method provided in an embodiment of this application.

[0021] Figure 2 This is another flowchart illustrating the workflow parallel scheduling method provided in this application embodiment.

[0022] Figure 3 This is a schematic diagram of the process of node feature aggregation in a convolutional neural network provided in this application embodiment.

[0023] Figure 4 This is a flowchart illustrating the DAG feature aggregation and global feature aggregation provided in the embodiments of this application.

[0024] Figure 5 This is a schematic diagram of the scheduling scheme generation process for the deep reinforcement learning network provided in the embodiments of this application.

[0025] Figure 6 This is a schematic diagram of the workflow parallel scheduling system provided in the embodiments of this application.

[0026] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0027] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0028] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The following description provides multiple embodiments of this application, which can be substituted or combined with each other. Therefore, this application can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this application should also be considered to include embodiments containing one or more other possible combinations of A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0029] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this application. Various processes or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0030] like Figure 1 and Figure 2 As shown in the embodiments of this application, the workflow parallel scheduling method based on a dynamic resource selection strategy includes at least the following steps: S1. Based on the temporal dependencies between tasks in the workflow, construct a DAG task flow graph. In this embodiment, the execution entity is the CPU of the controller used for parallel workflow scheduling, which can be a CPU used to execute tasks or a separate CPU. The DAG task flow graph is a directed acyclic graph (DAG). Each node represents a task and its workload information, and each edge represents the temporal relationship between two tasks. The temporal relationship between tasks indicates that a subsequent task can only begin after the preceding task is completed. Independent task paths in the workflow can form a task flow subgraph, that is, connecting a series of task nodes with temporal relationships to form a task flow subgraph. This results in one or more task flow subgraphs, and the set of these task flow subgraphs constitutes the DAG task flow graph including all tasks.

[0031] S2. Based on the graph convolutional neural network, node features are aggregated for each node to obtain multiple DAG task flow subgraphs. DAG feature aggregation and global feature aggregation operations are performed on each DAG task flow subgraph in sequence. The results of each DAG task flow subgraph, DAG feature aggregation and global feature aggregation operations are converted into target comprehensive feature vectors.

[0032] In one possible implementation, node features are aggregated based on a graph convolutional neural network to obtain multiple DAG task flow subgraphs, including: S11. Based on the topological structure between nodes of the DAG task flow graph, the task load information corresponding to the child node without a successor node is transmitted to the adjacent parent node in the form of an embedding vector through the graph neural network, and the feature is fused with the task load information of the parent node to obtain the embedding vector embedded in the parent node. S12. After fusing the node information features, the parent node is used as the child node. Repeat step S11 until each node in the DAG task flow graph has a corresponding embedding vector. Then, the branches corresponding to each node in the DAG task flow graph that do not have temporal relationships are used as DAG task flow subgraphs.

[0033] In the embodiments of this application, such as Figure 2 and Figure 3 As shown, child nodes without successor nodes are the sink nodes of the corresponding task paths. This avoids the influence of subsequent nodes on the embedding vector of that node; therefore, node feature aggregation begins from these nodes. Furthermore, each node in the DAG task flow graph embeds task load information, such as the amount of memory required to execute the task, the amount of network data transmitted, and the amount of CPU computation. This is crucial information for configuring computing resources for the task corresponding to the node. The process of performing node feature aggregation on each node of an independent task path branch in the DAG task flow graph that does not have a temporal relationship can be expressed by the following formula:

[0034] in, Indicates the parent node of the feature to be fused and embedded. The set of child nodes, This represents the initial embedding vector of the parent node to be fused and embedded. This represents the target embedding vector of the parent node to be fused and embedded. This indicates the number of the independent task path branch in the DAG task flow graph. This represents the feature extraction operation of a graph convolutional network. This represents the feature fusion operation of a graph convolutional network. In this patent, the graph convolutional network can be implemented using a multilayer perceptron (MLP), which is a multilayer perceptron with the same dimension as the embedded vector. The parent node can use this formula to fuse the features from all its child nodes with its own initial embedded vector to obtain a new embedded vector that contains the task load information and temporal information of all its child nodes.

[0035] In one possible implementation, DAG feature aggregation and global feature aggregation operations are sequentially performed on each DAG task flow subgraph, and the results of each DAG task flow subgraph, DAG feature aggregation, and global feature aggregation operations are converted into a target comprehensive feature vector, including: Add a local virtual parent node to each DAG task flow subgraph. The local virtual parent node is the common parent node of all nodes in the DAG task flow subgraph. The task load information of the parent node of each DAG task flow subgraph is passed to the corresponding local virtual parent node of the DAG task flow subgraph in the form of an embedding vector. The embedding vectors passed to the local virtual parent node are then fused to obtain the embedding vector embedded in the local virtual parent node. Add a common global virtual parent node to all local virtual parent nodes, and fuse the embedding vectors of each local virtual parent node to obtain the embedding vector of the global virtual parent node. The embedding vectors of all nodes, local virtual parent nodes, and global virtual parent nodes in each DAG task flow graph are converted into target comprehensive feature vectors.

[0036] In the embodiments of this application, such as Figure 4 As shown in the figure Indicates the first The parent node of a DAG task flow subgraph Embedded vector, Indicates the first The embedding vectors of the local virtual parent nodes corresponding to each DAG task flow subgraph are given by Z, where Z represents the embedding vector of the global virtual parent node. The initial embedding vectors of each node in the DAG task flow graph represent the individual task load characteristics of each node's corresponding task. The embedding vectors of the local virtual parent nodes represent the load and temporal characteristics of their respective independent task paths, while the embedding vectors of the global virtual parent nodes represent the load and temporal characteristics shared by all nodes. Concatenating the initial embedding vectors of each node with the embedding vectors of each virtual parent node yields a target comprehensive feature vector that integrates local and global features. This facilitates subsequent deep reinforcement learning networks in more accurately acquiring all task features of the workflow.

[0037] S3. Obtain the system environment information of the target operating system, and input the system environment information and the target comprehensive feature vector into the trained deep reinforcement learning network. In the embodiments of this application, such as Figure 2As shown, the system environment information is obtained from computing resources through the resource mapper of the task scheduling system. It can reflect real-time changes in the operating parameters of the CPU, memory, and information transmission network. For example, it can promptly obtain information such as whether the hardware of each computing resource is damaged, resource utilization, the number of components, and changes in the total available computing resources. The same computing resource may consist of multiple sub-computing resources; for example, memory is typically composed of multiple memory modules. Compared to general workflow parallel scheduling methods, this application can adjust the scheduling scheme in a timely manner based on the real-time operating parameters of computing resources, improving the dynamic adaptability to task load. Furthermore, inputting the system environment information as an embedded vector along with the target comprehensive feature vector into the deep reinforcement learning network can convert complex feature information into low-dimensional vectors, thereby improving the data processing efficiency of the deep reinforcement learning network.

[0038] S4. Constrain the decision-making process of the deep reinforcement learning network through the objective constraint algorithm to obtain the objective workflow scheduling scheme. The objective workflow scheduling scheme includes the execution path, execution time and scheduling results of computing resource allocation for each task.

[0039] In the embodiments of this application, such as Figure 5As shown, the deep reinforcement learning network of this application extracts and analyzes the comprehensive feature vectors of the target corresponding to each input task. Finally, it uses the softmax function to calculate the probability of each task being allocated to each computing resource. For example, the probability of allocating the first node task of the workflow to the third memory stick and the second CPU is the highest. Furthermore, the deep reinforcement learning network of this application can use an Actor-Critic policy network. The Actor network of the deep reinforcement learning network is used to generate the workflow scheduling scheme. The Actor network includes multiple sub-decision networks, including task scheduling sub-networks, CPU resource scheduling sub-networks, memory resource scheduling sub-networks, and network bandwidth resource scheduling sub-networks. The Critic network has only one sub-network, used to evaluate the merits of the current policy. After the input passes through each sub-network, the output is the probability of selecting each task and the computing resources allocated to that task. The task with the highest probability is selected for execution, and the actual execution information of the task is measured, including task completion time and computing resource utilization. The negative value of the task completion time is summed with the computing resource utilization as the reward signal. Finally, the reward signal is used as the penalty function for the reinforcement learning modules Actor and Critic networks. The network parameters are continuously updated through gradient descent to ensure that the deep learning network parameters are updated in the direction of minimizing task completion time and maximizing computational resource utilization, thereby achieving the goal of adjusting the scheduling scheme. Furthermore, the backbone networks of both the Actor and Critic networks in the deep reinforcement learning network can be composed of Transformer layers and MoE layers. The sub-networks of the non-backbone networks of the Actor and Critic networks can be MLP networks. Transformers can effectively capture long-range dependencies and global features between workflow tasks, while MoE layers can achieve specialized scheduling and dynamic resource selection for heterogeneous tasks through dynamic routing and expert sparse activation, improving model capacity and environmental adaptability. Therefore, using Transformer layers and MoE layers as the backbone network is beneficial for improving the scheduling capability of various tasks in the workflow.

[0040] In one possible implementation, the decision-making process of a deep reinforcement learning network is constrained by a target constraint algorithm, including: The process of generating workflow scheduling schemes by constraining deep reinforcement learning networks based on the EFT algorithm yields a first workflow scheduling scheme with the objective of minimizing the independent execution time of each task. The process of generating workflow scheduling schemes based on the CPEFT algorithm constrains the deep reinforcement learning network, resulting in a second workflow scheduling scheme with the objective of minimizing the execution time of tasks corresponding to key nodes in the DAG task flow graph, where each key node has at least two child nodes. The workflow scheduling scheme with the shortest total task execution time between the first and second workflow scheduling schemes will be selected as the target workflow scheduling scheme.

[0041] In this embodiment, using the EFT (Earliest Finish Time) algorithm to constrain deep reinforcement learning networks can lead to the network allocating computing resources to tasks based on the shortest individual task time during decision-making. This may result in a large number of physical machines being idle among similar computing resources, causing many critical node tasks (with numerous subsequent tasks) to be queued and executed on the same physical machine. Consequently, subsequent tasks based on critical node tasks are delayed, reducing the overall execution efficiency of the workflow. While the CPEFT (Critical-Path Earliest Finish Time) algorithm can effectively consider the impact of current critical node tasks and a series of subsequent tasks, it sacrifices a significant amount of time for non-critical paths when there are many simple, non-critical paths in the workflow, thus increasing the overall execution time of all tasks. Therefore, this application combines the EFT and CPEFT algorithms to constrain deep reinforcement learning networks, thereby effectively improving the scheduling efficiency of the workflow scheduling scheme and the stability of each computing resource in processing tasks.

[0042] Figure 6 A schematic diagram of the workflow parallel scheduling system based on a dynamic resource selection strategy provided in this application embodiment is shown below. Figure 6 As shown, the system includes at least: The task flow graph construction module is used to construct a DAG task flow graph based on the temporal dependencies between tasks in the workflow. The feature aggregation module is used to aggregate node features based on graph convolutional neural networks to obtain multiple DAG task flow subgraphs, and to perform DAG feature aggregation and global feature aggregation operations on each DAG task flow subgraph in sequence. The feature vector acquisition module is used to convert the results of each DAG task flow subgraph, DAG feature aggregation, and global feature aggregation operations into a target comprehensive feature vector. The feature vector concatenation module is used to obtain the environmental feature vector corresponding to the system environment information of the target operating system, and to concatenate the environmental feature vector with the target comprehensive feature vector. The scheduling scheme acquisition module is used to input the feature concatenation result into the trained deep reinforcement learning network, and to constrain the decision-making process of the deep reinforcement learning network through the target constraint algorithm to obtain the target workflow scheduling scheme.

[0043] like Figure 7 As shown, Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: a processor 701, a communications interface 702, a memory 703, and a communication bus 704. The processor 701, communications interface 702, and memory 703 communicate with each other via the communication bus 704. The processor 701 can call software instructions in the memory 703 to execute the methods described in the above embodiments.

[0044] Furthermore, the logical instructions in the aforementioned memory 703 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0045] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0046] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0047] It is understood that the processor in the embodiments of this application can be a CPU (Central Processing Unit), or other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0048] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, ROM (Read-only Memory), PROM (Programmable ROM), EPROM (Erasable PROM), EEPROM (Electrically Erasable EPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0049] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line DSL) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD (Solid State Disk)).

[0050] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0051] Those skilled in the art will readily understand that the above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A workflow parallel scheduling method based on a dynamic resource selection strategy, characterized in that, include: S1. Based on the temporal dependencies between tasks in the workflow, construct a DAG task flow graph. Each node of the DAG task flow graph can represent a task and the task load information of the task, and each edge represents the temporal relationship between two tasks. S2. Based on the graph convolutional neural network, node features are aggregated for each node to obtain multiple DAG task flow subgraphs. DAG feature aggregation and global feature aggregation operations are performed on each DAG task flow subgraph in sequence. The results of each DAG task flow subgraph, the DAG feature aggregation and global feature aggregation operations are converted into a target comprehensive feature vector. S3. Obtain the system environment information of the target operating system, and input the system environment information and the target comprehensive feature vector into the trained deep reinforcement learning network. The system environment information includes the real-time operating parameters of each computing resource. S4. Constrain the decision-making process of the deep reinforcement learning network through the objective constraint algorithm to obtain the objective workflow scheduling scheme, which includes the execution path, execution time and computing resource allocation of each task.

2. The workflow parallel scheduling method according to claim 1, characterized in that, The node feature aggregation based on the graph convolutional neural network yields multiple DAG task flow subgraphs, including: S11. Based on the topological structure between nodes of the DAG task flow graph, the task load information corresponding to the child node without a successor node is transmitted to the adjacent parent node in the form of an embedding vector through a graph neural network, and feature fusion is performed with the task load information of the parent node to obtain the embedding vector embedded in the parent node. S12. After fusing the node information features, the parent node is taken as the child node. Repeat step S11 until each node in the DAG task flow graph has a corresponding embedding vector. Then, the branches corresponding to each node in the DAG task flow graph that has the embedding vector but no temporal relationship are taken as DAG task flow subgraphs.

3. The workflow parallel scheduling method according to claim 1, characterized in that, The step of sequentially performing DAG feature aggregation on each of the DAG task flow subgraphs includes: Add a local virtual parent node to each of the DAG task flow subgraphs, where the local virtual parent node is the common parent node of all nodes in the DAG task flow subgraph; The task load information of the parent node of each DAG task flow subgraph is passed to the local virtual parent node corresponding to the DAG task flow subgraph in the form of an embedding vector. The embedding vectors passed to the local virtual parent node are then fused to obtain the embedding vector embedded in the local virtual parent node.

4. The workflow parallel scheduling method according to claim 3, characterized in that, The step of sequentially performing DAG feature aggregation and global feature aggregation operations on each of the DAG task flow subgraphs, and converting the results of each DAG task flow subgraph, the DAG feature aggregation, and the global feature aggregation operations into a target comprehensive feature vector, includes: Add a common global virtual parent node to all the local virtual parent nodes, and fuse the embedding vectors of each local virtual parent node to obtain the embedding vector embedded in the global virtual parent node; The embedding vectors of all nodes in each DAG task flow graph, each local virtual parent node, and the global virtual parent node are converted into target comprehensive feature vectors.

5. The workflow parallel scheduling method according to claim 1, characterized in that, The deep reinforcement learning network is an Actor-Critic policy network. The Actor network of the deep reinforcement learning network is used to generate workflow scheduling schemes. The Actor network includes multiple sub-decision networks, and the types of the sub-decision networks include task scheduling sub-network, CPU resource scheduling sub-network, memory resource scheduling sub-network, and network bandwidth resource scheduling sub-network.

6. The workflow parallel scheduling method according to claim 5, characterized in that, The backbone networks of the Actor network and Critic network of the deep reinforcement learning network are both composed of Transformer layers and MoE layers, and each of the sub-decision networks of the Actor network is an MLP network.

7. The workflow parallel scheduling method according to claim 1, characterized in that, The decision-making process of constraining the deep reinforcement learning network through the objective constraint algorithm includes: The process of generating a workflow scheduling scheme based on the EFT algorithm constrains the deep reinforcement learning network, resulting in a first workflow scheduling scheme with the objective of minimizing the independent execution time of each task. The process of generating a workflow scheduling scheme based on the CPEFT algorithm constrains the deep reinforcement learning network, resulting in a second workflow scheduling scheme with the objective of minimizing the execution time of the tasks corresponding to the key nodes of the DAG task flow graph, wherein the key nodes have at least two child nodes; The workflow scheduling scheme with the shortest total task execution time between the first and second workflow scheduling schemes shall be selected as the target workflow scheduling scheme.

8. A workflow parallel scheduling system based on a dynamic resource selection strategy, used to execute the method according to any one of claims 1-7, characterized in that, include: The task flow graph construction module is used to construct a DAG task flow graph based on the temporal dependencies between tasks in the workflow. The feature aggregation module is used to perform node feature aggregation on each node based on the graph convolutional neural network to obtain multiple DAG task flow subgraphs, and to perform DAG feature aggregation and global feature aggregation operations on each DAG task flow subgraph in sequence. The feature vector acquisition module is used to convert the results of each DAG task flow subgraph, the DAG feature aggregation, and the global feature aggregation operation into a target comprehensive feature vector. The feature vector concatenation module is used to obtain the environmental feature vector corresponding to the system environment information of the target operating system, and to concatenate the environmental feature vector with the target comprehensive feature vector. The scheduling scheme acquisition module is used to input the feature concatenation result into the trained deep reinforcement learning network, and constrain the decision-making process of the deep reinforcement learning network through the target constraint algorithm to obtain the target workflow scheduling scheme.

9. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the steps of the method according to any one of claims 1-7.

Images