Task scheduling method and device, medium and computer program product

Abstract

The invention relates to the technical field of communication, and particularly discloses a task scheduling method and device, a medium and a computer program product. Determining a first task execution cost after the current parallelizable to-be-allocated sub-tasks are allocated to the resource node and a second task execution cost corresponding to the next sub-task; parallel task execution cost is determined, optimization calculation is carried out with minimization of the parallel task execution cost as an optimization target and with the condition that second task execution cost corresponding to each next sub-task meets a cost constraint condition as a constraint condition, and target resource nodes corresponding to each to-be-allocated sub-task are obtained; therefore, with improvement of parallelism as a guide and guarantee of the performability of the subtasks in the next stage as a constraint, optimal scheduling is carried out on the current parallel subtasks, and the availability and execution efficiency of dynamic scheduling for complex tasks are improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a task scheduling method, device, medium and computer program product. Background Art

[0002] As technology develops, the complexity of application scenarios continues to increase, and business tasks become more complex and diverse. More and more work begins to focus on changing the task model, splitting the original monolithic application into a group of microservices with input and output dependencies to achieve more flexible business expansion capabilities. However, this model also leads to more challenges in task scheduling, namely the complex scheduling problem of how to allocate multiple tasks with dependencies to multiple resource nodes.

[0003] In response to this complex scheduling problem, the relevant technical field mainly proposes two solutions: static scheduling and dynamic scheduling. Among them, static scheduling is to generate a scheduling strategy for multiple subtasks of a business task at one time, but with the changing network environment and device status, the availability of static scheduling is extremely poor. Dynamic scheduling is to schedule the subtasks to be executed in real time as the task progresses, which can improve the availability of the scheduling strategy to a certain extent. However, due to the increasingly complex changes in the network environment and device status, the current dynamic scheduling strategy still has the problem of insufficient availability.

[0004] How to improve the availability of a distributed scheduling solution for tasks with dependencies is a technical problem that those skilled in the art need to solve. Summary of the invention

[0005] The object of the present invention is to provide a task scheduling method, device, medium and computer program product for improving the availability of a distributed scheduling solution for tasks with dependencies.

[0006] In order to solve the above technical problems, the present invention provides a task scheduling method, comprising: Get the task dependencies between multiple subtasks of the target task; According to the task dependency, determine the subtasks to be assigned that can be currently performed in parallel; Determine, according to the status information of each resource node in the target distributed system, a first task execution cost of the subtask to be assigned after the subtask to be assigned is assigned to the resource node and a second task execution cost of the next subtask of the subtask to be assigned; Determine the parallel task execution cost according to the first task execution cost of each of the subtasks to be assigned, take minimizing the parallel task execution cost as the optimization goal, take the second task execution cost of each of the next subtasks satisfying the cost constraint as the constraint condition, perform optimization calculation, and obtain the target resource node corresponding to each of the subtasks to be assigned; Allocate the subtask to be assigned to the target resource node for execution.

[0007] On the one hand, the first task execution cost is the sum of a first time for the resource node to obtain the computing parameters required for the subtask to be assigned and a second time for the resource node to execute the subtask to be assigned.

[0008] On the other hand, the parallel task execution cost is the sum of the first task execution costs of the subtasks to be assigned.

[0009] On the other hand, the parallel task execution cost is the maximum value of the first task execution costs of the subtasks to be assigned.

[0010] On the other hand, if the resource node has the task parameters of the subtask to be assigned, the first time is the time when the resource node obtains the input data of the subtask to be assigned; If the resource node does not have the task parameters of the subtask to be assigned, the first time is determined according to the time when the resource node obtains the task parameters of the subtask to be assigned and the time when the resource node obtains the input data of the subtask to be assigned.

[0011] On the other hand, the second task execution cost is the sum of a third time for the resource node executing the next subtask to obtain the calculation parameters of the next subtask and a fourth time for the resource node to execute the next subtask.

[0012] On the other hand, according to the status information of each resource node in the target distributed system, determining the first task execution cost of the subtask to be assigned after assigning the subtask to be assigned to the resource node and the second task execution cost of the next subtask of the subtask to be assigned includes: Determining, according to the status information of each of the resource nodes, the first task execution cost of the subtask to be assigned after the subtask to be assigned is assigned to the resource node; Determine a first number of resource nodes as candidate resource nodes corresponding to the subtasks to be assigned according to the order of the execution costs of the first tasks corresponding to the subtasks to be assigned from small to large; Determine the second task execution cost of the next subtask after allocating the to-be-allocated subtask to the candidate resource node.

[0013] On the other hand, it also includes: Determine a first number of resource nodes as candidate resource nodes corresponding to the subtasks to be assigned according to the order of the execution costs of the first tasks corresponding to the subtasks to be assigned from small to large; After the subtask to be assigned is assigned to the target resource node for execution, if the target resource node fails, the subtask to be assigned is migrated to another candidate resource node corresponding to the subtask to be assigned.

[0014] On the other hand, migrating the to-be-assigned subtask to another of the candidate resource nodes corresponding to the to-be-assigned subtask includes: Performing availability evaluation on the other multiple candidate resource nodes corresponding to the subtask to be assigned, and obtaining an availability evaluation value corresponding to each candidate resource node; Migrate the subtask to be assigned to the candidate resource node with the greatest availability.

[0015] On the other hand, the availability of the other multiple candidate resource nodes corresponding to the subtask to be assigned is evaluated to obtain the availability evaluation value corresponding to each candidate resource node, including: The availability of the candidate resource node is evaluated according to the resources required by the to-be-assigned subtask, the available resources of the candidate resource node, and the historical failure data of the candidate resource node to obtain the availability evaluation value of the candidate resource node.

[0016] On the other hand, the target distributed system is an edge computing system; The subtask to be assigned is the subtask that currently needs to be offloaded from the terminal device to the target edge computing device in the edge computing system.

[0017] On the other hand, the task dependencies between multiple subtasks of the target task are obtained, including: A directed acyclic graph corresponding to the target task is obtained, wherein the graph nodes in the directed acyclic graph correspond to the subtasks one-to-one, and the directed edges in the directed acyclic graph are used to represent the dependency relationship between the subtasks.

[0018] In order to solve the above technical problems, the present invention further provides a task scheduling device, comprising: Memory for storing computer programs; The processor is used to execute the computer program, and the computer program, when executed by the processor, implements the steps of any one of the task scheduling methods described above.

[0019] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the task scheduling method described in any one of the above items are implemented.

[0020] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the task scheduling method as described in any one of the above items are implemented.

[0021] The task scheduling method provided by the present invention has the beneficial effect that when dynamically scheduling according to the task dependency relationship between multiple subtasks of the target task, the first task execution cost corresponding to the subtask to be assigned and the second task execution cost corresponding to the next subtask of the subtask to be assigned are determined after the current parallel subtask to be assigned is assigned to the resource node according to the status information of each resource node in the target distributed system; the parallel task execution cost is determined according to the first task execution cost corresponding to each subtask to be assigned, and the minimization of the parallel task execution cost is taken as the optimization goal, and the second task execution cost corresponding to each next subtask satisfies the cost constraint condition as the constraint condition, and the optimization calculation is performed to obtain the target resource node corresponding to each subtask to be assigned, so as to optimize the scheduling of the current parallel subtasks with the improvement of parallelism as the guide and the executability of the subtasks in the next stage as the constraint, thereby improving the availability and execution efficiency of dynamic scheduling for complex tasks.

[0022] The task scheduling device, non-volatile storage medium and computer program product provided by the present invention have the above-mentioned beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of a task scheduling method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a directed acyclic graph of a target task provided by an embodiment of the present invention; Figure 3 A schematic diagram of a mapping relationship of candidate resource nodes provided in an embodiment of the present invention; Figure 4 A schematic diagram of a mapping relationship of target resource nodes provided by an embodiment of the present invention; Figure 5A structural diagram of a task scheduling device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The core of the present invention is to provide a task scheduling method, device, medium and computer program product for improving the availability of a distributed scheduling solution for tasks with dependencies.

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] To facilitate understanding of the technical solution provided by the embodiment of the present invention, some key terms and concepts used in the embodiment of the present invention are first explained here.

[0028] A distributed system is a system composed of multiple computers connected through communication lines. Therefore, each node in the distributed system contains its own processor and memory, and has the function of independently processing data. A large task can be divided into several subtasks, which can be executed on different nodes in the distributed system. Usually for users, a distributed system is a unified whole, and when executing user tasks, it uses the distributed system's intelligent agent to schedule tasks.

[0029] Depending on the design goals, a distributed system can be designed as a distributed storage system or a distributed computing system. Distributed storage systems are mainly used to handle storage business tasks, aiming to store and manage large amounts of data, and provide data persistence, reliability, and accessibility. Distributed computing systems are mainly used to provide computing business tasks, aiming to handle large-scale computing tasks in a distributed manner and improve computing efficiency and processing capabilities.

[0030] In the task scheduling scheme provided in the embodiment of the present invention, since the nodes in the distributed system provide the resources required to execute business tasks, these nodes are called resource nodes. A resource node is usually a computer or a server, and is configured as a computing resource node or a storage resource node or other focused resource nodes according to business needs.

[0031] The distributed system involved in the embodiment of the present invention may include an edge computing system.

[0032] Edge computing is a distributed computing scenario that places computing, storage, and network services close to the data source or user's location to reduce latency, improve response speed, and reduce the burden on central data centers. The data generated by mobile terminal users is increasing exponentially, and the requirements of various applications for the operating environment are also increasing, including higher computing intensity and lower latency requirements. In addition, processing private data requires that applications and services cannot excessively access user devices and upload them to the server for processing. In recent years, edge computing technology has developed rapidly in this context and has gradually been adopted by various industries and fields. It reduces latency by performing calculations close to the data source and provides scalability by distributing workloads to multiple edge devices. In addition, since the edge is closer to the user, the risk of privacy data leakage can be reduced. The main feature of edge computing is that it allows users nearby to offload tasks, bringing computing closer to the data source.

[0033] The task scheduling scheme provided by the embodiment of the present invention is a scheduling scheme for allocating complex tasks to a distributed system for processing. With the development of technology, the complexity of application scenarios continues to increase. More and more services are being decoupled. Through technologies such as containers, the original monolithic application is divided into a group of microservices with input and output dependencies, and the subtasks divided into complex tasks are scheduled to multiple target resources to achieve the best mapping between complex tasks and a group of resources, thereby improving the ability to serve terminal users.

[0034] In an embodiment of the present invention, a complex task refers to a task that can be split into multiple subtasks with input-output dependencies. Different subtasks can be assigned to the same or different resource nodes for execution. There is a dependency relationship between the subtasks, that is, the input data of some subtasks is the output data of other subtasks.

[0035] The input-output relationship between multiple subtasks belonging to the same complex task can be represented by a directed acyclic graph (DAG), so the above complex task can also be called a directed acyclic graph task. In a directed acyclic graph, "directed" means that each edge in the graph has a direction, that is, it points from one graph node to another graph node, and "acyclic" means that there is no cycle in the graph, that is, there is no path starting from a graph node, passing through a series of graph nodes and then returning to the original graph node. In an embodiment of the present invention, a graph node of a directed acyclic graph can be used to represent a subtask, and a directed edge of a directed acyclic graph can be used to represent the input-output dependency relationship between two subtasks. The direction of the edge represents the data transmission direction, that is, the execution order between subtasks.

[0036] The current scheduling technology for directed acyclic graph tasks focuses on task dependency, that is, static scheduling. Before the first subtask starts, a fixed scheduling strategy is formulated based on the current available resource set of the distributed system, which has poor robustness. This is because in real distributed systems, computing resources and network resources usually have certain dynamic characteristics and will be continuously updated over time. In addition, resource nodes may also have reduced ability to execute tasks due to task overload or other sudden failures.

[0037] Another more flexible scheduling scheme is dynamic scheduling, which not only considers the task structure, but also schedules the subtasks to be executed in real time as the task progresses. This can effectively avoid the problem of poor subtask scheduling performance. However, in the face of today's increasingly complex distributed system environment and increasingly complex tasks, the current dynamic scheduling scheme for directed acyclic graph tasks still has the problems of insufficient flexibility and poor usability.

[0038] In order to improve the availability of distributed scheduling schemes for tasks with dependencies, the task scheduling scheme provided by an embodiment of the present invention, when dynamically scheduling according to the task dependencies between multiple subtasks of a target task, determines, according to the status information of each resource node in the target distributed system, the first task execution cost corresponding to the subtask to be assigned after the current parallel subtask to be assigned is assigned to the resource node and the second task execution cost corresponding to the next subtask of the subtask to be assigned; determines the parallel task execution cost according to the first task execution cost corresponding to each subtask to be assigned, takes minimizing the parallel task execution cost as the optimization goal, takes the second task execution cost corresponding to each next subtask to satisfy the cost constraint as the constraint condition, performs optimization calculation, and obtains the target resource node corresponding to each subtask to be assigned, thereby optimizing and scheduling the current parallel subtasks with the improvement of parallelism as the guide and the guarantee of the executability of the subtasks in the next stage as the constraint, thereby improving the availability and execution efficiency of dynamic scheduling for complex tasks.

[0039] The task scheduling method provided by the embodiment of the present invention is described below.

[0040] Figure 1 A flowchart of a task scheduling method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a directed acyclic graph of a target task provided by an embodiment of the present invention.

[0041] like Figure 1 As shown, the task scheduling method provided by the embodiment of the present invention includes: S101: Obtaining task dependencies between multiple subtasks of a target task; S102: Determine the subtasks to be assigned in parallel according to the task dependency relationship; S103: determining, according to the status information of each resource node in the target distributed system, a first task execution cost of the subtask to be assigned after assigning the subtask to be assigned to the resource node and a second task execution cost of the next subtask of the subtask to be assigned; S104: Determine the parallel task execution cost according to the first task execution cost of each subtask to be assigned, take minimizing the parallel task execution cost as the optimization goal, take the second task execution cost of each next subtask to satisfy the cost constraint as the constraint condition, perform optimization calculation, and obtain the target resource node corresponding to each subtask to be assigned; S105: Allocate the subtask to be assigned to the target resource node for execution.

[0042] In the embodiment of the present invention, as introduced in the above concepts, the target distributed system can be any type of distributed system, such as a distributed computing system, a distributed storage system, an edge computing system, etc.

[0043] The task scheduling method provided by the embodiment of the present invention can be applied to one or more resource nodes in the target distributed system, and can also be applied to one or more computing devices outside the target distributed system.

[0044] In an embodiment of the present invention, each subtask of the target task can be a subtask oriented to the target distributed system, or some of the subtasks are subtasks oriented to the target distributed system, and the other subtasks are executed on the terminal device. Then in an application scenario of an embodiment of the present invention, the target distributed system is an edge computing system; the subtask to be assigned is a subtask of the target edge computing device that currently needs to be unloaded from the terminal device to the edge computing system.

[0045] In a specific implementation, for S101, the task dependency relationship between multiple subtasks of the target task is obtained, which at least includes the information of each subtask of the target task and the input-output dependency relationship between each subtask. As described in the above embodiment, the task dependency relationship between subtasks can be represented by a directed acyclic graph. Then, obtaining the task dependency relationship between multiple subtasks of the target task in S101 may include: obtaining a directed acyclic graph corresponding to the target task, the graph nodes in the directed acyclic graph correspond to the subtasks one by one, and the directed edges in the directed acyclic graph are used to represent the dependency relationship between the subtasks.

[0046] The directed acyclic graph of the target task can be expressed as .in, Indicates the target task subtasks, Indicates the dependency relationship between subtasks. The meaning of each edge can be: the output of the previous subtask will be used as the necessary input of the next subtask, otherwise the next subtask cannot be executed smoothly.

[0047] The task scheduling method provided in the embodiment of the present invention also adopts a dynamic scheduling method, that is, the subtasks to be executed next are scheduled in real time as the execution of the target task progresses. Different from the related art, the task scheduling method provided in the embodiment of the present invention first determines the parallel subtasks as the subtasks to be assigned in the current stage according to the task dependency relationship between the subtasks when scheduling the target task for the target distributed system.

[0048] Task segmentation can be achieved by segmenting the directed acyclic graph corresponding to the target task in two dimensions. In an embodiment of the present invention, the task segmentation of the first dimension can be defined as horizontal segmentation, which means starting from the first subtask in the directed acyclic graph, traversing the set C of all task paths that can reach the last subtask, and the subtasks in the set C have strong dependencies, and it is expected that there will be more data exchange costs. The task segmentation of the second dimension can be defined as vertical segmentation, which means that in the directed acyclic graph, according to the dependencies between the subtasks, the subsequent subtasks that process the same preceding subtask are included in the set L, and the subtasks in the set L can be executed concurrently in the same time period.

[0049] Assume that the target task can be divided into 9 subtasks, and the dependencies between these subtasks can be as follows: Figure 2 As shown, subtask 6 requires the output data of subtask 3 and the output data of subtask 5 as input, subtask 8 requires the output data of subtask 6 and the output data of subtask 7 as input, and so on; after executing subtask 1, subtask 2, subtask 4, and subtask 7 all require the output data of subtask 1 as input but have no dependency relationship with each other. Therefore, the moment when subtask 1 is completed is regarded as the beginning of the current stage, and subtask 2, subtask 4, and subtask 7 can be regarded as subtasks to be assigned and can be executed in parallel.

[0050] In some optional implementations of the present invention, determining the subtasks to be assigned in parallel according to the task dependency in S102 may include: determining all task paths from the first subtask to the last subtask according to the directed acyclic graph corresponding to the target task; and listing the first subtask corresponding to the current stage in each task path as the subtask to be assigned in the current stage. Figure 2 For example, subtask 1→subtask 2→subtask 3→subtask 6→subtask 8→subtask 9 can be considered as a task path, subtask 1→subtask 4→subtask 5→subtask 6→subtask 8→subtask 9 can be considered as a task path, and subtask 1→subtask 7→subtask 8→subtask 9 can be considered as a task path. If subtask 1 is completed in the current stage, subtask 2, subtask 4, and subtask 7 can be included in the subtasks to be assigned in the current stage.

[0051] Through observation, it can be seen that the number of subtasks on each task path of the directed acyclic graph task may be different, and for other subtasks except the first subtask, it is usually necessary to wait for all its predecessor subtasks to be executed before they can be executed. In other optional implementations of the embodiments of the present invention, S102 determines the subtasks to be assigned in parallel according to the task dependency, and may also include: after the first subtask is executed, the second subtask is determined according to the directed acyclic graph corresponding to the target task, and the first subtask and the second subtask are both subtasks belonging to multiple task paths; the subtasks between the first subtask and the second subtask that belong to the same task path are included in the same subtask group to be assigned, and each subtask group to be assigned is determined as a subtask that can be assigned in parallel at the current stage. That is to say, the subtasks at the intersection of multiple task paths are used as nodes for executing dynamic scheduling, and parallel scheduling is first performed on the subtask group to be assigned, and then the subtasks in the subtask group to be assigned are scheduled in sequence. Figure 2 For example, if the current stage is that subtask 1 has been completed, then subtask 1 is regarded as the first subtask, and subtask 6 can be regarded as the first subtask. Then subtask 2 and subtask 3 are included in one subtask group to be assigned, and subtask 4 and subtask 5 are included in another subtask group to be assigned; if the current stage is that subtask 1 has been completed, subtask 8 can also be regarded as the second subtask. Then subtask 2, subtask 3, and subtask 6 can be included in the first subtask group to be assigned, subtask 4, subtask 5, and subtask 6 can be included in the second subtask group to be assigned, and subtask 7 can be included in the third subtask group to be assigned.

[0052] For S103, the real-time status information of the target distributed system may be collected by a software and hardware detection device deployed in the target distributed system to determine the resources that can be provided for the subtasks to be assigned at the current stage.

[0053] In the embodiment of the present invention, the status information of the resource node may include but is not limited to one of basic environment status information and fault information.

[0054] The basic environment status information may include network topology information of the target distributed system, resource status information of resource nodes, and network status information between resource nodes.

[0055] The network topology information can include the resource nodes in the target distributed system and the connection relationship between resource nodes, which can be represented by a relationship diagram. Indicates. Among them, Represents the target distributed system resource nodes, Represents the edge formed by the interconnection between resource nodes.

[0056] The resource status information of the resource node may include information about the computing resources and storage resources on the resource node. Among them, the storage resources may include memory resources and persistent storage resources. The resource status information can be as follows: ; in, , , Represents resource nodes respectively The total amount of computing resources, available computing resources, and occupied computing resources are , , Represents resource nodes respectively The total amount of memory resources, available memory resources, and occupied memory resources are , , Represents resource nodes respectively The current total amount of persistent storage resources, the available amount of persistent storage resources, and the occupied amount of persistent storage resources.

[0057] The network status information between resource nodes is used to indicate the interconnection status between resource nodes. To represent resource nodes and resource nodes The interconnection status information between Represents a resource node To the resource node The actual bandwidth between Indicates the average delay level. The actual bandwidth capacity indicates the data transmission capacity, while the delay level is affected by the location of the resource nodes in the network topology, the link distance, and other external interferences that cause data transmission and reception delays. Therefore, Changes with system dynamics.

[0058] Fault information may include the type of fault that occurred on the resource node and the duration of the fault. Fault information It can be expressed as: ; in, Indicates the fault type. The state quantity may include resource node failure (which can be represented by 0), resource node failure (which can be represented by n, and the specific value of n represents the specific failure type), and resource node related network failure (which can be represented by e, and the specific value of e represents the specific network or resource node pointed to). Among them, the resource node related network may include a network composed of all resource nodes that have a connection relationship with the resource node. For other resource nodes directly connected to the resource node, other resource nodes interconnected with the resource node and other resource nodes unidirectionally pointed to by the resource node may be included in the resource node related network; for other resource nodes not directly connected to the resource node, a hop count threshold may be pre-set, and other resource nodes whose connection with the resource node is less than or equal to the hop count threshold may be included in the resource node related network, and other resource nodes whose connection with the resource node is greater than the hop count threshold may not be considered.

[0059] Indicates the fault duration, The state quantity may include the expected duration of the fault (which can be represented by T, and the specific value of T can be a specific duration) and the unknown duration of the fault (which can be represented by TE). The unknown duration of the fault means that the current fault of the resource node cannot be recovered and the duration of the fault cannot be determined.

[0060] By collecting historical data of each resource node, the regularity of the fault duration corresponding to each fault type can be obtained. Represents a resource node The actual fault type set that occurs on Represents a resource node The probability distribution model of the fault duration corresponding to the actual fault type that occurs on the network can be obtained by statistically analyzing historical data.

[0061] Then, in S103, the quantized perception target data can be obtained by perceiving the above state information. ,in, Represents the network topology information of the target distributed system, Represents a resource node Resource status information, Represents a resource node and resource nodes The interconnection status information between Represents a resource node Fault information.

[0062] The above sensing step can be performed periodically, and the most recently obtained state information can be obtained when the subtask to be assigned needs to be scheduled. The above sensing step can also be performed when the subtask to be assigned needs to be scheduled.

[0063] Then, according to the status information of each resource node in the target distributed system, the first task execution cost of the subtask to be assigned after the subtask to be assigned is assigned to the resource node is determined. The first task execution cost may include the cost of the resource node obtaining the task parameters and input data of the subtask to be assigned and the cost of the resource node executing the subtask to be assigned. Assuming that after the subtask to be assigned is assigned to a certain resource node, the second task execution cost of the next subtask of the subtask to be assigned can be determined according to the status information of each resource node in the target distributed system. The second task execution cost may include the cost of obtaining the task parameters and input data of the next subtask and the cost of executing the next subtask.

[0064] The first task execution cost may be determined according to at least one of a time cost, an energy consumption cost, and a service quality parameter.

[0065] The second task execution cost may be determined according to at least one of a time cost, an energy consumption cost, and a service quality parameter.

[0066] In S104, the parallel task execution cost is determined according to the first task execution cost of each subtask to be assigned in the current stage, which may include: taking the sum of the first task execution costs of each subtask to be assigned as the parallel task execution cost. If the first task execution cost includes a time cost, when calculating the parallel task execution time cost, the maximum value of the time costs corresponding to each subtask to be assigned may be used as the parallel task execution time cost.

[0067] In the embodiment of the present invention, in addition to minimizing the parallel task execution cost for the parallel subtasks to be assigned as the optimization goal, the feasibility of the dynamic scheduling scheme also needs to be considered, and the feasibility is guaranteed to at least include the constraints on the executability of the next subtask of the subtask to be assigned. Therefore, in the embodiment of the present invention, at least the second task execution cost of each next subtask satisfies the cost constraint as a constraint. According to the method for determining the second task execution cost, the cost constraint can be a constraint of at least one of the time cost, energy consumption cost, and service quality parameter.

[0068] After determining the optimization goal and constraints, optimization calculations are performed to solve a dynamic scheduling scheme for allocating each subtask to be assigned to the target resource node. This dynamic scheduling scheme has a lower task execution cost and obtains a higher reliability guarantee.

[0069] In S105, the dynamic scheduling solution obtained in S104 is executed. If the target task still has subsequent subtasks, the process returns to S102 for the subsequent subtasks.

[0070] The task scheduling method provided by the embodiment of the present invention, when dynamically scheduling according to the task dependency relationship between multiple subtasks of the target task, determines the first task execution cost corresponding to the subtask to be assigned after the current parallel subtask to be assigned is assigned to the resource node and the second task execution cost corresponding to the next subtask of the subtask to be assigned according to the status information of each resource node in the target distributed system; determines the parallel task execution cost according to the first task execution cost corresponding to each subtask to be assigned, takes minimizing the parallel task execution cost as the optimization goal, takes the second task execution cost corresponding to each next subtask to satisfy the cost constraint condition as the constraint condition, performs optimization calculation, and obtains the target resource node corresponding to each subtask to be assigned, thereby optimizing and scheduling the current parallel subtasks with the improvement of parallelism as the guide and the guarantee of the executability of the subtasks in the next stage as the constraint, thereby improving the availability and execution efficiency of dynamic scheduling for complex tasks.

[0071] Based on the above embodiments, the embodiments of the present invention further illustrate the establishment of optimization objectives and constraint conditions and the optimization calculation steps.

[0072] In the above embodiment of the present invention, it is introduced that the first task execution cost may be a task execution cost determined according to at least one of a time cost, an energy consumption cost and a service quality parameter.

[0073] In some optional implementations of the embodiments of the present invention, the first task execution cost is the sum of a first time for the resource node to obtain the calculation parameters required for the subtask to be assigned and a second time for the resource node to execute the subtask to be assigned.

[0074] On this basis, the parallel task execution cost is the sum of the first task execution costs of each subtask to be assigned. That is to say, the sum of the execution time of each subtask to be assigned (or subtask group to be assigned) can be used, because the execution time can also reflect the occupation of resources, so the time and value method is used to represent the parallel task execution cost at the current stage.

[0075] Alternatively, the parallel task execution cost is the maximum value of the first task execution costs of the subtasks to be assigned. That is, if the first task execution cost only considers the execution time, the maximum execution time of the subtasks to be assigned can be used as the parallel task execution cost.

[0076] When determining the execution cost of the first task according to the above method, if the resource node has the task parameters of the subtask to be assigned, the first time is the time when the resource node obtains the input data of the subtask to be assigned; if the resource node does not have the task parameters of the subtask to be assigned, the first time is determined according to the time when the resource node obtains the task parameters of the subtask to be assigned and the time when the resource node obtains the input data of the subtask to be assigned. That is to say, whether the time to obtain the task parameters needs to be considered is distinguished according to whether the resource node has the task parameters of the subtask to be assigned. The first time is determined according to the time when the resource node obtains the task parameters of the subtask to be assigned and the time when the resource node obtains the input data of the subtask to be assigned. The first time can be the sum of the time when the resource node obtains the task parameters of the subtask to be assigned and the time when the resource node obtains the input data of the subtask to be assigned, or the larger value of the time when the resource node obtains the task parameters of the subtask to be assigned and the time when the resource node obtains the input data of the subtask to be assigned can be used as the first time.

[0077] In some other optional implementations of the embodiments of the present invention, if the execution cost of the first task is determined based on time cost, energy cost and service quality parameters, the energy cost can be determined based on the data transmission process of the calculation parameters required for the subtask to be assigned and the computing resources, storage resources, network resources and power consumption required in the process of executing the subtask to be assigned. The service quality parameter can estimate the execution result of the resource node executing the subtask to be assigned based on the historical data of the resource node. If the subtask to be assigned is a model calculation task, the service quality parameter can be the model accuracy.

[0078] In the above embodiment of the present invention, the second task execution cost may be a task execution cost determined according to at least one of a time cost, an energy consumption cost and a service quality parameter.

[0079] In some optional implementations of the embodiments of the present invention, the second task execution cost may be the sum of the third time for the resource node executing the next subtask to obtain the calculation parameters of the next subtask and the fourth time for the resource node to execute the next subtask. The cost constraint condition may be determined for each next subtask, and the second task execution cost corresponding to the next subtask may be compared with the cost constraint condition.

[0080] On the basis of the above embodiment, S103 determines the first task execution cost of the subtask to be assigned after the subtask to be assigned is assigned to the resource node and the second task execution cost of the next subtask of the subtask to be assigned according to the status information of each resource node in the target distributed system. It can also include: determining the first task execution cost of the subtask to be assigned after the subtask to be assigned is assigned to the resource node according to the status information of each resource node; determining the first first number of resource nodes as candidate resource nodes corresponding to the subtask to be assigned in order from small to large of the first task execution costs corresponding to the subtask to be assigned; and determining the second task execution cost of the next subtask after the subtask to be assigned is assigned to the candidate resource node.

[0081] In an embodiment of the present invention, in order to further simplify the steps of determining the execution cost of the first task and the execution cost of the second task, and to improve the interpretability of the dynamic scheduling strategy generation process, a method of generating task scheduling strategies in stages is adopted. That is to say, in the task scheduling unloading strategy considering performance and reliability, the vertical segmentation set L is first used as the decision target for each stage, because the subtasks to be assigned belonging to the same group in the set L have a high degree of parallelism, and there is basically no need to consider the influence between each other. Therefore, the problem can be expressed as how to schedule all the subtasks to be assigned in the set L to a set of resource nodes N, and ensure that the group of subtasks to be assigned has a lower task execution cost and obtains a higher reliability guarantee. This step determines the set of candidate resource nodes corresponding to each subtask to be assigned.

[0082] Remember the path On The subtasks to be assigned are , which has Predecessor subtasks, The amount of data that needs to be transmitted by the predecessor subtask of the subtask to be assigned is , the amount of data expected to be generated is , then consider assigning the subtask to be Scheduling to resource nodes When the corresponding first task execution cost is It can be expressed by the following formula: ; in, Indicates the maximum value calculation. Indicates that subtasks will be assigned The task parameters are from the resource node Transfer to resource node The time cost required (for example, in the edge offloading scenario, it can be the time to offload the subtask to be assigned from the terminal device to the resource node in the target edge computing system), Indicates that subtasks will be assigned The required input data is from the resource node (i.e., subtasks to be assigned The resource node where the previous subtask is located) is transmitted to the resource node The time cost required, Indicates that Input data is transferred to the resource node The maximum time required, Represents a resource node Execute the assigned subtasks The second time.

[0083] Then, the first number of resource nodes can be determined as candidate resource nodes corresponding to the subtask to be assigned in the ascending order of the execution cost of the first task corresponding to the subtask to be assigned. To improve availability, the task requirements for assigning the subtask to be assigned to the resource node can be verified first, and the first K resource nodes that meet the task requirements can be used as candidate resource nodes for the subtask to be assigned: , ; in, Indicates the minimum value calculation. Indicates the cost threshold corresponding to the execution cost of the first task. K can be 3.

[0084] Summarize the candidate resource nodes corresponding to each subtask to be assigned, and obtain the candidate resource node set of all subtasks to be assigned to the current node ,Since there is a many-to-many mapping relationship between multiple subtasks and multiple resource nodes, considering the principle of optimal total system cost, when a resource node corresponds to multiple subtasks to be assigned, if all subtasks to be assigned cannot be satisfied at the same stage, it will be deleted from the candidate resource nodes of some subtasks to be assigned. The deletion method is: delete the subtasks to be assigned that occupy the least resources, so as to ensure that the subtasks to be assigned with higher resource requirements are optimally executed. Get the set of candidate resource nodes after the first round of screening .

[0085] Figure 3 A schematic diagram of a mapping relationship of candidate resource nodes provided in an embodiment of the present invention.

[0086] like Figure 3 As shown, assuming that the subtasks to be assigned to the current node include , , , , there are 8 resource nodes in the candidate resource node set. Figure 3 To be assigned subtasks The candidate resource nodes are resource nodes 1, 3, 6, and so on. All nodes marked " " is a candidate resource node.

[0087] Then, based on the first stage, it is necessary to determine the target resource nodes of all subtasks to be assigned from the candidate resource node set. At this time, for each subtask to be assigned, the execution status of the previous dependent subtasks belonging to the same horizontal partition set C is considered, and the subsequent dependent subtasks are kept consistent with the execution position of the subtask to be assigned as much as possible to reduce the data transmission cost.

[0088] However, due to the dynamic characteristics of the environment, the current optimal resource node may not meet the task requirements of the next stage. Therefore, the candidate resource node set obtained in the previous step Further sorting and screening are performed, and how to obtain the system decision that can produce the minimum cost and how to ensure that the subtasks in the next stage have sufficient resources as much as possible. The optimization objective function can be constructed as: ; The constraints are: . in, It represents the total cost of scheduling resource nodes for all subtasks to be assigned. The constraint condition represents the time cost required for the selected resource node to continue to execute the next subtask after executing the subtask to be assigned. Or the transmission time cost required to transmit the output data of the subtask to be assigned to the resource node that executes the next subtask None of the items in the above table exceeds the cost constraint of the next subtask. .

[0089] This optimization calculation problem is a combinatorial optimization problem, which is a typical NP-hard problem. A heuristic search algorithm can be used to find a solution that can meet the approximate optimal cost. The resource node obtained by the search algorithm is selected as the target resource node corresponding to the subtask to be assigned.

[0090] Figure 4 A schematic diagram of a mapping relationship of target resource nodes provided by an embodiment of the present invention.

[0091] like Figure 4 As shown, subtasks to be assigned The target resource node is resource node 3, and so on. " is the target resource node.

[0092] To further improve the availability of distributed scheduling solutions for tasks with dependencies, the task scheduling method provided in the embodiment of the present invention may also include: when a faulty resource node is detected in the target distributed system, fault-tolerant processing is performed on the subtasks in the faulty resource node.

[0093] In some optional implementations of the embodiments of the present invention, the task scheduling method provided by the embodiments of the present invention may also include: determining the first first number of resource nodes as candidate resource nodes corresponding to the subtask to be assigned in order from small to large of the execution cost of the first task corresponding to the subtask to be assigned; after the subtask to be assigned is assigned to the target resource node for execution, if the target resource node fails, migrating the subtask to be assigned to another candidate resource node corresponding to the subtask to be assigned.

[0094] That is, in the candidate resource node set obtained in the above steps, each candidate resource node that is not selected for the assigned subtask can be used as a fault-tolerant resource node set, and the resource nodes in the fault-tolerant resource node set can be used to provide migration nodes for fault-tolerant processing of faulty resource nodes.

[0095] In an embodiment of the present invention, migrating the subtask to be assigned to another candidate resource node corresponding to the subtask to be assigned may include: performing availability evaluation on multiple other candidate resource nodes corresponding to the subtask to be assigned to obtain availability evaluation values ​​corresponding to each candidate resource node; migrating the subtask to be assigned to the candidate resource node with the greatest availability. That is, the availability of multiple other candidate resource nodes corresponding to the subtask to be assigned may be measured by designing an availability evaluation function so as to select the candidate resource node with the greatest availability.

[0096] In an embodiment of the present invention, availability evaluation is performed on multiple other candidate resource nodes corresponding to the subtask to be assigned to obtain an availability evaluation value corresponding to each candidate resource node, which may include: performing availability evaluation on the candidate resource nodes based on the resources required by the subtask to be assigned, the available resources of the candidate resource nodes, and the historical fault data of the candidate resource nodes to obtain the availability evaluation value of the candidate resource nodes.

[0097] The subtask being executed by the faulty resource node is recorded as the pending subtask.

[0098] In some optional implementations of the embodiments of the present invention, the availability evaluation function can be expressed by the following formula: ; in, , , Represents the subtasks to be processed The required computing resources, storage resources, and network resources, , , Respectively represent the resource nodes being evaluated The corresponding computing, storage, and network resources that can be provided within the most recent time interval, and the smaller the ratio of these items, the better. Represents a resource node The recent statistics on the mean time between failures, It indicates the average failure-free time of all resource nodes in the target distributed system. The smaller the ratio is, the slower the resource nodes are. The failure rate is lower than the average level of the target distributed system.

[0099] , It represents a real number between 0 and 1, which is used to adjust the weights of different parts in the usability evaluation function.

[0100] Evaluate all resource nodes in the fault-tolerant resource node set and select the availability evaluation value The highest resource node, as the subtask to be processed Migrate nodes for fault tolerance and perform task migration.

[0101] The task scheduling method provided by the embodiment of the present invention, when dynamically scheduling tasks with dependencies, also uses resource nodes that are not selected when determining the candidate resource node set as fault-tolerant resource nodes to further improve the availability of distributed scheduling of complex tasks.

[0102] In addition to the fault-tolerant processing scheme introduced in the above-mentioned embodiment, in other optional implementation methods of the embodiments of the present invention, fault-tolerant processing of subtasks in faulty resource nodes may also include: using a trained fault-tolerant processing action generation model to generate multiple fault-tolerant processing actions for the subtasks to be processed according to the state information of the target distributed system at the current moment, performing availability evaluation on each fault-tolerant processing action, determining the actual fault-tolerant processing action according to the availability evaluation result, and executing the actual fault-tolerant processing action on the subtasks to be processed.

[0103] In an embodiment of the present invention, the status information may include but is not limited to basic environment status information, task status information and fault information. At the current moment, there may be multiple faulty resource nodes in the target distributed system, and there may be multiple pending subtasks on a faulty resource node. According to the device scheduling capability and system design, the pending subtasks can be scheduled one by one, or multiple pending subtasks can be scheduled at a time and completed in batches. Therefore, it is necessary to determine the priority of the pending subtasks and perform fault-tolerant processing of the pending subtasks in order from high to low priority. The method for determining the priority of the pending subtasks can be determined based on at least one of the task type, workload, and user-set parameters of the pending subtasks.

[0104] The fault-tolerant processing action generation model can be trained by reinforcement learning or deep reinforcement learning. The trained fault-tolerant processing action generation model is used to generate multiple fault-tolerant processing actions for the subtasks to be processed according to the current state information. During operation, the parameters of the fault-tolerant processing action generation model can also be fine-tuned according to the operating state of the target distributed system.

[0105] In order to improve the availability of fault-tolerant scheduling results, in an embodiment of the present invention, an action filter is designed to filter multiple fault-tolerant processing actions output by the fault-tolerant processing action generation model, based on pre-configured availability evaluation indicators, to ensure that the actual fault-tolerant processing actions available in the target distributed system at the current moment are filtered out and executed.

[0106] In an embodiment of the present invention, a method for determining the priority of a subtask to be processed may include: determining a fault impact parameter of a faulty resource node based on the fault type of the faulty resource node; determining a task evaluation parameter of the subtask to be processed based on the load status of the subtask to be processed on the faulty resource node; and determining a priority score of the subtask to be processed based on the fault impact parameter and the task evaluation parameter.

[0107] The embodiment of the present invention measures the priority of the subtask to be processed from two aspects: the fault impact of the fault resource node and the load status of the subtask to be processed on the fault resource node.

[0108] Among them, determining the fault impact parameter of the faulty resource node according to the fault type of the faulty resource node may include: determining the fault duration prediction value of the faulty resource node according to the historical operation data of the faulty resource node; and using the fault duration prediction value as the fault impact parameter.

[0109] In the above embodiment of the present invention, it is introduced that by counting the historical data of each resource node, the law of the fault duration corresponding to each fault type can be obtained. Then, determining the predicted value of the fault duration of the faulty resource node according to the historical data corresponding to the fault type of the faulty resource node may include: determining the expected value of the fault duration of the faulty resource node according to the fault type of the faulty resource node and the corresponding fault duration probability distribution model; using the expected value of the fault duration as the predicted value of the fault duration; wherein the fault duration probability distribution model corresponding to the fault type is determined according to the historical operation data of the resource node.

[0110] In the embodiment of the present invention, the expected value of the fault duration can be obtained by Calculated, that is, given Moment Resource Node Under the conditions of the actual fault types, the resource nodes are determined according to the probability distribution model of the fault duration corresponding to these fault types. The expected value of the fault duration.

[0111] In some optional implementations of the embodiments of the present invention, determining the task evaluation parameters of the subtask to be processed according to the load state of the subtask to be processed on the faulty resource node may include: taking the resource occupancy parameter of the subtask to be processed on the faulty resource node and the remaining execution time of the subtask to be processed on the faulty resource node as the load state of the subtask to be processed on the faulty resource node, and determining the task evaluation parameters according to the resource occupancy parameter and the remaining execution time. That is, the task evaluation parameters of the subtask to be processed may be determined by considering only the workload.

[0112] The step of determining the resource occupancy parameter may include: determining the resource occupancy parameter according to the amount of multiple types of resources occupied by the subtask to be processed on the faulty resource node. Determining the task evaluation parameter according to the resource occupancy parameter and the remaining execution time may include: taking the ratio of the resource occupancy parameter to the remaining execution time as the task evaluation parameter. Then the task evaluation parameter of the subtask to be processed can be calculated by the following formula: ; in, , , Respectively represent business tasks The amount of computing resources, memory resources, and persistent storage resources used by a single copy of Indicates business tasks The number of replicas running on the resource node. Indicates business tasks The occupation period of the current resource. Among them, is the same unit, , , , , can all be normalized parameters. It can be used to indicate the average amount of resources consumed by the pending subtasks on the faulty resource node per unit time.

[0113] In the embodiment of the present invention, the fault impact parameter may be a predicted fault duration of the faulty resource node, and the task evaluation parameter may be a resource consumption per unit time of the subtask to be processed at the faulty resource node. Then, determining the priority score of the subtask to be processed according to the fault impact parameter and the task evaluation parameter may include: taking the ratio of the resource consumption per unit time to the predicted fault duration as the priority score. The priority score of the subtask to be processed may be calculated by the following formula: ; in, express Time Fault Resource Node Pending subtasks on The priority score.

[0114] When comparing the priority scores, all pending subtasks on all faulty resource nodes in the target distributed system can be uniformly sorted, or they can be grouped according to the parallel capability of fault-tolerant scheduling and sorted by priority scores in the groups.

[0115] In some other optional implementations of the embodiments of the present invention, determining the task evaluation parameters of the subtask to be processed according to the load status of the subtask to be processed on the faulty resource node may also include: determining the task evaluation parameters according to the load status of the subtask to be processed on the faulty resource node and the task type priority parameter of the subtask to be processed. That is, when determining the priority score of the subtask to be processed, the task type priority of the subtask to be processed may also be combined, and specifically, the task type priority score of the subtask to be processed may be added to the priority score calculation formula of the subtask to be processed.

[0116] The training steps of the fault-tolerant processing action generation model may include: inputting sample state information into the fault-tolerant processing action generation model, and outputting sample fault-tolerant processing actions for the sample state information; after executing the sample fault-tolerant processing actions, obtaining updated sample state information; determining an environmental reward value based on the updated sample state information and a reward function; taking a group of sample fault-tolerant processing actions, sample fault-tolerant processing actions, updated sample state information, and environmental reward values ​​as a training sample; and using the training samples to update model parameters of the fault-tolerant processing action generation model.

[0117] In the embodiment of the present invention, the environmental reward value may adopt the fault-tolerant cost of executing the fault-tolerant processing action. In some optional implementations of the embodiment of the present invention, the fault-tolerant cost of executing the fault-tolerant processing action may be measured from the perspective of the time cost required to execute the fault-tolerant processing action, and the reward function may be a function of the fault-tolerant processing time of the task.

[0118] Fault-tolerant processing actions can be divided into three categories: task migration actions, task waiting actions, and task scaling actions. The time cost required to execute fault-tolerant processing actions can be determined by combining the service interruption time and transmission time caused by executing fault-tolerant processing actions. For the above three types of actions: Task migration action: Migrate the pending subtasks on the current faulty resource node to the resource node that is operating normally. The main time cost incurred in this process includes data transmission time and service recovery time.

[0119] Task waiting action: If the fault can be recovered in a short time, migration is not required and the pending subtasks are interrupted until the faulty resource node is recovered. The main cost of this process is the waiting time for service interruption.

[0120] Task scaling action: If the subtask to be processed has the same copy on other resource nodes, the services on other resource nodes can be scaled, and the service of the failed resource node will be terminated immediately. This process does not generate data migration, and the main cost is the service recovery time.

[0121] The fault-tolerant processing time may include a first processing time for a task migration action, a second processing time for a task waiting action, and a third processing time for a task scaling action; wherein the task migration action is to migrate the subtask to be processed from the faulty resource node to the first resource node, and the first processing time includes the data transmission time and the service recovery time at the first resource node; the task waiting action is to set the subtask to be processed to a service interruption state at the faulty resource node, and the second processing time is the service interruption waiting time; the task scaling action is to start the subtask to be processed at the second resource node where a copy of the subtask to be processed exists and terminate the subtask to be processed at the faulty resource node, and the third processing time is the service recovery time at the second resource node.

[0122] According to the training steps of the fault-tolerant processing action generation model and the types of fault-tolerant processing actions introduced in the embodiment of the present invention, the specific training steps of a fault-tolerant processing action generation model are introduced below.

[0123] set up The state parameters of the target distributed system observed at all times are ,in, express The basic environment status information of the target distributed system at all times (including network topology information, resource status information of resource nodes, and network status information between resource nodes), express The collection of various business tasks being run by all resource nodes in the target distributed system at any given moment. express Fault information of all faulty resource nodes in the target distributed system at all times.

[0124] set up The decision variable at time is ,in, Indicates the task migration action. Indicates that the task is waiting for action. Indicates a task scaling action.

[0125] Among them, you can configure and , When both are 0, it means that the pending subtasks will be migrated to the resource node On. When and , When both are 0, it means that the pending subtasks will be kept on the faulty resource node and wait, and no other operations will be performed. and , When both are 0, it means that the pending subtask will be processed on another resource node. Stretch on.

[0126] After executing the fault-tolerant processing action, the state information in the target distributed system is collected again, and the environmental reward value can be obtained , represents the time cost required to perform fault-tolerant processing actions to restore the service of the pending subtask. , Indicates the first processing time of the task migration action, Indicates the second processing time of the task waiting action. represents the third processing time of the task scaling action. Usually, since the decision variable is Only one type of fault-tolerant processing action can be executed within a period of time, and the time costs of the three types of fault-tolerant processing actions are mutually exclusive.

[0127] Construct an online decision generation network for deep reinforcement learning, that is, the initial fault-tolerant processing action generation model. The fault-tolerant processing action generation model can adopt a deep Q learning model (DQN). The initial fault-tolerant processing action generation model is , Generate model parameters for fault-tolerant processing actions. Input fault-tolerant processing action generation model, output fault-tolerant processing action After execution, the current stage environmental reward value is calculated through the reward function , the system enters the next observation state .

[0128] Multiple training samples are collected in the above way. Stored in the buffer pool D. When the number of training samples stored in the buffer pool D reaches the preset number of samples, the parameter update step of the fault-tolerant processing action generation model is entered.

[0129] In the process of updating the parameters of the fault-tolerant action generation model, M training samples are randomly selected from the cache pool D, and the target Q value is calculated according to the following formula: ; in, is a real coefficient between 0 and 1.

[0130] The loss function of the fault-tolerant action generation model can be: .

[0131] Update network parameters based on gradient back propagation , until the network converges. After convergence, the fault-tolerant processing action generation model outputs decision actions, namely fault-tolerant processing actions, based on the state information of the target distributed system at the current moment, and makes scheduling management decisions for the pending subtasks on the faulty resource nodes at different stages.

[0132] To ensure the availability of the fault-tolerant processing actions actually executed, the embodiment of the present invention first generates multiple fault-tolerant processing actions using a fault-tolerant processing action generation model, and then uses an action filter to filter out available actual fault-tolerant processing actions.

[0133] In the embodiment of the present invention, the availability evaluation is performed on each fault-tolerant processing action, and the actual fault-tolerant processing action is determined according to the availability evaluation result, which may include: the availability evaluation is performed on the fault-tolerant processing actions in descending order of the corresponding environmental reward values, until a fault-tolerant processing action that meets the availability evaluation condition is obtained as the actual fault-tolerant processing action. That is, the fault-tolerant processing action with the largest environmental reward value may be selected from the available fault-tolerant processing actions as the actual fault-tolerant processing action.

[0134] In a specific implementation, using an action filter to perform availability evaluation on a fault-tolerant processing action may include performing a security evaluation on the fault-tolerant processing action. If the fault-tolerant processing action meets the preset security conditions, then it is determined that the fault-tolerant processing action meets the availability evaluation conditions. Then, it can be set express The safety action indicator at the moment first sorts the fault-tolerant processing actions output by the fault-tolerant processing action generation model according to the environmental reward value from large to small, selects the first K fault-tolerant processing actions, and then checks whether they meet the availability evaluation conditions one by one according to the environmental reward value from large to small. If not, , until you select The fault-tolerant processing action is taken as the actual fault-tolerant processing action and executed.

[0135] The task scheduling method provided by the embodiment of the present invention, when dynamically scheduling tasks with dependencies, also provides a sustainable fault-tolerant processing solution for resource nodes executing subtasks, thereby further improving the availability of the distributed scheduling solution.

[0136] It should be noted that in the embodiments of the task scheduling method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for convenience of description are not the only implementation form of the task scheduling method provided in the embodiments of the present invention.

[0137] The above describes in detail various embodiments corresponding to the task scheduling method. On this basis, the present invention also discloses a task scheduling device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0138] The task scheduling device provided by the embodiment of the present invention may include: An acquisition unit, used to acquire task dependencies between multiple subtasks of a target task; A determination unit is used to determine the subtasks to be assigned that can be currently performed in parallel according to the task dependency relationship; and to determine the first task execution cost of the subtask to be assigned after assigning the subtask to the resource node and the second task execution cost of the next subtask of the subtask to be assigned according to the status information of each resource node in the target distributed system; An optimization calculation unit is used to determine the parallel task execution cost according to the first task execution cost of each subtask to be assigned, with minimizing the parallel task execution cost as the optimization goal and with the second task execution cost of each next subtask satisfying the cost constraint as the constraint condition, to perform optimization calculation and obtain the target resource node corresponding to each subtask to be assigned; The control unit is used to allocate the subtasks to be allocated to the target resource nodes for execution.

[0139] It should be noted that in each implementation of the task scheduling device provided in the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0140] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0141] Figure 5 A structural diagram of a task scheduling device provided by an embodiment of the present invention.

[0142] like Figure 5 As shown, the task scheduling device provided in an embodiment of the present invention includes: a memory 510, used to store a computer program 511; a processor 520, used to execute the computer program 511, and when the computer program 511 is executed by the processor 520, the steps of the task scheduling method provided in any of the above embodiments are implemented.

[0143] Among them, the processor 520 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 520 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array. The processor 520 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 520 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 520 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0144] The memory 510 may include one or more non-volatile storage media, which may be non-transitory. The memory 510 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 510 is at least used to store the following computer program 511, wherein the computer program 511, after being loaded and executed by the processor 520, can implement the relevant steps in the task scheduling method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 510 may also include an operating system 512 and data 513, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 512 may be Windows or other types of operating systems. Data 513 may include, but is not limited to, the data involved in the above method.

[0145] In some embodiments, the task scheduling device may further include a display screen 530 , a power supply 540 , a communication interface 550 , an input / output interface 560 , a sensor 570 , and a communication bus 580 .

[0146] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the task scheduling device, and may include more or fewer components than those shown in the figure.

[0147] The task scheduling device provided in the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the task scheduling method provided in the above embodiment, and the effect is the same as above.

[0148] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the task scheduling method provided in any one of the above embodiments can be implemented.

[0149] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0150] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the task scheduling method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0151] An embodiment of the present invention provides a computer program product, including a computer program, which implements the steps of the task scheduling method provided in any one of the above embodiments when executed by a processor.

[0152] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the task scheduling method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0153] The above is a detailed introduction to a task scheduling method, device, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0154] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A task scheduling method, characterized in that: include: Get the task dependencies between multiple subtasks of the target task; According to the task dependency, determine the subtasks to be assigned that can be currently performed in parallel; Determine, according to the status information of each resource node in the target distributed system, a first task execution cost of the subtask to be assigned after the subtask to be assigned is assigned to the resource node and a second task execution cost of the next subtask of the subtask to be assigned; Determine the parallel task execution cost according to the first task execution cost of each of the subtasks to be assigned, take minimizing the parallel task execution cost as the optimization goal, take the second task execution cost of each of the next subtasks satisfying the cost constraint as the constraint condition, perform optimization calculation, and obtain the target resource node corresponding to each of the subtasks to be assigned; Allocate the subtask to be assigned to the target resource node for execution.

2. The task scheduling method according to claim 1, characterized in that: The first task execution cost is the sum of a first time for the resource node to acquire the calculation parameters required for the subtask to be assigned and a second time for the resource node to execute the subtask to be assigned.

3. The task scheduling method according to claim 2, characterized in that: The parallel task execution cost is the sum of the first task execution costs of the subtasks to be assigned.

4. The task scheduling method according to claim 2, characterized in that: The parallel task execution cost is the maximum value of the first task execution costs of the subtasks to be assigned.

5. The task scheduling method according to claim 2, characterized in that: If the resource node has the task parameters of the subtask to be assigned, the first time is the time when the resource node obtains the input data of the subtask to be assigned; If the resource node does not have the task parameters of the subtask to be assigned, the first time is determined according to the time when the resource node obtains the task parameters of the subtask to be assigned and the time when the resource node obtains the input data of the subtask to be assigned.

6. The task scheduling method according to claim 1, characterized in that: The second task execution cost is the sum of a third time for the resource node executing the next subtask to obtain the calculation parameters of the next subtask and a fourth time for the resource node to execute the next subtask.

7. The task scheduling method according to claim 1, characterized in that: Determining, according to the status information of each resource node in the target distributed system, a first task execution cost of the subtask to be assigned after assigning the subtask to be assigned to the resource node and a second task execution cost of the next subtask of the subtask to be assigned, comprising: Determining, according to the status information of each of the resource nodes, the first task execution cost of the subtask to be assigned after the subtask to be assigned is assigned to the resource node; Determine a first number of resource nodes as candidate resource nodes corresponding to the subtasks to be assigned according to the order of the execution costs of the first tasks corresponding to the subtasks to be assigned from small to large; Determine the second task execution cost of the next subtask after allocating the to-be-allocated subtask to the candidate resource node.

8. The task scheduling method according to claim 1, characterized in that: Also includes: Determine a first number of resource nodes as candidate resource nodes corresponding to the subtasks to be assigned according to the order of the execution costs of the first tasks corresponding to the subtasks to be assigned from small to large; After the subtask to be assigned is assigned to the target resource node for execution, if the target resource node fails, the subtask to be assigned is migrated to another candidate resource node corresponding to the subtask to be assigned.

9. The task scheduling method according to claim 8, characterized in that: Migrating the to-be-assigned subtask to another of the candidate resource nodes corresponding to the to-be-assigned subtask includes: Performing availability evaluation on the other multiple candidate resource nodes corresponding to the subtask to be assigned, and obtaining an availability evaluation value corresponding to each candidate resource node; Migrate the subtask to be assigned to the candidate resource node with the greatest availability.

10. The task scheduling method according to claim 9, characterized in that: Performing availability evaluation on the other multiple candidate resource nodes corresponding to the subtask to be assigned to obtain an availability evaluation value corresponding to each candidate resource node, including: The availability of the candidate resource node is evaluated according to the resources required by the to-be-assigned subtask, the available resources of the candidate resource node, and the historical failure data of the candidate resource node to obtain the availability evaluation value of the candidate resource node.

11. The task scheduling method according to claim 1, characterized in that: The target distributed system is an edge computing system; The subtask to be assigned is the subtask that currently needs to be offloaded from the terminal device to the target edge computing device in the edge computing system.

12. The task scheduling method according to claim 1, characterized in that: Get the task dependencies between multiple subtasks of the target task, including: A directed acyclic graph corresponding to the target task is obtained, wherein the graph nodes in the directed acyclic graph correspond to the subtasks one-to-one, and the directed edges in the directed acyclic graph are used to represent the dependency relationship between the subtasks.

13. A task scheduling device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the task scheduling method according to any one of claims 1 to 12 are implemented.

14. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the task scheduling method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the task scheduling method according to any one of claims 1 to 12 are implemented.

Images