Distributed task scheduling method and apparatus, storage medium, and electronic device

Abstract

The present description provides a distributed task scheduling method and apparatus, a storage medium, and an electronic device. The method comprises: determining multiple candidate allocation strategies on the basis of tasks to be scheduled and state parameters of computing nodes of a computing platform, and separately determining a first objective function of each candidate allocation strategy on the basis of a preset allocation constraint condition; separately determining a second objective function of each candidate allocation strategy on the basis of whether the candidate allocation strategy meets the allocation constraint condition; determining fitness of the candidate allocation strategies on the basis of the first objective functions and the second objective functions, determining a target allocation strategy by means of an evolutionary algorithm, and on the basis of the target allocation strategy, scheduling the tasks to the computing nodes for computation.

Description

Distributed task scheduling method, device, storage medium and electronic device Technical Field

[0001] This specification relates to the field of communication technology, and in particular to a distributed task scheduling method, device, storage medium, and electronic device. Background Art

[0002] In recent years, with the development of internet technology, the use of distributed frameworks for task scheduling has become commonplace. In graph computing platforms, by assigning different tasks to different processors, each processor can simultaneously process its assigned tasks, improving task processing efficiency.

[0003] However, when graph computing platforms use traditional distributed task scheduling methods to handle large-scale tasks, they often focus too much on local optimization, resulting in load imbalance. For example, a server receives five tasks and assigns each task to a different compute node for processing. Since it is known that using compute node A to process tasks can shorten task processing time, the server assigns four tasks to compute node A and randomly assigns the remaining tasks to a compute node. The remaining compute nodes, which have not been assigned tasks, remain idle.

[0004] Although compute node A processes tasks quickly, it needs to handle multiple tasks, and each subsequent task must wait for the previous one to complete. This results in tasks not being processed in a timely manner, and some compute nodes not being effectively utilized, leading to load imbalance.

[0005] To this end, this specification provides a distributed task scheduling method.

[0006] Summary of the Invention

[0007] The present application provides a distributed task scheduling method, device, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0008] This application provides a distributed task scheduling method, including:

[0009] Determine the state parameters of multiple tasks to be scheduled and multiple computing nodes of the distributed computing platform;

[0010] Determining multiple candidate allocation strategies based on the tasks and the computing nodes, wherein each candidate allocation strategy includes an allocation relationship between the tasks and the computing nodes;

[0011] For each candidate allocation strategy, a first objective function is determined according to a preset allocation constraint; when the candidate allocation strategy satisfies the preset allocation constraint, a second objective function is determined according to a preset optimization objective; when the candidate allocation strategy does not satisfy the preset allocation constraint, a second objective function is determined according to a bias parameter and the preset optimization objective, wherein the bias parameter is used to increase the distance between the second objective function and the desired target;

[0012] According to the preset evolutionary algorithm, the following iterative process is performed until the evolution end condition is met: determining the fitness of each current candidate allocation strategy based on the first objective function and the second objective function of each current candidate allocation strategy, so as to adjust each current candidate allocation strategy to determine each next generation candidate allocation strategy;

[0013] Determine the target allocation strategy based on the fitness of each candidate allocation strategy adjusted when the evolution end condition is met;

[0014] According to the target allocation strategy, the tasks are dispatched to the computing nodes for computing and processing.

[0015] Optionally, determining the multiple tasks to be scheduled specifically includes:

[0016] Receive graph computation requests;

[0017] Determining, according to the graph computation request, a directed acyclic graph corresponding to the graph computation to be executed;

[0018] According to the directed acyclic graph, tasks to be scheduled and dependencies between tasks are determined.

[0019] Optionally, for each computing node, determining candidate allocation strategies based on the tasks and the computing nodes specifically includes:

[0020] According to the dependency relationship between the tasks, several candidate equal distribution strategies are generated for distributing the tasks to different computing nodes.

[0021] Optionally, the status parameters include at least: the communication resources of the computing node, the energy consumption of the computing node, the storage resources of the computing node, and one of the computing resources of the computing node, wherein the communication resources include frequency, bandwidth, real-time status, and real-time hardware parameters; the energy consumption includes power and heat loss; the storage resources include memory resources and external storage resources; and the computing resources include CPU resources and network resources.

[0022] Optionally, determining the first objective function according to the preset allocation constraint condition specifically includes:

[0023] Determine the number of tasks to be assigned to each computing node in the candidate allocation strategy;

[0024] Determine the number of computing nodes whose number of assigned tasks is not greater than a preset first value as a first constraint value;

[0025] A first sub-item of a first objective function is determined according to the first constraint value.

[0026] Optionally, determining the first objective function according to the preset allocation constraint condition further includes:

[0027] For each state parameter, predict the predicted state of each computing node when executing the task assigned by the candidate allocation strategy;

[0028] Determining an optimization objective function of the state parameter according to the predicted state and the corresponding state threshold;

[0029] According to the optimization objective function of each state parameter, a second sub-item of the first objective function is determined.

[0030] Optionally, before determining the second objective function, the method further includes:

[0031] Determine whether there is a candidate allocation strategy among the candidate allocation strategies that does not meet the preset allocation constraint conditions;

[0032] If so, the preset second value is determined to be the bias parameter of the second objective function.

[0033] Optionally, determining the second objective function includes:

[0034] When the candidate allocation strategy satisfies the preset allocation constraint condition, determining the second objective function as the total time consumed by each computing node when executing the task allocated by the candidate allocation strategy;

[0035] When the candidate allocation strategy does not satisfy the preset allocation constraint condition, the second objective function is determined to be the total time consumed by each computing node after the bias parameter is adjusted to execute the task allocated by the candidate allocation strategy.

[0036] Optionally, an iteration process specifically includes:

[0037] Determining the fitness of each current candidate allocation strategy according to the first objective function and the second objective function of each current candidate allocation strategy, and storing the fitness;

[0038] According to the fitness of each current candidate allocation strategy, a specified number of candidate allocation strategies are selected from the current candidate allocation strategies as a parent population;

[0039] Generating a recombination allocation strategy corresponding to each candidate allocation strategy in the parent population according to the parent population and the mutation parameter;

[0040] The candidate allocation strategies for the next population are determined according to the candidate allocation strategies and the recombination allocation strategies in the parent population.

[0041] Optionally, generating a recombination allocation strategy corresponding to each candidate allocation strategy in the parent population according to the parent population and the mutation parameter specifically includes:

[0042] Determine the preset transposed matrix, the initialized normal distribution vector, and the mutation intensity of the next generation population, and determine the mutation parameters;

[0043] For each candidate allocation strategy in the parent population, mutate the average allocation strategy of the parent population according to the mutation parameter to determine a mutation result;

[0044] Determining whether the mutation result exceeds a preset parameter range;

[0045] If yes, adjusting the mutation result according to the parameter range so that the mutation result does not exceed the parameter range as a recombination allocation strategy;

[0046] If not, the mutation result is used as the recombination allocation strategy;

[0047] The average allocation strategy is determined according to each candidate allocation strategy of the parent population.

[0048] Optionally, before generating the recombination allocation strategy corresponding to each candidate allocation strategy in the parent population according to the parent population and the mutation parameter, the method further includes:

[0049] Determine the difference between the number of iterations for the last adjustment of the mutation parameters and the number of iterations for the current adjustment;

[0050] Determining whether the difference reaches a preset round number difference;

[0051] If so, the first objective function gradient and the second objective function gradient are determined, and the mutation parameter is adjusted according to the determined gradients.

[0052] Optionally, adjusting the mutation parameter according to the determined gradient specifically includes:

[0053] When the gradient of the second objective function is greater than the first preset value, the gradient of the first objective function is (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value, adjusting the mutation parameter;

[0054] When the second objective function gradient value is (negative first preset value, first preset value], the first objective function gradient value is (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value or equal to 0, adjust the mutation parameter;

[0055] When the second objective function gradient takes a value of (negative first preset value, first preset value], and the first objective function gradient is less than or equal to the negative first preset value, adjusting the variation parameter, so that the variation parameter after adjustment is consistent with the variation parameter before adjustment;

[0056] When the gradient of the second objective function is less than or equal to the negative first preset value or greater than the first preset value, and the gradient of the first objective function is less than or equal to the negative first preset value or greater than the first preset value, the variation parameter is adjusted, and the adjusted variation parameter is consistent with the variation parameter before adjustment.

[0057] Optionally, the variation parameters include at least: a normal distribution vector and a variation intensity;

[0058] Adjusting the mutation parameters specifically includes:

[0059] Determine the pseudo-inverse matrix based on the identity matrix;

[0060] Determining a mutation vector according to a ratio of the mutation intensity of the previous generation population to the change of the candidate allocation strategy;

[0061] Determining a normal distribution vector according to the product of the pseudo inverse matrix and the variation vector;

[0062] Adjust the mutation intensity according to the preset maximum mutation intensity.

[0063] Optionally, determining the fitness of each current candidate allocation strategy according to the first objective function and the second objective function of each current candidate allocation strategy specifically includes:

[0064] Determining the recombination weights corresponding to the candidate allocation strategies in the parent population according to the positions of the current candidate allocation strategies in the population;

[0065] Using the initial weight as the weight of the second objective function, and determining the weight of the first objective function according to the weight of the second objective function, wherein the weight of the second objective function and the weight of the first objective function are normalized;

[0066] The fitness of each current candidate allocation strategy is determined according to the first objective function and the second objective function of each current candidate allocation strategy and the corresponding weights.

[0067] Optionally, the method further includes:

[0068] Determine the difference between the number of iterations for the last adjustment of the mutation parameters and the number of iterations for the current adjustment;

[0069] Determining whether the difference reaches a preset round number difference;

[0070] If so, adjusting the weight of the second objective function;

[0071] Wherein, when the gradient of the second objective function is greater than the first preset value, the gradient value of the first objective function is (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value, the weight of the second objective function is determined to be any weight value;

[0072] When the second objective function gradient value is (negative first preset value, first preset value], the first objective function gradient value is (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value, the weight of the second objective function is determined to be any weight value;

[0073] When the second objective function gradient value is (negative first preset value, first preset value), the first objective function gradient value is (negative first preset value, first preset value), and the optimal solution of the first objective function is equal to 0, the weight of the second objective function is adjusted, and the weight of the second objective function after adjustment is consistent with the weight of the second objective function before adjustment;

[0074] When the second objective function gradient takes a value of (negative first preset value, first preset value], and the first objective function gradient is less than or equal to the negative first preset value, determining the weight of the second objective function according to the maximum weight, the initial weight, the last adjusted weight of the second objective function, and the adaptive strength;

[0075] When the gradient of the second objective function is less than or equal to the negative first preset value, and the gradient of the first objective function is less than or equal to the negative first preset value or greater than the first preset value, the weight of the second objective function is determined based on the maximum weight, the initial weight, the last adjusted weight of the second objective function, and the adaptive strength.

[0076] This specification provides a distributed task scheduling device, including:

[0077] The parameter module determines the state parameters of the multiple tasks to be scheduled and the multiple computing nodes of the distributed computing platform; the candidate allocation module determines multiple candidate allocation strategies based on the tasks and the computing nodes, wherein each candidate allocation strategy includes an allocation relationship between the tasks and the computing nodes; the first constraint module determines the first objective function for each candidate allocation strategy based on the preset allocation constraint conditions; the second constraint module determines the second objective function based on the preset optimization goal when the candidate allocation strategy meets the preset allocation constraint conditions; when the candidate allocation strategy does not meet the preset allocation constraint conditions, it determines the second objective function based on the bias parameter and the preset Optimize the target and determine the second objective function, wherein the bias parameter is used to increase the distance between the second objective function and the desired target; the optimization module executes the following iterative process according to the preset evolutionary algorithm until the evolution end condition is met: determine the fitness of each current candidate allocation strategy according to the first objective function and the second objective function of each current candidate allocation strategy, so as to adjust each current candidate allocation strategy to determine each next generation candidate allocation strategy; determine the target allocation strategy according to the fitness of each candidate allocation strategy adjusted when the evolution end condition is met; and schedule the tasks to the computing nodes for computational processing according to the target allocation strategy.

[0078] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed task scheduling method is implemented.

[0079] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned distributed task scheduling method when executing the program.

[0080] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0081] In the distributed task scheduling method provided in this specification, the scheduling center determines the status parameters of the computing nodes of the task to be scheduled and the computing platform, and determines the candidate allocation strategy; determines the first objective function based on the candidate allocation strategy and the preset allocation constraints; determines the second objective function based on the different situations in which the candidate allocation strategy satisfies the allocation constraints; determines the fitness of the candidate allocation strategy according to the first objective function and the second objective function according to the evolutionary algorithm to adjust the candidate allocation strategy; until the evolution end condition is met, determines the target allocation strategy based on the adjusted fitness of the candidate allocation strategy, and schedules the task to the computing node for calculation according to the target allocation strategy.

[0082] It can be seen from the above method that, since the candidate allocation strategies that do not meet the allocation constraints are not directly abandoned, but the strong constraints are converted into objective functions, the restrictions of the constraints on optimization are reduced, so that the candidate allocation strategies can be optimized in a larger range, the accuracy of determining the target allocation strategy is improved, and the utilization rate of computing nodes in distributed task scheduling scenarios is higher and the load is more balanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0084] FIG1 is a flow chart of a distributed task scheduling method provided in this specification;

[0085] FIG2 is a reference table for adjusting the fitness of an evolutionary strategy provided in this specification;

[0086] FIG3 is a weight reference table for adjusting the evolutionary strategy provided in this specification;

[0087] FIG4 is a flow chart of a task scheduling process for a graph computing platform provided in this specification;

[0088] FIG5 is a schematic diagram of a distributed task scheduling device provided in this specification;

[0089] FIG6 is a schematic diagram of the device corresponding to FIG1 provided in this specification. DETAILED DESCRIPTION

[0090] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0091] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0092] FIG1 is a flow chart of a distributed task scheduling method in this specification, which specifically includes the following steps.

[0093] S101: Determine state parameters of multiple tasks to be scheduled and multiple computing nodes of a distributed computing platform.

[0094] Distributed task scheduling is a multitasking method that assigns tasks to multiple different computing nodes. Typically, this assignment occurs at the distributed computing platform's dispatch center, typically a server, but it can also be other devices that perform the same or similar functions. For ease of description, we'll use a server as an example.

[0095] In one or more embodiments of this specification, the server must first determine the tasks that currently need to be scheduled, that is, the tasks to be scheduled. After receiving a task, it can be split into several subtasks as tasks to be scheduled. Of course, there is no limit to the number of tasks received, that is, it can also be allocated after receiving multiple tasks. As long as the received tasks can exist independently and / or be split independently, distributed task scheduling can be performed on each allocated and / or split task. In order to determine how to allocate, generally speaking, a target allocation strategy will be adopted to try to balance the efficiency and load of each computing node, so that each computing node can achieve the highest working efficiency. Therefore, the server should have the ability to collect the status parameters of each computing node and implement the collection of the status parameters of each computing node.

[0096] Specifically, because the state parameters required for different distributed task scheduling by the server vary, the state parameters that need to be collected are also different. Generally speaking, the state parameters of each computing node should include at least one of the following: the computing node's communication resources, the computing node's energy consumption, the computing node's storage resources, and the computing node's computing resources. Communication resources include, for example, frequency, bandwidth, real-time status, and real-time hardware parameters, which can be extracted from the same type of parameters; energy consumption includes, for example, power and heat loss, which can be extracted from the same type of parameters; storage resources include, for example, memory resources and external storage resources, which can be extracted from the same type of parameters; and computing resources include, for example, CPU resources and network resources, which can be extracted from the same type of parameters.

[0097] Since the graph calculation requests received by the server are different, the directed acyclic graph corresponding to the graph calculation to be executed can be determined based on each graph calculation request. Then, based on the directed acyclic graph corresponding to each graph calculation, the tasks to be scheduled are determined. Since in the graph calculation request, there are often correlations between the tasks to be scheduled, especially one task often has a great impact on the execution conditions before and after another task. For example, A and Task B require similar computing power, and Task B is a task that can only be performed based on the data obtained by Task A. Traditional allocation methods, such as Shortest Job First (SJF) or Least Load First (LLF), may result in two tasks being assigned to two different computing nodes with similar computing power at the same time, or assigning another task to another different computing node with similar computing power when one task is executed. Regardless of the above allocation method, the computing node performing Task B will be in a waiting state until the computing node performing Task A is completed, which greatly reduces the efficiency of task execution. Therefore, in the embodiment of this specification, before performing task allocation, the dependency relationship between the tasks can be determined first.

[0098] Furthermore, in the embodiments of this specification, the server can also determine information such as the computational complexity, input data size, output data size, and estimated execution time of each task to evaluate the execution effect of each task on the distributed computing platform.

[0099] S103: Determine multiple candidate allocation strategies based on the tasks and the computing nodes, where each candidate allocation strategy includes an allocation relationship between the tasks and the computing nodes.

[0100] In one or more embodiments of this specification, the server first determines the tasks to be scheduled output in S101 and generates candidate allocation strategies based on the locations of the computing nodes in the distributed computing platform and the dependencies of the tasks to be scheduled. In each candidate allocation strategy, there is no limit on the number of scheduling tasks assigned to each computing node, no order requirement, and no dependencies need to be met.

[0101] S105: For each candidate allocation strategy, determine a first objective function according to preset allocation constraints.

[0102] In one or more embodiments of the present specification, after determining each candidate allocation strategy, the server may determine a corresponding first objective function for each candidate allocation strategy, wherein the first objective function is determined according to a preset allocation constraint.

[0103] Specifically, in this specification, the server may pre-store allocation constraints. These allocation constraints are used to restrict which tasks are assigned to which computing nodes. The pre-set allocation constraints may include one or more items. This specification does not limit the specific allocation constraints and may be set as needed. Generally, allocation constraints can be set based on manual experience.

[0104] In one or more embodiments of this specification, the allocation constraint may limit the number of tasks that can be assigned to a computing node. Assume that the allocation constraint is to limit the assignment of only one task to a computing node. The server may then determine the number of tasks assigned to each computing node in the candidate allocation strategy, and determine the number of computing nodes whose assigned number of tasks is no greater than a preset first value as a first constraint value. The allocation constraint may be expressed using the following formula:

[0105] Among them, x ij It is used to indicate whether the i-th task is assigned to the j-th computing node, x ij =1 means that the i-th task is assigned to the j-th computing node, x ij = 0 means that the i-th task is not assigned to the j-th computing node, then x j Indicates the number of tasks assigned to the jth computing node. If x j If it is greater than 1, the candidate allocation strategy does not meet the allocation constraints.

[0106] In one or more embodiments of this specification, the allocation constraint may limit the total time taken by each computing node to complete the assigned task to no more than a preset total time. The allocation constraint may be expressed using the following formula:

[0107] Σ(x ij ×T ij )≤makespan, that is

[0108] Among them, T ij represents the execution time of task i on processor j, then Σ(x ij ×T ij ) represents the total time it takes for each computing node to complete each assigned task, and makespan represents the preset total time. The preset total time can be set as needed and is not limited in this manual.

[0109] In one or more embodiments of this specification, the allocation constraint may limit the communication volume required by each computing node to complete the assigned task to no more than a preset total communication volume. The allocation constraint may be expressed using the following formula:

[0110] Σ(x ik ×xjl ×C ij )≤C max ,Right now

[0111] Among them, (x ik ×x jl ×C ij ) represents the communication volume required by computing nodes k and l to transmit the data of task i and task j when task i is assigned to computing node k and task j is assigned to computing node l. max Indicates the preset total communication volume.

[0112] In one or more embodiments of this specification, the allocation constraint condition may limit the total energy consumption required by each computing node to complete the assigned task to not exceed a preset total energy consumption. The allocation constraint condition may be expressed using the following formula:

[0113] Σ(x ij ×E ij )≤E jmax ,Right now

[0114] Among them, E ij represents the energy consumption of computing node j executing task i, E jmax represents the maximum energy consumption allowed by computing node j.

[0115] In one or more embodiments of this specification, the allocation constraint may limit the memory required by each computing node to complete the assigned task to no more than a preset total memory usage. The allocation constraint may be expressed using the following formula:

[0116] Σ(x ij ×M ij )≤M jmax ,Right now

[0117] Among them, M ij represents the memory usage of computing node j executing task i, M jmax Indicates the maximum memory usage allowed for compute node j.

[0118] Of course, in this specification, the above allocation constraints can be used individually or in combination, and the specific allocation constraints can be set as needed. In this specification, for the convenience of description, the following uses the above allocation constraints as an example, and the allocation constraints include:

[0119] However, the aforementioned allocation constraints are strong constraints. If the allocation constraints are used directly to determine the allocation strategy, any candidate allocation strategy that does not meet any of the allocation constraints will no longer be considered, resulting in a smaller search space for subsequent optimization and the possibility of disconnected search spaces. In reality, it is possible that a candidate allocation strategy that does not meet a certain constraint may be superior overall. For example, there are allocation strategies A and B. A completes the computing task half an hour earlier than B, but does not meet the preset energy consumption constraints, which will cause a computing node to overheat for a short period of time. B fully meets all constraints. In this case, strategy A is preferred and strategy B is discarded.

[0120] Therefore, in this specification, for each candidate allocation strategy, the server may determine a first objective function according to preset allocation constraints.

[0121] Specifically, in the above allocation constraints, The first allocation constraint is a constraint on the number of tasks assigned to a computing node, while the other allocation constraints are constraints on the resource consumption of the computing node. Determine the first sub-item of the first objective function. That is, the server may first determine the number of tasks assigned to each computing node in the candidate allocation strategy, then determine the number of computing nodes whose assigned task number is greater than a preset first value as a first constraint value; and determine the first sub-item of the first objective function based on the first constraint value. This is shown in the following formula:

[0122] Furthermore, based on the state parameters, the predicted state of each computing node when executing the assigned task according to the candidate allocation strategy is predicted. Determine the second sub-term of the first objective function as shown in the following formula:

[0123] Add the first sub-term and the second sub-term to get the first objective function, as shown in the following formula:

[0124] Through the above-mentioned first objective function, the strong constraints of each allocation constraint condition are converted into the expectation of the first objective function, so that the server can search in a larger search space and determine the final allocation strategy.

[0125] S107: When the candidate allocation strategy satisfies the preset allocation constraint, a second objective function is determined according to the preset optimization goal; when the candidate allocation strategy does not satisfy the preset allocation constraint, a second objective function is determined according to the bias parameter and the preset optimization goal, where the bias parameter is used to increase the distance between the second objective function and the desired goal.

[0126] In one or more embodiments of this specification, the above step S105 converts the allocation constraint used to screen the allocation strategy into a first objective function. However, there may be multiple candidate allocation strategies that meet the preset allocation constraint, so it is necessary to set a second objective function, such as minimizing the total time required to complete each task. The second objective function can be expressed by the following formula: max =max(C1,C2,…,C M ),

[0127] Right now

[0128] Among them, C M represents the total time taken by computing node M to complete each task, C max Indicates the total time required to complete the task by the candidate allocation strategy.

[0129] Furthermore, because the server does not screen the candidate allocation strategies determined in step S103 based on the aforementioned allocation constraints, each candidate allocation strategy determined may fully satisfy the allocation constraints, partially satisfy the allocation constraints, or even fail to satisfy the allocation constraints at all. Therefore, for candidate allocation strategies in different scenarios, the server can further determine a second objective function by adding a bias parameter to target candidate allocation strategies that do not satisfy the preset allocation constraints. This ensures that the target allocation strategy ultimately determined is optimized, for example, to minimize the total time required to complete each task while maximizing the satisfaction of the allocation constraints.

[0130] Specifically, based on the second objective function set in this step to minimize the total time required to complete each task, if all candidate allocation strategies determined in S103 satisfy the allocation constraint, the server can directly determine the second objective function. The second objective function can be expressed as the following formula:

[0131] Right now

[0132] in, Indicates the total time required for candidate allocation strategies to complete tasks, Indicates that all candidate allocation strategies satisfy the allocation constraints.

[0133] In the case where part of the candidate allocation strategy determined in S103 does not satisfy the allocation constraint, the server may add a bias parameter to the part of the candidate allocation strategy that does not satisfy the allocation constraint to determine a second objective function. The second objective function may be expressed as follows:

[0134] Where C represents the added bias parameter, Indicates that the candidate allocation strategy does not satisfy the allocation constraints.

[0135] In the case that none of the candidate allocation strategies determined in S103 satisfy the allocation constraint, the server may add a bias parameter to the portion of the candidate allocation strategy that satisfies the allocation constraint to determine a second objective function. The second objective function may be expressed as follows:

[0136] Right now

[0137] This allocation method utilizes a matrix adaptation evolution strategy based on auxiliary and equivalent objectives. This strategy can simultaneously address multiple allocation objectives, enabling the target allocation strategy to achieve global load balancing. As the name suggests, this strategy requires the simultaneous calculation of both the auxiliary objective (the first objective function in S105) and the equivalent objective (the second objective function in S107).

[0138] S109: According to the preset evolutionary algorithm, iterate the following process until the evolution end condition is met, and determine the target allocation strategy according to the fitness of each candidate allocation strategy: determine the fitness of each current candidate allocation strategy according to the first objective function and the second objective function of each current candidate allocation strategy, so as to adjust each current candidate allocation strategy to determine each next generation candidate allocation strategy.

[0139] In one or more embodiments of this specification, the server may employ a preset evolutionary algorithm to determine the fitness of each candidate allocation strategy and adjust each candidate allocation strategy during each iteration, thereby searching the search space for the optimal allocation strategy. When the evolutionary algorithm iteration meets an evolution termination condition, the target allocation strategy is determined based on the determined fitness of each candidate allocation strategy.

[0140] Specifically, for each iteration of the server, the initial candidate allocation strategies in the iteration are treated as individuals to be evolved in the population. Based on the first objective function and the second objective function of each candidate allocation strategy, the fitness of each candidate allocation strategy is determined and stored. The fitness is determined using the following formula:

[0141] in, represents the first objective function obtained by S105, represents the second objective function obtained by S107.

[0142] Secondly, according to the fitness of each candidate allocation strategy, a specified number of candidate allocation strategies are selected from each candidate allocation strategy as the parent population. Determine the parent population, where λ represents each candidate allocation strategy.

[0143] Then, based on the parent population and preset mutation parameters, a recombination allocation strategy corresponding to each candidate allocation strategy in the parent population is generated. Specifically, for each candidate allocation strategy in the parent population, the server may determine a mutation value range and a mutation direction based on the preset mutation parameters. That is, the server may determine a random value range for the mutation parameter based on a preset initial mutation strength and a preset maximum mutation strength, and a random mutation direction based on a preset effective population size, a preset search path learning rate, and a preset evolutionary path. Thus, a recombination allocation strategy for the candidate allocation strategy is generated.

[0144] Finally, the candidate allocation strategies in the next population are determined according to the candidate allocation strategies and the recombinant allocation strategies in the parent population.

[0145] That is, for each candidate allocation strategy in the parent population, its recombination allocation strategy is determined, and after adjustment based on the recombination allocation strategy, the adjusted candidate allocation strategy is used as the candidate allocation strategy in the next population.

[0146] Repeat the above iterative process until the evolution end condition is met, and then determine the target allocation strategy based on the fitness of each stored candidate allocation strategy.

[0147] It should also be noted that during each iteration of one or more embodiments of this specification, because the second objective function is used to minimize the total time consumption of each task, when all candidate allocation strategies satisfy the allocation constraints, i.e., when the first objective function values ​​are all 0, the optimization goal is to obtain a candidate allocation strategy with the lowest fitness. In this case, the server may select the allocation strategy with the lowest fitness as the target allocation strategy based on the fitness of each stored candidate allocation strategy. Alternatively, when not all candidate allocation strategies satisfy the allocation constraints, i.e., when the first objective function values ​​are not all 0, the server may sort the candidate allocation strategies in ascending order of fitness, determine the candidate allocation strategies whose difference from the top candidate allocation strategy is within a preset third value, and then determine the candidate allocation strategy with the first objective function value of 0 as the target allocation strategy. In other words, if the candidate allocation strategy with the lowest fitness fails to satisfy the allocation constraints, the server may also determine a candidate allocation strategy that satisfies the allocation constraints and whose fitness is not significantly different from the candidate allocation strategy with the lowest fitness (i.e., the difference is within the preset third value) as the target allocation strategy.

[0148] S111: According to the target allocation strategy, each task is dispatched to each computing node for computing processing.

[0149] In one or more embodiments of this specification, the server must first determine the target allocation strategy output by S109, and then allocate the tasks to be scheduled determined in S101 to the computing nodes determined in S101 through the target allocation strategy according to the target allocation strategy, thereby completing distributed task scheduling.

[0150] Based on the distributed task scheduling method provided in Figure 1, the server determines the state parameters of the tasks to be scheduled and the computing nodes of the computing platform, determines the candidate allocation strategy, and determines the first objective function based on the candidate allocation strategy and the preset allocation constraints; determines the second objective function according to different situations in which the candidate allocation strategy satisfies the allocation constraints; determines the fitness of the candidate allocation strategy according to the first objective function and the second objective function according to the evolutionary algorithm, so as to adjust the candidate allocation strategy until the evolution end condition is met, determines the target allocation strategy according to the adjusted fitness of the candidate allocation strategy, and schedules the task to the computing node for calculation according to the target allocation strategy.

[0151] It can be seen from the above method that, since the candidate allocation strategies that do not meet the allocation constraints are not directly abandoned, but the strong constraints are converted into the first objective function, the restrictions of the constraints on the optimization are reduced, so that the candidate allocation strategies can be optimized in a larger range, and the accuracy of determining the target allocation strategy is improved, so that in the distributed task scheduling scenario, the utilization rate of computing nodes is higher and the load is more balanced.

[0152] Furthermore, when determining the candidate allocation strategies for the next population in step S109, after determining the parent population of the current iteration, recombinant parent individuals are determined based on the candidate allocation strategies in the parent population. This allows the generation of some candidate allocation strategies for the next population based on the recombinant parent individuals.

[0153] Specifically, first, the server can use the formula Determine the recombinant parent individuals of this round of iteration. Where t represents the evolutionary generation (i.e., the number of iterations), μ represents the number of parent individuals, and i:μ represents the i-th individual in the parent population. represents the recombination weight of the i-th individual in the parent population in the t-th generation. The recombination weight of the i-th individual is expressed as follows:

[0154] Secondly, determine how to generate preliminary candidate allocation strategies for the next generation population based on the recombined parent individuals through mutation. The server can determine the preliminary candidate allocation strategies for the next generation population according to the following formula.

[0155] in, represents the mutation vector, σ [t] Represents the mutation intensity of the tth generation. M is the transposed matrix of the identity matrix I, represents a standard normal distribution vector.

[0156] Furthermore, when generating preliminary candidate allocation strategies for the next generation population, the generated preliminary candidate allocation strategies may differ significantly from those of the recombinant parent individuals. These significantly different candidate allocation strategies are considered abnormal and can result in excessively long "steps" in the optimization process, reducing efficiency. Therefore, for x values ​​outside the parameter range, the server can adjust and re-determine the candidate allocation strategies for the next generation population.

[0157] Specifically, the server may use the following formula to re-determine the candidate allocation strategy for the next generation population.

[0158] The KeepRange function is used to adjust the preliminary candidate allocation strategies that exceed the parameter range to candidate allocation strategies that do not exceed the parameter range. Since it is a relatively commonly used function, this specification will not further describe this function.

[0159] Furthermore, if If the mutation parameter is adjusted, it means that the mutation parameter may be too large, and the server can also adjust the mutation vector. The mutation vector is determined based on the mutation intensity of the previous generation population and the ratio of the candidate allocation strategy changes. The following formula is used to adjust the mutation vector:

[0160] in, Indicates the change of candidate allocation strategy, is the variation vector before adjustment. The normal distribution vector is adjusted using the following formula:

[0161] Among them, M -1 is the pseudo-inverse matrix of the transposed matrix. The process of determining the pseudo-inverse matrix can be expressed as follows: -1 =PseudoInverse(M [t] ).

[0162] In addition, in one or more embodiments of the present specification, in order to reduce the occurrence of falling into a local optimal solution and improve the optimization efficiency, the server can adjust the basic parameters for determining the mutation parameters according to a preset number of iteration rounds.

[0163] Specifically, first, the server may screen out a specified number of candidate allocation strategies from the candidate allocation strategies according to the fitness of each candidate allocation strategy as a parent population.

[0164] Next, the server determines the difference between the last iteration number for adjusting the mutation parameters and the current iteration number. Whether the mutation parameters need adjustment can be determined over multiple iterations. For example, the fitness of a candidate allocation strategy derived from a single round of iterative optimization cannot determine whether it has become "stuck." To implement periodic adjustment of the mutation parameters, the server can determine during each iteration whether the "period" for adjusting the mutation parameters has been reached. This requires calculating the difference in iteration numbers.

[0165] The server then determines whether the difference has reached a preset number of rounds. If so, the server determines that the adjustment period has arrived, determines the gradient of the first objective function and the gradient of the second objective function, and adjusts the mutation parameter based on the determined gradients. If not, the mutation parameter is not adjusted.

[0166] When adjusting the mutation parameters, the server may adopt different adjustment methods according to different situations.

[0167] Specifically, the adjustment methods that can be adopted in different situations can be referred to Figure 2. Figure 2 is a reference table for adjusting the fitness of the evolution strategy provided in this specification, in which the second objective function gradient (Δe [t] ) changes, the first objective function gradient (Δv [t] ) changes and the first objective function of the candidate allocation strategy in round t The optimal solution (or historical optimal solution) determines the mutation parameter transpose matrix (M[t]) and the mutation parameter mutation intensity (σ [t])How to adjust.

[0168] When the second objective function gradient is greater than the first preset value, it indicates that the search is moving faster towards the second objective function. If the first objective function gradient is (negative first preset value, first preset value), and the optimal solution of the first objective function of the candidate allocation strategy in round t is greater than the first preset value, it indicates that the search is also moving faster towards the first objective function. The server adjusts the mutation parameter according to the preset parameters. The second objective function gradient can be determined by the server according to the following formula:

[0169] The first objective function gradient of the server can be determined according to the following formula:

[0170] When the second objective function gradient takes the value of (minus the first preset value, the first preset value), it indicates that the search has stagnated in the first target direction. If the first objective function gradient takes the value of (minus the first preset value, the first preset value), and the optimal solution of the first objective function of the candidate allocation strategy in the tth round is greater than the first preset value, it indicates that the current population is not close to the feasible solution area, and the server adjusts the mutation parameter according to the preset parameters.

[0171] When the second objective function gradient takes the value of (minus the first preset value, the first preset value), it indicates that the search has stagnated in the first target direction. If the first objective function gradient takes the value of (minus the first preset value, the first preset value), and the optimal solution of the first objective function of the candidate allocation strategy in the tth round is equal to 0, it indicates that a feasible solution has been found, and the server adjusts the mutation parameters according to the preset parameters.

[0172] When the second objective function gradient takes the value of (negative first preset value, first preset value), it indicates that the search has stagnated in the first target direction. If the first objective function gradient is less than or equal to the negative first preset value, it indicates that the search is accelerating away from the feasible solution region. In this case, the server does not need to adjust the mutation parameter, or in other words, the server adjusts the mutation parameter, but the adjusted mutation parameter is consistent with the mutation parameter before adjustment.

[0173] When the gradient of the second objective function is less than or equal to the negative first preset value, it indicates that the search is slowing down in the direction of the second objective function. If the gradient of the first objective function is less than or equal to the negative first preset value or greater than the first preset value, it indicates that the search is accelerating away from the feasible solution region. In this case, the server does not need to adjust the mutation parameter. Alternatively, the server adjusts the mutation parameter, but the adjusted mutation parameter remains the same as the pre-adjusted mutation parameter.

[0174] In addition, in one or more embodiments of the present specification, since the importance of the first objective function and the second objective function may be different, when the server determines the fitness in step S109, it can also determine the fitness of each candidate allocation strategy based on the first objective function and the second objective function of each candidate allocation strategy, as well as the corresponding weights.

[0175] The fitness can be further determined accurately by adding weights. The fitness is determined using the following formula:

[0176] Among them, ω is the weight Where U(-θ,θ) means that if the weight is negative, it is negated and recalculated.

[0177] Furthermore, similar to the reason for adjusting the mutation parameters periodically as mentioned above, the server may also adjust the weights periodically.

[0178] Specifically, the adjustment methods that can be used in different situations can be referred to Figure 3. Figure 3 is a reference table for adjusting the weight of the evolution strategy provided in this specification, and Figure 3 is a reference table for adjusting the fitness of the evolution strategy provided in this specification, wherein the second objective function gradient (Δe [t] ) changes, the first objective function gradient (Δv [t] ) changes and the first objective function of the candidate allocation strategy in round t Optimal solution, determine the weight of the second objective function (b [t] )How to adjust.

[0179] First, the server determines the difference between the number of iterations of the last adjustment of the mutation parameter and the number of the current iteration. Then, the server determines whether the difference reaches a preset number of rounds difference. If so, the server adjusts the weight of the second objective function; otherwise, the server determines not to adjust the weight.

[0180] When the gradient of the second objective function is greater than the first preset value, it indicates that the search is moving faster in the direction of the second objective function. If the gradient of the first objective function is (negative of the first preset value, the first preset value), and the optimal solution of the first objective function of the candidate allocation strategy in the tth round is greater than the first preset value, it indicates that the search is also moving faster in the direction of the first objective function, then the server determines the weight of the second objective function to be any weight value.

[0181] When the gradient value of the second objective function is (negative first preset value, first preset value), it indicates that the search is stagnant in the first target direction. If at the same time the gradient value of the first objective function is (negative first preset value, first preset value), and the optimal solution of the first objective function of the candidate allocation strategy in the tth round is greater than the first preset value, it indicates that the current population is not close to the feasible solution area, then the server determines the weight of the second objective function to be any weight value.

[0182] When the gradient value of the second objective function is (negative first preset value, first preset value), it indicates that the search is stagnant in the first target direction. If the gradient value of the first objective function is (negative first preset value, first preset value), and the optimal solution of the first objective function of the candidate allocation strategy in the tth round is equal to 0, it indicates that a feasible solution has been found, then the server does not adjust the weight of the second objective function, or in other words, the server adjusts the weight of the second objective function, and the weight of the adjusted second objective function is consistent with the weight of the second objective function before adjustment.

[0183] When the gradient of the second objective function takes the value of (negative first preset value, first preset value), it indicates that the search has stagnated in the first target direction. If the gradient of the first objective function is less than or equal to the negative first preset value, it indicates that the search is accelerating away from the feasible solution area. The server determines the weight of the second objective function based on the maximum weight, the initial weight, the last adjusted second objective function, and the adaptive strength. This operation can help the population accelerate the search towards the feasible solution area.

[0184] When the gradient of the second objective function is less than or equal to the negative first preset value, the search is slowing down towards the second objective function. If the gradient of the first objective function is less than or equal to the negative first preset value or greater than the first preset value, the search is accelerating away from the feasible solution region. The server determines the weight of the second objective function based on the maximum weight, the initial weight, the last adjusted second objective function, and the adaptive strength. This operation can help the population accelerate the search towards the feasible solution region.

[0185] It should be noted that the server weight adjustment method in this manual is based on the number of rounds for the purpose of better target allocation strategy. If the adjustment is not based on the number of rounds, the server direct adjustment method is still feasible, but it will have an uncertain impact on the output target allocation strategy.

[0186] It should be noted that in this step, when the server selects a specified number of candidate allocation strategies from each candidate allocation strategy as the parent population, the formula uses a value of 1 / 3. However, this value is not necessarily fixed, but rather a preset value. When using the methods described in this specification, those skilled in the art can adjust this preset value based on specific circumstances to obtain a better target allocation strategy. This specification does not impose any specific restrictions on this preset value.

[0187] Based on the above task allocation process, this specification also provides a schematic diagram of task allocation in the graph computing scenario, as shown in Figure 4. The server of the graph computing platform can determine the graph computing task to be executed, and determine the tasks included in the graph computing task, namely, task A, task B, task C, and task D. Among them, there is a dependency relationship b between task A and task B, there is a dependency relationship a between task A and task C, there is a dependency c between task B and task C, and there is a dependency relationship d between task C and task D. It should be noted that this solution does not limit whether the above dependency relationships a to d are of the same type.

[0188] The graph computing platform generates candidate allocation strategies based on each received task. These candidate allocation strategies may include those that satisfy the allocation constraints or those that do not. Furthermore, the evolutionary algorithm corresponding to step S109 is used to iteratively determine several allocation strategies, from Allocation Strategies 1 to Allocation Strategies 9. Assuming Allocation Strategies 8 is optimal, it is selected as the target allocation strategy.

[0189] Finally, the tasks are assigned to computing nodes A, B, C, D, and E according to the target allocation strategy, wherein the computing nodes can communicate with each other.

[0190] The above is a distributed task scheduling method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding distributed task scheduling device, as shown in Figure 5. Figure 5 is a structural diagram of the distributed task scheduling device provided in this specification, including:

[0191] Parameter module 200, determining state parameters of multiple tasks to be scheduled and multiple computing nodes of the distributed computing platform;

[0192] A candidate allocation module 204 determines multiple candidate allocation strategies based on the tasks and the computing nodes, wherein each candidate allocation strategy includes an allocation relationship between each task and each computing node;

[0193] The first constraint module 206 determines a first objective function for each candidate allocation strategy according to a preset allocation constraint condition;

[0194] The second constraint module 208 determines a second objective function according to a preset optimization objective when the candidate allocation strategy satisfies the preset allocation constraint; and determines a second objective function according to a bias parameter and the preset optimization objective when the candidate allocation strategy does not satisfy the preset allocation constraint, wherein the bias parameter is used to increase the distance between the second objective function and the desired target;

[0195] The optimization module 209 performs the following iterative process according to the preset evolutionary algorithm until the evolutionary end condition is met: determining the fitness of each current candidate allocation strategy based on the first objective function and the second objective function of each current candidate allocation strategy, so as to adjust each current candidate allocation strategy to determine each next generation candidate allocation strategy;

[0196] Determining a target allocation strategy based on the fitness of each candidate allocation strategy adjusted when the evolution end condition is met;

[0197] The scheduling module 210 schedules each task to each computing node for computing processing according to the target allocation strategy.

[0198] Optionally, the parameter module 200 is used to receive a graph calculation request; determine a directed acyclic graph corresponding to the graph calculation to be executed based on the graph calculation request; determine the tasks to be scheduled and the dependencies between the tasks based on the directed acyclic graph.

[0199] Optionally, the candidate allocation module 204 is configured to generate several candidate allocation strategies for respectively allocating the tasks to different computing nodes according to the dependency relationships between the tasks.

[0200] Optionally, for each computing node, the status parameter includes at least one of: communication resources of the computing node, energy consumption of the computing node, storage resources of the computing node, and computing resources of the computing node. The communication resources include frequency, bandwidth, real-time status, and real-time hardware parameters; the energy consumption includes power and heat loss; the storage resources include memory resources and external storage resources; and the computing resources include CPU resources and network resources.

[0201] Optionally, the first constraint module 206 is used to determine the number of tasks assigned to each computing node in the candidate allocation strategy; determine the number of computing nodes whose assigned number of tasks is not greater than a preset first value as a first constraint value; and determine the first sub-item of the first objective function based on the first constraint value.

[0202] Optionally, the first constraint module 206 is used to predict, for each state parameter, the predicted state of each computing node when executing the task assigned by the candidate allocation strategy; determine the optimization objective function of the state parameter based on the predicted state and the corresponding state threshold; and determine the second sub-item of the first objective function based on the optimization objective function of each state parameter.

[0203] Optionally, the first constraint module 206 is configured to determine whether there is a candidate allocation strategy among the candidate allocation strategies that does not satisfy a preset allocation constraint condition; if so, determine a preset second value as a bias parameter of the second objective function.

[0204] Optionally, the second constraint module 208 is used to determine the second objective function as the total time consumed by each computing node when the candidate allocation strategy satisfies the preset allocation constraint conditions; when the candidate allocation strategy does not satisfy the preset allocation constraint conditions, to determine the second objective function as the total time consumed by each computing node when the bias parameter is adjusted to execute the task assigned by the candidate allocation strategy.

[0205] Optionally, an iterative process in the optimization module 209 includes determining and storing the fitness of each current candidate allocation strategy based on the first objective function and the second objective function of each current candidate allocation strategy; screening a specified number of candidate allocation strategies from each current candidate allocation strategy as the parent population based on the fitness of each current candidate allocation strategy; generating a recombination allocation strategy corresponding to each candidate allocation strategy in the parent population based on the parent population and the variation parameter; and determining each candidate allocation strategy of the next population based on each candidate allocation strategy in the parent population and each recombination allocation strategy.

[0206] Optionally, the optimization module 209 is used to determine a preset transposed matrix, an initialized normal distribution vector, and the mutation intensity of the next generation population to determine the mutation parameters; for each candidate allocation strategy in the parent population, the average allocation strategy of the parent population is mutated according to the mutation parameters to determine the mutation result; it is judged whether the mutation result exceeds the preset parameter range; if so, the mutation result is adjusted according to the parameter range so that the mutation result does not exceed the parameter range as a recombination allocation strategy; if not, the mutation result is used as the recombination allocation strategy; wherein the average allocation strategy is determined based on the candidate allocation strategies of the parent population.

[0207] Optionally, the optimization module 209 is used to determine the difference between the number of iterations for the last adjustment of the mutation parameter and the current number of iterations; determine whether the difference reaches a preset number of rounds; if so, determine the first objective function gradient and the second objective function gradient, and adjust the mutation parameter according to the determined gradient.

[0208] Optionally, the optimization module 209 is configured to adjust the variation parameter when the gradient of the second objective function is greater than a first preset value, the gradient of the first objective function is (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value; adjust the variation parameter when the gradient of the second objective function is (negative first preset value, first preset value], the gradient of the first objective function is (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value or equal to 0; adjust the variation parameter when the gradient of the second objective function is (negative first preset value, first preset value], and the gradient of the first objective function is less than or equal to the negative first preset value, so that the adjusted variation parameter is consistent with the variation parameter before adjustment; adjust the variation parameter when the gradient of the second objective function is less than or equal to the negative first preset value or greater than the first preset value, and the gradient of the first objective function is less than or equal to the negative first preset value or greater than the first preset value, so that the adjusted variation parameter is consistent with the variation parameter before adjustment.

[0209] Optionally, the mutation parameters include at least a normal distribution vector and a mutation intensity. Optimization module 209 is configured to determine a pseudo-inverse matrix based on the identity matrix; determine the mutation vector based on the ratio of the mutation intensity of the previous generation population to the change in candidate allocation strategy; determine a normal distribution vector based on the product of the pseudo-inverse matrix and the mutation vector; and adjust the mutation intensity based on a preset maximum mutation intensity.

[0210] Optionally, the optimization module 209 is used to determine the reorganization weights corresponding to each candidate allocation strategy in the parent population according to the position of each current allocation method in the population; use the initial weight as the weight of the second objective function, and determine the weight of the first objective function according to the weight of the second objective function, wherein the weight of the second objective function and the weight of the first objective function are normalized; determine the fitness of each current candidate allocation strategy according to the first objective function and the second objective function of each current candidate allocation strategy, and the corresponding weights.

[0211] Optionally, the optimization module 209 is also used to determine the difference between the number of iterations of the last adjustment of the mutation parameter and the current number of iterations; determine whether the difference reaches the preset number of rounds; if so, adjust the weight of the second objective function. Wherein, when the gradient of the second objective function is greater than the first preset value, the gradient of the first objective function takes the value of (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value, the weight of the second objective function is randomly determined within the preset weight value range; when the gradient of the second objective function takes the value of (negative first preset value, first preset value], the gradient of the first objective function takes the value of (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value, the weight of the second objective function is randomly determined within the preset weight value range; when the gradient of the second objective function takes the value of (negative first preset value, first preset value], the gradient of the first objective function takes the value of (negative first preset value, first preset value], and the first objective function When the optimal solution is equal to 0, adjust the weight of the second objective function, and the weight of the adjusted second objective function is consistent with the weight of the second objective function before adjustment; when the gradient value of the second objective function is (negative first preset value, first preset value], and the gradient of the first objective function is less than or equal to the negative first preset value, determine the weight of the second objective function according to the maximum weight, the initial weight, the last adjusted weight of the second objective function, and the adaptive strength; when the gradient of the second objective function is less than or equal to the negative first preset value, and the gradient of the first objective function is less than or equal to the negative first preset value or greater than the first preset value, determine the weight of the second objective function according to the maximum weight, the initial weight, the last adjusted weight of the second objective function, and the adaptive strength.

[0212] This specification also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the distributed task scheduling method provided in FIG. 1 above.

[0213] This specification also provides a schematic structural diagram of the electronic device shown in Figure 6. As shown in Figure 6, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the distributed task scheduling method described in Figure 1 above. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0214] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0215] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0216] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0217] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0218] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0219] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0220] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0221] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0222] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0223] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0224] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0225] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0226] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0227] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0228] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0229] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of this application.

Claims

1. A distributed task scheduling method, characterized in that: include: Determine the state parameters of multiple tasks to be scheduled and multiple computing nodes of the distributed computing platform; Determine multiple candidate allocation strategies according to the tasks and the computing nodes, wherein each candidate allocation strategy includes an allocation relationship between the tasks and the computing nodes; For each candidate allocation strategy, Determining a first objective function according to a preset allocation constraint; When the candidate allocation strategy satisfies the preset allocation constraint condition, determining a second objective function according to the preset optimization goal; When the candidate allocation strategy does not satisfy the preset allocation constraint, a second objective function is determined according to the bias parameter and the preset optimization target, wherein the bias parameter is used to increase the distance between the second objective function and the desired target; According to the preset evolutionary algorithm, the following iterative process is performed until the evolution end condition is met: according to the first objective function and the second objective function of each current candidate allocation strategy, the fitness of each current candidate allocation strategy is determined to adjust each current candidate allocation strategy to determine each next generation candidate allocation strategy; Determining a target allocation strategy according to the fitness of each candidate allocation strategy adjusted when the evolution end condition is met; According to the target allocation strategy, each task is dispatched to each computing node for computing processing.

2. The method according to claim 1, characterized in that Determining the multi-tasks to be scheduled specifically includes: Receive graph computation request; Determine, according to the graph computation request, a directed acyclic graph corresponding to the graph computation to be executed; According to the directed acyclic graph, the tasks to be scheduled and the dependencies between the tasks are determined.

3. The method according to claim 1, characterized in that For each computing node, the state parameter includes at least one of: communication resources of the computing node, energy consumption of the computing node, storage resources of the computing node, and computing resources of the computing node. Among them, the communication resources include frequency, bandwidth, real-time status and real-time hardware parameters; the energy consumption includes power and heat loss; the storage resources include memory resources and external storage resources; the computing resources include CPU resources and network resources.

4. The method according to claim 3, characterized in that Determining a first objective function according to the preset allocation constraint condition specifically includes: Determine the number of tasks to be assigned to each computing node in the candidate allocation strategy; Determine the number of computing nodes whose number of assigned tasks is not greater than a preset first value as a first constraint value; A first sub-item of a first objective function is determined according to the first constraint value.

5. The method according to claim 4, characterized in that Determining a first objective function according to the preset allocation constraint condition also includes: For each state parameter, predict the predicted state of each computing node when executing the task assigned by the candidate allocation strategy; Determining an optimization objective function of the state parameter according to the predicted state and the corresponding state threshold; According to the optimization objective function of each state parameter, a second sub-item of the first objective function is determined.

6. The method according to any one of claims 1 to 5, characterized in that: Before determining the second objective function, the method further includes: Determine whether there is a candidate allocation strategy that does not meet the preset allocation constraint condition among the candidate allocation strategies; If so, the preset second value is determined as the bias parameter of the second objective function.

7. The method according to claim 6, characterized in that Determining the second objective function includes: When the candidate allocation strategy satisfies the preset allocation constraint condition, the second objective function is determined as The total time consumed when executing the tasks assigned by the candidate allocation strategy; When the candidate allocation strategy does not satisfy the preset allocation constraint condition, the second objective function is determined to be the total time consumed when each computing node executes the task allocated by the candidate allocation strategy after the bias parameter is adjusted.

8. The method according to any one of claims 1 to 7, characterized in that: An iteration process specifically includes: Determine the fitness of each current candidate allocation strategy according to the first objective function and the second objective function of each current candidate allocation strategy, and store the fitness of each current candidate allocation strategy; According to the fitness of each current candidate allocation strategy, a specified number of candidate allocation strategies are selected from each current candidate allocation strategy as a parent population; Generating a recombination allocation strategy corresponding to each candidate allocation strategy in the parent population according to the parent population and the variation parameter; According to the candidate allocation strategies and the recombination allocation strategies in the parent population, the candidate allocation strategies of the next population are determined.

9. The method according to claim 8, characterized in that According to the parent population and the variation parameter, a recombination allocation strategy corresponding to each candidate allocation strategy in the parent population is generated, specifically comprising: Determine the preset transposed matrix, the initialized normal distribution vector and the mutation intensity of the next generation population, and determine the mutation parameters; For each candidate allocation strategy in the parent population, mutate the average allocation strategy of the parent population according to the mutation parameter to determine a mutation result; Determining whether the variation result exceeds a preset parameter range; If yes, adjusting the variation result according to the parameter range so that the variation result does not exceed the parameter range as a recombination allocation strategy; If not, the variation result is used as a recombination allocation strategy; The average allocation strategy is determined according to each candidate allocation strategy of the parent population.

10. The method according to claim 8 or 9, characterized in that Before generating the recombination allocation strategy corresponding to each candidate allocation strategy in the parent population according to the parent population and the variation parameter, the method further comprises: Determine the difference between the number of iterations for adjusting the mutation parameters last time and the number of iterations currently; Determine whether the difference reaches a preset round number difference; If so, determine the first objective function gradient and the second objective function gradient, and adjust the variation parameter according to the determined gradients.

11. The method according to claim 10, characterized in that Adjusting the mutation parameter according to the determined gradient specifically includes: When the second objective function gradient is greater than the first preset value, the first objective function gradient takes a value of (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value, adjusting the mutation parameter; When the second objective function gradient value is (negative first preset value, first preset value], the first objective function gradient value is (negative first preset value, first preset value], and the first objective function optimal solution is greater than the first preset value or equal to 0, adjust the mutation parameter; When the second objective function gradient takes a value of (negative first preset value, first preset value], and the first objective function gradient is less than or equal to the negative first preset value, adjusting the variation parameter, and the adjusted variation parameter is consistent with the variation parameter before the adjustment; When the gradient of the second objective function is less than or equal to the negative first preset value or greater than the first preset value, and the gradient of the first objective function is less than or equal to the negative first preset value or greater than the first preset value, the variation parameter is adjusted, and the adjusted variation parameter is consistent with the variation parameter before adjustment.

12. The method according to claim 10 or 11, characterized in that The variation parameters at least include: a normal distribution vector and a variation intensity; Adjusting the variation parameters specifically includes: Determine a pseudo-inverse matrix based on the identity matrix; Determining a mutation vector according to a ratio of the mutation intensity of the previous generation population to the change of the candidate allocation strategy; Determine a normal distribution vector according to the product of the pseudo inverse matrix and the variation vector; Adjust the mutation intensity according to the preset maximum mutation intensity.

13. The method according to any one of claims 8 to 12, characterized in that: Determining the fitness of each current candidate allocation strategy according to the first objective function and the second objective function of each current candidate allocation strategy specifically includes: Determine the recombination weights corresponding to the candidate allocation strategies in the parent population according to the positions of the current candidate allocation strategies in the population; Using the initial weight as the weight of the second objective function, and determining the weight of the first objective function according to the weight of the second objective function, wherein the weight of the second objective function and the weight of the first objective function are normalized; The fitness of each current candidate allocation strategy is determined according to the first objective function and the second objective function of each current candidate allocation strategy and the corresponding weights.

14. The method according to claim 13, characterized in that The method further comprises: Determine the difference between the number of iterations for adjusting the mutation parameters last time and the number of iterations currently; Determine whether the difference reaches a preset round number difference; If yes, adjusting the weight of the second objective function; Wherein, when the gradient of the second objective function is greater than the first preset value, the gradient value of the first objective function is (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value, the weight of the second objective function is randomly determined within the preset weight value range; When the second objective function gradient value is (negative first preset value, first preset value], the first objective function gradient value is (negative first preset value, first preset value], and the optimal solution of the first objective function is greater than the first preset value, the weight of the second objective function is randomly determined within the preset weight value range; When the second objective function gradient value is (negative first preset value, first preset value], the first objective function gradient value is (negative first preset value, first preset value], and the optimal solution of the first objective function is equal to 0, the weight of the second objective function is adjusted, and the weight of the second objective function after adjustment is consistent with the weight of the second objective function before adjustment; When the second objective function gradient takes a value of (negative first preset value, first preset value], and the first objective function gradient is less than or equal to the negative first preset value, determining the weight of the second objective function according to the maximum weight value, the initial weight, the weight of the second objective function adjusted last time, and the adaptive strength; When the gradient of the second objective function is less than or equal to the negative first preset value, and the gradient of the first objective function is less than or equal to the negative first preset value or greater than the first preset value, the weight of the second objective function is determined according to the maximum weight, the initial weight, the last adjusted weight of the second objective function, and the adaptive strength.

15. A distributed task scheduling device, characterized in that: include: A parameter module, which determines the state parameters of multiple tasks to be scheduled and multiple computing nodes of the distributed computing platform; A candidate allocation module, which determines multiple candidate allocation strategies according to the tasks and the computing nodes, wherein each candidate allocation strategy includes an allocation relationship between the tasks and the computing nodes; A first constraint module, for each candidate allocation strategy, determines a first objective function according to a preset allocation constraint condition; A second constraint module, when the candidate allocation strategy satisfies the preset allocation constraint condition, determines a second objective function according to a preset optimization goal; When the candidate allocation strategy does not satisfy the preset allocation constraint, a second objective function is determined according to the bias parameter and the preset optimization target, wherein the bias parameter is used to increase the distance between the second objective function and the desired target; The optimization module performs the following iterative process according to the preset evolutionary algorithm until the evolution end condition is met: determining the fitness of each current candidate allocation strategy according to the first objective function and the second objective function of each current candidate allocation strategy, so as to adjust each current candidate allocation strategy to determine each next generation candidate allocation strategy; Determining a target allocation strategy according to the fitness of each candidate allocation strategy adjusted when the evolution end condition is met; The scheduling module schedules each task to each computing node for computing processing according to the target allocation strategy.

16. A computer storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 14 is implemented.

Images