A scheduling method and system for improving execution efficiency based on time and task state

Abstract

The application relates to the technical field of task scheduling, and provides a scheduling method and system for improving execution efficiency based on time and task state, which comprises the following steps: abstracting a task scheduling scene into a directed acyclic graph carrying external attributes; constructing a reinforcement learning environment comprising a Q table, a state space and an action space; constructing a reward function, wherein the reward function comprises target state rewards, time penalties, parallel task rewards and error penalties; when a dependent set of actions has an intersection with an action set contained in a current action, reducing a current Q value; and when the Q table is iteratively trained to convergence, outputting a corresponding action selection strategy as a task scheduling scheme. The application is based on the antecedent state of time and a task, assisted by DAG and reinforcement learning, and optimized according to the characteristics of the environment, so that the efficiency of task scheduling is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of task scheduling, in particular to a scheduling method and system for improving execution efficiency based on time and task state. BACKGROUND

[0002] In current business scenarios and task scenarios, there are two aspects of execution prerequisites for tasks: one is that there is a dependency relationship between tasks in the system, and the other is that the tasks are affected by other pre-task outside the system. Each task has its own external pre-task, and different external pre-tasks have different execution times. The present application abstracts this execution time as an attribute, namely the latest readiness time. The subsequent task can be executed only after the pre-task is executed, and the latest readiness time of the pre-external task is the earliest execution time of the current task. Not only are there external pre-tasks in the environment, but there is also a certain dependency relationship between tasks in the system. Since the time dependency of the external system is a parameter not controlled by the system, and there is a slight difference in the execution time of the internal task, the task executor in the environment appears in the form of a cluster and can execute multiple tasks in parallel. In order to ensure that the tasks can be executed under the correct dependency relationship, the scheduling and dispatching of the tasks are often completed by manual calculation and manual timing, which increases the labor cost and reduces the work efficiency. SUMMARY

[0003] In order to help solve the above technical problems, the present application provides a scheduling method and system for improving execution efficiency based on time and task state.

[0004] In a first aspect, the present application provides a scheduling method for improving execution efficiency based on time and pre-task state, which adopts the following technical solution:

[0005] A scheduling method for improving execution efficiency based on time and pre-task state, comprising the following steps:

[0006] S1: abstracting a task scheduling scenario as a directed acyclic graph carrying external attributes;

[0007] S2: constructing a reinforcement learning environment including a Q table, a state space, and an action space, comprising:

[0008] State space construction: a binary array with a length of n is used to describe the execution state of n tasks, an executed task is 1, and an unexecuted task is 0, and the state value S t It can be expressed in the following way:

[0009] wherein array is a binary array, and i is an array index;

[0010] Action space construction: 0 to 2 n-1 integer and perform bit operations to generate subset combinations, forming a set of actions , A stores all action subsets, wherein , i is greater than or equal to 0 and less than or equal to 2 n -1;

[0011] S3: Construct a reward function, which includes a target state reward, a time penalty, a parallel task reward, and an error penalty;

[0012] S4: Perform Q table training based on the reward function, the Q table includes action values Q(s, a), when the dependent set of actions has an intersection with the action set contained in the current action, the current Q(s, a) value is reduced:

[0013] ,

[0014] Wherein, Dep(a) represents the dependent set of action a, A st represents the action set contained in the current action, as a decision parameter, is a positive integer, represents an empty set;

[0015] S5: When the Q table is iteratively trained to convergence, output the corresponding action selection strategy as a task scheduling scheme.

[0016] Preferably, S1 includes:

[0017] The nodes in the directed acyclic graph represent task units, and the directed edges represent the dependency constraints and execution order between tasks. The external attribute is the pre-external task attribute of the task, which includes the earliest execution time of the internal task.

[0018] Preferably, S2 includes:

[0019] Convert the state array to an integer through binary mapping:

[0020] .

[0021] Preferably, S3 includes:

[0022] Design a reward function:

[0023] ,

[0024] Where s is the current state, s' is the next state, is the target state reward, when is the state of all task completion ; represents the time penalty, is a negative coefficient, to distinguish between action errors and dependency errors, is a positive value less than 1; represents a parallel task reward, is a positive coefficient, error penalty : including dependency errors and action errors , t represents a discrete time step count variable, which is a positive integer.

[0025] In a second aspect, the application provides a scheduling system for improving execution efficiency based on time and previous task state, which adopts the following technical solution:

[0026] A scheduling system for improving execution efficiency based on time and previous task state, wherein the scheduling method for improving execution efficiency based on time and previous task state is adopted as described in any of the first aspect, comprising:

[0027] A task information processing module for executing the S1;

[0028] A reinforcement learning environment construction module for executing the S2;

[0029] A reward function construction module for executing the S3;

[0030] A Q table training module for executing the S4;

[0031] A policy generation module for executing the S5.

[0032] In summary, the application is based on time and the previous state of the task, assisted by DAG and reinforcement learning, and optimized according to the characteristics of the environment, thereby improving the efficiency of task scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a flowchart of a scheduling method for improving execution efficiency based on time and previous task state of the application;

[0034] Figure 2 is a first schematic diagram of a directed acyclic graph of the application;

[0035] Figure 3 is a second schematic diagram of a directed acyclic graph of the application;

[0036] Figure 4 is a first schematic diagram of a training process of the application;

[0037] Figure 5 is a second schematic diagram of a training process of the application;

[0038] Figure 6 is a third schematic diagram of a training process of the application;

[0039] Figure 7 Figure 1 is a first schematic diagram of training results of the present application;

[0040] Figure 8 Figure 2 is a second schematic diagram of training results of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described by examples in conjunction with the accompanying drawings in which the structure and principles of the present application are apparent to those skilled in the art. It should be understood that the specific embodiments described herein are merely illustrative of the present application and should not be considered limiting of the present application.

[0042] A scheduling method based on time and pre-task state to improve execution efficiency, comprising the following steps:

[0043] S1: Abstract the task scheduling scene as a directed acyclic graph carrying external attributes. S1 includes: the nodes in the directed acyclic graph represent task units, the directed edges represent the dependency constraints and execution order between tasks, and the external attributes are pre-external task attributes of the task, including the earliest execution time of the internal task.

[0044] S2: Construct a reinforcement learning environment including Q table, state space, action space and reward function, including:

[0045] State space construction: describe the execution state of n tasks with a binary array of length n, executed tasks are 1, unexecuted tasks are 0, and the state value S t is expressed as:

[0046] where array is a binary array and i is the array index.

[0047] Convert the state array to an integer by binary mapping:

[0048] .

[0049] Action space construction: the action space list is , A stores all action subsets, where i is greater than or equal to 0 and less than or equal to 2 n-1 .

[0050] S3: Construct a reward function, including target state reward, time penalty, parallel task reward and error penalty. S3 includes: design the reward function:

[0051] ,

[0052] where s is the current state, s' is the next state, is the target state reward, when for all tasks triggered at time representing a temporal penalty, is a negative coefficient, is a dependency error penalty, for distinguishing action errors and dependency errors, is a positive value less than 1; representing a parallel task reward, is a positive coefficient, error penalty : including dependency errors and action errors , t represents a discretized time step count variable, which is a positive integer, representing the number of discrete time units experienced from the initial state to the current state, for quantifying the timing nodes of the interaction between the agent and the environment in the task scheduling process under the reinforcement learning framework.

[0053] S4: When the dependency set of the action has an intersection with the action set contained in the current action, reduce the current Q value:

[0054] ,

[0055] where Dep(a) represents the dependency set of action a, A st represents the action set contained in the current action, as a decision parameter, is a positive integer, represents an empty set.

[0056] S5: When the Q table is iteratively trained to convergence, output the corresponding action selection strategy as a task scheduling scheme.

[0057] Specifically, there is a formal abstract mapping relationship between task scheduling and DAG. In a computing system, the execution process of a complex task can be modeled by a DAG, where nodes represent task units and directed edges represent dependency constraints and execution order between tasks. The topological structure of the DAG naturally avoids circular dependency problems, ensuring the logical feasibility of the task sequence.

[0058] As shown in FIG. 1, Figure 1 the present application abstracts the environment as a DAG graph with external structure, and the external structure is the pre-external task attribute of the task. Nodes 1, 2, 3, 4, 5, and 6 represent internal tasks of the system, and nodes a, b, c, d, e, and f represent external tasks of the system. The solid line represents the dependency relationship between internal tasks, and task 6 can be executed only after tasks 1 and 2 are executed. The dashed line represents the dependency relationship between internal tasks and external tasks, and the restriction condition of the external task on the internal task is represented by time t. Time t represents the latest output time of the external task, which is also the earliest execution time of the internal task. In the figure, t1 is the latest output time of external task a, which is also the earliest execution time of internal task 1.

[0059] The state space is the set of all possible states in the environment. In the system of this application, the state space is defined according to whether a task is executed within the system. Since there are multiple tasks in the system, an array is used to specifically describe the states of all tasks. If the number of tasks is n, then an array of length n is given. Each of them a i This indicates whether the i-th task has been executed; 1 indicates that it has been executed, and 0 indicates that it has not been executed.

[0060] In reinforcement learning frameworks, action space and state space are core mathematical objects that formally define the interaction between an agent and its environment. Their rigor directly affects the modeling accuracy of Markov decision processes. The state space contains all possible states and is defined as follows:

[0061] .

[0062] Based on the number of tasks n, the size of the state space is... .

[0063] To improve mathematical operability, reduce computational complexity, and enhance the enumerability of states, this application performs numerical transformation on the state space, establishing an array and a S... t The transformation relationship between them, S t This is used to represent the state at time t. Based on a binary conversion method, the state array is mapped to integers, completing the transformation from a two-dimensional space to a finite set. The formula is as follows:

[0064] .

[0065] `array` is a binary array representing state `x`, where `i` is the array index, ranging from 0 to `n-1`. This concisely expresses the process of converting the array into a state: multiplying each array element by its corresponding weight (a power of 2) and summing the results gives the final state value.

[0066] In order to clearly and directly display the current environment state during the calculation process, it is necessary not only to convert the array into integers, but also to perform the reverse conversion:

[0067] ,

[0068] Among them, S t This represents the current state value that needs to be reversed, array is the array of states to be transformed, and i represents the array index. Each element represents the state of the corresponding task, and the conversion from integers to binary arrays is achieved based on AND and left shift operations.

[0069] Action space defines the set of actions that an agent can perform in a given state, which directly affects the state immediate reward. In the current environment of the present application, there are multiple actuators that can work simultaneously, so multiple tasks can be performed synchronously without additional constraints. Therefore, when setting the action space, the present application considers the list of all possible subsets and organizes them into a result set containing multiple integer lists. All subsets are generated by enumerating integers from 0 to 2 n -1 and performing bit operations on each integer to select elements, and the action space list is as follows:

[0070] .

[0071] A stores all action subsets, where

[0072] ,

[0073] Here n is the number of elements in the original set, and it can be seen that 2 n is the total number of subsets. i is the index of the current subset, representing a specific subset, which helps the present application to traverse all subsets, i.e. from 0 to 2 n -1, each i can be represented as an n-bit binary number in binary, where each bit corresponds to an element in the original set. j is an index of an element in the original set, i.e. from 0 to n-1, which helps the present application to generate the i-th subset, check whether the j-th element of the i-th subset is 1, and if it is 1, add it to the current subset.

[0074] Given the current state s, the action a taken, the next state s' moved to, and the immediate reward r, the updating rule of the Q table can be expressed by the following mathematical expression:

[0075] ,

[0076] where is the Q value of taking action a in the current state s. is the learning rate, which controls the degree of fusion of new and old knowledge. It should be noted that in the present application, the learning rate only appears once, and after the first "+" from left to right in the above expression , the rest of the letters similar to those outside are actions a. r is the immediate reward, reflecting the immediate income after taking the current action. is the discount factor, which determines the current value of future rewards. is the maximum Q value that can be obtained by selecting the best action a' in the next state s'.

[0077] When the current system is executed, the execution time of the action is affected by many factors, such as the amount of incremental data on the day, the amount of inventory data in the current year, the current machine state, and the like, so the focus is not on the execution of the task, and the application focuses on the initiation of the task. The system itself aims to complete all task executions, and the target state is The application uses the parameter to represent the reward for walking into the target state, which is a larger positive value. However, using only the final goal of "all tasks initiated for execution" as a reward is not enough. As the tasks are executed step by step, small immediate rewards will be obtained.

[0078] Due to the complexity of the dependencies and execution conditions in the system, in addition to the reward mechanism, there is also a corresponding punishment mechanism. The system is a data processing transfer system with multiple upstream and downstream systems, and the output times of the upstream systems are also different. The upstream system provides the latest output time, so the external dependency is called time dependency. The system itself has internal dependencies and can only start execution after the dependent task is completed. At the same time, it is required that the tasks in the current execution period cannot be repeatedly executed. These situations are referred to as task dependencies. Due to the inability to execute time dependencies and task dependencies, a dependency error penalty is defined.

[0079] There is also a native error action that should be punished. As shown in Figure 3 , in this micro scenario in Figure 3 , there are two actions in the action space, red and gray. Among them, the red action can be successfully executed when it is performed, as there is no relationship between task 1 and task 2. Of course, whether it will ultimately be carried out also depends on factors such as the pre-time requirement. However, the two tasks in the gray action have a direct relationship, and this kind of relationship within the same action should be avoided through training. Therefore, for this situation, the penalty coefficient is set to a larger value. The application defines this situation as an action error penalty .

[0080] and are used to define the penalty situation, so they are negative numbers. Here the application defines an error penalty to adjust the weights of different penalty types. When an action is executed and the next state is generated, different values are generated, which are used to represent the various types of deductions involved in the external dependencies not executed, internal dependencies not executed, repeated execution, and the like, are assigned different values when they are generated to distinguish the punishment intensity.

[0081] At the same time, the present application hopes to complete all tasks in the shortest possible time, and the smaller the number of time steps used, the better, so the design parameters for the number of time steps , represent the time penalty, is negative. Since the task execution in the current environment has parallelism, the design parameter is the action length reward coefficient, which encourages the parallel tasks contained in the action to be as many as possible under the executable premise, which can save time, is a small positive value.

[0082] Combining the above various factors, the optimized reward function is as follows:

[0083] ,

[0084] where s is the current state, s' is the next state, is the subject performing the action, is used to distinguish between action error penalty and dependency error penalty, and is a positive value less than 1.

[0085] The optimized reward function proposed by the present application involves multiple parameters, which are often determined by experience at this stage. Wanting to determine the parameters through the program needs more profound theory and practice exploration, and may make the readability of the program worse when put into engineering, and the operation cost is higher. In order to improve the intuitiveness of the method, the present application explores another optimization process.

[0086] In the above method, the initial probability of each mapping of s and a in the Q table of the present application is equal. Now the present application considers reflecting the difference in the reward function in the Q table. Let Q(s, a) represent the Q value of taking action a in state s, A represent the action space, and S represent the state space,

[0087] Dep(a) represents the dependency set of action a, represents the action set contained in the current action. Before training, the known Q table content is preprocessed, if action a has dependencies, and these dependencies have intersection with the current selected action, that is,

[0088] ,

[0089] then update the current Q value, reduce the original Q value by , as a decision parameter, is a positive integer. Can be described as:

[0090] .

[0091] In actual operation, the optimization of the Q table can be combined with the optimization of the reward function. The optimization of the Q table can be more used for correction of obviously unreasonable actions, and in initialization, the possibility of task selection is reduced from the root. The optimization of the reward function can be used for more uncertain cases to play a guiding role in the process of searching for a strategy.

[0092] In summary, the method of the application provides a good way to solve the problem of task scheduling work with external dependencies and pre-time conditions. The application can train an efficient scheduling decision support execution according to known environmental conditions. The strategy of optimizing the reward mechanism is highly adaptive to the current system. The optimization of the Q table method is a high-efficiency way for beginners. At the same time, the application provides innovative inspiration, which helps the application to open up ideas in future work and solve more problems with the help of reinforcement learning methods. When the application provides a set of task information, the program can give a good execution strategy after multiple rounds of learning. Figures 4 to 6 , as shown in the environment, there are 5 nodes, each of which has the earliest execution time (external dependency) and the dependency relationship between the nodes (internal dependency), and 100 rounds of dependency training process. After training, the Q table result can be seen. In order to show the Q table more clearly, the data of the application is displayed in the table. Finally, according to the Q table, the test of the model is carried out, and the optimal execution strategy of task 2→task 1, 4→task 0, 2 is obtained.

[0093] For the optimization of the reward mechanism, the application designs a simple use case to verify the optimization with 5 tasks. The control group is based on the current environment, and the reward is based on the pursuit of the target state and the less time. The invalid action is handled by a unified penalty term. After 200 rounds of training, the application compares the round in which the user reaches the optimal solution. It can be found that the optimized method can get a good strategy in a relatively early time and remain stable.

[0094] As shown in Figure 7 , from the execution result, it can be seen that using the optimized reward function, the scheduling strategy can be obtained in a relatively low number of training rounds. The control group gets the optimal strategy in the 136th round, and the subsequent fluctuation appears. The optimized reward function gets the optimal strategy in the 74th round and the subsequent rounds are relatively stable.

[0095] The experiment in the foregoing is extended, and the optimized Q table is added. As shown in Figure 8 , from the execution result, it can be seen that using the preprocessed Q table, not only the number of unknown parameters in the method is effectively reduced, but also the optimization effect is obvious. The optimal strategy is obtained in a relatively early time and is relatively stable.

[0096] benchmark represents the performance of the base algorithm without any optimization during the reinforcement learning training process. The blue line in the figure shows its performance changes with the training episode, which is used for performance comparison with other optimization algorithms.

[0097] optReward shows the performance changes of the algorithm after the reward function is optimized. In reinforcement learning, "reward" represents the immediate feedback value obtained by the agent through actions, and the optimized reward function can more effectively guide the algorithm to learn an efficient strategy.

[0098] optQ reflects the optimization process of Q value in the Q learning algorithm. Q value represents the expected long-term return of taking a certain action in a specific state, and the optimized Q value can accelerate the convergence of the algorithm and improve the quality of the strategy.

[0099] The horizontal axis represents the training process of the reinforcement learning algorithm, and each "episode" represents a complete training cycle from the initial state to the terminal state. As the value of the horizontal axis increases, the performance evolution of the algorithm in continuous training is shown. The vertical axis represents time.

[0100] The application also proposes a scheduling system based on time and pre-task state to improve execution efficiency. The system uses the above scheduling method. The system includes:

[0101] a task information processing module for executing S1;

[0102] a reinforcement learning environment construction module for executing S2;

[0103] a reward function construction module for executing S3;

[0104] a Q table training module for executing S4;

[0105] a strategy generation module for executing S5.

Claims

1. A scheduling method for improving execution efficiency based on time and pre-task state, characterized in that, The method comprises the following steps: S1: abstracting a task scheduling scenario as a directed acyclic graph carrying external attributes; S1 comprises: The nodes in the directed acyclic graph represent task units, the directed edges represent the dependency constraints and execution order between tasks, and the external attributes are pre-external task attributes of the tasks, including the earliest execution time of the internal tasks; S2: constructing a reinforcement learning environment comprising a Q table, a state space and an action space, comprising: State space construction: The execution state of n tasks is described by a binary array of length n, with executed tasks as 1 and unexecuted tasks as 0, state value S t Is expressed by the following way: where array is a binary array, i is an array index; Action space construction: Enumerate 0 to 2ⁿ-1 integers and perform bitwise operations to generate subset combinations, forming the action set , A stores all action subsets, where , i is greater than or equal to 0 and less than or equal to 2 n -1; S3: constructing a reward function, the reward function comprising a target state reward, a time penalty, a parallel task reward and an error penalty; S3 comprises: Designing a reward function: ， where s is the current state, s' is the next state, is the reward for the goal state, when is the all-task completion state is triggered; represents the time penalty, is a negative coefficient, is the dependency error penalty, used to distinguish action errors and dependency errors, is a positive value less than 1; represents the parallel task reward, is a positive coefficient, the error penalty : including dependency errors and action errors , t represents a discrete time step count variable, which is a positive integer; S4: training the Q table based on the reward function, the Q table comprising action values Q(s, a), and when the dependency set of an action has an intersection with the action set contained in the current action, the current Q(s, a) value is reduced: ， wherein Dep(a) represents a dependency set of action a, A st represents a set of actions contained in the current action, is a positive integer as a decision parameter, represents an empty set; S5: when the Q table iterative training converges, outputting a corresponding action selection strategy as a task scheduling scheme.

2. The method of claim 1, wherein S2 Comprise: Converting the state array into an integer through binary mapping: 。 3. A scheduling system for improving execution efficiency based on time and pre-task state, characterized by, The scheduling method based on time and pre-task state promotion of execution efficiency according to claim 1 or 2, comprising: A task information processing module for performing S1; A reinforcement learning environment construction module for performing S2; A reward function construction module for performing S3; A Q table training module for performing S4; A strategy generation module for performing S5.

Images