Dynamic task scheduling method for heterogeneous multi-machine systems based on multi-agent reinforcement learning

Abstract

A dynamic task scheduling method for heterogeneous multi-machine systems based on multi-agent reinforcement learning includes the following steps: 1. Inputting task information and decomposing all tasks into multiple atomic tasks with different capability requirements that can be executed by a single robot; 2. Establishing an adhesion utility evaluation model and heterogeneous priorities to create a mathematical optimization model; 3. Establishing a local perception domain model and a Markov decision model based on the sequential decision-making process of scheduling decision points; 4. Building a heterogeneous multi-machine system scheduling environment and combining multiple advanced training strategies to learn a deep reinforcement learning solver for heterogeneous multi-machine task scheduling through the interaction between the environment and the agents; 5. Using the deep reinforcement learning solver, sequentially outputting actions for each scheduling decision point for the scheduling instance to obtain a dynamic scheduling solution. This method can handle the uncertainty in the task execution process by inserting dynamic scheduling decision points for dynamic events, thereby achieving fast and robust dynamic task scheduling.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent task scheduling, and in particular to a dynamic task scheduling method for a heterogeneous multi-machine system based on multi-agent reinforcement learning. Background Art

[0002] In recent years, with the rapid development of robotics, its application has become increasingly widespread across various sectors. In the industrial sector, collaborative robots have garnered widespread attention, with key enterprises actively promoting intelligent manufacturing models in an effort to further enhance the level of intelligence and production efficiency in the manufacturing industry. In the service sector, mobile robots have garnered significant attention from industry leaders due to the continued integration of mobile robotics with technologies such as the mobile internet and big data processing. Innovative applications are also emerging in the field of specialized robotics, such as post-disaster rescue and complex terrain exploration. However, as robotic application scenarios become increasingly complex and the variety and number of tasks required increase, the earlier single-robot approach is no longer sufficient. Multi-robot system solutions offer greater flexibility, scalability, and collaborative capabilities. In many cases, heterogeneous robot systems with diverse functions and roles offer advantages. However, the resulting challenge is how to better coordinate different robots to jointly complete more complex overall tasks while ensuring safety. Therefore, efficient and effective task scheduling for heterogeneous multi-robot systems is crucial.

[0003] The task scheduling problem for heterogeneous multi-robot systems can be reduced to a subproblem of MRTA (Multi-Robot Task Allocation) extended in the time dimension. This problem remains NP-hard, meaning that it becomes nearly impossible to find an exact solution within a reasonable timeframe as the problem scale increases. Common approaches to address this problem include heuristic algorithms, such as empirically designed heuristic rules that allocate tasks based on principles such as earliest completion time or shortest execution time. While these approaches are somewhat adaptable to problems of varying scales, their performance is limited, often resulting in significant idle time windows for robots or processing machines, leading to overall task inefficiency. Metaheuristic algorithms, such as genetic algorithms, particle swarm optimization algorithms, and ant colony algorithms, along with their variants, employ the fundamental principle of iteratively searching through an encoded space of candidate solutions using different paradigms to ultimately arrive at an optimal or suboptimal solution. While these approaches generally yield good solutions for small problem sizes, their computational time increases exponentially with the problem size, and they are not robust to the dynamic uncertainties inherent in real-world scenarios.

[0004] In most cases, the above studies oversimplify the modeling of actual scenarios, mostly based on the assumption of isomorphism for multi-machine systems, do not explore the complex constraints that actually exist between tasks in depth, and ignore the dynamics and uncertainties that are widely present in actual scenarios. In addition, it is difficult to achieve an ideal level of real-time performance. Therefore, a heterogeneous multi-robot task scheduling method is needed that can adapt to dynamic environments and produce a better solution within a reasonable time. Summary of the Invention

[0005] The present invention provides a dynamic task scheduling method for a heterogeneous multi-machine system based on multi-agent reinforcement learning to solve the technical problems mentioned in the background technology.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] The present invention provides a dynamic task scheduling method for a heterogeneous multi-machine system based on multi-agent reinforcement learning, which specifically includes the following steps:

[0008] S1. Input task information, decompose all tasks into multiple atomic tasks with different capability requirements that can be executed by a single robot according to task complexity and constraint relationships, and establish a constraint relationship matrix to describe the complex constraint relationships between tasks;

[0009] S2. Input robot information and establish an adhesion utility evaluation model and heterogeneous priority based on the differences in the three dimensions of robot capability, speed, and size, as well as the constraint relationship matrix. Furthermore, a mathematical optimization model is created with maximizing the adhesion utility as the objective function.

[0010] S3. Establish a local perception domain model based on the robot's characteristics, and establish a Markov decision model based on the sequential decision process of the scheduling decision points;

[0011] S4. Based on the above modeling and deep reinforcement learning paradigm, we build a heterogeneous multi-machine system scheduling environment. By combining multiple advanced training strategies, we learn a deep reinforcement learning solver for heterogeneous multi-machine task scheduling through the interaction between the environment and the agent.

[0012] S5. Through the deep reinforcement learning solver, considering dynamic task changes and robot failures, the actions of the scheduling instance are sequentially output at each scheduling decision point to obtain a dynamic scheduling solution.

[0013] Furthermore, the S1 specifically includes the following steps:

[0014] S11. Input task information, and classify and decompose the input task information according to task complexity and constraint relationships to obtain multiple atomic tasks;

[0015] S12. Establish a constraint relationship matrix between multiple atomic tasks, where the constraint relationship matrix includes matrix elements kj and matrix elements jk;

[0016] The values ​​of matrix elements kj and jk are both 2, indicating that there is a hard constraint between atomic tasks k and j: "atomic tasks k and j must be assigned to the same robot." A hard constraint is a constraint that must be satisfied.

[0017] If the values ​​of matrix element kj and matrix element jk are both -2, it means that there is a hard constraint between atomic task k and atomic task j that "atomic task k and atomic task j cannot be assigned to the same robot";

[0018] If the matrix element kj is 1 and the matrix element jk is -1, it means that atomic task k must be completed before atomic task j starts;

[0019] If the values ​​of matrix elements kj and jk are decimals between (0, 1) or (-1, 0), there is a soft constraint between atomic tasks k and j, which means that after atomic tasks k and j are assigned to the same robot, the utility will be increased or decreased by a degree of β. The soft constraint indicates that when it exists, it will have a positive or negative impact on the execution of a specific atomic task.

[0020] Furthermore, the implementation of S11 is as follows:

[0021] For composite tasks composed of multiple independent tasks, they are directly split into independent subtasks. For complex tasks with dependencies between subtasks, they are decomposed into subtasks with different constraints based on the dependencies. Regardless of whether the subtasks are independent or have constraints, they must ultimately be decomposed to a granularity that can be executed and completed by a single robot, called an atomic task. Through task classification and decomposition, multiple atomic tasks are obtained.

[0022] The basic information of each atomic task is represented by a five-tuple<pos,utility,exe_time,ability_need,state> , where pos represents the location of the atomic task, utility represents the direct utility of completing the atomic task, exe_time represents the execution time required for the robot to complete the atomic task, ability_need represents the ability requirement required to complete the atomic task, and state represents whether the atomic task has been assigned and to which robot it has been assigned.

[0023] Furthermore, the S2 specifically includes the following steps:

[0024] S21. Input robot attributes. The basic information of the robot in the heterogeneous system can be represented by a five-tuple.<cur_pos,left_time,speed,ability,size> , where cur_pos represents the current position of the robot, left_time represents the remaining running time of the robot, speed represents the speed of the robot, ability represents the ability level of the robot to complete the atomic task, and size represents the size of the robot;

[0025] S22. Based on the differences in the three dimensions of robot capability, speed, and size and the constraint relationship matrix, establish an adhesion utility evaluation model;

[0026] S23, based on the three heterogeneous attributes of capability, speed and size, and according to the current state of the robot, a dynamic heterogeneous priority list is constructed to obtain the heterogeneous priority;

[0027] S24. Create a mathematical optimization model with maximizing the adhesion utility as the objective function.

[0028] Furthermore, the adhesion utility evaluation model in S22 is specifically as follows:

[0029]

[0030] Where U ij represents the direct utility of robot i completing atomic task j, represents the adhesion utility obtained by robot i after completing all assigned atomic tasks; α ij represents the hard constraint satisfaction of atomic task j assigned to robot i; β ij represents the soft constraint satisfaction when assigning atomic task j to robot i; J i Represents the set of all atomic tasks assigned to robot i.

[0031] Furthermore, the mathematical optimization model in S24 is specifically as follows:

[0032] Objective function:

[0033]

[0034] Constraints:

[0035] (1) A single robot can only perform one atomic task at a time;

[0036]

[0037] in, is a binary variable, indicating Whether atomic task j is assigned to robot i at the moment, I represents the set of all robots; J represents the set of all atomic tasks; It means any;

[0038] (2) An atomic task can only be assigned to one robot at a time;

[0039]

[0040] Among them, x ij Refers to the allocation of atomic task j. If atomic task j is assigned to robot i, then x ij is equal to 1, otherwise 0;

[0041] (3) The time it takes for the robot to complete all assigned atomic tasks must be less than T ddl ;

[0042]

[0043] Among them, t ij represents the execution time required for robot i to perform atomic task j, represents the time required for the robot to travel to the location of atomic task j, represents the waiting time of the robot at atomic task j, T ddl Indicates the final time of atomic task scheduling;

[0044] (4) The robot's capabilities must match the capabilities required by the assigned atomic tasks;

[0045]

[0046] Among them, c ij is a binary variable representing whether the capability of robot i matches the requirements of atomic task j;

[0047] (5) If there are timing constraints between the assigned atomic tasks, the timing constraints must be met;

[0048]

[0049] Among them, δ jk is a binary variable representing the order constraint between atomic tasks j and k; x nk Indicates whether the atomic task k is assigned to robot n; s nk represents the time when robot n starts to perform atomic task k: x mj Indicates whether the atomic task j is assigned to robot m; f mj represents the time point when robot m completes atomic task j; R represents a set of real numbers.

[0050] Furthermore, the S3 specifically includes the following steps:

[0051] S31. Determine the appropriate size of the local perception field LPF through a comprehensive evaluation of the robot's speed, capabilities, and remaining resources. The radius of the local perception field LPF is R lpf The calculation formula is:

[0052] R lpf =ω r ×T lim ×speed

[0053] Among them, ω r is the adjustable coefficient, T lim is the maximum available time of the robot, and speed is the speed of the robot;

[0054] Then the radius R of the local receptive field LPF is lpf The utility of the visible atomic tasks is evaluated to screen out feasible atomic tasks, and three indicators of local state information in the state space of the Markov model are calculated. The three indicators are the attractiveness of the special atomic tasks to the robot in the local perception domain att_s, the attractiveness of the sequential atomic tasks to the robot in the local perception domain att_t, and the attractiveness of the closest atomic tasks to the robot in the local perception domain att_g;

[0055] At this point, the local perception domain model is obtained;

[0056] S32, and establishing a Markov decision model based on the sequential decision process of the scheduling decision point; the Markov decision model includes a state space, an action space, a reward function, and a state transition;

[0057] The state space mainly consists of three parts: the individual state information of the decision-making robot, local state information, and global state information. The individual state information of the decision-making robot comes from the robot's attributes; the local state information includes the atomic task information most relevant to the robot;

[0058] The action space includes two types of heuristic rules: priority rules and exploration rules. Both act on available atomic tasks within the local perception domain. Priority rules focus more on obtaining short-term utility. Exploration rules are composite actions that focus more on obtaining long-term potential utility and may not necessarily result in specific atomic tasks that match the rule. Priority rules refer to three priority actions: 1. Select the nearest idle task that meets the capacity constraint; 2. Select the nearest idle task with special capacity requirements; 3. Select the highest priority idle task that meets the timing constraint.

[0059] The structure of the reward function mainly consists of three parts: the basic reward when the action is effective, the additional encouragement when different actions are effective, and the penalty when different actions are ineffective;

[0060] State transition means that every time a time step reaches a scheduling decision point, a state transition will be triggered. According to the current heterogeneous priority list, the next robot will be selected as the decision-making robot. The next state is obtained based on the robot state and the specific state of the environment. This process continues until the remaining time of the robot is used up. At this time, the robot will exit the priority list and no longer participate in the sequential decision. When all robots have insufficient remaining running time, they enter the termination state.

[0061] Furthermore, the reward function in S32 is specifically as follows:

[0062]

[0063] Where r represents the reward function; u is the utility of the atomic task selected by the action, t used The total time used to complete the atomic task, w1, w2 and w3 respectively represent the Boolean values ​​of the three priority actions in S32, and w represents the Boolean value of whether the exploration action is effective.

[0064] Furthermore, the S4 specifically includes the following steps:

[0065] S41. Build a heterogeneous multi-machine system scheduling environment based on the adhesion utility evaluation model, mathematical optimization model, local perception domain model, Markov decision model, and deep reinforcement learning paradigm;

[0066] S42, introduce the behavior network and target network, where the behavior network is used to select actions, and the target network is used to evaluate the Q value of the action. The Q value is the state-action value, which represents the expected value of the long-term cumulative return of taking a specific action in a specific state, and the target value is is the target network's estimate of the Q value; the target value The calculation formula is as follows:

[0067]

[0068] Where γ is the discount coefficient, θ t is the parameter of the t-time walk behavior network, θ - is the parameter of the target network; Q(.) represents the behavior network or target network; at the target value In the formula, Q(.) in the brackets represents the behavior network, and Q(.) outside the brackets represents the target network; t represents the reward value of the environment feedback at the tth time step; s t+1 Indicates the state of the t+1th time step; a t+1represents the action at time step t+1;

[0069] The parameter relationship between the behavior network and the target network is agreed upon. The target network that calculates the target Q value only follows the behavior network for soft updates through the above parameter relationship. The specific parameter relationship is:

[0070] θ - =λ·θ t +(1-λ)·θ -

[0071] Among them, λ is a fixed value, and λ<<1;

[0072] S43. Utilize the Dueling DQN training strategy and, by changing the internal architecture of the behavior network and target network, decompose the target network's direct prediction of the Q value into the state value and the action advantage value relative to the state value to improve the accuracy of the state-action value assessment. Further train the behavior network and target network in a heterogeneous multi-machine system scheduling environment until the reward curve of the reward function converges, and obtain the trained behavior network and target network. Use the trained behavior network and target network as a deep reinforcement learning solver.

[0073] Furthermore, the S5 specifically includes the following steps:

[0074] S51. Input the state information of the scheduling instance at the initial moment into the deep reinforcement learning solver. The deep reinforcement learning solver outputs an action. The specific atomic task allocation is determined according to the heuristic rule corresponding to the action. The atomic task and robot attributes are changed accordingly, and the state is updated. The above process is repeated to obtain a static scheduling solution.

[0075] S52. Determine the information of different scheduling decision points based on the static scheduling plan and the execution status of the atomic tasks at the current moment. If a dynamic event occurs, insert a dynamic scheduling decision point. According to the relevant information of the dynamic scheduling decision point, input the corresponding moment status in order according to the heterogeneous priority, and then call the deep reinforcement learning solver to solve in real time to obtain several candidate solutions. Select the most suitable robot-atomic task allocation among the candidate solutions, update the scheduling plan, and obtain the dynamic scheduling plan.

[0076] Beneficial effects of the present invention:

[0077] The present invention provides a dynamic task scheduling method for heterogeneous multi-machine systems based on multi-agent reinforcement learning. By establishing an adhesion utility evaluation model for the complex constraint relationships between tasks and a Markov decision model based on local perception domain description, a heterogeneous multi-machine system scheduling environment is built. Then, based on the reinforcement learning training paradigm, a double adversarial Q network (i.e., a behavior network and a target network using an adversarial network architecture) is trained to obtain a deep reinforcement learning solver. The scheduling process based on the deep reinforcement learning solver can quickly obtain an efficient scheduling solution for the scheduling problem, and can handle the uncertainty in the task execution process by inserting dynamic scheduling decision points for dynamic events, thereby realizing fast and robust dynamic task scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of the present invention;

[0079] Figure 2 A schematic diagram of a problem instance generated based on atomic task information in an embodiment of the present invention;

[0080] Figure 3 A schematic diagram of a dynamic heterogeneous priority list generated in an embodiment of the present invention;

[0081] Figure 4 Schematic diagram of the coverage of different robot perception domains in an embodiment of the present invention;

[0082] Figure 5 This is a diagram of the overall training framework for multi-agent reinforcement learning that combines multiple training strategies in the present invention;

[0083] Figure 6 Schematic diagram of the execution status of each robot in an embodiment of the present invention;

[0084] Figure 7 This is a specific flow chart of S5 in the present invention. DETAILED DESCRIPTION

[0085] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present invention. However, the present invention may be implemented in many other forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present disclosure.

[0086] Reference Figure 1 The present invention provides a method for dynamic task scheduling in a heterogeneous multi-machine system based on multi-agent reinforcement learning, which specifically includes the following steps:

[0087] S1. Input task information, decompose all tasks into multiple atomic tasks with different capability requirements that can be executed by a single robot according to task complexity and constraint relationships, and establish a constraint relationship matrix to describe the complex constraint relationships between tasks;

[0088] S2. Input robot information and establish an adhesion utility evaluation model and heterogeneous priority based on the differences in the three dimensions of robot capability, speed, and size, as well as the constraint relationship matrix. Furthermore, a mathematical optimization model is created with maximizing the adhesion utility as the objective function.

[0089] S3. Establish a local perception domain model based on the robot's characteristics, and establish a Markov decision model based on the sequential decision process of the scheduling decision points;

[0090] S4. Based on the above modeling and deep reinforcement learning paradigm, we build a heterogeneous multi-machine system scheduling environment. By combining multiple advanced training strategies, we learn a deep reinforcement learning solver for heterogeneous multi-machine task scheduling through the interaction between the environment and the agent.

[0091] S5. Through the deep reinforcement learning solver, considering dynamic task changes and robot failures, the actions of the scheduling instance are sequentially output at each scheduling decision point to obtain a dynamic scheduling solution.

[0092] In some embodiments, the step S1 specifically includes the following steps:

[0093] S11. Input task information, and classify and decompose the input task information according to task complexity and constraint relationships to obtain multiple atomic tasks;

[0094] S12. Establish a constraint relationship matrix between multiple atomic tasks, where the constraint relationship matrix includes matrix elements kj and matrix elements jk;

[0095] The values ​​of matrix elements kj and jk are both 2, indicating that there is a hard constraint between atomic tasks k and j: "atomic tasks k and j must be assigned to the same robot." A hard constraint is a constraint that must be satisfied.

[0096] If the values ​​of matrix element kj and matrix element jk are both -2, it means that there is a hard constraint between atomic task k and atomic task j that "atomic task k and atomic task j cannot be assigned to the same robot";

[0097] If the matrix element kj is 1 and the matrix element jk is -1, it means that atomic task k must be completed before atomic task j starts;

[0098] If the values ​​of matrix elements kj and jk are decimals between (0, 1) or (-1, 0), there is a soft constraint between atomic tasks k and j, which means that after atomic tasks k and j are assigned to the same robot, the utility will be increased or decreased by a degree of β. The soft constraint indicates that when it exists, it will have a positive or negative impact on the execution of a specific atomic task.

[0099] C represents the constraint relationship matrix that describes the constraints between all atomic tasks. Based on the known atomic task information, problem instances can be generated such as Figure 2 In the figure, black nodes represent warehouses; black arrows represent timing constraints, pointing from predecessor atomic tasks to successor atomic tasks; digital nodes represent atomic tasks, their size represents the utility level, and the internal numbers represent the ability requirements to complete the atomic task.

[0100] In some embodiments, the implementation of S11 is as follows:

[0101] For composite tasks composed of multiple independent tasks, they are directly split into independent subtasks. For complex tasks with dependencies between subtasks, they are decomposed into subtasks with different constraints based on the dependencies. Regardless of whether the subtasks are independent or have constraints, they must ultimately be decomposed to a granularity that can be executed and completed by a single robot, called an atomic task. Through task classification and decomposition, multiple atomic tasks are obtained.

[0102] The basic information of each atomic task is represented by a five-tuple<pos,utility,exe_time,ability_need,state> , where pos represents the location of the atomic task, utility represents the direct utility of completing the atomic task, exe_time represents the execution time required for the robot to complete the atomic task, ability_need represents the ability requirement required to complete the atomic task, and state represents whether the atomic task has been assigned and to which robot it has been assigned.

[0103] In some embodiments, the step S2 specifically includes the following steps:

[0104] S21. Input robot attributes. The basic information of the robot in the heterogeneous system can be represented by a five-tuple.<cur_pos,left_time,speed,ability,size> , where cur_pos represents the current position of the robot, left_time represents the remaining running time of the robot, speed represents the speed of the robot, ability represents the ability level of the robot to complete the atomic task, and size represents the size of the robot;

[0105] S22. Due to the different allocation schemes, the atomic tasks assigned to each robot satisfy the constraints differently. The robot's completion of the atomic task contributes to the total utility of the system U IJ The contributions of U IJ ≠∑ i∈I ∑ j∈J U ij As shown in the formula, the total system utility is no longer simply equal to the sum of the direct utilities of all robots completing atomic tasks. Therefore, it is necessary to establish an adhesion utility model to reflect the impact of the complex constraints between atomic tasks on the utility obtained by robots completing atomic tasks when different atomic tasks are assigned. The establishment process is as follows:

[0106] Based on the differences in the three dimensions of robot capability, speed, and size and the constraint relationship matrix, an adhesion utility evaluation model is established;

[0107] S23, based on the three heterogeneous attributes of capability, speed and size, and according to the current state of the robot, a dynamic heterogeneous priority list is constructed to obtain the heterogeneous priority; Figure 3 ,This list provides the basic basis for the decision order of subsequent dynamic scheduling;

[0108] S24. Create a mathematical optimization model with maximizing the adhesion utility as the objective function.

[0109] In some embodiments, the adhesion utility evaluation model in S22 is specifically as follows:

[0110]

[0111] Where, α ij ∈{0,1},β ij ∈(-1,1), U ij represents the direct utility of robot i completing atomic task j, represents the adhesion utility obtained by robot i after completing all assigned atomic tasks. Its calculation result needs to consider the satisfaction of the completed atomic tasks with respect to both hard constraints and soft constraints; α ij represents the satisfaction of the hard constraints of the atomic task j assigned to robot i, and is 1 if and only if the hard constraints are satisfied; β ij Represents the soft constraint satisfaction when assigning atomic task j to robot i. Its value range is (-1, 1). It is 0 if and only if the atomic task is an atomic task with no dependencies. i Represents the set of all atomic tasks assigned to robot i

[0112] In some embodiments, the mathematical optimization model in S24 is specifically as follows:

[0113] Objective function:

[0114]

[0115] Constraints:

[0116] (1) A single robot can only perform one atomic task at a time;

[0117]

[0118] in, is a binary variable, indicating Whether atomic task j is assigned to robot i at the moment, I represents the set of all robots; J represents the set of all atomic tasks; It means any;

[0119] (2) An atomic task can only be assigned to one robot at a time;

[0120]

[0121] Among them, x ij Refers to the allocation of atomic task j. If atomic task j is assigned to robot i, then x ij is equal to 1, otherwise 0;

[0122] (3) The time it takes for the robot to complete all assigned atomic tasks must be less than T ddl ;

[0123]

[0124] Among them, t ij represents the execution time required for robot i to perform atomic task j, represents the time required for the robot to travel to the location of atomic task j, represents the waiting time of the robot at atomic task j, T ddl Indicates the final time of atomic task scheduling;

[0125] (4) The robot's capabilities must match the capabilities required by the assigned atomic tasks;

[0126]

[0127] Among them, c ij is a binary variable representing whether the capability of robot i matches the requirements of atomic task j;

[0128] (5) If there are timing constraints between the assigned atomic tasks, the timing constraints must be met;

[0129]

[0130] Among them, δ jk is a binary variable representing the order constraint between atomic tasks j and k; x nk Indicates whether the atomic task k is assigned to robot n; s nk represents the time when robot n starts to perform atomic task k: x mj Indicates whether the atomic task j is assigned to robot m; f mj represents the time point when robot m completes atomic task j; R represents a set of real numbers.

[0131] Furthermore, the present invention relies on the following premises and assumptions:

[0132] 1) All robots are initially in the charging dock area and fully charged;

[0133] 2) All tasks were not initially assigned to any robot;

[0134] 3) Once the robot starts executing a task, it cannot be interrupted unless a fault occurs.

[0135] In some embodiments, S3 specifically includes the following steps:

[0136] S31. Determine the appropriate size of the local perception field (LPF) by comprehensively evaluating the robot's speed, capabilities, and remaining resources. Evaluate the utility of different types of available atomic tasks within the local perception field (LPF) and calculate their attractiveness as part of the subsequent Markov decision model. The radius R of the local perception field (LPF) is lpf The calculation formula is:

[0137] R lpf =ω r ×T lim ×speed

[0138] Among them, ω r is the adjustable coefficient, T lim is the maximum available time of the robot, speed is the robot speed; Figure 4 ,The coverage of the perception domain of different robots is different;

[0139] Then the radius R of the local receptive field LPF is lpfThe utility of the visible atomic tasks is evaluated to screen out feasible atomic tasks, and three indicators of local state information in the state space of the Markov model are calculated. The three indicators are the attractiveness of the special atomic tasks to the robot in the local perception domain att_s, the attractiveness of the sequential atomic tasks to the robot in the local perception domain att_t, and the attractiveness of the closest atomic tasks to the robot in the local perception domain att_g;

[0140] At this point, the local perception domain model is obtained;

[0141] S32, and establishing a Markov decision model based on the sequential decision process of the scheduling decision point; the Markov decision model includes a state space, an action space, a reward function, and a state transition;

[0142] Only time points that meet the following conditions can be used as scheduling decision points: 1. The initial moment when atomic tasks begin to be assigned; 2. The moment when a robot is idle; 3. The moment when a dynamic event occurs. At each scheduling decision point, utility evaluation is performed in sequence according to the order of the current heterogeneous priority list, and atomic tasks are assigned based on the reinforcement learning solver.

[0143] The following octets are available in the state space:

[0144] <(left_time,speed,ability,size),(att_s,att_t,att_g),left_g> represents the robot, which mainly includes three parts: individual state information of the decision-making robot, local state information and global state information. The individual state information of the decision-making robot comes from the robot attributes; left_time is the remaining available time of the robot, speed is the speed of the robot, ability is the ability of the robot, and size is the size of the robot; local state information includes the atomic task information most relevant to the robot; att_s represents the attractiveness of the special atomic task in the local perception domain to the robot, att_t represents the attractiveness of the sequential atomic task in the local perception domain to the robot, and att_g represents the attractiveness of the closest atomic task in the local perception domain to the robot; global state information will affect the exploration-utilization balance of the robot's decision-making, and left_g is the proportion of unassigned atomic tasks to all atomic tasks;

[0145] The action space includes two types of heuristic rules, namely priority rules and exploration rules. Both act on the available atomic tasks in the local perception domain. Priority rules focus more on obtaining short-term utility. Priority rules refer to three priority actions, namely: 1. Select the nearest idle atomic task that meets the capability constraint; 2. Select the nearest idle atomic task with special capability requirements; 3. Select the idle atomic task with the highest priority that meets the timing constraint; the exploration rule is a compound action that focuses more on obtaining long-term potential utility, and may not necessarily obtain a specific atomic task that matches the rule; the steps are as follows: first, expand the radius of the local perception domain to the limit radius R_lim, which can be The limit LPF is obtained. The distance from the visible atomic task within the limit LPF to the robot's location is used as the dynamic radius R_dym. The target LPF is then obtained with the corresponding visible atomic task as the center and R_dtm as the radius. The common area obtained by the limit LPF and the target LPF can be used to obtain several candidate exploration areas. The number of idle and available atomic tasks in all candidate exploration areas (which can be called density) is then calculated. Finally, the closest available atomic task in the exploration area with the highest density is selected. The motivation for designing the action space is to achieve a balance between exploration and utilization for the agent, so as to better handle atomic task scheduling in dynamic situations, without falling into short-sighted greed or excessive exploration.

[0146] The structure of the reward function mainly consists of three parts: the basic reward when the action is effective, the additional encouragement when different actions are effective, and the penalty when different actions are ineffective;

[0147] State transition means that every time a time step reaches a scheduling decision point, a state transition will be triggered. According to the current heterogeneous priority list, the next robot will be selected as the decision-making robot. The next state is obtained based on the robot state and the specific state of the environment. This process continues until the remaining time of the robot is used up. At this time, the robot will exit the priority list and no longer participate in the sequential decision. When all robots have insufficient remaining running time, they enter the termination state.

[0148] In some embodiments, the reward function in S32 is specifically as follows:

[0149]

[0150] Where r represents the reward function; u is the utility of the atomic task selected by the action, t used The total time used to complete the atomic task, w represents the Boolean value of whether the exploration action is effective, and is currently only 1 when the exploration action is effective; w1, w2 and w3 represent the Boolean values ​​of the three priority actions in S32, and are 0 if and only if a priority rule action is selected but no atomic task allocation occurs.

[0151] In some embodiments, reference Figure 5 , the S4 specifically includes the following steps:

[0152] S41. Build a heterogeneous multi-machine system scheduling environment based on the adhesion utility evaluation model, mathematical optimization model, local perception domain model, Markov decision model, and deep reinforcement learning paradigm;

[0153] The machine that learns and executes decisions in a heterogeneous multi-machine system scheduling environment is called an agent, and all external objects that interact with the agent are called the environment. The specific interaction between the agent and the environment in a heterogeneous multi-machine system scheduling environment is as follows: At each time step t, the agent observes the environment state s t , and select action a based on strategy π t When the environment receives an action, a state transition will occur, and the quality of the selected action will be fed back to the agent as a reward r t , and transfer to the new state s t+1 Through the interaction between the agent and the environment, the trajectory from the initial state to the final state (s0, a0, r0, s1, a1, r1, ...) can be obtained;

[0154] S42, introduce the behavior network and target network. The behavior network and target network are Q networks with the same initial parameters but different update methods. The behavior network is used to select actions, and the target network is used to evaluate the Q value of actions. The Q value is the state-action value, which represents the expected value of the long-term cumulative reward of taking a specific action in a specific state, and the target value is the state-action value. is the target network's estimate of the Q value; the target value The calculation formula is as follows:

[0155]

[0156] Where γ is the discount coefficient, θ t is the parameter of the t-time walk behavior network, θ - is the parameter of the target network; Q(.) represents the behavior network or target network; at the target value In the formula, Q(.) in the brackets represents the behavior network, and Q(.) outside the brackets represents the target network; t represents the reward value of the environment feedback at the tth time step; s t+1 Indicates the state of the t+1th time step; a t+1 represents the action at time step t+1;

[0157] The parameter relationship between the behavior network and the target network is agreed upon. The target network that calculates the target Q value only follows the behavior network for soft updates through the above parameter relationship. The specific parameter relationship is:

[0158] θ - =λ·θt +(1-λ)·θ -

[0159] Among them, λ is a fixed value, and λ<<1;

[0160] S43. Utilize the Dueling DQN training strategy and, by changing the internal architecture of the behavior network and target network, decompose the target network's direct prediction of the Q value into the state value and the action advantage value relative to the state value to improve the accuracy of the state-action value assessment. Further train the behavior network and target network in a heterogeneous multi-machine system scheduling environment until the reward curve of the reward function converges, that is, the reward curve gradually becomes stable, and the parameter updates of the behavior network and target network almost stagnate at this stage. Obtain the trained behavior network and target network, and use the trained behavior network and target network as a deep reinforcement learning solver.

[0161] In some embodiments, the step S5 specifically includes the following steps:

[0162] S51. Input the state information of the scheduling instance at the initial moment into the deep reinforcement learning solver. The deep reinforcement learning solver outputs an action. The specific atomic task allocation is determined according to the heuristic rule corresponding to the action. The atomic task and robot attributes are changed accordingly, and the state is updated. The above process is repeated to obtain a static scheduling solution.

[0163] According to the static scheduling scheme, robots execute atomic tasks over time according to their respective atomic task assignments, e.g. Figure 6 As shown, the robot may be in one of the three states of moving, waiting, and executing at different time points. During the entire atomic task execution process, it will determine in real time whether there is a dynamic event;

[0164] Specifically, the S51 includes the following steps:

[0165] S511, first generate a scheduling case based on the atomic task and robot attributes;

[0166] S512. Update the heterogeneous priority according to the scheduling case; and confirm the decision robot based on the current heterogeneous priority order;

[0167] S513. Obtain a state vector based on the individual, local, and global states of the decision-making robot, call a deep reinforcement learning solver to solve the state vector, and then confirm the atomic task allocation based on the action output by the deep reinforcement learning solver;

[0168] S514. Update the heterogeneous multi-machine system scheduling environment related information based on the above information;

[0169] S515: Determine whether termination is required. If so, output and obtain a static scheduling solution; if not, proceed to S512.

[0170] S52. Determine the information of different scheduling decision points based on the static scheduling plan and the execution status of the atomic tasks at the current moment. If a dynamic event occurs, insert a dynamic scheduling decision point. According to the relevant information of the dynamic scheduling decision point, input the corresponding moment status in order according to the heterogeneous priority, and then call the deep reinforcement learning solver to solve in real time to obtain several candidate solutions. Select the most suitable robot-atomic task allocation among the candidate solutions, update the scheduling plan, and obtain the dynamic scheduling plan.

[0171] Specifically, the S52 includes the following steps:

[0172] S521. The robot executes the atomic task according to the static scheduling plan or the updated scheduling plan;

[0173] S522. During the execution of the atomic task, it is determined in real time whether there is a dynamic event. If so, the process proceeds to S523; if not, the process proceeds to S527.

[0174] S523: Insert a dynamic scheduling decision point and confirm the relevant information at that moment;

[0175] S524: Update the heterogeneous priority, confirm the candidate robots, and then obtain the corresponding state vector according to the candidate robots;

[0176] S525. Calling a deep reinforcement learning solver to solve different state vectors of multiple candidate robots to obtain a candidate action set;

[0177] S526, selecting the optimal allocation from the candidate action set;

[0178] S527, updating the scheduling plan;

[0179] S528. Determine whether the execution is completed. If yes, then end and output the dynamic scheduling plan. If not, enter S521.

[0180] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but they must be based on the fact that they can be implemented by a person of ordinary skill in the art. If the combination of technical solutions is mutually inconsistent or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

Claims

1. A dynamic task scheduling method for heterogeneous multi-machine systems based on multi-agent reinforcement learning, characterized in that: The specific steps include: S1. Input task information, decompose all tasks into multiple atomic tasks with different capability requirements that can be executed by a single robot based on task complexity and constraint relationships, and establish a constraint relationship matrix; S2. Input robot information and establish an adhesion utility evaluation model and heterogeneous priority based on the differences in the three dimensions of robot capability, speed, and size, as well as the constraint relationship matrix. Furthermore, a mathematical optimization model is created with maximizing the adhesion utility as the objective function. S3. Establish a local perception domain model based on the robot's characteristics, and establish a Markov decision model based on the sequential decision process of the scheduling decision points; S4. Based on the above modeling and deep reinforcement learning paradigm, we build a heterogeneous multi-machine system scheduling environment. By combining multiple advanced training strategies, we learn a deep reinforcement learning solver for heterogeneous multi-machine task scheduling through the interaction between the environment and the agent. S5. Using a deep reinforcement learning solver, taking into account dynamic task changes and robot failures, the actions are sequentially output at each scheduling decision point for the scheduling instance to obtain a dynamic scheduling solution. The adhesion utility evaluation model is as follows: Where U ij represents the direct utility of robot i completing atomic task j, represents the adhesion utility obtained by robot i after completing all assigned atomic tasks; α ij represents the hard constraint satisfaction of atomic task j assigned to robot i; β ij represents the soft constraint satisfaction when assigning atomic task j to robot i; J i Represents the set of all atomic tasks assigned to robot i.

2. The method for dynamic task scheduling in a heterogeneous multi-machine system according to claim 1, characterized in that: The S1 specifically includes the following steps: S11. Input task information, and classify and decompose the input task information according to task complexity and constraint relationships to obtain multiple atomic tasks; S12. Establish a constraint relationship matrix between multiple atomic tasks, where the constraint relationship matrix includes matrix elements kj and matrix elements jk; The values ​​of matrix elements kj and jk are both 2, indicating that there is a hard constraint between atomic tasks k and j: "atomic tasks k and j must be assigned to the same robot." A hard constraint is a constraint that must be satisfied. If the values ​​of matrix elements kj and jk are both -2, it means that there is a hard constraint between atomic tasks k and j that "atomic tasks k and j cannot be assigned to the same robot"; If the matrix element kj is 1 and the matrix element jk is -1, it means that atomic task k must be completed before atomic task j starts; If the values ​​of matrix elements kj and jk are decimals between (0, 1) or (-1, 0), there is a soft constraint between atomic task k and atomic task j, which means that after atomic task k and atomic task j are assigned to the same robot, the utility will be increased or decreased by a degree of β. The soft constraint means that when it exists, it will have a positive or negative impact on the execution of a specific atomic task.

3. The method for dynamic task scheduling in a heterogeneous multi-machine system according to claim 2, characterized in that: The implementation of S11 is as follows: For composite tasks composed of multiple independent tasks, they are directly split into independent subtasks. For complex tasks with dependencies between subtasks, they are decomposed into subtasks with different constraints based on the dependencies. Regardless of whether the subtasks are independent or have constraints, they are ultimately decomposed to a granularity that can be executed and completed by a single robot, called an atomic task. Through task classification and decomposition, multiple atomic tasks are obtained. The basic information of each atomic task is represented by a five-tuple<pos,utility,exe_time,ability_need,state> , where pos represents the location of the atomic task, utility represents the direct utility of completing the atomic task, exe_time represents the execution time required for the robot to complete the atomic task, ability_need represents the ability requirement required to complete the atomic task, and state represents whether the atomic task has been assigned and to which robot it has been assigned.

4. The method for dynamic task scheduling in a heterogeneous multi-machine system according to claim 1, wherein: The S2 specifically includes the following steps: S21. Input robot attributes. The basic information of the robot in the heterogeneous system can be represented by a five-tuple.<cur_pos,left_time,speed,ability,size> , where cur_pos represents the current position of the robot, left_time represents the remaining running time of the robot, speed represents the speed of the robot, ability represents the ability level of the robot to complete the atomic task, and size represents the size of the robot; S22. Based on the differences in the three dimensions of robot capability, speed, and size and the constraint relationship matrix, establish an adhesion utility evaluation model; S23, based on the three heterogeneous attributes of capability, speed and size, and according to the current state of the robot, a dynamic heterogeneous priority list is constructed to obtain the heterogeneous priority; S24. Create a mathematical optimization model with maximizing the adhesion utility as the objective function.

5. The method for dynamic task scheduling in a heterogeneous multi-machine system according to claim 4, characterized in that: The mathematical optimization model in S24 is specifically as follows: Objective function: Constraints: (1) A single robot can only perform one atomic task at a time; in, is a binary variable, indicating Whether atomic task j is assigned to robot i at the moment, I represents the set of all robots; J represents the set of all atomic tasks; It means any; (2) An atomic task can only be assigned to one robot at a time; Among them, x ij Refers to the allocation of atomic task j. If atomic task j is assigned to robot i, then x ij is equal to 1, otherwise 0; (3) The time it takes for the robot to complete all assigned atomic tasks must be less than T ddl ; Among them, t ij represents the execution time required for robot i to perform atomic task j, represents the time required for the robot to travel to the location of atomic task j, represents the waiting time of the robot at atomic task j, T ddl Indicates the final time of atomic task scheduling; (4) The robot's capabilities must match the capabilities required by the assigned atomic tasks; Among them, c ij is a binary variable representing whether the capability of robot i matches the requirements of atomic task j; (5) If there are timing constraints between the assigned atomic tasks, the timing constraints must be met; Among them, δ jk is a binary variable representing the order constraint between atomic tasks j and k; x nk Indicates whether the atomic task k is assigned to robot n; s nk represents the time when robot n starts to perform atomic task k: x mj Indicates whether the atomic task j is assigned to robot m; f mj represents the time point when robot m completes atomic task j; R represents a set of real numbers.

6. The method for dynamic task scheduling in a heterogeneous multi-machine system according to claim 1, wherein: The S3 specifically includes the following steps: S31. Determine the appropriate size of the local perception field LPF through a comprehensive evaluation of the robot's speed, capabilities, and remaining resources. The radius of the local perception field LPF is R lpf The calculation formula is: R lpf =ω r ×T lim ×speed Among them, ω r is the adjustable coefficient, T lim is the maximum available time of the robot, and speed is the speed of the robot; Then the radius R of the local receptive field LPF is lpf The utility of the visible atomic tasks is evaluated to screen out feasible atomic tasks, and three indicators of local state information in the state space of the Markov model are calculated. The three indicators are the attractiveness of the special atomic tasks to the robot in the local perception domain att_s, the attractiveness of the sequential atomic tasks to the robot in the local perception domain att_t, and the attractiveness of the closest atomic tasks to the robot in the local perception domain att_g; At this point, the local perception domain model is obtained; S32, and establishing a Markov decision model based on the sequential decision process of the scheduling decision point; the Markov decision model includes a state space, an action space, a reward function, and a state transition; The state space mainly consists of three parts: individual state information of the decision-making robot, local state information, and global state information. The individual state information of the decision-making robot comes from the robot attributes. The action space includes two types of heuristic rules: priority rules and exploration rules. Both of them act on the available atomic tasks in the local perception domain. The priority rule refers to three priority actions:

1. Select the nearest idle task that meets the capacity constraint; 2. Select the nearest idle task with special capacity requirements; 3. Select the highest priority idle task that meets the timing constraint. The structure of the reward function mainly consists of three parts: the basic reward when the action is effective, the additional encouragement when different actions are effective, and the penalty when different actions are ineffective; State transition means that every time a time step reaches a scheduling decision point, a state transition will be triggered. According to the current heterogeneous priority list, the next robot will be selected as the decision-making robot. The next state is obtained based on the robot state and the specific state of the environment. This process continues until the remaining time of the robot is used up. At this time, the robot will exit the priority list and no longer participate in the sequential decision. When all robots have insufficient remaining running time, they enter the termination state.

7. The method for dynamic task scheduling in a heterogeneous multi-machine system according to claim 6, characterized in that: The reward function in S32 is specifically as follows: Where r represents the reward function; u is the utility of the atomic task selected by the action, t used The total time used to complete the atomic task, w1, w2 and w3 respectively represent the Boolean values ​​of the three priority actions in S32; w represents the Boolean value of whether the exploration action is effective.

8. The method for dynamic task scheduling in a heterogeneous multi-machine system according to claim 1, wherein: The S4 specifically includes the following steps: S41. Build a heterogeneous multi-machine system scheduling environment based on the adhesion utility evaluation model, mathematical optimization model, local perception domain model, Markov decision model, and deep reinforcement learning paradigm; S42, introduce the behavior network and target network, where the behavior network is used to select actions, and the target network is used to evaluate the Q value of the action. The Q value is the state-action value, which represents the expected value of the long-term cumulative return of taking a specific action in a specific state, and the target value is is the target network's estimate of the Q value; the target value The calculation formula is as follows: Where γ is the discount coefficient, θ t is the parameter of the t-time walk behavior network, θ - is the parameter of the target network; Q(.) represents the behavior network or target network; at the target value In the formula, Q(.) in the brackets represents the behavior network, and Q(.) outside the brackets represents the target network; t represents the reward value of the environment feedback at the tth time step; s t+1 Indicates the state of the t+1th time step; a t+1 represents the action at time step t+c; The parameter relationship between the behavior network and the target network is agreed upon. The target network that calculates the target Q value only follows the behavior network for soft updates through the above parameter relationship. The specific parameter relationship is: i - =λ·θ t +(1-λ)·θ - Among them, λ is a fixed value, and λ<<1; S43. Utilize the Dueling DQN training strategy and, by changing the internal architecture of the behavior network and target network, decompose the target network's direct prediction of the Q value into the state value and the action advantage value relative to the state value to improve the accuracy of the state-action value assessment. Further train the behavior network and target network in a heterogeneous multi-machine system scheduling environment until the reward curve of the reward function converges, and obtain the trained behavior network and target network. Use the trained behavior network and target network as a deep reinforcement learning solver.

9. The method for dynamic task scheduling of a heterogeneous multi-machine system according to any one of claims 1 to 8, characterized in that: The S5 specifically includes the following steps: S51. Input the state information of the scheduling instance at the initial moment into the deep reinforcement learning solver. The deep reinforcement learning solver outputs an action. The specific atomic task allocation is determined according to the heuristic rule corresponding to the action. The atomic task and robot attributes are changed accordingly, and the state is updated. The above process is repeated to obtain a static scheduling solution. S52. Determine the information of different scheduling decision points based on the static scheduling plan and the execution status of the atomic tasks at the current moment. If a dynamic event occurs, insert a dynamic scheduling decision point. According to the relevant information of the dynamic scheduling decision point, input the corresponding moment status in order according to the heterogeneous priority, and then call the deep reinforcement learning solver to solve in real time to obtain several candidate solutions. Select the most suitable robot-atomic task allocation among the candidate solutions, update the scheduling plan, and obtain the dynamic scheduling plan.

Images