Multi-robot dynamic task arrangement method and device, terminal equipment and storage medium

Abstract

The invention discloses a multi-robot dynamic task arrangement method and device, terminal equipment and a storage medium, and the method comprises the steps: obtaining a to-be-distributed task, and distributing the to-be-distributed task to a plurality of robots according to the types of the robots; task parameters of all the robots are obtained; for each robot, creating a Q table of the current robot according to the task parameters of the current robot; the Q table of the current robot is updated according to the Q tables of the other robots except the current robot, and the updated Q table of the current robot is obtained; determining updated task parameters of the robots according to the updated Q tables of the robots; and inputting the updated task parameters of each robot into a task allocation model, so that the task allocation model outputs a task allocation strategy of each robot. According to the invention, the task allocation efficiency of multiple robots can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot task allocation, and in particular to a multi-robot dynamic task scheduling method, device, terminal equipment and storage medium. Background Art

[0002] Multi-robot task allocation is a core problem in robotic control. It focuses on how to optimally schedule multiple robots while satisfying complex constraints to maximize system performance and efficiency. These robots may have different functions and performance metrics, and may perform different types of tasks. Multi-robot task allocation is an NP-hard problem. When the environment is complex, it is generally impossible to achieve optimal task allocation within a limited timeframe.

[0003] Existing multi-robot task allocation algorithms, such as multi-agent reinforcement learning algorithms, have been widely used in the field of multi-robot task allocation and have achieved great results. However, multi-agent reinforcement learning algorithms mainly solve the multi-robot task allocation problem by inputting all task parameters of all robots into the task allocation model at one time to make task allocation decisions. This method requires establishing Q-tables and state spaces for all robots, but ignores the correlation between the same assigned tasks in the Q-tables between robots when establishing the Q-tables. As a result, all Q-tables and established state spaces need to be processed one by one when processing the task allocation model, resulting in low task allocation efficiency. Summary of the Invention

[0004] Embodiments of the present invention provide a multi-robot dynamic task scheduling method, apparatus, terminal device, and storage medium, which can improve the efficiency of multi-robot task allocation.

[0005] An embodiment of the present invention provides a multi-robot dynamic task scheduling method, comprising:

[0006] Obtain tasks to be assigned and assign them to several robots according to robot types;

[0007] Obtaining task parameters for each robot; wherein the task parameters include: task sequence, current position information, target position information required to move to perform each task in the task sequence, and task status of each task in the task sequence;

[0008] For each robot, create a Q table for the current robot based on the task parameters of the current robot; wherein the Q table includes the reward values of the current robot performing different assigned tasks under the current task parameters;

[0009] The Q table of the current robot is updated based on the Q tables of the other robots except the current robot to obtain an updated Q table of the current robot. When updating the Q table of the current robot based on the Q tables of the other robots except the current robot, if the same assigned task is in the Q tables of multiple robots at the same time, the maximum reward value in the Q tables of all robots under the same assigned task is used to update the reward value under the corresponding assigned task in the Q table of the current robot;

[0010] Determine the updated task parameters of each robot based on the updated Q table of each robot;

[0011] The updated task parameters of each robot are input into the task allocation model so that the task allocation model outputs the task allocation strategy of each robot.

[0012] Furthermore, the construction of the task allocation model includes:

[0013] Obtain historical updated task parameters and historical task allocation strategies for several robots;

[0014] Constructing a state space of multiple robots based on each historical updated task parameter, constructing an action space of multiple robots based on each historical task allocation strategy, and constructing a reward function based on the state space and action space;

[0015] An initial task allocation model is constructed, and the initial task allocation model is iteratively trained using the state space and action space until the loss function of the initial task allocation model is minimized, thereby generating the task allocation model; wherein the loss function is the inverse of the reward function.

[0016] Furthermore, before inputting the updated task parameters of each robot into the task allocation model, the method further includes:

[0017] Determine the number of pending decision-making task allocation strategies for each robot based on the updated task parameters of each robot;

[0018] The total number of the robots' pending task allocation strategies is obtained according to the number of pending task allocation strategies of each robot;

[0019] If the total number of pending task allocation strategies is greater than the total number of robots, the worst strategy elimination operation is repeated until the total number of pending task allocation strategies is equal to the total number of robots;

[0020] The updated task parameters of each robot corresponding to the total number of task allocation strategies to be decided is equal to the total number of robots are used as the updated task parameters of each robot input into the task allocation model;

[0021] The worst strategy elimination operation includes:

[0022] For each robot, calculate the benefits of the current robot's strategy for allocating tasks to be decided;

[0023] The strategy for allocating tasks to be decided that has the lowest benefit among all robots is selected as the strategy;

[0024] When the selected strategy is not the only pending task allocation strategy for the corresponding robot, the selected strategy is eliminated, and the total number of pending task allocation strategies after elimination is calculated;

[0025] When the selected strategy is the only pending task allocation strategy for the corresponding robot, the pending task allocation strategies with the lowest benefits other than the selected strategy and not the only pending task allocation strategy for the corresponding robot are eliminated, and the total number of pending task allocation strategies after elimination is calculated.

[0026] Furthermore, after the task allocation model outputs the task allocation strategy for each robot, the following steps are also included:

[0027] Control each robot to perform tasks according to the task allocation strategy, and update the task parameters of each robot after the task is completed.

[0028] Based on the above method embodiment, the present invention provides a corresponding device embodiment;

[0029] An embodiment of the present invention provides a multi-robot dynamic task scheduling device, comprising: a task pre-assignment module, a task parameter acquisition module, a task parameter update module and a task assignment module;

[0030] The task pre-assignment module is used to obtain tasks to be assigned and assign the tasks to a number of robots according to robot types;

[0031] The task parameter acquisition module is used to obtain the task parameters of each robot; wherein the task parameters include: task sequence, current position information, target position information required to move to perform each task in the task sequence, and task status of each task in the task sequence;

[0032] The task parameter updating module is configured to create a Q-table for each robot based on the task parameters of the current robot; wherein the Q-table includes reward values for the current robot when performing different assigned tasks under the current task parameters; update the Q-table of the current robot based on the Q-tables of the robots other than the current robot to obtain an updated Q-table for the current robot; wherein, when updating the Q-table of the current robot based on the Q-tables of the robots other than the current robot, if the same assigned task is simultaneously in the Q-tables of several robots, update the reward value for the corresponding assigned task in the Q-table of the current robot with the maximum value of the reward values in the Q-tables of all robots under the same assigned task; and determine the updated task parameters of each robot based on the updated Q-tables of each robot;

[0033] The task allocation module is used to input the updated task parameters of each robot into the task allocation model so that the task allocation model outputs the task allocation strategy of each robot. Furthermore, it also includes: a task allocation model construction module;

[0034] The task allocation model construction module is used to obtain historical updated task parameters and historical task allocation strategies of several robots;

[0035] Constructing a state space of multiple robots based on each historical updated task parameter, constructing an action space of multiple robots based on each historical task allocation strategy, and constructing a reward function based on the state space and action space;

[0036] An initial task allocation model is constructed, and the initial task allocation model is iteratively trained using the state space and action space until the loss function of the initial task allocation model is minimized, thereby generating the task allocation model; wherein the loss function is the inverse of the reward function.

[0037] Furthermore, it also includes: a strategy elimination module;

[0038] Before inputting the updated task parameters of each robot into the task allocation model, it also includes:

[0039] Determine the number of pending decision-making task allocation strategies for each robot based on the updated task parameters of each robot;

[0040] The total number of the robots' pending task allocation strategies is obtained according to the number of pending task allocation strategies of each robot;

[0041] If the total number of pending task allocation strategies is greater than the total number of robots, the worst strategy elimination operation is repeated until the total number of pending task allocation strategies is equal to the total number of robots;

[0042] The updated task parameters of each robot corresponding to the total number of task allocation strategies to be decided is equal to the total number of robots are used as the updated task parameters of each robot input into the task allocation model;

[0043] The worst strategy elimination operation includes:

[0044] For each robot, calculate the benefits of the current robot's strategy for allocating tasks to be decided;

[0045] The strategy for allocating tasks to be decided that has the lowest benefit among all robots is selected as the strategy;

[0046] When the selected strategy is not the only pending task allocation strategy for the corresponding robot, the selected strategy is eliminated, and the total number of pending task allocation strategies after elimination is calculated;

[0047] When the selected strategy is the only pending task allocation strategy for the corresponding robot, the pending task allocation strategies with the lowest benefits other than the selected strategy and not the only pending task allocation strategy for the corresponding robot are eliminated, and the total number of pending task allocation strategies after elimination is calculated.

[0048] Furthermore, it also includes: a control and update module;

[0049] The control and updating module is used to control each robot to perform a task according to the task allocation strategy, and to update the task parameters of each robot after the task is completed.

[0050] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for dynamic task scheduling of multiple robots described in the above-mentioned embodiment of the invention is implemented.

[0051] Another embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the multi-robot dynamic task scheduling method described in the above-mentioned embodiment of the invention.

[0052] The following beneficial effects are achieved by implementing the present invention:

[0053] The present invention provides a multi-robot dynamic task scheduling method, apparatus, terminal device and storage medium. The task scheduling method obtains tasks to be assigned and assigns the tasks to be assigned to several robots according to robot type, and then further obtains task parameters of each robot. For each robot, a Q table of the current robot is created according to the task parameters of the current robot, and then the Q table of the current robot is updated according to the Q tables of the remaining robots except the current robot to obtain an updated Q table of the current robot, and the updated task parameters of each robot are determined according to the updated Q table of each robot; by constructing the Q table of each robot based on the task parameters of each robot, and then updating the Q table of each robot through the reward value corresponding to each assigned task in the Q table of each robot, and then determining the updated task parameters of each robot based on the updated Q table, the problem of overlapping task parameters between different robots is avoided, and the efficiency of generating task allocation strategies is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The figure is a flow chart of a multi-robot dynamic task scheduling method provided by one embodiment of the present invention.

[0055] Figure 2 This is a flowchart of a Nash equilibrium calculation provided by an embodiment of the present invention.

[0056] Figure 3 The figure is a schematic structural diagram of a multi-robot dynamic task scheduling device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0060] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0061] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0063] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0064] In the description of the embodiments of the present application, unless otherwise expressly specified or limited, technical terms such as "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components or interactions between two components. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0065] like Figure 1 FIG. 1 is a multi-robot dynamic task scheduling method provided by an embodiment of the present invention, comprising:

[0066] Step S1: Obtain tasks to be assigned, and assign the tasks to several robots according to robot types;

[0067] Step S2: Obtaining task parameters of each robot; wherein the task parameters include: task sequence, current position information, target position information required to move to perform each task in the task sequence, and task status of each task in the task sequence;

[0068] Step S3: For each robot, create a Q-table for the current robot based on the task parameters of the current robot; wherein the Q-table includes the reward values of the current robot for performing different assigned tasks under the current task parameters; update the Q-table of the current robot based on the Q-tables of the robots other than the current robot to obtain an updated Q-table for the current robot; wherein, when updating the Q-table of the current robot based on the Q-tables of the robots other than the current robot, if the same assigned task is in the Q-tables of several robots at the same time, update the reward value for the corresponding assigned task in the Q-table of the current robot based on the maximum value of the reward values in the Q-tables of all robots under the same assigned task; determine the updated task parameters of each robot based on the updated Q-tables of each robot;

[0069] Step S4: inputting the updated task parameters of each robot into the task allocation model, so that the task allocation model outputs the task allocation strategy of each robot.

[0070] Regarding step S1, obtaining the tasks to be assigned, the tasks to be assigned can be tasks automatically generated in the platform associated with each robot or tasks manually created by the relevant operator. The obtained tasks to be assigned are assigned to m robots according to robot type. Robot types can be categorized as drones, robot dogs, and humanoid robots, among others. Preferably, the process of obtaining the tasks to be assigned is performed dynamically in real time. This can be done by obtaining the tasks to be assigned in real time through motion capture, or by obtaining the tasks to be assigned in real time through a timed acquisition method.

[0071] For step S2, after the tasks to be assigned are assigned to each robot, the task sequence of the robot to which the tasks are assigned is updated, that is, the task parameters of each robot will be updated. At this time, it is necessary to further obtain the task parameters of each robot. The task parameters include: task sequence, current position information, target position information required to move to execute each task in the task sequence, and task status of each task in the task sequence.

[0072] For step S3, for each robot, observe the task parameters of other robots, determine whether the executable action of the current robot is shared by other robots based on the task parameters of other robots, update the task parameters of the current robot based on the task parameters of the remaining robots, and obtain the updated task parameters of the current robot. Specifically, after obtaining the task parameters of the robot, a state space is established based on the task parameters of each robot. The first column of the matrix corresponding to the state space S is the state of m robots, and columns 2 to n+1 are the states of n tasks to be assigned. According to the corresponding state in the state space, that is, the task parameters corresponding to each robot, its own Q table is created. The table includes i rows and j columns, where the value of the i-th row and j-th column is Q(s i,1 ,a i,j ), the Q value is initialized to the reward value R(s i,1 ,a i,j ), where s i,1 Represents the state of the i-th robot, including the current position information of the i-th robot, the task sequence and the task status of each task in the task sequence, a i,j Indicates that the i-th robot performs the action corresponding to the j-th task. It should be noted that the Q table created by each robot only stores its own state and action, and does not store the state and action of other robots. The Q tables of the robots other than the current robot are compared with the Q table of the current robot to determine whether the same executable action is shared by other robots. If the action corresponding to the j-th task performed by the i-th robot is shared by other robots, the Q value of the action is updated to the maximum reward value of all robots sharing the action. For example, robot s 1,1 ,s 2,1 ,s 3,1 All three robots can perform action j of task j k , the Q values of the three robots are and The maximum value in With this Update the value of and The updated Q tables of the three robots are obtained by using the values of . Then, the updated task parameters of each robot are determined based on the updated Q tables of the robots.

[0073] In step S4, the updated task parameters of each robot are input into the task allocation model, so that the task allocation model outputs the task allocation strategy of each robot.

[0074] The construction of the task allocation model includes:

[0075] Obtain historical updated task parameters and historical task allocation strategies for several robots;

[0076] Constructing a state space of multiple robots based on each historical updated task parameter, constructing an action space of multiple robots based on each historical task allocation strategy, and constructing a reward function based on the state space and action space;

[0077] An initial task allocation model is constructed, and the initial task allocation model is iteratively trained using the state space and action space until the loss function of the initial task allocation model is minimized, thereby generating the task allocation model; wherein the loss function is the inverse of the reward function.

[0078] Specifically, a corresponding state space is constructed according to the updated task parameters of each robot, and the corresponding state space is input into the task allocation model. The task allocation model outputs the action space corresponding to the updated task parameters of each robot, and the task state of each task corresponding to the updated task parameters of each robot is determined according to the parameters in the action space; wherein, the task state of each task corresponding to the updated task parameters of each robot is the task allocation strategy of each robot mentioned above.

[0079] Before training and generating the task allocation model, the network framework of the present invention needs to be constructed.

[0080] First, we need to define the state space. Assuming there are m robots of different types and n tasks to be assigned, the state space S can be represented by a matrix of size m*(n+1). The first column of the matrix corresponding to the state space S is the state of the m robots, and columns 2 to n+1 are the states of the n tasks to be assigned. It can be expressed as:

[0081]

[0082] In the present invention, task allocation is mainly mobile task allocation. Taking mobile task allocation as an example, the first column in the state space S is m robot states, and the robot state includes the robot's current position information, task sequence and the task state of each task in the task sequence; the values of columns 2 to n+1 in the state space S are the mobile task states corresponding to each robot, and the mobile task state includes the target position information required to move to execute each task in the task sequence and the task state of each task in the task sequence. The task state of each task in the task sequence refers to whether the task is executable.

[0083] Define the action space. The action space A is the set of actions performed by each robot in a specific state s, that is, the actions performed by each robot after being assigned to a task in a specific state s. Indicates that the mth robot performs the action j corresponding to the jth task k , the value of k is determined according to the number of actions that can be executed in the jth task.

[0084] The action space A can be expressed as:

[0085]

[0086] Define the reward function. The reward function R is the total benefit of executing the corresponding action a in the action space A under a specific state s in the state space S. It can be expressed as:

[0087]

[0088] Among them, the function d() represents the i-th robot S i,1 Execute an action The benefit minus the cost of executing the current task; constraints represents the constraints, constraints=min f(x1,x2,x3,…x n ); where x n It can refer to the length of the task path executed, the number of actions performed, power consumption, and execution time, etc.

[0089] Preferably, since the robot's assigned tasks are updated in real time, a state transition function needs to be defined. The state transition function T represents the transition from executing action a in state s to a new state S. n , the state transition function can be expressed as:

[0090] T(s,a)=S n

[0091] After completing the definitions of the above state space, action space, reward function and state transition function, it is necessary to further design a neural network to solve the task allocation strategy.

[0092] Build an initial task allocation model based on the Transformer neural network. The Transformer neural network was originally designed to process sequence-to-sequence tasks such as machine translation. However, due to its powerful feature extraction and pattern recognition capabilities, it has also been applied to combinatorial optimization problems.

[0093] Obtain the historical updated task parameters and historical task allocation strategies of several robots. These historical data can be obtained through the database corresponding to each robot or the database of the same management platform for each robot. Based on the state space architecture defined above, a multi-robot state space s containing the states of several robots is constructed according to the historical updated task parameters and the above state space architecture. t , construct the multi-robot action space a according to each historical task allocation strategy and the above action space architecture t, where the historical task allocation strategy is essentially a set of mapping relationships between robots, tasks, and actions, including robot ID, task ID, and the action corresponding to the task. That is, the action performed by each robot under each task can be determined according to the historical task allocation strategy; according to the state space s t and action space a t Construct reward function R(s t ,a t ). The inverse of the reward function is used as the loss function of the initial task assignment model.

[0094] It should be noted that the state space s here t and action space a t After obtaining the historical updated task parameters and historical task allocation strategies of several robots, the parameters are filled in according to the framework of the state space S and action space A constructed above, that is, in essence, the state space s t and action space a t It is the different numerical filling forms of the state space S and the action space A under the same architecture. The state space s t The structure is completely consistent with the state space S, and the action space a t It is identical in structure to action space A.

[0095] In the state space s t Assign the input of the model to the initial task, in the action space a t The output of the model is assigned to the initial task, with the reward function R(s t ,a t ) as the loss function of the initial task assignment model, and iteratively train the initial task assignment model until the loss function of the initial task assignment model is minimized, and then generate a task assignment model. At this time, the task assignment model also establishes a state space s t To obtain the action space a of the maximum reward R t relationship.

[0096] Preferably, when obtaining the historical updated task parameters of several robots, the historical task parameters of the robots need to be obtained first, and then each robot creates its own Q table according to the corresponding state in the state space, that is, the historical task parameters. The table includes i rows and j columns, wherein the value of the i-th row and j-th column is Q(s i,1 ,a i,j ), the Q value is initialized to the reward value R(s i,1 ,a i,j ), where s i,1 Represents the state of the i-th robot, including the current position information of the i-th robot, the task sequence and the task status of each task in the task sequence, a i,jIndicates that the i-th robot performs the action corresponding to the j-th task. It should be noted that the Q table created by each robot only stores its own state and action, and does not store the state and action of other robots. The Q tables of the robots other than the current robot are compared with the Q table of the current robot to determine whether the same executable action is shared by other robots. If the action corresponding to the j-th task performed by the i-th robot is shared by other robots, the Q value of the action is updated to the maximum reward value of all robots sharing the action. For example, robot s 1,1 ,s 2,1 ,s 3,1 All three robots can perform action j of task j k , the Q values of the three robots are and The maximum value in With this Update the value of and The values of are used to obtain the updated Q tables of the three robots.

[0097] After updating each robot's Q-table, a game model is established. This model includes the participants, their strategy space, and the payoff of each strategy. In game theory, a Nash equilibrium refers to a situation where all participants adopt a certain strategy. In this situation, changing any strategy by any participant will not increase the total payoff. In this method, the task allocation strategy of each robot is optimized by iteratively removing the worst strategy and combining it with a task allocation model. In this game model, there are m robots as participants, the strategy space is the executable actions in each robot's Q-table, and the payoff of each strategy is the Q value of executing the corresponding action in each robot's Q-table.

[0098] In a preferred embodiment, before inputting the updated task parameters of each robot into the task allocation model, the method further includes: determining the number of pending task allocation strategies for each robot based on the updated task parameters of each robot; obtaining the total number of pending task allocation strategies for each robot based on the number of pending task allocation strategies for each robot; if the total number of pending task allocation strategies is greater than the total number of robots, repeatedly performing the worst strategy elimination operation until the total number of pending task allocation strategies is equal to the total number of robots; using the updated task parameters of each robot corresponding to the time when the total number of pending task allocation strategies is equal to the total number of robots as the updated task parameters of each robot input into the task allocation model;

[0099] Among them, the worst strategy elimination operation includes: for each robot, calculating the benefits of each pending task allocation strategy of the current robot; taking the pending task allocation strategy with the lowest benefit among all robots as the selected strategy; when the selected strategy is not the only pending task allocation strategy for the corresponding robot, eliminating the selected strategy, and calculating the total number of pending task allocation strategies after elimination; when the selected strategy is the only pending task allocation strategy for the corresponding robot, eliminating the pending task allocation strategies with the lowest benefits other than the selected strategy and which are not the only pending task allocation strategy for the corresponding robot, and calculating the total number of pending task allocation strategies after elimination.

[0100] Specifically, such as Figure 2 Figure 2 shows a Nash equilibrium calculation flow chart provided by the present invention. First, the number of pending task allocation strategies for each robot is determined based on its Q table. The total number of pending task allocation strategies for each robot is then calculated based on the number of pending task allocation strategies. If the total number of pending task allocation strategies is greater than the total number of robots, the termination condition is not met. The worst strategy elimination operation is then performed, eliminating the worst strategies until the total number of pending task allocation strategies equals the total number of robots.

[0101] The worst strategy elimination operation includes: calculating the benefits of each pending task allocation strategy for each robot, that is, calculating the Q value of the corresponding action of each pending task allocation strategy for each robot in the Q table of the current robot, and selecting the pending task allocation strategy with the lowest Q value for all robots as the selected strategy. If the selected strategy is not the only pending task allocation strategy for the corresponding robot, that is, after eliminating the selected strategy, there are still task allocation strategies that can be executed in the corresponding robot, then the selected strategy is eliminated, and the total number of pending task allocation strategies after elimination is calculated. The calculation of the total number is used to determine whether to continue the next worst strategy elimination operation. If the selected strategy is the only pending task allocation strategy for the corresponding robot, then if the selected strategy is deleted, the corresponding robot will lack an executable strategy. Therefore, if the selected strategy is the only pending task allocation strategy for the corresponding robot, the pending task allocation strategy with the lowest benefit other than the selected strategy and not the only pending task allocation strategy for the corresponding robot is eliminated, and the total number of pending task allocation strategies after elimination is calculated. Through this worst strategy elimination operation, each robot contains a task allocation strategy to be decided, and the task allocation strategy combinations to be processed in the input model are reduced, thereby reducing the computational complexity of the model.

[0102] After completing the strategy elimination until the termination condition is met, that is, the total number of pending task allocation strategies is equal to the total number of robots and each robot contains a pending task allocation strategy, the state space s is constructed based on the state of each robot after the termination condition is met.t , the state space s t Input into the task allocation model, and solve the output action space a through the task allocation model t , the output action space a t is the Nash equilibrium solution of the game model, which is in the state space s t The optimal solution for the task assignment of each robot is t The optimal task allocation strategy for each robot.

[0103] In a preferred embodiment, after the task allocation model outputs the task allocation strategy for each robot, the method further includes: controlling each robot to execute the task according to the task allocation strategy, and updating the task parameters of each robot after the task is completed.

[0104] Specifically, according to the task allocation strategy, the tasks to be assigned are reallocated to the task sequence of the corresponding robots, and each robot is controlled to perform tasks according to its own task sequence. After completing the corresponding tasks, the state of each robot in the state space S is updated according to the state transfer function T defined above, that is, the task parameters of each robot in the state space S are updated.

[0105] Based on the above method embodiments, the present invention provides corresponding device embodiments.

[0106] like Figure 3 As shown, an embodiment of the present invention provides a multi-robot dynamic task scheduling device, including: a task pre-assignment module, a task parameter acquisition module, a task parameter update module and a task assignment module;

[0107] The task pre-assignment module is used to obtain tasks to be assigned and assign the tasks to a number of robots according to robot types;

[0108] The task parameter acquisition module is used to obtain the task parameters of each robot; wherein the task parameters include: task sequence, current position information, target position information required to move to perform each task in the task sequence, and task status of each task in the task sequence;

[0109] The task parameter updating module is configured to create a Q-table for each robot based on the task parameters of the current robot; wherein the Q-table includes reward values for the current robot when performing different assigned tasks under the current task parameters; update the Q-table of the current robot based on the Q-tables of the robots other than the current robot to obtain an updated Q-table for the current robot; wherein, when updating the Q-table of the current robot based on the Q-tables of the robots other than the current robot, if the same assigned task is simultaneously in the Q-tables of several robots, update the reward value for the corresponding assigned task in the Q-table of the current robot with the maximum value of the reward values in the Q-tables of all robots under the same assigned task; and determine the updated task parameters of each robot based on the updated Q-tables of each robot;

[0110] The task allocation module is used to input the updated task parameters of each robot into the task allocation model, so that the task allocation model outputs the task allocation strategy of each robot.

[0111] In a preferred embodiment, it further comprises: a task allocation model building module;

[0112] The task allocation model construction module is used to obtain historical updated task parameters and historical task allocation strategies of several robots;

[0113] Constructing a state space of multiple robots based on each historical updated task parameter, constructing an action space of multiple robots based on each historical task allocation strategy, and constructing a reward function based on the state space and action space;

[0114] An initial task allocation model is constructed, and the initial task allocation model is iteratively trained using the state space and action space until the loss function of the initial task allocation model is minimized, thereby generating the task allocation model; wherein the loss function is the inverse of the reward function.

[0115] In a preferred embodiment, it further comprises: a policy elimination module;

[0116] Before inputting the updated task parameters of each robot into the task allocation model, it also includes:

[0117] Determine the number of pending decision-making task allocation strategies for each robot based on the updated task parameters of each robot;

[0118] The total number of the robots' pending task allocation strategies is obtained according to the number of pending task allocation strategies of each robot;

[0119] If the total number of pending task allocation strategies is greater than the total number of robots, the worst strategy elimination operation is repeated until the total number of pending task allocation strategies is equal to the total number of robots;

[0120] The updated task parameters of each robot corresponding to the total number of task allocation strategies to be decided is equal to the total number of robots are used as the updated task parameters of each robot input into the task allocation model;

[0121] The worst strategy elimination operation includes:

[0122] For each robot, calculate the benefits of the current robot's strategy for allocating tasks to be decided;

[0123] The strategy for allocating tasks to be decided that has the lowest benefit among all robots is selected as the strategy;

[0124] When the selected strategy is not the only pending task allocation strategy for the corresponding robot, the selected strategy is eliminated, and the total number of pending task allocation strategies after elimination is calculated;

[0125] When the selected strategy is the only pending task allocation strategy for the corresponding robot, the pending task allocation strategies with the lowest benefits other than the selected strategy and not the only pending task allocation strategy for the corresponding robot are eliminated, and the total number of pending task allocation strategies after elimination is calculated.

[0126] In a preferred embodiment, it further comprises: a control update module;

[0127] The control and updating module is used to control each robot to perform a task according to the task allocation strategy, and to update the task parameters of each robot after the task is completed.

[0128] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0129] Those skilled in the art will clearly understand that for the sake of convenience and brevity, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0130] Based on the above method embodiment, the present invention provides a corresponding terminal device embodiment.

[0131] An embodiment of the present invention provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a multi-robot dynamic task scheduling method as described in any one of the present inventions is implemented.

[0132] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0133] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, connecting various parts of the entire terminal device using various interfaces and lines.

[0134] The memory can be used to store the computer program, and the processor realizes various functions of the terminal device by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0135] Based on the above method embodiment, the present invention provides a corresponding storage medium embodiment.

[0136] An embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a multi-robot dynamic task allocation method described in any one of the present inventions.

[0137] The storage medium is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. The computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0138] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-robot dynamic task scheduling method, characterized in that: include: Obtain tasks to be assigned and assign them to several robots according to robot types; Obtaining task parameters for each robot; wherein the task parameters include: task sequence, current position information, target position information required to move to perform each task in the task sequence, and task status of each task in the task sequence; For each robot, create a Q table for the current robot based on the task parameters of the current robot; wherein the Q table includes the reward values of the current robot performing different assigned tasks under the current task parameters; The Q table of the current robot is updated based on the Q tables of the other robots except the current robot to obtain an updated Q table of the current robot. When updating the Q table of the current robot based on the Q tables of the other robots except the current robot, if the same assigned task is in the Q tables of multiple robots at the same time, the maximum reward value in the Q tables of all robots under the same assigned task is used to update the reward value under the corresponding assigned task in the Q table of the current robot; Determine the updated task parameters of each robot based on the updated Q table of each robot; The updated task parameters of each robot are input into the task allocation model so that the task allocation model outputs the task allocation strategy of each robot.

2. A multi-robot dynamic task scheduling method according to claim 1, characterized in that: The construction of the task allocation model includes: Obtain historical updated task parameters and historical task allocation strategies for several robots; Constructing a state space of multiple robots based on each historical updated task parameter, constructing an action space of multiple robots based on each historical task allocation strategy, and constructing a reward function based on the state space and action space; An initial task allocation model is constructed, and the initial task allocation model is iteratively trained using the state space and action space until the loss function of the initial task allocation model is minimized, thereby generating the task allocation model; wherein the loss function is the inverse of the reward function.

3. A multi-robot dynamic task scheduling method according to claim 2, characterized in that: Before inputting the updated task parameters of each robot into the task allocation model, it also includes: Determine the number of pending decision-making task allocation strategies for each robot based on the updated task parameters of each robot; The total number of the robots' pending task allocation strategies is obtained according to the number of pending task allocation strategies of each robot; If the total number of pending task allocation strategies is greater than the total number of robots, the worst strategy elimination operation is repeated until the total number of pending task allocation strategies is equal to the total number of robots; The updated task parameters of each robot corresponding to the total number of task allocation strategies to be decided is equal to the total number of robots are used as the updated task parameters of each robot input into the task allocation model; The worst strategy elimination operation includes: For each robot, calculate the benefits of the current robot's strategy for allocating tasks to be decided; The strategy for allocating tasks to be decided that has the lowest benefit among all robots is selected as the strategy; When the selected strategy is not the only pending task allocation strategy for the corresponding robot, the selected strategy is eliminated, and the total number of pending task allocation strategies after elimination is calculated; When the selected strategy is the only pending task allocation strategy for the corresponding robot, the pending task allocation strategies with the lowest benefits other than the selected strategy and not the only pending task allocation strategy for the corresponding robot are eliminated, and the total number of pending task allocation strategies after elimination is calculated.

4. A multi-robot dynamic task scheduling method as claimed in claim 3, characterized in that: After the task allocation model outputs the task allocation strategy for each robot, it also includes: Control each robot to perform tasks according to the task allocation strategy, and update the task parameters of each robot after the task is completed.

5. A multi-robot dynamic task scheduling device, characterized in that: include: Task pre-assignment module, task parameter acquisition module, task parameter update module and task assignment module; The task pre-assignment module is used to obtain tasks to be assigned and assign the tasks to a number of robots according to robot types; The task parameter acquisition module is used to obtain the task parameters of each robot; wherein the task parameters include: task sequence, current position information, target position information required to move to perform each task in the task sequence, and task status of each task in the task sequence; The task parameter updating module is configured to create a Q-table for each robot based on the task parameters of the current robot; wherein the Q-table includes reward values for the current robot when performing different assigned tasks under the current task parameters; update the Q-table of the current robot based on the Q-tables of the robots other than the current robot to obtain an updated Q-table for the current robot; wherein, when updating the Q-table of the current robot based on the Q-tables of the robots other than the current robot, if the same assigned task is simultaneously in the Q-tables of several robots, update the reward value for the corresponding assigned task in the Q-table of the current robot with the maximum value of the reward values in the Q-tables of all robots under the same assigned task; and determine the updated task parameters of each robot based on the updated Q-tables of each robot; The task allocation module is used to input the updated task parameters of each robot into the task allocation model, so that the task allocation model outputs the task allocation strategy of each robot.

6. A multi-robot dynamic task scheduling device as claimed in claim 5, characterized in that: Also includes: Task allocation model building module; The task allocation model construction module is used to obtain historical updated task parameters and historical task allocation strategies of several robots; Constructing a state space of multiple robots based on each historical updated task parameter, constructing an action space of multiple robots based on each historical task allocation strategy, and constructing a reward function based on the state space and action space; An initial task allocation model is constructed, and the initial task allocation model is iteratively trained using the state space and action space until the loss function of the initial task allocation model is minimized, thereby generating the task allocation model; wherein the loss function is the inverse of the reward function.

7. A multi-robot dynamic task scheduling device according to claim 6, characterized in that: Also includes: Strategy culling module; Before inputting the updated task parameters of each robot into the task allocation model, it also includes: Determine the number of pending decision-making task allocation strategies for each robot based on the updated task parameters of each robot; The total number of the robots' pending task allocation strategies is obtained according to the number of pending task allocation strategies of each robot; If the total number of pending task allocation strategies is greater than the total number of robots, the worst strategy elimination operation is repeated until the total number of pending task allocation strategies is equal to the total number of robots; The updated task parameters of each robot corresponding to the total number of task allocation strategies to be decided is equal to the total number of robots are used as the updated task parameters of each robot input into the task allocation model; The worst strategy elimination operation includes: For each robot, calculate the benefits of the current robot's strategy for allocating tasks to be decided; The strategy for allocating tasks to be decided that has the lowest benefit among all robots is selected as the strategy; When the selected strategy is not the only pending task allocation strategy for the corresponding robot, the selected strategy is eliminated, and the total number of pending task allocation strategies after elimination is calculated; When the selected strategy is the only pending task allocation strategy for the corresponding robot, the pending task allocation strategies with the lowest benefits other than the selected strategy and not the only pending task allocation strategy for the corresponding robot are eliminated, and the total number of pending task allocation strategies after elimination is calculated.

8. The multi-robot dynamic task scheduling device according to claim 7, characterized in that: Also includes: Control update module; The control and updating module is used to control each robot to perform a task according to the task allocation strategy, and to update the task parameters of each robot after the task is completed.

9. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for dynamic task scheduling of multiple robots according to any one of claims 1 to 4 is implemented.

10. A storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the multi-robot dynamic task scheduling method according to any one of claims 1 to 4.

Images