A scale-independent unmanned cluster distributed collaborative task allocation method and system

Abstract

The present invention belongs to the technical field of multi-UAV agent task allocation, and relates to a scale-independent unmanned cluster distributed collaborative task allocation method. It adopts a centralized training distributed deployment form. Each agent relies on its own perception and local communication to obtain time-sensitive new tasks and information about adjacent agents, and uses MF-MADQN to perform distributed collaborative task allocation. The present invention defines a task candidate queue and a task execution queue in each agent. Time-sensitive new targets can be included in the task candidate queue at any time to participate in the allocation, and tasks that really need to be executed are moved into the task execution queue. In addition, the method of the present invention embeds a mean field mechanism, which equates the interaction of each agent with its adjacent agents to interaction with a virtual agent. Therefore, it can cope with the "dimensionality trap" phenomenon of neural networks when large-scale clusters and agents increase or decrease, thereby effectively solving the problem of large-scale unmanned cluster collaborative task allocation in unknown environments with time-sensitive new targets and agent increase or decrease scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-UAV intelligent agent task allocation, and specifically relates to a scale-independent unmanned cluster distributed collaborative task allocation method and system. Background Art

[0002] With the development of modern information communications and artificial intelligence, the application of autonomous unmanned intelligent agents is becoming increasingly widespread. Unmanned swarms solve the problem of a single drone / vehicle's inability to handle complex tasks. They possess high robustness and survivability, and are widely used in both military and civilian fields. When an unmanned swarm performs a task, it must first determine the sequence of tasks required of each drone / vehicle. Therefore, the problem of collaborative task allocation among multiple unmanned intelligent agents becomes crucial for the application of unmanned swarms. Traditional centralized task allocation methods have difficulties in responding to new, time-sensitive targets and the increase or decrease in the number of intelligent agents. The central controller has limited communication coverage, high computational overhead, and is prone to system-wide paralysis due to single-point failures. In recent years, distributed collaborative task allocation methods, due to their decentralized and self-organizing nature, have become a research hotspot for collaborative task allocation among multiple unmanned intelligent agents.

[0003] Existing distributed multi-UAV agent task allocation methods can be divided into two categories: one is based on consensus and auction mechanisms. Each agent uses local communication and combines its own and neighboring information to continuously negotiate or auction tasks, ultimately reaching a unified task allocation solution. While these methods can handle time-sensitive targets and scenarios with agent additions and subtractions, they carry a heavy communication burden, requiring numerous communications to reach a unified task allocation solution. Furthermore, as the number of agents and tasks to be assigned increases, the number of communications increases significantly, making them unsuitable for large-scale multi-UAV collaborative task allocation scenarios.

[0004] Another type is based on multi-agent reinforcement learning. Each unmanned device acts as an agent, using its own information and neighborhood information obtained through local communication as its state. It selects actions based on the trained strategy and ultimately completes collaborative task decisions. This type of method can also cope with new time-sensitive targets and has a low communication burden. It can also ensure the efficiency of its task decision-making in unknown environments. However, this method requires pre-training, and the original trained model becomes invalid when the number of agents changes. Moreover, when applied to large-scale unmanned cluster collaborative task allocation, the neural network suffers from the input and output "dimensionality trap" problem. Therefore, existing methods have difficulty solving the problem of large-scale unmanned cluster collaborative task allocation in unknown environments, with new time-sensitive targets and the increase or decrease of agents. Summary of the Invention

[0005] The purpose of the present invention is to provide a scale-independent unmanned cluster distributed collaborative task allocation method and system, which solves the problem of difficulty in large-scale unmanned cluster collaborative task allocation under time-sensitive new targets and scenarios of increase or decrease of intelligent agents.

[0006] The present invention is achieved through the following technical solutions:

[0007] The present invention discloses a scale-independent unmanned cluster distributed collaborative task allocation method, comprising the following steps:

[0008] Step 1: Scene information and agent initialization:

[0009] Each agent A i Correspondingly, there is a task candidate queue C i and task execution queue M i , where each agent's task candidate queue C i Save the tasks to be assigned, and the tasks are in the task execution queue M i The order in is agent A i The order in which tasks are executed;

[0010] For any task t j ,j∈{1,…,N t}, whose information is P j ={x j ,y j}, indicating that the task t j If there is a time-sensitive new target, the time-sensitive new task information is P new ={x new ,y new} and include it in its own task candidate queue C i middle;

[0011] Step 2: Establish UAV communication topology and local communication model;

[0012] Step 3: Use the mean field mechanism to establish the state space, action space, and reward function of each agent in the task allocation process:

[0013] Step 4: Train the MF-MADQN network to obtain a trained MF-MADQN network.

[0014] Step 5: Use the trained MF-MADQN network to perform multi-UAV agent collaborative task allocation. For a given multi-UAV collaborative task allocation scenario, each UAV agent A i Need to execute queue M according to the assigned tasks i Starting from the starting point, strike the corresponding targets in sequence;

[0015] In step 3, the reward function needs to calculate its own reward function r i and adopting joint rewards considering virtual agents Among them, each agent’s own reward function r i By maintaining rewards Receive Rewards Transfer Rewards Redundant action penalty It consists of four parts;

[0016] Agent A i When performing action a i The subsequent self-reward function r i The calculation formula is as follows:

[0017] r i =r i 0 +r i 1 +r i 2 +r i 3 ;

[0018] Maintenance reward r i 0 Represents the agent A i Candidate task queue C i The reward for maintaining the decision in the task selection, receiving reward r i 1 Represents the agent A i Candidate task queue C i The reward for the task selection decision in the task selection; the transfer reward r i 2 Represents the agent A i Candidate task queue C i The reward for the task selection transfer decision in , and the redundant action penalty r i 3 For agent A i Candidate task queue C i Punishment for hollow tasks but action decisions are receive and transfer;

[0019] Since the goal of multi-agent reinforcement learning in a cooperative situation is to maximize global benefits, if the agent only considers its own rewards, it will fall into selfishness. Therefore, the joint reward function is defined in the following form:

[0020]

[0021] β∈[0,1] represents the global reward adjustment coefficient. When β approaches 0, it means that each agent pays more attention to its own benefits and is more selfish; when β approaches 1, it means that each agent pays more attention to cooperation to improve the global benefits. i For agent A i A set of neighboring agents with which communication can be established.

[0022] Furthermore, step 2 is specifically as follows:

[0023] Assume that each drone agent A i The communication radius is Then for any UAV agent A i and A j , when the following equation is satisfied:

[0024]

[0025] Represents agent A i Can obtain agent A j Status information w j ;

[0026] in For agent A j Current location, For agent A j The number of tasks currently to be executed, L j For agent A j Remaining distance

[0027] Therefore, agent A i The obtained neighboring agent information set is expressed as satisfy Among them D i For agent A i A set of neighboring agents with which communication can be established.

[0028] Furthermore, in step 3, the state space establishment process of each agent is as follows:

[0029] The state of each agent is determined by its own task candidate queue information C i And its own running status w i It consists of two parts, namely:

[0030] s i = <C i ,w i >

[0031] In the state space definition, the mean field mechanism is used to make the interaction between the agent and all adjacent agents equivalent to the interaction with a virtual agent, and to share the local state. iThe joint state of is represented as:

[0032]

[0033] Among them D i For agent A i A set of neighboring agents with which communication can be established.

[0034] Furthermore, in step 3, the process of establishing the action space of each agent is as follows:

[0035] In the task allocation process, each agent A i It is necessary to determine the candidate sequence C of the task i For any task t j ∈C i , the agent’s decision d for this task j There are three types: 0-maintain, 1-accept and 2-transfer, i.e. d j ∈{0,1,2};

[0036] Then each agent A i The action space is:

[0037]

[0038] Among them, any d k i ∈a i Represents agent A i For task t k i ∈C i decision-making;

[0039] Based on the mean field mechanism, we get agent A i The virtual agent action formed by all adjacent agents as follows:

[0040]

[0041] Among them D i For agent A i A set of neighboring agents with which communication can be established.

[0042] Furthermore, if the decision is to maintain, then after this time step, task t j Still remain in Agent A i Task candidate queue C i middle;

[0043] If the decision is to accept, then after this time step, task t j From Agent A i Task candidate queue C iTransfer to task execution queue M i , use the greedy algorithm to calculate and insert M one by one i The accumulated distance at each position of the queue is j Insert the position with the shortest accumulated distance;

[0044] If the decision is to transfer, then after this time step, task t j From Agent A i Task candidate queue C i Transfer to the task candidate queue of any adjacent agent and wait for the decision in the next time step.

[0045] Further, let represents the number of tasks in the candidate task queue that need to perform maintenance actions, then the maintenance reward r i 0 The calculation formula is as follows:

[0046]

[0047] make Represents the set of tasks in the candidate task queue that need to perform the receiving action, then the receiving reward r i 1 The calculation formula is as follows:

[0048]

[0049] Where R k Represents task t k The task execution benefit is α, the distance calculation coefficient is set as dis{M i} represents agent A i Execute M in sequence i The total flight distance of the mission, Indicates the tasks to be received Follow the greedy algorithm to insert the shortest cumulative distance into M i The subsequent task sequence;

[0050] make represents the number of tasks in the candidate task queue that need to perform transfer actions, then the transfer reward r i 2 The calculation formula is as follows:

[0051]

[0052] represents the number of empty tasks but the action decision is to receive and transfer, then the redundant action penalty r i 3 The calculation formula is as follows:

[0053]

[0054] Furthermore, in step 4, the training method of the MF-MADQN network specifically includes the following steps:

[0055] Step 401: Initialize the agent neural network parameters;

[0056] Step 402: Initialize the task allocation scenario;

[0057] Step 403: Each agent makes an action decision;

[0058] Each agent A i The joint state obtained based on the current time step As the input of the MF-MADQN network, the current agent A is obtained i Action a i ;

[0059] Step 404: Each agent performs an action to update the task candidate queue and the task execution queue;

[0060] Then use the method of establishing the action space of each agent in step 1 to obtain its own reward r i , and generate the next state w i ';

[0061] Step 405: Each agent communicates with adjacent agents to obtain information about the adjacent agents.

[0062] Each agent A i Obtain the actions, immediate rewards, and next state of each adjacent agent, and then calculate the virtual agent actions separately Joint reward function Next joint state

[0063] Step 406: Store the experience in a buffer;

[0064] The experience gained by each agent in step 405 is expressed in the form of a 5-tuple. Save in the experience buffer;

[0065] If the number of experiences in the experience buffer is less than the number of batches required for one training session, the process returns to step 403 to continue acquiring new experiences. If the number of experiences in the experience buffer is sufficient, the process extracts experiences of a specified number of batches using experience-first playback, and then proceeds to step 407 for training.

[0066] Step 407: Perform neural network training;

[0067] According to the experience extracted in step 406, the target value of each sampling experience is calculated respectively

[0068]

[0069] The parameters of the action network are then act The gradient is given by the following loss function ξ(φ act ) to obtain the action network parameters φ act Update with learning rate τ in the form of minimizing loss;

[0070]

[0071] Then the parameters of the target network φ tar Soft update according to update rate δ;

[0072] φ tar =δφ act +(1-δ)φ tar ;

[0073] Among them, δ<<1, represents the target network parameter value φ tar The training process is now complete with slow changes.

[0074] Step 408: Each agent performs termination detection: If any agent A in the cluster i Task candidate queue C i If it is not empty, then return to step 403 and continue the task allocation process for the next time step; if all agents have no candidate tasks, it means that the current scene has been allocated, then return to step 401 to reinitialize, and the training process ends when the specified number of training times is achieved.

[0075] Furthermore, in step 5, the trained MF-MADQN network is used to perform collaborative task allocation among multiple UAV agents, which specifically includes the following steps:

[0076] Step 501: Each agent loads A i The trained neural network parameters are converted to N according to the needs of the scene. t Targets are randomly injected into each agent A without duplication i Task candidate queue C i middle;

[0077] Step 502: Use the same methods as in steps 403, 404, and 405 to make action decisions, update status, and obtain the status of adjacent agents;

[0078] When each agent A i Discover new targets when time is short new When the time-sensitive new task information P new ={x new,y new} Join its own task candidate queue C i , and then participate in the allocation process;

[0079] Step 503: Each agent performs termination detection: If any agent A in the cluster i Task candidate queue C i If it is not empty, then return to step 403 and continue the task assignment process for the next time step; if all agents have no candidate tasks, it means that all tasks have been assigned.

[0080] The present invention also discloses a scale-independent unmanned cluster distributed collaborative task allocation system for implementing the scale-independent unmanned cluster distributed collaborative task allocation method, comprising a plurality of intelligent agents A i , each agent A i Correspondingly, there are task candidate queue module, task execution queue module, MF-MADQN network module, local communication module, self-perception module and task execution module;

[0081] The task candidate queue module is used to store the task candidate queue C i The task execution queue module is used to store the task execution queue M i ; Among them, each agent task candidate queue C i Save the tasks to be assigned, task execution queue M i Save A i The task to be executed, and the task execution queue M i The order in is agent A i The order in which tasks are executed;

[0082] For any task t j ,j∈{1,…,N t}, whose information is P j ={x j ,y j}, indicating that the task t j location;

[0083] The local communication module is used to establish the UAV communication topology and local communication model. Only UAVs within a limited communication range can communicate locally.

[0084] The self-perception module is used to determine the i Call its own local sensors to analyze the time-sensitive new task information P new ={x new ,y new} and include it in its own task candidate queue C i middle;

[0085] The MF-MADQN network module adopts the DQN network infrastructure, with an action network and a goal network. All agents share an experience buffer pool to store experience information. The mean field mechanism is used to establish the state space, action space, and reward function of each agent in the task allocation process.

[0086] The execution module is used to use the trained MF-MADQN network to perform multi-UAV agent collaborative task allocation. For a given multi-UAV collaborative task allocation scenario, each UAV agent A i Need to execute queue M according to the assigned tasks i Starting from the starting point, strike the corresponding targets in turn.

[0087] Compared with the prior art, the present invention has the following beneficial technical effects:

[0088] This paper employs a distributed collaborative task allocation method (MF-MADQN) based on multi-agent reinforcement learning. This method employs a centralized training and distributed deployment model. Each agent relies on its own perception and local communication to acquire information about new tasks and neighboring agents, and then uses MF-MADQN to perform distributed collaborative task allocation. Because the method incorporates a mean field mechanism, equating the interactions between each agent and its neighbors to interactions with a virtual agent, it can address the "dimensionality trap" phenomenon of neural networks in large-scale clusters and when agents are added or subtracted. This effectively solves the problem of large-scale unmanned cluster collaborative task allocation in unknown environments, with time-sensitive new targets, and when agents are added or subtracted.

[0089] First, each intelligent agent distributes collaborative tasks in a distributed manner through local communication, avoiding the need for continuous access to a single central controller, expanding the activity range of the unmanned cluster, and avoiding the problem of system-wide paralysis caused by single-point failure of the controller; in addition, when a time-sensitive target is discovered, each intelligent agent can rely on its own perception to join its own candidate queue in real time, and then perform task allocation with other intelligent agents without going through the lengthy process of "task return-central controller reallocation-new result reception", effectively improving the ability to respond to time-sensitive targets, while traditional centralized methods generally have poor reallocation capabilities; finally, by embedding the mean field mechanism into the neural network, the number of adjacent intelligent agents and the communication topology can change dynamically during the task allocation process, effectively coping with the scenario of increase or decrease of intelligent agents, while centralized methods require task reallocation.

[0090] Compared to existing distributed unmanned cluster collaborative task allocation methods, the method proposed in this paper utilizes a multi-agent reinforcement learning mechanism. This reduces the number of communications required during the allocation process, eliminating the need for extensive communication required by consensus-based, auction-based methods. This significantly reduces the communication overhead associated with large-scale unmanned cluster collaborative task allocation. Compared to traditional multi-agent reinforcement learning methods, our proposed method incorporates a mean field mechanism, equating each agent's interactions with other adjacent agents to interactions with a virtual agent. This effectively addresses challenges such as the large number of adjacent agents in large-scale clusters and the dynamic increase and decrease of agents. It eliminates the input-output "dimensionality trap" problem, and the embedded mean field mechanism ensures that the original model remains usable even when the size of the unmanned cluster changes, eliminating the need for retraining. This method exhibits superior generalization capabilities and engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 This is a block diagram of the architecture of the method of the present invention.

[0092] Figure 2 Flowchart of the method of the present invention.

[0093] Figure 3 Schematic diagram of the mean field mechanism of the method of the present invention.

[0094] Figure 4 Schematic diagram of the task queue structure of each agent in the method of the present invention.

[0095] Figure 5 This is a flow chart of the neural network training process of the method of the present invention.

[0096] Figure 6 The present invention provides a task allocation process flow chart. DETAILED DESCRIPTION

[0097] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following is a further detailed description with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. That is, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments.

[0098] The components described and illustrated in the drawings and embodiments of the present invention may be arranged and designed in a variety of different configurations. Therefore, the detailed description of the embodiments of the present invention provided in the following drawings is not intended to limit the scope of the claimed invention, but merely represents a selected embodiment of the present invention. All other embodiments derived by those skilled in the art based on the drawings and embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.

[0099] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0100] In order to solve the problem of difficulty in allocating large-scale unmanned cluster collaborative tasks under time-sensitive new targets and scenarios of agent increase and decrease, the present invention provides a scale-independent unmanned cluster distributed collaborative task allocation method.

[0101] The method proposed in this invention is executed in a distributed manner in each agent. The main steps are as follows:

[0102] Step 1: Initialize scene information and each agent. i Through the task candidate queue C i and task execution queue M i To buffer tasks, each agent's task candidate queue stores tasks to be assigned, and tasks are stored in the task execution queue M i The order in is agent A i The task execution order. For any task t j ,j∈{1,…,N t}, whose information is P j ={x j ,y j}, indicating that the task point t j For the task allocation scenario, except for the time-sensitive new target, the location of each task point is known.

[0103] Step 2: Establish the UAV communication topology and local communication model. i , obtain the state information w of the surrounding intelligent agents according to the communication radius of the communication device it carries j . in For agent A j Current location, For agent A j The number of tasks currently to be executed, L j For agent A j The remaining distance. Therefore, agent A i The obtained neighboring agent information set is expressed as satisfy Among them D i For agent A i A set of neighboring agents with which communication can be established.

[0104] Step 3: The state space, action space, and reward function of each agent in the task allocation process are established using the mean field mechanism. The state s of each agent is i By its own task candidate queue information C i And its own running status w iThe mean field mechanism is used to make the interaction between the agent and all adjacent agents equivalent to the interaction with a virtual agent, and local state sharing is performed to obtain the joint state of the virtual agent. The agent has three decisions for each task in the task candidate queue: 0-maintain, 1-accept and 2-transfer, thus obtaining the agent action space a i Joint actions with virtual agents Similarly, the reward function calculates its own reward function r i and considering the joint reward of the virtual agent

[0105] Step 4: Train the proposed MF-MADQN network. This method requires centralized training. The training process includes initializing the agent neural network parameters, initializing the task allocation scenario, selecting actions for each agent, executing actions, communicating with neighboring agents, obtaining neighboring agent information, storing experience in a buffer, training the neural network, and performing termination detection for each agent.

[0106] Step 5: Use the trained MF-MADQN network to perform collaborative task allocation across multiple UAV agents. This method operates in a distributed manner across each agent, with each agent relying on local communication to complete collaborative task allocation. The task allocation process includes loading neural network parameters, loading task information, making action decisions for each agent, updating state, acquiring the state of neighboring agents, and detecting the termination of each agent.

[0107] The method of the present invention is further described in detail below with reference to the accompanying drawings and embodiments of the present invention. u UAV agents attack N t Take the multi-UAV collaborative task allocation scenario with a target as an example. Figure 1 The specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0108] like Figure 2 As shown, the present invention includes the following steps:

[0109] Step 1: Scene information and agent initialization:

[0110] Each agent A i ,i∈{1,…,N u} has a task candidate queue C i and a task execution queue M i To buffer tasks, the queue lengths are N C and N M , where N MThe maximum number of tasks that can be performed by the UAV should not exceed N. Ci and N Mi Represents the task candidate queue C i and task execution queue M i The actual number of tasks in the task execution queue M i The order in is agent A i The order in which tasks are executed.

[0111] For any task t j ,j∈{1,…,N t}, whose information is P j ={x j ,y j}, indicating that the task point t j In addition, if there is a time-sensitive new target t new , whose information is P new ={x new ,y new}, with the same meaning.

[0112] Step 2: Establish UAV communication topology and local communication model:

[0113] Each drone agent A i , the communication radius of the communication device it carries is Then for any UAV agent A i and A j , if and only if the following equation is satisfied:

[0114]

[0115] Represents agent A i Can obtain agent A j Running status information w j . in For agent A j Current location, For agent A j The number of tasks currently to be executed, L j For agent A j The remaining distance. Therefore, agent A i The obtained neighboring agent information set is expressed as satisfy Among them D i For agent A i A set of neighboring agents with which communication can be established.

[0116] Step 3: For a given multi-UAV collaborative task allocation scenario, each UAV agent A iNeed to execute queue M according to the assigned tasks i Starting from the starting point, the corresponding targets are attacked in sequence. Therefore, the state space, action space, and reward function of each agent in the task allocation process are defined as follows:

[0117] Step 301: Establish the state space of each agent;

[0118] The state of each agent is determined by its own task candidate queue information C i And its own running status w i It consists of two parts, namely

[0119] s i = <C i ,w i >

[0120] Since the number of neighboring agents may change during the task allocation process, and the number of neighboring agents in a large-scale cluster is huge, it is difficult to train and deploy neural networks. Therefore, in the definition of state space, the present invention uses Figure 3 The mean field mechanism shown solves the above problem and treats the interaction between the agent and all adjacent agents as equivalent to the interaction with a virtual agent, sharing local states. Therefore, agent A i The joint state can be expressed as:

[0121]

[0122] Step 302: Establish the action space of each agent;

[0123] Each agent A i It is necessary to make decisions on the tasks in its candidate task sequence. For any task t j ∈C i , the agent’s decision d for this task j There are three types: 0-maintain, 1-accept and 2-transfer, i.e. d j ∈{0,1,2}. If the decision is to maintain, then the task t after this time step j Still remain in Agent A i Task candidate queue C i If the decision is to accept, then the task t after this time step j From Agent A i Task candidate queue C i Transfer to task execution queue M i , use the greedy algorithm to calculate and insert M one by one i The accumulated distance at each position of the queue is t j Insert the position with the shortest cumulative distance; if the decision is to transfer, then the task t after this time step j From Agent Ai Task candidate queue C i Transfer to the task candidate queue of a random adjacent agent and wait for the decision in the next time step. Figure 4 The task queue structure and corresponding action operations of the present invention are further demonstrated.

[0124] According to the decision of each task, agent A i The action space is defined as

[0125]

[0126] Among them, any d k i ∈a i Represents agent A i For task t k i ∈C i Decision, task candidate sequence C i The length is N C .

[0127] Similarly, the present invention is based on the mean field mechanism to share local actions and obtain the agent A i The virtual agent action formed by all adjacent agents as follows:

[0128]

[0129] Step 303: Establish a reward function for each agent;

[0130] Each agent A i You can get your own reward r at each time step i and the local state s i The corresponding self-reward function r i It needs to be calculated using the following method:

[0131] According to the definition of action space, agent A i Task candidate sequence task candidate queue C i Each task in is given a unique decision, so its own reward r i By maintaining the reward r i 0 Receive reward r i 1 , transfer reward r i 2 , redundant action penalty r i 3 It consists of four parts.

[0132] Maintenance reward r i 0Represents the agent A i Candidate task queue C i The reward obtained by maintaining the decision of task selection in represents the number of tasks in the candidate task queue that need to perform maintenance actions, then the maintenance reward r i 0 The calculation formula is as follows:

[0133]

[0134] Receive reward r i 1 Represents the agent A i Candidate task queue C i The task selection in receives the reward for the decision, Represents the set of tasks in the candidate task queue that need to perform the receiving action, and then receive the reward The calculation formula is as follows:

[0135]

[0136] Where R k Represents task t k The task execution benefit is α, the distance calculation coefficient is set as dis{M i} represents agent A i Execute M in sequence i The total flight distance of the mission, Indicates that The tasks to be received are inserted into M in turn according to the greedy algorithm i The task sequence after the position corresponding to the shortest cumulative distance.

[0137] Transfer Rewards Represents the agent A i Candidate task queue C i The reward obtained by the task selection transfer decision in represents the number of tasks in the candidate task queue that need to perform transfer actions, then the transfer reward r i 2 The calculation formula is as follows:

[0138]

[0139] Redundant action penalty r i 3 For agent A i Candidate task queue C i Penalty for hollow tasks but action decisions are receive and transfer, represents the number of empty tasks but the action decision is to receive and transfer, then the redundant action penalty r i 3The calculation formula is as follows:

[0140]

[0141] Therefore, agent A i When performing action a i After the self reward r i The calculation formula is as follows:

[0142] r i =r i 0 +r i 1 +r i 2 +r i 3

[0143] Since the goal of multi-agent reinforcement learning in a cooperative situation is to maximize global benefits, if the agent only considers its own rewards, it will fall into self-interest. Therefore, the present invention defines the reward function in the following form:

[0144]

[0145] Here β∈[0,1] represents the global reward adjustment coefficient. When β approaches 0, it means that each agent pays more attention to its own benefits and is more selfish; when β approaches 1, it means that each agent pays more attention to cooperation to improve the global benefits.

[0146] Step 4: Train the proposed MF-MADQN network. The process is as follows Figure 5 As shown:

[0147] Step 401: Initialize the agent neural network parameters;

[0148] Initialize each agent A i Each agent adopts the DQN network infrastructure, with an action network and a target network. All agents share an experience buffer pool to store experience information.

[0149] Step 402: Initialize the task allocation scenario;

[0150] Initialize the task allocation scenario and assign all N t Targets are randomly injected into each agent A without duplication i Task candidate queue C i middle.

[0151] Step 403: Each agent makes action decisions;

[0152] Each agent A i According to the joint state obtained at this time step As the input of the neural network, we get the current agent A i Action a i .

[0153] Step 404: Each agent performs an action and updates the task candidate queue and task execution queue;

[0154] Each agent A i Execute action a i , for its own task candidate queue C i All tasks in the process are processed, and each task is processed according to action a i The decision is to perform one of the following actions: maintain, receive, or transfer.

[0155] If it is a maintenance action, the task will still remain in the agent's task candidate queue after this time step;

[0156] If it is a receiving action, then after this time step, the task is transferred from the agent's task candidate queue to the task execution queue, and its location is obtained by calculating the shortest cumulative distance position using the greedy algorithm;

[0157] If it is a transfer action, the task will be transferred from the agent's task candidate queue to the task candidate queue of any adjacent agent after this time step, waiting for the decision in the next time step.

[0158] Then use the method in step 303 to get your own reward r i , and generate the next self-state observation w i ′.

[0159] Step 405: Each agent communicates with adjacent agents to obtain information about the adjacent agents;

[0160] Each agent A i Obtain the action, immediate reward, and next local state of each adjacent agent, and then calculate the following steps:

[0161] Step 406: Store the experience into a buffer;

[0162] The experience gained by each agent in step 405 is expressed in the form of a 5-tuple. Save in the experience buffer. If the number of experiences in the experience buffer is less than the number of batches required for one training session, return to step 403 to continue acquiring new experiences. If the number of experiences in the experience buffer is sufficient, extract the specified number of experiences in the form of experience priority playback and execute step 407 for training.

[0163] Step 407: Perform neural network training;

[0164] According to the experience extracted in step 406, the target value of each sampling experience is calculated respectively

[0165]

[0166] The parameters of the action network are then act The gradient is given by the following loss function ξ(φ act ) to obtain the action network parameters φ act The update is done with a learning rate τ in the form of minimizing the loss.

[0167]

[0168] Then the parameters of the target network φ tar Soft update according to update rate δ.

[0169] φ tar =δφ act +(1-δ)φ tar

[0170] Among them, δ<<1, represents the target network parameter value φ tar The changes will be slow, thereby improving the stability of the neural network training process, and the training process is completed.

[0171] Step 408: Each agent performs termination detection. If any agent A in the cluster i Task candidate queue C i If it is not empty, then return to step 403 and continue the task allocation process for the next time step; if all agents have no candidate tasks, it means that the current scene has been allocated, then return to step 401 to reinitialize, and the training process ends when the specified number of training times is achieved.

[0172] Step 5: Use the trained MF-MADQN network to perform multi-UAV agent collaborative task allocation. The process is as follows: Figure 6 As shown:

[0173] Step 501: Each agent loads the trained neural network parameters and sets N t Targets are randomly injected into each agent A without duplication i Task candidate queue C i middle.

[0174] Step 502: Use the same methods as in steps 403, 404, and 405 to make action decisions, update status, and obtain the status of adjacent agents. i Discover new targets when time is shortnew When the task point information P is directly new ={x new ,y new} Join its own task candidate queue C i , and then participate in the allocation process;

[0175] Step 503: Each agent performs termination detection. If any agent A in the cluster i Task candidate queue C i If it is not empty, then return to step 403 and continue the task allocation process for the next time step; if all agents have no candidate tasks, it means that all tasks have been allocated and the algorithm ends.

[0176] In summary, the present invention designs a distributed collaborative task allocation method (MF-MADQN) based on multi-agent reinforcement learning, which adopts the form of centralized training and distributed deployment. Each agent obtains the surrounding new task information and the information of the adjacent agents through its own perception and local communication, and uses MF-MADQN to perform distributed collaborative task allocation. When a time-sensitive new target is discovered, the agent can directly include it in its own task candidate queue to participate in task allocation without the need for reallocation, and has good time-sensitive target response capabilities. In addition, since the method of the present invention embeds the mean field mechanism, the interaction between each agent and its adjacent agents is equivalent to interaction with a virtual agent, so it can cope with large-scale clusters and agent increase and decrease scenarios. At the same time, the communication requirements of this method are relatively small, the scale of the drone can be dynamically changed, and the parameters are easy to deploy. It has good engineering application value and can effectively deal with the problem of large-scale unmanned cluster collaborative task allocation in unknown environments when there are time-sensitive new targets and agent increase and decrease scenarios.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A scale-independent unmanned cluster distributed collaborative task allocation method, characterized in that: The following steps are involved: Step 1: Scene information and agent initialization: Each agent A i Correspondingly, there is a task candidate queue C i and task execution queue M i , where each agent's task candidate queue C i Save the tasks to be assigned, and the tasks are in the task execution queue M i The order in is agent A i The order in which tasks are executed; For any task t j ,j∈{1,…,N t }, whose information is P j ={x j ,y j }, indicating that the task t j If there is a time-sensitive new target, the time-sensitive new task information is P new ={x new ,y new } and include it in its own task candidate queue C i middle; Step 2: Establish UAV communication topology and local communication model; Step 3: Use the mean field mechanism to establish the state space, action space, and reward function of each agent in the task allocation process: Step 4: Train the MF-MADQN network to obtain a trained MF-MADQN network. Step 5: Use the trained MF-MADQN network to perform multi-UAV agent collaborative task allocation. For a given multi-UAV collaborative task allocation scenario, each UAV agent A i Need to execute queue M according to the assigned tasks i Starting from the starting point, strike the corresponding targets in sequence; In step 3, the reward function needs to calculate its own reward function r i and adopting joint rewards considering virtual agents Among them, each agent’s own reward function r i By maintaining rewards Receive Rewards Transfer Rewards Redundant action penalty It consists of four parts; Agent A i When performing action a i The subsequent self-reward function r i The calculation formula is as follows: r i ＝r i 0 +r i 1 +r i 2 +r i 3 ； Maintenance reward r i 0 Represents the agent A i Candidate task queue C i The reward for maintaining the decision in the task selection, receiving reward r i 1 Represents the agent A i Candidate task queue C i The reward for the task selection decision in the task selection; the transfer reward r i 2 Represents the agent A i Candidate task queue C i The reward for the task selection transfer decision in , and the redundant action penalty r i 3 For agent A i Candidate task queue C i Punishment for hollow tasks but action decisions are receive and transfer; Since the goal of multi-agent reinforcement learning in a cooperative situation is to maximize global benefits, if the agent only considers its own rewards, it will fall into selfishness. Therefore, the joint reward function is defined in the following form: β∈[0,1] represents the global reward adjustment coefficient. When β approaches 0, it means that each agent pays more attention to its own benefits and is more selfish; when β approaches 1, it means that each agent pays more attention to cooperation to improve the global benefits. i For agent A i A set of neighboring agents with which communication can be established.

2. The scale-independent unmanned cluster distributed collaborative task allocation method according to claim 1 is characterized in that: Step 2 is as follows: Assume that each drone agent A i The communication radius is Then for any UAV agent A i and A j , when the following equation is satisfied: Represents agent A i Can obtain agent A j Status information w j ; in For agent A j Current location, For agent A j The number of tasks currently to be executed, L j For agent A j Remaining distance Therefore, agent A i The obtained neighboring agent information set is expressed as satisfy Among them D i For agent A i A set of neighboring agents with which communication can be established.

3. The scale-independent unmanned cluster distributed collaborative task allocation method according to claim 1 is characterized in that: In step 3, the state space of each agent is established as follows: the state of each agent is determined by its own task candidate queue information C i And its own running status w i It consists of two parts, namely: s i =<C i ,w i > In the state space definition, the mean field mechanism is used to make the interaction between the agent and all adjacent agents equivalent to the interaction with a virtual agent, and to share the local state. i The joint state of is represented as: Among them D i For agent A i A set of neighboring agents with which communication can be established.

4. The scale-independent unmanned cluster distributed collaborative task allocation method according to claim 1 is characterized in that: In step 3, the process of establishing the action space of each agent is as follows: In the task allocation process, each agent A i It is necessary to determine the candidate sequence C of the task i For any task t j ∈C i , the agent’s decision d for this task j There are three types: 0-maintain, 1-accept and 2-transfer, i.e. d j ∈{0,1,2}; Then each agent A i The action space is: Among them, any Represents agent A i Task decision-making; Based on the mean field mechanism, we get agent A i The virtual agent action formed by all adjacent agents as follows: Among them D i For agent A i A set of neighboring agents with which communication can be established.

5. The scale-independent unmanned cluster distributed collaborative task allocation method according to claim 4 is characterized in that: If the decision is to maintain, then after this time step, task t j Still remain in Agent A i Task candidate queue C i middle; If the decision is to accept, then after this time step, task t j From Agent A i Task candidate queue C i Transfer to task execution queue M i , use the greedy algorithm to calculate and insert M one by one i The accumulated distance at each position of the queue is j Insert the position with the shortest cumulative distance; If the decision is to transfer, then after this time step, task t j From Agent A i Task candidate queue C i Transfer to the task candidate queue of any adjacent agent and wait for the decision in the next time step.

6. The scale-independent unmanned cluster distributed collaborative task allocation method according to claim 1 is characterized in that: make represents the number of tasks in the candidate task queue that need to perform maintenance actions, then the maintenance reward r i 0 The calculation formula is as follows: make Represents the set of tasks in the candidate task queue that need to perform the receiving action, then the receiving reward r i 1 The calculation formula is as follows: Where R k Represents task t k The task execution benefit is α, the distance calculation coefficient is set as dis{M i } represents agent A i Execute M in sequence i The total flight distance of the mission, Indicates the tasks to be received Follow the greedy algorithm to insert the shortest cumulative distance into M i The subsequent task sequence; make represents the number of tasks in the candidate task queue that need to perform transfer actions, then the transfer reward r i 2 The calculation formula is as follows: represents the number of empty tasks but the action decision is to receive and transfer, then the redundant action penalty r i 3 The calculation formula is as follows:

7. The scale-independent unmanned cluster distributed collaborative task allocation method according to claim 1 is characterized in that: In step 4, the training method of the MF-MADQN network specifically includes the following steps: Step 401: Initialize the agent neural network parameters; Step 402: Initialize the task allocation scenario; Step 403: Each agent makes an action decision; Each agent A i The joint state obtained based on the current time step As the input of the MF-MADQN network, the current agent A is obtained i Action a i ; Step 404: Each agent performs an action, updates the task candidate queue and the task execution queue; then uses the method of establishing the action space of each agent in step 1 to obtain its own reward r i , and generate the next state w i '; Step 405: Each agent communicates with its neighboring agents to obtain neighboring agent information; each agent A i Obtain the actions, immediate rewards, and next state of each adjacent agent, and then calculate the virtual agent actions separately Joint reward function Next joint state Step 406: Store the experience in a buffer; The experience gained by each agent in step 405 is expressed in the form of a 5-tuple. Save in the experience buffer; If the number of experiences in the experience buffer is less than the number of batches required for one training, then return to step 403 to continue acquiring new experiences. If the number of experiences in the experience buffer is sufficient, then the experience of the specified batch number is extracted in the form of experience priority playback, and step 407 is executed for training. Step 407: Perform neural network training; According to the experience extracted in step 406, the target value of each sampled experience is calculated respectively The parameters of the action network are then act The gradient is given by the following loss function ξ(φ act ) to obtain the action network parameters φ act Update with learning rate τ in the form of minimizing loss; Then the parameters of the target network φ tar Soft update according to update rate δ; f tar =df act +(1-d)φ tar ; Among them, δ<<1, represents the target network parameter value φ tar The training process is now complete with slow changes. Step 408: Each agent performs termination detection: If any agent A in the cluster i Task candidate queue C i If it is not empty, then return to step 403 and continue the task allocation process for the next time step; if all agents have no candidate tasks, it means that the current scene has been allocated, then return to step 401 to reinitialize, and the training process ends when the specified number of training times is achieved.

8. The scale-independent unmanned cluster distributed collaborative task allocation method according to claim 7 is characterized in that: In step 5, the trained MF-MADQN network is used to perform collaborative task allocation among multiple UAV agents. The specific steps include: Step 501: Each agent loads A i The trained neural network parameters are converted to N according to the needs of the scene. t Targets are randomly injected into each agent A without duplication i Task candidate queue C i Step 502: sequentially adopt the same method as in steps 403, 404, and 405 to perform action decision-making, state update, and adjacent agent state acquisition; When each agent A i Discover new targets when time is short new When the time-sensitive new task information P new ={x new ,y new } Join its own task candidate queue C i , and then participate in the allocation process; Step 503: Each agent performs termination detection: If any agent A in the cluster i Task candidate queue C i If it is not empty, then return to step 403 and continue the task assignment process for the next time step; if all agents have no candidate tasks, it means that all tasks have been assigned.

9. A scale-independent unmanned cluster distributed collaborative task allocation system that implements the scale-independent unmanned cluster distributed collaborative task allocation method according to any one of claims 1 to 8, characterized in that: Including multiple agents A i , each agent A i Correspondingly, there are task candidate queue module, task execution queue module, MF-MADQN network module, local communication module, self-perception module and task execution module; The task candidate queue module is used to store the task candidate queue C i The task execution queue module is used to store the task execution queue M i ; Each agent task candidate queue C i Save the tasks to be assigned, task execution queue M i Save A i The task to be executed, and the task execution queue M i The order in is agent A i The order in which tasks are executed; For any task t j ,j∈{1,…,N t }, whose information is P j ={x j ,y j }, indicating that the task t j location; The local communication module is used to establish the UAV communication topology and local communication model. Only UAVs within a limited communication range can communicate locally. The self-perception module is used to determine the i Call its own local sensors to analyze the time-sensitive new task information P new ={x new ,y new } and include it in its own task candidate queue C i middle; The MF-MADQN network module adopts the DQN network infrastructure, with an action network and a goal network. All agents share an experience buffer pool to store experience information. The mean field mechanism is used to establish the state space, action space, and reward function of each agent in the task allocation process. The execution module is used to use the trained MF-MADQN network to perform multi-UAV agent collaborative task allocation. For a given multi-UAV collaborative task allocation scenario, each UAV agent A i Need to execute queue M according to the assigned tasks i Starting from the starting point, strike the corresponding targets in turn.

Images