Multi-agent remote machine autonomous collaboration method and system

Abstract

The invention discloses a multi-agent remote autonomous collaboration method and system, and the method comprises the steps: initializing an environment based on an agent cluster, a dynamic target and region, and a task between an agent and the target; constructing a reinforcement learning action space based on the initialized environment; building an agent action network for calculating a local Qa value based on the reinforcement learning action space; building a Mixer network for calculating a global Qtot value based on the reinforcement learning action space; constructing action logic of red and blue parties, calculating a local Qa value through an agent action network, and calculating a global Qtot value through a Mixer network based on the local Qa value; determining the next action based on the global Qtot value; repeating the steps to determine the next action, and completing task allocation and execution. According to the method, it can be ensured that multiple agents autonomously, cooperatively and preferentially go to the nearest task target point, and meanwhile, the method is not affected by target dynamic change and rapid position adjustment.

Description

Technical Field

[0001] The present invention belongs to the field of multi-agent cluster control, and specifically relates to a multi-agent ad hoc autonomous collaboration method and system. Background Art

[0002] In the process of multi-agent collaborative tasks, task allocation is a key technology that determines how the agents respond to the appearance and position changes of dynamic targets.

[0003] Traditional target allocation algorithms are usually designed for static targets or task allocation based on prior information. However, in modern complex environments, targets may change dynamically, adjust positions quickly, or become obstacles to the agent's task execution. In these cases, traditional target allocation algorithms may not achieve optimal task allocation and execution. Current target allocation algorithms usually focus on static targets, which rely on prior target information and static allocation strategies. These methods may perform poorly when dealing with dynamic targets because they cannot effectively adapt to changing situations. The target allocation algorithms in the prior art usually adopt the following methods:

[0004] Nearest neighbor assignment algorithm: One of the traditional target assignment methods, the agent selects the target closest to it for assignment. This method is simple and intuitive, but may lead to uneven distribution of tasks.

[0005] Weighted allocation algorithms: Some existing techniques introduce the concepts of target weights and task priorities to consider the relative importance of tasks.

[0006] Max-min assignment algorithm: This algorithm aims to ensure even distribution of tasks, but still faces the challenge of dynamic target allocation.

[0007] In recent years, the development trend in the field of technology shows that task allocation algorithms need to better adapt to dynamic goals and autonomous collaborative scenarios. Emerging technologies and methods, such as reinforcement learning, game theory, mixed integer linear programming, etc., are gradually being introduced into the field of intelligent agent task allocation to cope with more challenging situations. These trends emphasize the following directions:

[0008] Adaptability and intelligence: The task allocation algorithm needs to be more adaptable and be able to adjust the task allocation strategy in real time according to the emergence and behavior changes of dynamic targets.

[0009] Ad hoc task capability: Considering the ad hoc task environment, the algorithm needs stronger ad hoc response capability to ensure the effectiveness and reliability of task allocation and execution.

[0010] Big data and real-time: Technologies for processing large amounts of data and sensor information need to continue to advance to support real-time task allocation decisions.

[0011] With the rapid development of intelligent technology, the market demand for intelligent agents is also increasing. Multi-agent cluster systems have broad application prospects in the fields of smart home, fire rescue, intelligent manufacturing, etc. In order to meet the growing market demand, it is necessary to provide a more efficient, safer and more powerful multi-agent cluster solution. Based on this, the present invention proposes a multi-agent autonomous collaboration method. Summary of the invention

[0012] The purpose of the present invention is to provide a multi-agent ad hoc autonomous collaboration method and system to achieve dynamic target allocation and task execution of the agents, which is suitable for ad hoc dynamic environments.

[0013] The technical solution for achieving the purpose of the present invention is:

[0014] A multi-agent ad hoc autonomous collaboration method, comprising:

[0015] Step 1, initialize the environment based on the agent cluster, dynamic targets and regions, and the tasks between agents and targets;

[0016] Step 2: construct a reinforcement learning action space based on the initialized environment;

[0017] Step 3: Based on the reinforcement learning action space, build a local Q calculation a The agent action network of the value;

[0018] Step 4: Build a global Q calculation based on the reinforcement learning action space tot Mixer network of values;

[0019] Step 5: Construct the action logic for both the red and blue parties and calculate the local Q through the agent action network a Value, based on local Q a Value, calculate the global Q through the Mixer network tot value;

[0020] Step 6: Based on the global Q tot The value determines the next action;

[0021] Step 7: Repeat steps 5-6 to complete task allocation and execution.

[0022] Furthermore, the step 1 specifically includes: first constructing the distribution of agents and targets, denoting the number of agents as n_agents, the number of targets as n_opponents, the distance between the agents and the center point as view_radius, the reward obtained by the agents each time they go to the target point and perform a task as kill_reward, and the width of the grid as grid_width; according to the grid width, constructing a reinforcement learning grid, the grid size is grid_width*grid_width; in the grid, the agents are distributed with a random center point, randomly distributed in a square with a side length of view_radius, and the target area is randomly distributed in the entire grid.

[0023] Furthermore, the reinforcement learning action space includes the movement actions of the agent itself and the actions for completing specific tasks, wherein the movement actions of the agent itself include five actions: moving up, moving down, moving left, moving right and staying still. The number of actions for performing specific tasks is consistent with the number of specific tasks, and each specific action represents the completion of a specific task.

[0024] Furthermore, the agent action network adopts a GRU structure.

[0025] Furthermore, the agent action network includes a cascaded first fully connected layer, a recurrent neural network, and a second fully connected layer, wherein:

[0026] The input dimension of the first fully connected layer is the dimension of the observation value of the reinforcement learning action space, and the output dimension is the hidden layer dimension;

[0027] The recurrent neural network includes multiple GRU units;

[0028] The input of the second fully connected layer is the output of the recurrent neural network, the dimension is the hidden layer dimension, and the output is the local Q of the action space a Value, dimension is the dimension of action space.

[0029] Furthermore, the reset gate and update gate of the GRU unit are:

[0030] z t =sigmoid(W z x t +U z h t-1 )

[0031] r t =sigmoid(W r x t +U r h t-1 )

[0032]

[0033] Among them, z t Update the gate weight parameters for GRU, r t Reset the gate weight parameters for GRU, h t is the hidden layer weight of the recurrent neural network at time step t, is the candidate hidden layer weight, W z , U z , W r , U r is the learnable linear transformation parameter, x t is the output of the first fully connected layer, sigmoid and tanh are activation functions.

[0034] Furthermore, the Mixer network includes a weight matrix module, a bias neural network and an output neural network, the weight matrix module includes two dense neural networks to generate two weight matrices W1 and W2; the bias neural network is a dense neural network; the output neural network includes two dense neural networks.

[0035] Furthermore, the loss function of the Mixer network update is:

[0036] loss = TDError = Q tot (evaluate)-(r+γQ tot (target))

[0037] The Mixer network receives the Q value of the behavior selected by each Agent network in state s as input, and outputs Q tot (evaluate), the Mixer network takes the Q value of the behavior selected by each Agent network in state s' as input, and outputs Q tot (target).

[0038] Furthermore, the ε-greedy strategy is adopted to determine the next action.

[0039] A multi-agent ad hoc autonomous collaborative system, comprising:

[0040] The environment initialization unit initializes the environment based on the agent cluster, dynamic targets and regions, and the tasks between the agents and the targets;

[0041] A reinforcement learning action space construction unit, which constructs the reinforcement learning action space based on the initialized environment;

[0042] Agent action network building unit, based on reinforcement learning action space, builds calculation local Q aThe agent action network of the value;

[0043] Mixer network building unit, based on reinforcement learning action space, builds a global Q calculation tot Mixer network of values;

[0044] Local Q a Value and Global Q tot The value calculation unit builds the action logic of the red and blue parties and calculates the local Q through the agent action network a Value, based on local Q a Value, calculate the global Q through the Mixer network tot value

[0045] The next action determination unit is based on the global Q tot The value determines the next action.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention introduces the QMix reinforcement learning algorithm, builds an agent action network and a Mixer network, so that a multi-agent cluster can perceive and respond to the appearance and position changes of dynamic targets in real time, so as to achieve a smarter and more adaptable task allocation and execution strategy; the present invention can ensure that multiple agents autonomously and collaboratively go to the task target point, and at the same time, are not affected by the dynamic changes of the target and the rapid adjustment of the position, and is particularly suitable for dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the agent action network.

[0048] Figure 2 This is a diagram of the Mixer network. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The implementation of the multi-agent autonomous collaborative system mainly includes the following steps:

[0051] (1) Initialize the environment. In the process of multi-agent autonomous collaborative execution of tasks, agent clusters, dynamic targets and regions are involved. There are many factors between agents and targets that affect the development of tasks. First, construct the distribution of agents and targets. The number of agents is n_agents, the number of targets is n_opponents, the distance between the agent and the center point is view_radius, the reward the agent gets each time it goes to the target point and performs a task is kill_reward, and the width of the grid is grid_width. According to the grid width, a reinforcement learning grid is constructed with a grid size of grid_width*grid_width; in the grid, the agents are distributed with a random center point, randomly distributed in a square with a side length of view_radius, and the target area is randomly distributed in the entire grid.

[0052] (2) Constructing the action space of reinforcement learning. The action space of the agent includes the movement of the agent itself and the actions to complete specific tasks; the movement of the agent itself includes five actions: moving up, moving down, moving left, moving right, and staying still. The number of actions to perform tasks is the same as the number of tasks, and each action represents the completion of a specific task.

[0053] (3) Build the agent action network. The agent action network consists of three parts: fully connected layer-recurrent neural network-fully connected layer; the input dimension of the first fully connected layer MLP is the dimension of the observation value of the reinforcement learning environment, and the output dimension is the hidden layer dimension agent_hidden_dim. The recurrent neural network part consists of multiple GRU units. The reset gate and update gate formulas of GRU are as follows, where is the observation value of the agent at the current moment, is the action taken by the agent at the last moment, x t for and The output obtained by the first fully connected layer MLP is W z , U z , W r , U r is the learnable linear transformation parameter, z t Update the gate weight parameters for GRU, r t Reset the gate weight parameters for GRU, h t is the hidden layer weight of the recurrent neural network at time step t, are the alternative hidden layer weights, sigmoid and tanh are the activation functions:

[0054] z t =sigmoid(W z xt +U z h t-1 )

[0055] r t =sigmoid(W r x t +U r h t-1 )

[0056]

[0057] The input dimension of the third fully connected layer is the hidden layer dimension agent_hidden_dim, and the output dimension is the dimension of the action space.

[0058] (4) Build a Mixer network, such as Figure 2 As shown. The Mixer network consists of four parts, including a neural network that generates weight matrices W1 and W2, a bias b1, and an output neural network. The weight generation network consists of two parts, including two dense neural networks. The input dimension of the first dense neural network is the product of the dimension of Q-value and the number of agents, and the output dimension is the embedding dimension mixer_embed_dim of the Mixer network. The input dimension of the second dense neural network is the output dimension of the first dense neural network, and the output dimension is the hidden layer dimension mixer_hidden_dim of the Mixer network; the bias network consists of a dense neural network, with an input dimension of the Q-value dimension and the number of agents, and an output dimension of the hidden layer dimension mixer_hidden_dim of the Mixer network; the output network consists of two dense neural networks. The input dimension of the first dense neural network is the product of the dimension of Q-value and the number of agents, and the output dimension is the hidden layer dimension mixer_hidden_dim of the Mixer network. The input dimension of the second dense neural network is the hidden layer dimension mixer_hidden_dim of the Mixer network, and the output dimension is 1.

[0059] (5) Construct the action logic of the red and blue parties. The red party is the agent that performs the task, and the blue party is the task target. The red party is controlled by the actions generated by the reinforcement learning algorithm; the blue party moves randomly in the environment; when the blue party appears within a grid of the red party, the red party will perform the task action.

[0060] (6) Calculate local Q aSample the red team's action and the reward reward obtained in the environment after taking the corresponding action, concatenate the unique hot vector of the agent ID, the agent position, the unique hot vector of the task target ID, and the target position and input them into the Agent network; the Agent network calculates the local Q of each agent a .

[0061] (7) Calculate the global Q tot . The Q output of each Agent network a Spliced ​​into a one-bit vector, input into the Mixer network, and get the global Q tot . Use the following formula to calculate the loss and update the network:

[0062] loss = TDError = Q tot (evaluate)-(r+γQ tot (target))

[0063] Where Q tot (evaluate) The network receives the Q value of each agent network's selected behavior in state s as input and outputs Q tot (evaluate). Q tot The (target) network receives the Q value of the behavior selected by each Agent network in state s' as input and outputs Q tot (target).

[0064] (8) According to the global Q value Q tot Sample the next action. The strategy for sampling actions is ε-greedy.

[0065] A * ←argmax a (Q tot ,a)

[0066] π(a|s)←1-∈+∈ / ∣A∣ifa=A *

[0067] π(a|s)←∈ / |A|if a≠A *

[0068] Among them, A * is the optimal action, |A| is the number of all actions, ∈ is the probability of randomly selecting an unknown action, π(a|s) is the action selection strategy, a is the currently selected action, and argmax is the method of finding the independent variable when maximizing the function value.

[0069] A multi-agent ad hoc autonomous collaborative system, comprising:

[0070] The environment initialization unit initializes the environment based on the agent cluster, dynamic targets and regions, and the tasks between the agents and the targets;

[0071] Agent network building unit. The Agent network adopts the Gated Recurrent Unit (GRU) structure to realize the sequence modeling and state memory of the agent. The parameters and structure of the GRU network allow the agent to better capture time series information in the game confrontation environment.

[0072] Mixer network building unit, the Mixer network integrates the local Q values ​​of multiple agents to generate the global Q value of v; based on the global Q value, the Mixer network can coordinate the decisions of multiple drones to maximize the global Q value. The structure and parameters of the Mixer network support knowledge sharing and collaboration among agents;

[0073] Local Q a Value and Global Q tot The value calculation unit builds the action logic of the red and blue parties and calculates the local Q through the agent action network a Value, based on local Q a Value, calculate the global Q through the Mixer network tot value;

[0074] The next action determination unit is based on the global Q tot The value determines the next action.

[0075] The system may also include multiple technical modules such as distributed global key data synchronization, environmental perception sharing system, communication protocol, data processing, task planning and path planning to support better operation of the entire system and realize multiple functions such as dynamic target allocation, balanced task allocation, path planning, and on-the-spot task situation decision-making. Since distributed global key data synchronization, environmental perception sharing system, communication protocol, data processing, task planning and path planning adopt existing technologies, they will not be repeated here.

[0076] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A multi-agent autonomous collaboration method, characterized in that: include: Step 1, initialize the environment based on the agent cluster, dynamic targets and regions, and the tasks between agents and targets; Step 2: construct a reinforcement learning action space based on the initialized environment; Step 3: Based on the reinforcement learning action space, build a local Q calculation a The agent action network of the value; Step 4: Build a global Q calculation based on the reinforcement learning action space tot Mixer network of values; Step 5: Construct the action logic for both the red and blue parties and calculate the local Q through the agent action network a Value, based on local Q a Value, calculate the global Q through the Mixer network tot value; Step 6: Based on the global Q tot The value determines the next action; Step 7: Repeat steps 5-6 to complete task allocation and execution.

2. A multi-agent autonomous collaboration method according to claim 1, characterized in that: The step 1 specifically includes: first constructing the distribution of agents and targets, where the number of agents is n_agents, the number of targets is n_opponents, the distance between the agent and the center point is view_radius, the reward obtained by the agent each time it goes to the target point and performs a task is kill_reward, and the width of the grid is grid_width; according to the grid width, constructing a reinforcement learning grid, the grid size is grid_width*grid_width; in the grid, the agents are distributed with a random center point, randomly distributed in a square with a side length of view_radius, and the target area is randomly distributed in the entire grid.

3. The multi-agent autonomous collaboration method according to claim 1, characterized in that: The reinforcement learning action space includes the movement actions of the agent itself and the actions to complete specific tasks, wherein the movement actions of the agent itself include five actions of moving up, moving down, moving left, moving right and staying still. The number of actions to perform specific tasks is consistent with the number of specific tasks, and each specific action represents the completion of a specific task.

4. A multi-agent autonomous collaboration method according to claim 1, characterized in that: The agent action network adopts a GRU structure.

5. A multi-agent spontaneous autonomous collaboration method according to claim 4, characterized in that: The agent action network includes a cascaded first fully connected layer, a recurrent neural network, and a second fully connected layer, wherein: The input dimension of the first fully connected layer is the dimension of the observation value of the reinforcement learning action space, and the output dimension is the hidden layer dimension; The recurrent neural network includes multiple GRU units; The input of the second fully connected layer is the output of the recurrent neural network, the dimension is the hidden layer dimension, and the output is the local Q of the action space a Value, dimension is the dimension of action space.

6. A multi-agent spontaneous autonomous collaboration method according to claim 5, characterized in that: The reset gate and update gate of the GRU unit are: z t =sigmoid(W z x t +U z h t-1 ) r t =sigmoid(W r x t +U r h t-1 ) Among them, z t Update the gate weight parameters for GRU, r t Reset the gate weight parameters for GRU, h t is the hidden layer weight of the recurrent neural network at time step t, is the candidate hidden layer weight, W z , U z , W r , U r is the learnable linear transformation parameter, x t is the output of the first fully connected layer, sigmoid and tanh are activation functions.

7. The multi-agent autonomous collaboration method according to claim 1, characterized in that: The Mixer network includes a weight matrix module, a bias neural network and an output neural network. The weight matrix module includes two dense neural networks to generate two weight matrices W1 and W2; the bias neural network is a dense neural network; and the output neural network includes two dense neural networks.

8. A multi-agent spontaneous autonomous collaboration method according to claim 7, characterized in that: The loss function of the Mixer network update is: loss=TDError=Q tot (evaluate)-(r+γQ tot (target)) The Mixer network receives the Q value of the behavior selected by each Agent network in state s as input, and outputs Q tot (evaluate), the Mixer network takes the Q value of the behavior selected by each Agent network in state s' as input, and outputs Q tot (target).

9. The multi-agent autonomous collaboration method according to claim 1, characterized in that: The ε-greedy strategy is used to determine the next action.

10. A multi-agent autonomous collaborative system for implementing any of the methods of claims 1-9, characterized in that: include: The environment initialization unit initializes the environment based on the agent cluster, dynamic targets and regions, and the tasks between the agents and the targets; A reinforcement learning action space construction unit, which constructs the reinforcement learning action space based on the initialized environment; Agent action network building unit, based on reinforcement learning action space, builds calculation local Q a The agent action network of the value; Mixer network building unit, based on reinforcement learning action space, builds a global Q calculation tot Mixer network of values; Local Q a Value and Global Q tot The value calculation unit builds the action logic of the red and blue parties and calculates the local Q through the agent action network a Value, based on local Q a Value, calculate the global Q through the Mixer network tot value The next action determination unit is based on the global Q tot The value determines the next action.

Images