Multi-agent collaborative control method and system based on deep reinforcement learning, and medium

Abstract

This invention relates to a multi-agent cooperative control method, system, and medium based on deep reinforcement learning. The method includes: an agent acquiring and initializing its current state; inputting the current state into an Actor network; receiving a reward and the next state transitioned to from the current state from the environment; initializing an experience buffer; initializing sampling probabilities and storing the trajectory data and importance of the current action in the experience buffer; selecting a preset number of trajectories from the experience buffer based on their importance exceeding a preset threshold; updating the Actor network and the Critic network; and using the updated model to interact with the environment to obtain a trajectory. This invention addresses the problem of environmental instability caused by an increase in the number of agents, leading to difficulties in multi-agent cooperative action spaces.

Description

Technical Field

[0001] This invention relates to the field of multi-agent cooperative control methods. In particular, it relates to a multi-agent cooperative control method, system, and medium based on deep reinforcement learning. Background Technology

[0002] Intelligent robotics, as an emerging research discipline, encompasses knowledge from multiple disciplines such as robot kinematics, biosimulation, artificial intelligence, and sensor technology, playing a vital role in social development and the advancement of human life. Intelligent robots autonomously complete difficult and complex industrial tasks by acquiring and processing external information. They can also adapt to changes in the environment, establishing and revising new environmental models to perform various tasks. As an important tool for future societal development, intelligent robot technology plays a prominent role in many fields. For example, the application of intelligent robots in manufacturing can significantly increase output and has successfully promoted the development of intelligent production systems and the future of intelligent living for humanity.

[0003] With the rapid development of intelligent robots, they are indispensable in all walks of life, especially in high-risk jobs, complex working environments, and tasks that humans cannot complete, such as logistics handling and underwater exploration. In previous robot systems, relatively perfect control algorithms existed for individual robots, and with the maturity of machine vision and embedded technologies, individual robot systems have become increasingly intelligent. However, with the development of life and technology, research on collaborative control between robots has become particularly important. Simply put, the collaborative ability between robots means that each intelligent machine agent forms a large system, with multiple agents working together to perceive, make decisions, and perfectly complete a task. Therefore, compared to traditional, single-robot control algorithms, researchers are more eager to study multi-intelligent control algorithms.

[0004] Traditional robot control methods involve establishing very rigorous and precise mathematical models of the robot, calculating forward and inverse kinematic parameters based on the task, and then designing the robot's motion path. This traditional control method is computationally intensive, has limited application to a single task, and lacks versatility.

[0005] Overcoming the shortcomings of traditional multi-agent control methods, such as high computational complexity, limited task scope, and low versatility, is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a multi-agent cooperative control method, system, and medium based on deep reinforcement learning. It aims to address the environmental instability caused by the increasing number of agents, which leads to difficulties for multi-agents in joint action spaces.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a multi-agent cooperative control method based on deep reinforcement learning, for a multi-agent system comprising multiple agents, each agent being configured with an Actor network and a Critic network for generating actions and evaluating the quality of those actions, comprising:

[0008] Initialization steps: Each agent obtains its current state, inputs it into the Actor network, generates a corresponding current action, and the environment returns the reward obtained by the agent for performing the current action and the next state to which it will transition; the experience buffer pool is initialized, which is used to record the experience of all agents; the agent initializes the sampling probability representing the importance of the trajectory of the current action, and stores the trajectory data and the importance of the current action in the experience buffer pool;

[0009] Data acquisition steps: The agent selects a preset number of batches of trajectories whose importance exceeds a preset threshold from the experience buffer pool according to the importance of the experience.

[0010] Model update steps: Based on the experience data in the experience buffer pool, update the Actor network and the Critic network of each agent, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training and update of the Actor network;

[0011] Data update steps: The agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory, calculates the importance of the current trajectory, and then updates the experience buffer pool with the current trajectory and the corresponding importance.

[0012] To achieve the above objectives, in a second aspect, the present invention provides a multi-agent cooperative control system based on deep reinforcement learning, for use in a multi-agent system comprising multiple agents, each agent being configured with an Actor network and a Critic network for generating actions and evaluating the quality of those actions, comprising:

[0013] An initialization module is used for each agent to obtain and initialize its current state, input the current state into the Actor network, generate the corresponding current action, and have the environment return the reward obtained by the agent for performing the current action and the next state to be transitioned to. An experience buffer pool is initialized to record the experience of all agents. The agent initializes the sampling probability representing the importance of the trajectory of the current action and stores the trajectory data and the importance of the current action in the experience buffer pool.

[0014] Data acquisition module: used by the intelligent agent to select a preset number of batches of trajectories whose importance exceeds a preset threshold from the experience buffer pool according to the importance of experience;

[0015] Model update module: used to update the Actor network and the Critic network of each agent based on the experience data in the experience buffer pool, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training and update of the Actor network;

[0016] Data update module: This module is used by the agent to interact with the environment using the updated Actor network and Critic network to obtain a trajectory, calculate the importance of the current trajectory, and then update the experience buffer pool with the current trajectory and the corresponding importance.

[0017] To achieve the above objectives, in a third aspect, the present invention also relates to a computer-readable storage medium storing instructions that, when executed, perform the aforementioned multi-agent cooperative control method based on deep reinforcement learning.

[0018] The present invention relates to a multi-agent cooperative control method, system, and medium based on deep reinforcement learning, which has the following advantages compared to the prior art:

[0019] By designing a deterministic policy gradient algorithm, we can address the environmental instability caused by the increase in the number of agents, which leads to certain difficulties for multi-agents in the joint action space.

[0020] (1) Traditional reinforcement learning algorithms cannot adapt to dynamic environments by simply changing the policies of each agent due to the instability of the environment. Furthermore, the gradient variance increases with the number of agents, which fails to meet the convergence conditions of reinforcement learning. This invention learns the policy information of other agents while training its own action policy, and can complete tasks in a competitive and cooperative relationship.

[0021] (2) Traditional reinforcement learning works well in single-agent systems, but in multi-agent systems, due to the complexity of the environment, each agent cannot collect information from other agents, leading to the failure of training the model. This invention effectively improves the policy stability of multi-agent systems through policy integration, and this network structure is also applicable to complex environments of cooperation, competition, or a mixture of both.

[0022] (3) This invention explores the communication methods between agents in complex environments, integrates all agent strategies, improves the stability and robustness of the algorithm, and can be applied to both cooperative and competitive relationships between agents.

[0023] (4) The present invention determines the priority of each piece of empirical data based on the TD error. Each sampling uses empirical data with higher priority to update the network, thereby improving the quality of training data and thus improving the model convergence speed and the cooperative performance of the agent.

[0024] (5) In addition, a Critic network based on attention mechanism was used to learn cooperation between agents, enabling agents to dynamically select surrounding information and to a certain extent realize the scalability of the multi-agent system. Attached Figure Description

[0025] Figure 1 This is a flowchart of a multi-agent cooperative control method based on deep reinforcement learning according to Embodiment 1 of the present invention.

[0026] Figure 2 This is a schematic diagram of the MADDPG algorithm, a multi-agent cooperative control method based on deep reinforcement learning, in Embodiment 1 of the present invention.

[0027] Figure 3 This is the intrinsic motivation module of a multi-agent cooperative control method based on deep reinforcement learning in Embodiment 1 of the present invention;

[0028] Figure 4 This is a block diagram illustrating the principle of a multi-agent cooperative control method based on deep reinforcement learning, as described in Embodiment 1 of the present invention.

[0029] Figure 5 This is a schematic diagram of a multi-agent cooperative control system based on deep reinforcement learning in Embodiment 2 of the present invention. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0031] Example 1

[0032] Please refer to the following: A multi-agent cooperative control method based on deep reinforcement learning. Figures 1-2For multi-agent systems comprising multiple agents, each agent is equipped with an Actor network and a Critic network for generating actions and evaluating the quality of those actions, including: S10-S40

[0033] S10: Initialization steps: Each agent obtains the current state and inputs it into the Actor network to generate the corresponding current action. The environment returns the reward obtained by the agent for executing the current action and the next state to be transitioned to. The experience buffer pool is initialized to record the experience of all agents. The agent initializes the sampling probability representing the importance of the trajectory of the current action and stores the trajectory data and the importance of the current action in the experience buffer pool.

[0034] S20: Data acquisition step: The agent selects a preset number of batches of trajectories with an importance exceeding a preset threshold from the experience buffer pool according to the importance of the experience.

[0035] S30: Model update steps: Based on the empirical data in the experience buffer pool, update the Actor network and Critic network of each agent. The Critic network of each agent observes the situation of the entire multi-agent system and guides the training and update of the Actor network.

[0036] S40: Data update step: The agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory, calculates the importance of the current trajectory, and then updates the experience buffer pool with the current trajectory and the corresponding importance.

[0037] Preferably, before the agent acquires and initializes the current state, inputs the current state into the Actor network, and generates the corresponding current action, the process also includes:

[0038] The modeling of a deep reinforcement learning model consists of a seven-tuple <N,S,O,A,R,P,γ>, where:

[0039] N represents the set of agents, N = {1, 2, ..., n};

[0040] S represents the joint state space of the agents;

[0041] A represents the joint action space of all agents, A = {A1, A2, ..., A} n}, A n This represents the action space taken by agent n;

[0042] O represents the observation set of the agent;

[0043] R represents the reward function of the agent, r(s) t ,a t R represents the reward obtained by the agent when performing an action in the state at time t;

[0044] P represents the state transition probability function;

[0045] γ represents the discount factor, γ∈[0,1]. The smaller the value, the more important the immediate reward is; the larger the value, the more important the future reward is.

[0046] Preferably, the agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory, specifically including:

[0047] Identify a target agent from among multiple agents performing a target task;

[0048] According to the Actor network, a value function is used to obtain the target trajectory and the expected action of the target trajectory from the updated policy set among multiple agents, so as to control the target agent to execute the expected action. The expected action is to call the action corresponding to at least one policy from the policy set. The value function is related to the incentive value function of the target agent and the interaction value function of the other agents among the multiple agents other than the target agent relative to the target agent. The incentive value function of the target agent is the sum of the external incentive value function and the internal incentive value function of the target agent. The interaction value function is related to the expected difference between the action-value function of the other agents among the multiple agents after the transition and the action-value function calculated by counterfactual calculation under the condition of ignoring the state and action of the agent.

[0049] Preferably, the desired action for a target agent among multiple agents to obtain a target trajectory from an updated policy set, based on a value function of the Actor network, includes:

[0050] Determine the interaction value function of the target agent among multiple agents;

[0051] The value function of the target intelligent agent is determined based on the interaction value function and the incentive value function;

[0052] Based on the value function, obtain the desired action corresponding to a policy from the policy set, and control the target agent to execute the desired action according to the policy.

[0053] Preferably, the desired action of the target agent among multiple agents to obtain the target trajectory from the updated policy set is determined by a value function of the Actor network. Specifically, an intrinsic motivation model and a value function of the Actor network are used to obtain the target trajectory and the desired action of the target trajectory from the updated policy set for the target agent among multiple agents. The intrinsic motivation model includes three neural networks: a fixed target network that sets the prediction problem, a prediction network trained based on data collected by the agents, and a policy network. The fixed target network generates a constant output for a given state, the prediction network is used to predict the output of the target network, and the policy network determines the next action of the agent.

[0054] Preferably, the Critic network of each agent observes the situation of the entire multi-agent system, and also includes the Critic network of each agent observing the actions and states of other agents;

[0055] The Critic network guides the training of the Actor network, specifically including:

[0056] Each agent's Actor network updates its policy based on the agent's local observations and feedback from the agent's Critic network. The Critic network provides accurate deterministic joint action value estimates to guide the training and updates of the Actor network.

[0057] In this embodiment, the update of the Critic network in S30 is specifically as follows:

[0058] The MADDPG algorithm includes the following steps:

[0059] The set of policies for all agents is π. i ={π1,π2,...,π n}, and the expected return J[θ] of agent i i ] = E[R i As shown in the formula:

[0060]

[0061] Where, p μ It is the state distribution of the agent. It is a deterministic joint action value function, whose input is the joint action (a1,...,a) of all agents. N ) and joint observation x=(o1,...,o N ), output a Q value.

[0062] Consider a continuous policy μ of N agents. i This is further extended to a deterministic policy, with the policy gradient as shown in the formula:

[0063]

[0064] Where D is the experience buffer pool;

[0065] Deterministic joint action value function The update method is shown in the following formula:

[0066]

[0067] In one instance, the MADDPG algorithm may include the following steps:

[0068] Step 1: Set the number of training iterations N;

[0069] Step 2 initializes a random action sampling process;

[0070] Step 3: Initialize the state values;

[0071] Step 4: Set the number of training rounds;

[0072] Step 5: For each agent, select a random process action with added noise;

[0073] Step 6 returns the set of actions, rewards, and next state values ​​for all agents;

[0074] Step 7: Store the current state, reward, action set, and next state into the experience pool;

[0075] Step 8: Update for each agent's round;

[0076] Step 9: Randomly select the smallest batch of data from the experience pool;

[0077] Step 10: Calculate the loss function;

[0078] Step 11: Update the policy network using gradient policies;

[0079] Step 12: The agent's round ends;

[0080] Step 13: Update the target network parameters;

[0081] Step 14 ends;

[0082] Step 15 training is complete.

[0083] In this embodiment: After each agent completes the S40 data update step, it continues to execute S20-S40 sequentially according to the preset number of training rounds until the number of training rounds is reached, and then updates the target network parameters of the multi-agent system.

[0084] In this embodiment, as Figure 2As shown, the MADDPG algorithm is employed. The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is a typical centralized training-step-by-step execution method. The algorithm's principle diagram is shown below. Figure 2 As shown, each agent's Critic network can observe the entire multi-agent system, including the actions and states of other agents. Therefore, the Critic network can evaluate the impact of each agent's actions on the entire system. Through centralized training, the Critic network can provide accurate Q-value estimates to guide the training of the Actor network. Each agent's Actor network updates its policy based on its local observations and feedback from the Critic network to maximize the agent's expected reward. During the execution phase, the MADDPG algorithm only uses the Actor network with local observations to take actions. By using centralized training and step-by-step execution, the MADDPG algorithm fully utilizes global information to guide the training process, improving training efficiency and performance. Simultaneously, maintaining each agent's local observations and independent actions during the execution phase ensures the algorithm's real-time performance and scalability.

[0085] This invention also designs a multi-agent intrinsic motivation module and adds it to the MADDPG algorithm. By using curiosity reward as the agent's intrinsic reward, it compensates for the agent's insufficient exploration motivation.

[0086] The reward sparsity problem is addressed by designing an overall reward function based on curiosity and environmental rewards. The agent can learn excellent search strategies, exhibiting greater advantages in convergence speed and stability, thus improving the agent's efficiency in searching for targets.

[0087] Multi-intelligent intrinsic motivation modules such as Figure 3 As shown, it comprises three neural networks: a fixed target network that sets the prediction problem, a prediction network trained on data collected by the agent, and a policy network. The fixed target network generates a constant output for a given state, the prediction network predicts the output of the target network, and the policy network determines the agent's next action. The target network and the prediction network have the same structure, both using convolutional neural networks (CNNs) and fully connected layers for feature transformation. The feature expressions are respectively derived from f... N (t+1) and f' N (t+1) represents the prediction error between the feature encodings of the target network and the prediction network at time t+1, which is used as the agent's intrinsic curiosity reward at time t. As in the formula

[0088] The intrinsic motivation module calculates the difference between the outputs of the target network and the prediction network, enabling the agent to generate greater intrinsic rewards for unfamiliar states. This drives the agent to minimize the error between the actual state and the predicted state, effectively solving the reward sparsity problem and enabling the agent to explore the environment more quickly and thus find the target.

[0089] Multi-agent deep reinforcement learning (MADDPG) is an effective method for solving the multi-agent cooperative goal search problem. Its core idea is centralized training and distributed execution: during the training phase, the Critic network can obtain the observation states of all agents to guide the Actor network training; during the execution phase, the agents only use the Actor network to take actions. However, due to the reward sparsity problem in the multi-agent cooperative goal search process, the agents' exploration motivation is insufficient, leading to difficulties in convergence of the MADDPG model. To address this, an intrinsic motivation module is introduced based on the MADDPG algorithm, such as... Figure 4 As shown. By introducing an intrinsic motivation module, agent i can, based on the current environmental state S i (t) generates an intrinsic motivational reward for curiosity r i C (t), rewarding intrinsic curiosity r i C (t) and environmental reward r i e (t) collectively serve as the overall reward for agent i during its target search process. This approach not only increases the agent's exploration motivation and addresses the reward sparsity problem, but also prevents overexploration. Multi-agent training is a process of finding the policy that produces the maximum average expected total reward, as shown in the formula:

[0090]

[0091] p represents the state distribution of the agents, N represents the total number of agents, and π i This represents the search strategy of agent i.

[0092] The algorithm employs a centralized training and distributed execution framework. Each agent has four neural networks: an Actor network, a Critic network, a Target Actor network, and a Target Critic network. During the training phase, the Critic network obtains the joint observation state and overall reward from all agents in the environment to guide the training of the Actor network. During the execution phase, the agent only uses the Actor network to take actions.

[0093] Each agent uses its gradient parameter θ i To train its policy π i The policy gradient update formula is as follows:

[0094]

[0095] This represents the lumped action-value function, whose inputs are the actions of all agents and some state information of the agents, and whose output is the Q-value of agent i. x = (o1,...,o2) N ) is the collective observation of all agents.

[0096] The deterministic action policy π obtained for agent i i The loss function provided by the following formula is used for centralized evaluation and optimization.

[0097]

[0098] Where x is the agent's state information after being updated by the Critic network, and γ is the discount factor. and π' represents the deterministic action value of agent i and the deterministic action value after being updated by the Critic network, respectively. i o represents the updated policy of agent i. i a represents the observation state of agent i. i This represents the action taken by agent i.

[0099] To train a multi-agent system, we use the sum of two reward functions as the overall reward function. One is the environment reward r. e (t), the other is curiosity intrinsic motivation reward r C (t). The multi-agent cooperative goal search process is transformed into a Markov decision process. In each time step t, agent i will determine the target based on the current state S. i (t) Receives a total reward r i (t) and take an action a i (t). And based on the environmental change dynamics model p i (S i (t+1)|S i (t),a i (t) transforms the current state into the next state. Each agent aims to maximize its final expected reward, and the reward of agent i at time t is given by the formula...

[0100]

[0101] (x i (t),yi (t)) and (x tar (t),y tar (t) represents the positions of agent i and the target to be searched at time t, respectively.

[0102] Example 2

[0103] like Figure 5 As shown, a multi-agent cooperative control system based on deep reinforcement learning is used in a multi-agent system comprising multiple agents. Each agent is equipped with an Actor network and a Critic network for generating actions and evaluating the quality of those actions, including:

[0104] The initialization module 51 is used for each agent to obtain the current state, input the current state into the Actor network, generate the corresponding current action, and return the reward obtained by the agent for performing the current action and the next state to be transitioned to by the environment; initialize the experience buffer pool, which is used to record the experience of all agents; initialize the sampling probability representing the importance of the trajectory of the current action of the agent, and store the trajectory data of the current action and the trajectory importance of the current action in the experience buffer pool;

[0105] Data acquisition module 52: used by the agent to select a preset number of batches of trajectories whose importance exceeds a preset threshold from the experience buffer pool according to the importance of experience;

[0106] Model update module 53: used to update the Actor network and Critic network of each agent based on the empirical data in the experience buffer pool, wherein the Critic network of each agent observes the situation of the entire multi-agent system and guides the training and update of the Actor network.

[0107] Data Update Module 54: This module is used by an agent to interact with the environment using the updated Actor and Critic networks to obtain a trajectory, calculate the importance of the current trajectory, and then update the experience buffer pool with the current trajectory and its corresponding importance.

[0108] The implementation process and effects of the multi-agent cooperative control system based on deep reinforcement learning in this embodiment are the same as those of the multi-agent cooperative control method based on deep reinforcement learning described in Embodiment 1, and will not be repeated here.

[0109] Example 3

[0110] This invention relates to a computer-readable storage medium storing instructions that, when executed, produce a multi-agent cooperative control method based on deep reinforcement learning, as described in Embodiment 1. The execution process and effects are the same as those of the multi-agent cooperative control method based on deep reinforcement learning described in Embodiment 1, and will not be repeated here.

[0111] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0112] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A multi-agent cooperative control method based on deep reinforcement learning, characterized in that, For a multi-agent system comprising multiple agents, each agent is equipped with an Actor network and a Critic network for generating actions and evaluating the quality of those actions, including: Initialization steps: Each agent obtains its current state, inputs it into the Actor network, generates a corresponding current action, and the environment returns the reward obtained by the agent for performing the current action and the next state to which it will transition; the experience buffer pool is initialized, which is used to record the experience of all agents; the agent initializes the sampling probability representing the importance of the trajectory of the current action, and stores the trajectory data and the importance of the current action in the experience buffer pool; Data acquisition steps: The agent selects a preset number of batches of trajectories whose importance exceeds a preset threshold from the experience buffer pool according to the importance of the experience. Model update steps: Based on the experience data in the experience buffer pool, update the Actor network and the Critic network of each agent, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training and update of the Actor network; Data update steps: The agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory, calculates the importance of the current trajectory, and then updates the experience buffer pool with the current trajectory and the corresponding importance. The agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory. Prior to this, the process includes: using an intrinsic motivation model and a value function of the Actor network to obtain a target trajectory and the expected action of the target trajectory from the updated policy set for the target agent among the multiple agents. The intrinsic motivation model includes three neural networks: a fixed target network that sets the prediction problem, a prediction network trained based on data collected by the agent, and a policy network. The fixed target network generates a constant output for a given state, the prediction network predicts the output of the target network, and the policy network determines the agent's next action.

2. The multi-agent cooperative control method based on deep reinforcement learning according to claim 1, characterized in that, Before the agent acquires and initializes the current state, inputs the current state into the Actor network, and generates the corresponding current action, the process also includes: The modeling of the deep reinforcement learning model consists of a seven-tuple <N,S,O,A,R,P,γ>, where: N represents the set of agents, N = {1, 2, ..., n}; S represents the joint state space of the agents; A represents the joint action space of all agents, A = {A1, A2, ..., A} n }, A n This represents the action space taken by agent n; O represents the observation set of the agent; R represents the reward function of the agent, r(s) t ,a t R represents the reward obtained by the agent when performing an action in the state at time t; P represents the state transition probability function; γ represents the discount factor, γ∈[0,1]. The smaller the value, the more important the immediate reward is; the larger the value, the more important the future reward is.

3. The multi-agent cooperative control method based on deep reinforcement learning according to claim 1, characterized in that, The agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory, specifically including: Determine a target intelligent agent from among the multiple intelligent agents performing a target task; According to a value function of the Actor network, the target agent among the plurality of agents obtains the target trajectory and the expected action of the target trajectory from the updated policy set, so as to control the target agent to execute the expected action, wherein the expected action is to call the action corresponding to at least one policy from the policy set; wherein the value function is related to the incentive value function of the target agent and the interaction value function of the other agents among the plurality of agents other than the target agent relative to the target agent, wherein the incentive value function of the target agent is the sum of the extrinsic incentive value function and the intrinsic incentive value function of the target agent, and the interaction value function is related to the expected difference between the action-value function of the other agents among the plurality of agents after the transition and the action-value function calculated by counterfactual calculation under the condition of ignoring the state and action of the agent.

4. The multi-agent cooperative control method based on deep reinforcement learning according to claim 3, characterized in that, The step of obtaining the desired action of the target agent from the updated policy set based on a value function of the Actor network includes: Determine the interaction value function of the target intelligent agent among the plurality of intelligent agents; The value function of the target agent is determined based on the interaction value function and the incentive value function; Based on the value function, a desired action corresponding to a policy is obtained from the policy set, and the target agent is controlled to execute the desired action according to the policy.

5. The multi-agent cooperative control method based on deep reinforcement learning according to claim 1, characterized in that, The Critic network of each agent observes the situation of the entire multi-agent system, and also includes the Critic network of each agent observing the actions and states of the other agents; The Critic network guides the training of the Actor network, specifically including: Each agent's Actor network updates its policy based on the agent's local observations and feedback from the agent's Critic network, wherein the Critic network provides accurate deterministic joint action value estimates to guide the training and updates of the Actor network.

6. The multi-agent cooperative control method based on deep reinforcement learning according to claim 5, characterized in that... The update of the Critic network specifically involves: The set of policies for all agents is π. i ={π1,π2,...,π n }, and the expected return J[θ] of agent i i ] = E[R i As shown in the formula: Where, p μ It is the state distribution of the agent. It is a deterministic joint action value function, whose input is the joint action (a1,...,a) of all agents. N ) and joint observation x=(o1,...,o N Output a Q value; Consider a continuous policy μ of N agents. i This is further extended to a deterministic policy, with the policy gradient as shown in the formula: Wherein, D is the experience buffer pool; The deterministic joint action value function The update method is shown in the following formula:

7. The multi-agent cooperative control method based on deep reinforcement learning according to claim 1, characterized in that: After each agent completes the data update step, it continues to execute the data acquisition step, model update step, and data update step in sequence according to the preset number of training rounds until the number of training rounds is reached, and then updates the target network parameters of the multi-agent system.

8. A multi-agent cooperative control system based on deep reinforcement learning, characterized in that, For a multi-agent system comprising multiple agents, each agent is equipped with an Actor network and a Critic network for generating actions and evaluating the quality of those actions, including: An initialization module is used for each agent to obtain and initialize its current state, input the current state into the Actor network, generate the corresponding current action, and have the environment return the reward obtained by the agent for performing the current action and the next state to be transitioned to. An experience buffer pool is initialized to record the experience of all agents. The agent initializes the sampling probability representing the importance of the trajectory of the current action and stores the trajectory data and the importance of the current action in the experience buffer pool. Data acquisition module: used by the intelligent agent to select a preset number of batches of trajectories whose importance exceeds a preset threshold from the experience buffer pool according to the importance of experience; Model update module: used to update the Actor network and the Critic network of each agent based on the experience data in the experience buffer pool, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training and update of the Actor network; Data update module: used by the agent to interact with the environment using the updated Actor network and Critic network to obtain a trajectory, calculate the importance of the current trajectory, and then update the experience buffer pool with the current trajectory and the corresponding importance. The agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory. Prior to this, the process includes: using an intrinsic motivation model and a value function of the Actor network to obtain a target trajectory and the expected action of the target trajectory from the updated policy set for the target agent among the multiple agents. The intrinsic motivation model includes three neural networks: a fixed target network that sets the prediction problem, a prediction network trained based on data collected by the agent, and a policy network. The fixed target network generates a constant output for a given state, the prediction network predicts the output of the target network, and the policy network determines the agent's next action.

9. A computer-readable storage medium, characterized in that: The storage medium stores instructions that, when executed, perform a multi-agent cooperative control method based on deep reinforcement learning as described in any one of claims 1-7.

Images