Multi-agent cooperative control method and system based on deep reinforcement learning, and medium

Abstract

The invention relates to a multi-agent cooperative control method and system based on deep reinforcement learning and a medium, the multi-agent cooperative control method based on deep reinforcement learning comprises the following steps: an agent obtains an initialized current state, inputs the current state into an Actor network, and initializes the current state; the environment returns the reward obtained by executing the current action in the current state and the transferred next state to the agent; initializing an experience buffer pool; the intelligent agent initializes the sampling probability and stores the trajectory data of the current action and the trajectory importance degree of the current action into an experience buffer pool; the intelligent agent selects a preset number of batches of tracks with the importance degree exceeding a preset threshold value from the experience buffer pool according to the importance degree of the experience; the Actor network and the Critic network are updated; and interacting with the environment by utilizing the updated model to obtain a track. According to the invention, the problem that the multi-agent has certain difficulty in the joint action space due to the unstable environment caused by the increase of the number of the agents is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent collaborative control methods, and in particular to a multi-agent collaborative control method, system and medium based on deep reinforcement learning. Background Art

[0002] As an emerging research discipline, intelligent robots cover a variety of disciplines such as robot kinematics, biosimulation, artificial intelligence technology, sensor technology, etc., and play an important role in the development of society and the progress of human life. Intelligent robots can autonomously complete difficult and tedious industrial tasks by acquiring and processing external information. At the same time, they can also establish and modify new environmental models to complete various tasks as the environment changes. As an important tool for the future development of society, intelligent robot technology plays a prominent role in many fields. For example, the application of intelligent robots in the manufacturing field can efficiently increase production, and successfully promote the development of intelligent production systems and the intelligent life of humans in the future.

[0003] With the rapid development of intelligent robots, all walks of life cannot do without intelligent robots, especially in complex working environments such as logistics handling and seabed exploration, as well as tasks that humans cannot complete. In previous robot systems, there have been relatively perfect control algorithms for single robot control. At the same time, machine vision and embedded technologies have matured, and single robot systems have become more and more intelligent. However, with the development of life and technology, research on collaborative control between robots is particularly important. Simply put, the collaborative ability between robots means that each machine agent forms a large system, and multiple agents work together to perceive, make decisions, and perfectly complete a task. Therefore, compared with traditional, single robot control algorithms, researchers are more eager to study multi-intelligent control algorithms.

[0004] In the past, traditional robot control methods all established very strict and precise mathematical models for robots, calculated the forward and inverse kinematic parameters according to the work tasks, and designed the robot's movement path. This traditional control method has a very large amount of calculation, and the task is single and not very versatile.

[0005] How to overcome the shortcomings of traditional multi-agent control methods, such as large amount of computation, single task and low versatility, is an urgent problem to be solved. Summary of the invention

[0006] The present invention provides a multi-agent cooperative control method, system and medium based on deep reinforcement learning to solve the problem of environmental instability caused by the increase in the number of agents, which leads to certain difficulties for multiple agents in the joint action space.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a multi-agent collaborative control method based on deep reinforcement learning, which is used for a multi-agent system including multiple agents, each of which is provided with an Actor network and a Critic network for generating actions and evaluating the quality of the actions, including:

[0008] Initialization step: each agent obtains the initialized current state, inputs the current state into the Actor network, generates the corresponding current action, and the environment returns the reward obtained by executing the current action and the next state to be transferred to the agent in the current state; initializes the experience buffer pool, which is used to record the experience of all agents; the agent initializes the sampling probability representing the importance of the trajectory of the current action, and stores the trajectory data of the current action and the trajectory importance of the current action in the experience buffer pool;

[0009] Data collection step: the agent selects a preset number of batches of trajectories with an importance exceeding a preset threshold from the experience buffer pool according to the importance of the experience;

[0010] Model updating step: based on the experience data in the experience buffer pool, updating the Actor network and the Critic network of each agent, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training update of the Actor network;

[0011] Data updating step: the agent uses the updated Actor network and the Critic network to interact with the environment to obtain a trajectory, calculates the importance of the current trajectory, and then updates the experience buffer pool with the current trajectory and the corresponding importance.

[0012] To achieve the above object, in a second aspect, the present invention provides a multi-agent collaborative control system based on deep reinforcement learning, which is used for a multi-agent system including multiple agents, each of which is provided with an Actor network and a Critic network for generating actions and evaluating the quality of the actions, including:

[0013] The initialization module is used for each agent to obtain the initialization current state, input the current state into the Actor network, generate the corresponding current action, and the environment returns the reward obtained by the agent for executing the current action and the next state to be transferred to the current state; initialize the experience buffer pool, which is used to record the experience of all agents; the agent initializes the sampling probability representing the importance of the trajectory of the current action, and stores the trajectory data of the current action and the trajectory importance of the current action in the experience buffer pool;

[0014] Data collection module: used for the agent to select a preset number of batches of trajectories with an importance exceeding a preset threshold from the experience buffer pool according to the importance of the experience;

[0015] Model updating module: used for updating the Actor network and the Critic network of each agent based on the experience data in the experience buffer pool, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training update of the Actor network;

[0016] Data update module: used for the intelligent agent to interact with the environment using the updated Actor network and the Critic network to obtain a trajectory, calculate the importance of the current trajectory, and then update the experience buffer pool with the current trajectory and the corresponding importance.

[0017] To achieve the above objectives, in a third aspect, the present invention also relates to a computer-readable storage medium, in which instructions are stored, and when the instructions are run, the above-mentioned multi-agent collaborative control method based on deep reinforcement learning is executed.

[0018] The present invention relates to a multi-agent collaborative control method, system and medium based on deep reinforcement learning, which has the following beneficial effects compared with the prior art:

[0019] A deterministic policy gradient algorithm is designed to solve the environmental instability caused by the increase in the number of agents, which leads to certain difficulties for multiple agents in the joint action space.

[0020] (1) For traditional reinforcement learning algorithms, due to the instability of the environment, it is impossible to adapt to the dynamic environment by simply changing the strategy of each intelligent agent, and the strategy gradient will increase with the increase in the number of intelligent agents, resulting in failure to meet the convergence conditions of reinforcement learning. While training and learning its own action strategy, the present invention also learns the strategy information of other intelligent agents and can complete tasks in a competitive and cooperative relationship.

[0021] (2) Traditional reinforcement learning works well in the field of single-agent systems, but in the field of multiple agents, due to the complexity of the environment, each agent cannot collect information from other agents, resulting in the failure of the training model. The present invention effectively improves the strategy stability of the multi-agent system through strategy integration, and this network structure is also applicable to complex environments of cooperation, competition, or a mixture of the two.

[0022] (3) The present invention explores the communication mode between intelligent agents in a complex environment, integrates all intelligent agent strategies, and improves the stability and robustness of the algorithm. At the same time, it can be applied to both the cooperative relationship between intelligent agents and the competitive relationship between intelligent agents.

[0023] (4) The present invention calculates the priority of each piece of experience data based on the TD error, and uses experience data with higher priority to update the network each time it samples, thereby improving the quality of training data and thus improving the model convergence speed and the cooperative performance of the intelligent agent.

[0024] (5) In addition, a critic network based on the attention mechanism is used to learn the cooperation between agents, so that the agents can dynamically select surrounding information and achieve the scalability of the multi-agent system to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a method flow chart of a multi-agent collaborative control method based on deep reinforcement learning in Embodiment 1 of the present invention;

[0026] Figure 2 This is a schematic diagram of the MADDPG algorithm of a multi-agent collaborative control method based on deep reinforcement learning in Example 1 of the present invention;

[0027] Figure 3 It is an intrinsic motivation module of a multi-agent collaborative control method based on deep reinforcement learning in embodiment 1 of the present invention;

[0028] Figure 4 It is a principle block diagram of a collaborative control method of a multi-agent collaborative control method based on deep reinforcement learning in Embodiment 1 of the present invention;

[0029] Figure 5 It is a structural diagram of a multi-agent collaborative control system based on deep reinforcement learning in Example 2 of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0031] Embodiment 1

[0032] A multi-agent collaborative control method based on deep reinforcement learning, see Figure 1-Figure 2, used for multi-agent systems including multiple agents, each agent sets up an Actor network and a Critic network to generate actions and evaluate the quality of actions, including: S10-S40

[0033] S10: Initialization step: Each agent obtains the initialized current state, inputs the current state into the Actor network, generates the corresponding current action, and the environment returns the reward obtained by the agent for executing the current action in the current state and the next state to be transferred to; initializes the experience buffer pool, which is used to record the experience of all agents; the agent initializes the sampling probability representing the importance of the trajectory of the current action, and stores the trajectory data of the current action and the trajectory importance of the current action in the experience buffer pool;

[0034] S20: Data collection step: The agent selects a preset number of batches of trajectories whose importance exceeds a preset threshold from the experience buffer pool according to the importance of the experience;

[0035] S30: Model update step: based on the experience data in the experience buffer pool, update the Actor network and Critic network of each agent, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training update of the Actor network;

[0036] S40: Data update step: The agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory, calculates the importance of the current trajectory, and then updates the experience buffer pool with the current trajectory and the corresponding importance.

[0037] Preferably, before the agent obtains and initializes the current state, inputs the current state into the Actor network, and generates the corresponding current action, it also includes:

[0038] The modeling of the deep reinforcement learning model is composed of a seven-tuple <N, S, O, A, R, P, γ>, where:

[0039] N represents the set of agents, N = {1, 2, ... n};

[0040] S represents the joint state space of the agents;

[0041] A represents the joint action space of all agents, A={A1,A2,...,A n}, A n represents the action space taken by agent n;

[0042] O represents the observation set of the agent;

[0043] R represents the agent’s reward function, r(s t ,a t )∈R represents the reward obtained by the agent for performing an action in the state at time t;

[0044] P represents the state transition probability function;

[0045] γ represents the discount factor, γ∈[0,1]. The smaller the value, the more emphasis is placed on immediate rewards, and the larger the value, the more emphasis is placed on future rewards.

[0046] Preferably, the agent uses the updated Actor network and Critic network to interact with the environment to obtain a trajectory, specifically including:

[0047] Determine a target agent from a plurality of agents that perform a target task;

[0048] According to a value function of the Actor network, a target agent among multiple agents obtains a target trajectory and an expected action of the target trajectory from an updated strategy set to control the target agent to perform the expected action, wherein the expected action is an action corresponding to at least one strategy called from the strategy set; wherein the value function is related to the incentive value function of the corresponding target agent, and the interaction value function of other agents among the multiple agents except the target agent relative to the target agent, wherein the incentive value function of the target agent is the sum of the external incentive value function and the internal incentive value function of the target agent, and the interaction value function is related to the expected difference between the action-value function of the other agents among the multiple agents except the target agent after the transfer and the action-value function obtained by counterfactually calculating under the condition of ignoring the state and action of the agent.

[0049] Preferably, obtaining a desired action of a target trajectory for a target agent among the multiple agents from an updated strategy set according to a value function of the Actor network includes:

[0050] Determine the interaction value function of the target agent among multiple agents;

[0051] Determine the value function of the target agent based on the interaction value function and the incentive value function;

[0052] According to the value function, a strategy corresponding to the expected action is obtained from the strategy set, and the target agent is controlled to perform the expected action according to the strategy.

[0053] Preferably, a target agent among multiple agents obtains an expected action of a target trajectory from an updated strategy set based on a value function of an Actor network, specifically: an intrinsic motivation model and a value function of an Actor network are used to obtain a target trajectory and an expected action of the target trajectory from an updated strategy set for a target agent among multiple agents, wherein the intrinsic motivation model includes three neural networks: one is a fixed target network for setting a prediction problem, one is a prediction network trained based on data collected by the agent, and the other is a policy network, wherein the fixed target network generates a constant output for a given state, the prediction network is used to predict the output of the target network, and the policy network determines the next action of the agent.

[0054] Preferably, the Critic network of each agent observes the situation of the entire multi-agent system, and also includes that the Critic network of each agent observes the actions and states of other agents;

[0055] The Critic network guides the training of the Actor network, including:

[0056] The Actor network of each agent updates its strategy based on the local observations of the agent and the feedback from the agent's Critic network, where the Critic network provides accurate and deterministic joint action value estimates to guide the training updates of the Actor network.

[0057] In this embodiment, in S30: the updating of the Critic network is specifically as follows:

[0058] The MADDPG algorithm consists of the following steps:

[0059] The strategy set of all agents is π i ={π1,π2,...,π n}, and the expected return J[θ i ]=E[R i ], as shown in the formula:

[0060]

[0061] Among them, p μ is the state distribution of the agent, is a deterministic joint action-value function whose input is the joint action of all agents (a1,...,a N ) and the joint observation x=(o1,...,o N ), output a Q value.

[0062] Consider N agents with a continuous strategy μ i , and expand it into a deterministic strategy, the policy gradient is shown in the formula:

[0063]

[0064] Among them, D is the experience buffer pool;

[0065] Deterministic Joint Action-Value Function The update method is shown in the following formula:

[0066]

[0067] In one example, the MADDPG algorithm may include the following steps:

[0068] Step 1: Set the number of loop training times N;

[0069] Step 2 initializes a random action sampling process;

[0070] Step 3: Initialize the state value;

[0071] Step 4: Set the number of training rounds;

[0072] Step 5: For each agent, select a random process action with added noise;

[0073] Step 6 returns the action set, reward, and next state value of all agents;

[0074] Step 7 stores the current state, reward, action set, and next state into the experience pool;

[0075] Step 8 is updated for each agent round;

[0076] Step 9 randomly selects the minimum batch data from the experience pool;

[0077] Step 10: Calculate the loss function;

[0078] Step 11: Gradient strategy updates the strategy network;

[0079] Step 12: The agent round ends;

[0080] Step 13: Update target network parameters;

[0081] Step 14: End of round;

[0082] Step 15: Training is complete.

[0083] In this embodiment: after each agent completes the data update step S40, S20-S40 are continuously executed in a loop according to a preset number of training rounds until the number of training rounds is reached, and then the target network parameters of the multi-agent system are updated.

[0084] In this embodiment, if Figure 2As shown in the figure, the MADDPG algorithm is used. The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is a typical centralized training-step-by-step execution method. The algorithm principle diagram is shown in Figure 2 As shown in the figure, the Critic network of each agent can observe the situation of the entire multi-agent system, including the action information, state information, etc. of other agents. Therefore, the Critic network can evaluate the impact of the actions taken by each agent on the entire multi-agent system. By using centralized training, the Critic network can provide accurate Q-value estimates to guide the training of the Actor network. The Actor network of each agent updates its strategy based on its own local observations and the feedback from the Critic network to maximize the expected reward return of the agent. In the execution phase, the MADDPG algorithm only uses the Actor network with local observations to take actions. By using centralized training and step-by-step execution, the MADDPG algorithm makes full use of global information to guide the training process, improving the efficiency and performance of training. At the same time, the local observations and independent actions of each agent are maintained in the execution phase, ensuring the real-time and scalability of the algorithm.

[0085] The present invention also designs a multi-agent intrinsic motivation module and adds it to the MADDPG algorithm. Curiosity rewards are used as the intrinsic rewards of the agents to make up for the problem of insufficient exploration motivation of the agents.

[0086] By designing an overall reward function based on curiosity reward and environment reward to solve the reward sparsity problem, the agent can learn a good search strategy, which has greater advantages in convergence speed and stability, and improves the efficiency of the agent's search for targets.

[0087] Multi-agent intrinsic motivation module Figure 3 As shown in the figure, there are three neural networks: one is a fixed target network for setting prediction problems, one is a prediction network trained based on the data collected by the agent, and the other is a policy network. The fixed target network generates a constant output for a given state, the prediction network is used to predict the output of the target network, and the policy network determines the next action of the agent. The target network and the prediction network have the same structure, both using convolutional neural networks (CNN) and fully connected layers for feature transformation, and the feature expressions are respectively represented by f N (t+1) and f' N (t+1) represents the prediction error between the feature encoding of the target network and the prediction network at time t+1 as the intrinsic curiosity reward of the agent at time t As the formula

[0088] The intrinsic motivation module calculates the difference between the output of the target network and the prediction network, so that the agent can generate greater intrinsic rewards for unfamiliar states, driving the agent to minimize the error between the actual state and the predicted state, effectively solving the reward sparsity problem, and enabling the agent to explore the environment more quickly and search for the target.

[0089] Multi-agent deep reinforcement learning is an effective method to solve the problem of multi-agent collaborative target search, based on the MADDPG algorithm. The central idea is centralized training and decentralized execution, that is, in the training phase, the Critic network can obtain the observation status of all agents to guide the Actor network training, and in the execution phase, the agent only uses the Actor network to take actions. However, due to the sparse reward problem in the multi-agent collaborative target search process, the agent's exploration motivation is insufficient, making it difficult for the MADDPG model to converge. Based on the MADDPG algorithm, an intrinsic motivation module is introduced, such as Figure 4 By introducing the intrinsic motivation module, agent i can i (t) Generate a curiosity intrinsic motivation reward r i C (t), rewarding intrinsic curiosity r i C (t) and environmental reward r i e (t) are used together as the overall reward for the target search process of agent i. This not only increases the exploration motivation of the agent and solves the problem of sparse rewards, but also prevents the generation of excessive exploration behavior. Multi-agent training is a process of finding a strategy that can produce the maximum average total return expectation, as shown in the formula:

[0090]

[0091] p is the state distribution of the agent, N represents the total number of agents, π i Represents the search strategy of agent i.

[0092] Using a centralized training and decentralized execution algorithm framework, each agent has four neural networks: Actor network, Critic network, Target Actor network, and Target Critic network. During the training phase, the Critic network can obtain the joint observation state and overall reward of all agents from the environment to guide the training of the Actor network. During the execution phase, the agent only uses the Actor network to take actions.

[0093] Each agent uses its gradient parameters θ i To train its strategy π i , the policy gradient update formula is as follows

[0094]

[0095] represents the centralized action value function, whose input is the actions of all agents and some state information of the agents, and the output is the Q value of agent i. N ) is the centralized observation value of all agents.

[0096] The deterministic action strategy π obtained for agent i i , the loss function provided by the following formula is used for centralized evaluation optimization,

[0097]

[0098] Among them, x is the state information of the agent after being updated by the Critic network, γ is the discount factor, and Represent the deterministic action value of agent i and the deterministic action value after being updated by the Critic network. π' i represents the updated strategy of agent i, o i represents the observed state of agent i, a i Represents the action taken by agent i.

[0099] The sum of two reward functions is set as the overall reward function to train multi-agents, one is the environment reward r e (t), the other is curiosity intrinsic motivation reward r C (t). The multi-agent collaborative target search process is converted into a Markov decision process. In each time step t, agent i will i (t) Receive a block reward r i (t) and take an action a i (t). And according to the environmental transformation dynamics model p i (S i (t+1)|S i (t),a i (t)), transforming the current state into the next state. The goal of each agent is to maximize its final expected reward. The reward of agent i at time t is as follows:

[0100]

[0101] (x i (t),yi (t)) and (x tar (t),y tar (t)) represent the positions of agent i and the target to be searched at time t respectively.

[0102] Embodiment 2

[0103] like Figure 5 As shown, a multi-agent cooperative control system based on deep reinforcement learning is used for a multi-agent system including multiple agents, each of which is equipped with an Actor network and a Critic network for generating actions and evaluating the quality of the actions, including:

[0104] Initialization module 51 is used for each agent to obtain the initialization current state, input the current state into the Actor network, generate the corresponding current action, and the environment returns the reward obtained by the agent for executing the current action in the current state and the next state to be transferred to; initialize the experience buffer pool, which is used to record the experience of all agents; the agent initializes the sampling probability representing the importance of the trajectory of the current action, and stores the trajectory data of the current action and the trajectory importance of the current action in the experience buffer pool;

[0105] Data collection module 52: used for the agent to select a preset number of batches of trajectories with importance exceeding a preset threshold from the experience buffer pool according to the importance of the experience;

[0106] Model updating module 53: used to update the Actor network and Critic network of each agent based on the experience data in the experience buffer pool, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training update of the Actor network;

[0107] Data update module 54: It is used for the intelligent agent to interact with the environment using the updated Actor network and Critic network to obtain a trajectory, calculate the importance of the current trajectory, and then update the experience buffer pool with the current trajectory and the corresponding importance.

[0108] The implementation process, method and effect of a multi-agent collaborative control system based on deep reinforcement learning in this embodiment are the same as those of a multi-agent collaborative control method based on deep reinforcement learning described in Example 1, and will not be repeated here.

[0109] Embodiment 3

[0110] The present invention relates to a computer-readable storage medium, which stores instructions. When the instructions are executed, a multi-agent collaborative control method based on deep reinforcement learning is implemented. The runtime implementation process method and effect are the same as the multi-agent collaborative control method based on deep reinforcement learning described in Example 1, and will not be repeated here.

[0111] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0112] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-agent collaborative control method based on deep reinforcement learning, characterized in that: A multi-agent system including multiple agents, each of which is provided with an Actor network and a Critic network for generating actions and evaluating the quality of the actions, including: Initialization step: each agent obtains the initialized current state, inputs the current state into the Actor network, generates the corresponding current action, and the environment returns the reward obtained by executing the current action and the next state to be transferred to the agent in the current state; initializes the experience buffer pool, which is used to record the experience of all agents; the agent initializes the sampling probability representing the importance of the trajectory of the current action, and stores the trajectory data of the current action and the trajectory importance of the current action in the experience buffer pool; Data collection step: the agent selects a preset number of batches of trajectories with an importance exceeding a preset threshold from the experience buffer pool according to the importance of the experience; Model updating step: based on the experience data in the experience buffer pool, updating the Actor network and the Critic network of each agent, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training update of the Actor network; Data updating step: the agent uses the updated Actor network and the Critic network to interact with the environment to obtain a trajectory, calculates the importance of the current trajectory, and then updates the experience buffer pool with the current trajectory and the corresponding importance.

2. According to the multi-agent collaborative control method based on deep reinforcement learning in claim 1, it is characterized in that: The agent obtains and initializes the current state, inputs the current state into the Actor network, and generates the corresponding current action, and also includes: The modeling of the deep reinforcement learning model is composed of a seven-tuple <N, S, O, A, R, P, γ>, where: N represents the set of agents, N = {1, 2, ... n}; S represents the joint state space of the agents; A represents the joint action space of all agents, A={A1,A2,...,A n }, A n represents the action space taken by agent n; O represents the observation set of the agent; R represents the agent’s reward function, r(s t ,a t )∈R represents the reward obtained by the agent for performing an action in the state at time t; P represents the state transition probability function; γ represents the discount factor, γ∈[0,1]. The smaller the value, the more emphasis is placed on immediate rewards, and the larger the value, the more emphasis is placed on future rewards.

3. According to the multi-agent collaborative control method based on deep reinforcement learning in claim 1, it is characterized in that: The agent uses the updated Actor network and the Critic network to interact with the environment to obtain a trajectory, specifically including: Determine a target agent from the plurality of agents performing a target task; According to a value function of the Actor network, a target agent among the multiple agents obtains a target trajectory and an expected action of the target trajectory from an updated strategy set to control the target agent to perform the expected action, wherein the expected action is an action corresponding to at least one strategy called from the strategy set; wherein the value function is related to the incentive value function of the corresponding target agent, and the interaction value function of other agents among the multiple agents except the target agent relative to the target agent, wherein the incentive value function of the target agent is the sum of the external incentive value function and the internal incentive value function of the target agent, and the interaction value function is related to the expected difference between the action-value function of the other agents among the multiple agents except the target agent after the transfer and the action-value function obtained by counterfactually calculating under the condition of ignoring the state and action of the agent.

4. The multi-agent collaborative control method based on deep reinforcement learning according to claim 3 is characterized in that: The step of obtaining the desired action of the target trajectory from the updated strategy set for a target agent among the multiple agents according to a value function of the Actor network includes: Determining an interaction value function of a target agent among the plurality of agents; Determine the value function of the target agent according to the interaction value function and the incentive value function; According to the value function, a strategy corresponding to the expected action is obtained from the strategy set, and the target agent is controlled to perform the expected action according to the strategy.

5. The multi-agent collaborative control method based on deep reinforcement learning according to claim 1 is characterized in that: The Critic network of each agent observes the situation of the entire multi-agent system, and also includes that the Critic network of each agent observes the actions and states of other agents; The Critic network guides the training of the Actor network, specifically including: The Actor network of each of the agents performs strategy updates based on the local observations of the agent and feedback from the Critic network of the agent, wherein the Critic network provides accurate deterministic joint action value estimates to guide the training updates of the Actor network.

6. A multi-agent collaborative control method based on deep reinforcement learning according to claim 5, characterized in that : The update of the Critic network is specifically as follows: The strategy set of all agents is π i ={π1,π2,...,π n }, and the expected return J[θ i ]=E[R i ], as shown in the formula: Among them, p μ is the state distribution of the agent, is a deterministic joint action-value function whose input is the joint action of all agents (a1,...,a N ) and the joint observation x=(o1,...,o N ), output a Q value. Consider N agents with a continuous strategy μ i , and expand it into a deterministic strategy, the policy gradient is shown in the formula: Wherein, D is the experience buffer pool; The deterministic joint action-value function The update method is shown in the following formula:

7. The multi-agent collaborative control method based on deep reinforcement learning according to claim 1, characterized in that: After each agent completes the data updating step, the data collection step, the model updating step and the data updating step are continuously executed in sequence according to a preset number of training rounds until the number of training rounds is reached, and then the target network parameters of the multi-agent system are updated.

8. The multi-agent collaborative control method based on deep reinforcement learning according to claim 3 is characterized in that: The method further includes: obtaining a target trajectory and an expected action of the target trajectory from an updated strategy set for a target agent among the multiple agents based on a value function of the Actor network, wherein the intrinsic motivation model and a value function of the Actor network are used to obtain a target trajectory and an expected action of the target trajectory from an updated strategy set for a target agent among the multiple agents, wherein the intrinsic motivation model includes three neural networks: one is a fixed target network for setting a prediction problem, one is a prediction network trained based on data collected by the agent, and the other is a policy network, wherein the fixed target network generates a constant output for a given state, the prediction network is used to predict the output of the target network, and the policy network determines the next action of the agent.

9. A multi-agent collaborative control system based on deep reinforcement learning, characterized in that: A multi-agent system including multiple agents, each of which is provided with an Actor network and a Critic network for generating actions and evaluating the quality of the actions, including: The initialization module is used for each agent to obtain the initialization current state, input the current state into the Actor network, generate the corresponding current action, and the environment returns the reward obtained by the agent for executing the current action and the next state to be transferred to the current state; initialize the experience buffer pool, which is used to record the experience of all agents; the agent initializes the sampling probability representing the importance of the trajectory of the current action, and stores the trajectory data of the current action and the trajectory importance of the current action in the experience buffer pool; Data collection module: used for the agent to select a preset number of batches of trajectories with an importance exceeding a preset threshold from the experience buffer pool according to the importance of the experience; Model updating module: used for updating the Actor network and the Critic network of each agent based on the experience data in the experience buffer pool, wherein the Critic network of each agent observes the situation of the entire multi-agent system, and the Critic network guides the training update of the Actor network; Data update module: used for the intelligent agent to interact with the environment using the updated Actor network and the Critic network to obtain a trajectory, calculate the importance of the current trajectory, and then update the experience buffer pool with the current trajectory and the corresponding importance.

10. A computer-readable storage medium, characterized in that: The storage medium stores instructions, which, when executed, execute a multi-agent collaborative control method based on deep reinforcement learning as described in any one of claims 1-8.

Images