Unmanned system cluster control method based on decentralized multi-agent reinforcement learning

Abstract

The invention belongs to the technical field of unmanned system cluster control, and particularly relates to an unmanned system cluster control method based on decentralized multi-agent reinforcement learning, and the method comprises the steps: constructing an Actor-Critic model of each agent in a target unmanned system cluster, and carrying out the decentralized multi-agent reinforcement learning training of the Actor-Critic model; in the training process, an experience replay buffer area is configured for each agent and used for storing actions, states and rewards of the agent at all moments; each agent transmits actions, states and rewards of the agent and Critic network parameter updating gradients to neighbor agents in a communication mode according to the topological structure between the agents, a replay buffer area of the agent is used for storing the actions, states and rewards of the agent and the neighbor agents, and actions, states and rewards of neighbors are used for training Critic network parameters of the agent; the neighbor Critic network parameter updating gradient is used for self Critic network parameter training. According to the invention, the stability and robustness of unmanned system cluster control can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned system cluster control, and more specifically, relates to a method for controlling an unmanned system cluster based on decentralized multi-agent reinforcement learning. Background Art

[0002] An unmanned system cluster refers to a networked collaborative system composed of multiple unmanned platforms (such as unmanned aerial vehicles, unmanned boats, etc.). These unmanned systems achieve coordinated cooperation in perception, decision-making, and execution through communication. The significance of unmanned system clusters covers multiple fields such as technology, economy, military, and society. In fact, the group-based, hierarchical, autonomous, and intelligent unmanned system cluster is an important development direction for future unmanned systems. Compared with a single unmanned system, an unmanned system cluster can utilize its scale advantage to complete complex tasks and has stronger survivability and cost advantages. For example, when some unmanned system nodes fail, other unmanned system nodes can replace them to complete the predetermined tasks, making the cluster system have high fault tolerance and self-healing ability.

[0003] Regarding the defects and deficiencies of existing solutions for realizing the cluster control of unmanned systems using multi-agent reinforcement learning, conventional multi-agent reinforcement learning algorithms usually consider adopting the paradigm of centralized training with decentralized execution (CTDE). Centralized training with decentralized execution means using some global information that individual agents cannot see during training to achieve better training effects, and not using this information during execution. Each agent directly acts according to its own policy to achieve the effect of decentralized execution. The algorithm of centralized training with decentralized execution can effectively utilize global information during training to achieve better and more stable training effects. At the same time, when performing policy model inference, only local information can be used, making the algorithm have a certain scalability. However, there are some limitations in the existing mainstream CTDE paradigm. First, as the number of agents increases, the amount of information that the central controller needs to process will increase exponentially, eventually leading to the curse of dimensionality and bringing huge time and space overheads to the training of the entire system. Second, the centralized assumption of training is unrealistic in many real-world scenarios. In fact, CTDE needs to maintain a completely centralized function during the training process and observe the global information of all other agents, which is a very strong assumption and cannot be satisfied in the vast majority of multi-agent problems in real life, such as the construction of unmanned driving systems, the distributed scheduling and optimization of power grids, and the cooperation tasks of large-scale unmanned systems (such as drones and unmanned boats), and so on. In addition, the existence of the central controller also reduces the ability of the system to resist malicious attacks, and such a training system is not robust. More importantly, completely centralized training treats the multi-agent problem as a single-agent problem, which goes against the original intention of using multi-agent learning algorithms: to explore the implementation of distributed AI.

[0004] Therefore, for the cluster control of unmanned systems, a new multi-agent reinforcement learning paradigm is urgently needed. Summary of the Invention

[0005] In view of the above defects or improvement requirements of the prior art, the present invention provides a method for cluster control of unmanned systems based on decentralized multi-agent reinforcement learning, aiming to improve the stability and robustness of the cluster control of unmanned systems.

[0006] To achieve the above object, according to one aspect of the present invention, there is provided a method for cluster control of unmanned systems based on decentralized multi-agent reinforcement learning, including:

[0007] Training phase: Construct a neural network Actor-Critic model that includes an Actor policy network and a Critic network for each agent in the target unmanned system cluster, and perform decentralized multi-agent reinforcement learning training on it;

[0008] Among them, during the model training process of each agent, configure an experience replay buffer for each agent, and this replay buffer is used to store its own actions, states, and rewards at each moment; in each training of each agent, according to the current inter-agent topology, communicate with neighbor agents to transfer its own actions, states, and rewards at the current moment and the Critic network parameter update gradient. Then, the replay buffer of this agent is also used to store the actions, states, and rewards of the neighbor agents collected and received. The actions, states, and rewards of the neighbor agents collected are used for the training of its own Critic network parameters to achieve decentralized learning; the Critic network parameter update gradients of the neighbor agents collected and received are used for the training of its own Critic network parameters to achieve the consistency among the Critic networks of each agent;

[0009] Execution phase: Each agent uses the trained Actor policy network to execute tasks.

[0010] Furthermore, the experience replay buffer adopts a first-in-first-out (FIFO) queue mode. When it is full, the oldest data is evicted and new data is allowed to enter.

[0011] Furthermore, during the training process, the Deep Deterministic Policy Gradient (DDPG) method is used to determine the Actor policy network and the Critic network.

[0012] Furthermore, the parameter update gradient of the Actor policy network of the i-th agent is expressed as:

[0013]

[0014] In the formula, represents the policy network of the i-th agent; represents the parameters in the policy network; represents the loss function of the policy network of the i-th agent; represents the mathematical expectation following distribution, represents the collectible tuple of the experience replay buffer of the i-th agent for short; represents the partial derivative with respect to θ i , represents the partial derivative with respect to ; a i represents the action of the i-th agent, oi represents the observation of the $i$-th agent, represents the partial derivative with respect to $a$ i ; represents the Q-function of the $i$-th agent jointly with its neighboring agents ; represents the current and next observations of the $i$-th agent jointly with its neighboring agents ; represents the current action of the $i$-th agent jointly with its neighboring agents ; represents the action output by the $i$-th agent based on the observation.

[0015] Furthermore, the parameter update gradient of the Critic network of the $i$-th agent is expressed as:

[0016]

[0017] where $\theta$ i represents the parameters of the Critic network corresponding to the $i$-th agent, represents the loss function of the Critic network under $\theta$ i ; represents the mathematical expectation following the distribution, is an intermediate variable;

[0018]

[0019] In the formula, $r$ i , $r$ j represent the reward values of agent $i$ and agent $j$; $\alpha$ i,j represents the trust degree between agents $i$ and $j$, which is a hyperparameter; $\gamma$ represents the discount factor; represents the Q-function of the $i$-th agent, and $\mu'(\theta)$ represents the corresponding policy.

[0020] Furthermore, the Critic network of each agent updates its parameters in the following manner:

[0021]

[0022] where represents the preset parameter transfer weight between the $i$-th agent and its $j$-th neighboring agent, which is related to time $t$ and the current inter-agent topological structure diagram ; $\theta$ i,t represents the parameters of the Critic network corresponding to the $i$-th agent at time $t$; represents the partial derivative with respect to $\theta$ i,t ; represents $\theta$ i,tThe loss function of the lower Critic network; β represents the hyperparameter of gradient descent.

[0023] According to another aspect of the present invention, there is provided an electronic device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the method described above are implemented.

[0024] According to another aspect of the present invention, there is provided a computer-readable storage medium, where the computer-readable storage medium includes a stored computer program, and when the computer program is run by a processor, the device where the storage medium is located is controlled to execute the steps of the method described above.

[0025] According to another aspect of the present invention, there is provided a computer program product, including a computer program or instruction, and when the computer program or instruction is executed by a processor, the steps of the method described above are implemented.

[0026] Generally speaking, compared with the prior art by the above technical solution conceived by the present invention, the technical solution provided by the present invention mainly has the following beneficial effects:

[0027] 1. The present invention proposes a method for controlling an unmanned system cluster based on decentralized multi-agent reinforcement learning, which transforms the decentralized multi-agent reinforcement learning problem into a decentralized distributed optimization problem, decomposes the Critic network, and each agent neural network model is a neural network Actor-Critic model including an Actor policy network and a Critic network. Then, during the model training process of each agent, an experience replay buffer is configured for each agent, and the replay buffer is used to store the actions, states, and rewards of the agent at each moment; in each training of each agent, each agent only transmits its own information at the current moment to the neighboring agents during the communication phase, including its own actions, states, rewards, and the update gradient of the Critic network parameters. These information are only transmitted to the agents that are topologically connected to itself and will not be transmitted to other agents. The actions, states, and rewards of the collected neighboring agents are used for the training of the Critic network parameters of itself to achieve decentralized learning; the update gradients of the Critic network parameters of the collected neighboring agents are used for the training of the Critic network parameters of itself to achieve the consistency between the Critic networks of each agent through the "consensus" of the gradients. Therefore, through the application of decentralized distributed optimization, the connection of the network topology between agents, and the realization of collaborative multi-agent reinforcement learning, the present invention can solve the problems of the existing multi-agent reinforcement learning-based unmanned system cluster control scheme, such as unstable performance indicators, physical implementation, high hardware requirements, and insufficient system robustness.

[0028] 2. The present invention further proposes that during the training process, the Actor policy network and the Critic network are determined by using the method of Deep Deterministic Policy Gradient (DDPG), which has unique advantages in continuous control tasks. By combining deterministic policies, experience replay, and target networks, efficient and stable learning is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 FIG. is a comparison diagram of the CTDE and DTDE architectures of the multi-agent reinforcement learning algorithm provided by the embodiment of the present invention;

[0030] Figure 2 FIG. is a schematic diagram of the decentralized multi-agent reinforcement learning algorithm provided by the embodiment of the present invention;

[0031] Figure 3 FIG. is a schematic diagram of the internal computing module of the agent provided by the embodiment of the present invention;

[0032] Figure 4 FIG. is a schematic diagram of the dynamic process of the networked decentralized partially observable Markov process provided by the embodiment of the present invention;

[0033] Figure 5 FIG. is a schematic diagram of the application of the unmanned boat cluster environment provided by the embodiment of the present invention;

[0034] Figure 6 FIG. is a schematic diagram of the application of the UAV cluster environment provided by the embodiment of the present invention;

[0035] Figure 7 FIG. is a schematic diagram of the topological relationship of eight agent clusters provided by the embodiment of the present invention;

[0036] Figure 8 FIG. is a test result diagram provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] In order to make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] Embodiment 1

[0039] A method for controlling an unmanned system cluster based on decentralized multi-agent reinforcement learning includes:

[0040] Training phase: Construct a neural network Actor-Critic model including an Actor policy network and a Critic network for each agent in the target unmanned system cluster, and perform decentralized multi-agent reinforcement learning training on it;

[0041] Among them, during the model training process of each agent, an experience replay buffer is configured for each agent, and this replay buffer is used to store its own actions, states, and rewards at each moment; in each training of each agent, according to the current inter-agent topology, the agent communicates with its neighbor agents to transfer its own actions, states, and rewards at the current moment, as well as the gradient of the Critic network parameter update. Then, the replay buffer of this agent is also used to store the actions, states, and rewards of the neighbor agents it collects and receives. The actions, states, and rewards of the neighbor agents it collects are used for the training of its own Critic network parameters to achieve decentralized learning; the gradient of the Critic network parameter update of the neighbor agents it collects and receives is used for the training of its own Critic network parameters to achieve the consistency between the Critic networks of each agent;

[0042] Execution phase: Each agent uses the trained Actor policy network to execute tasks.

[0043] This embodiment introduces a completely decentralized multi-agent reinforcement learning paradigm, that is, decentralized training and decentralized execution (Decentralized Training Decentralized Execution, which can be abbreviated as DTDE), in order to solve various problems of CTDE in the control of unmanned system clusters. Figure 1 Shows the comparison between the multi-agent reinforcement learning algorithms CTDE and DTDE architectures. It is clearly visible that the CTDE algorithm has a central controller, while the DTDE algorithm does not have a central controller.

[0044] Using theoretical analysis, the decentralized multi-agent reinforcement learning problem is transformed into a decentralized distributed optimization problem. Thus, the method of this embodiment decomposes the Critic network. Each agent neural network model is a neural network Actor-Critic model including an Actor policy network and a Critic network, which gives theoretical guarantee to this scheme. In addition, for the actual training of each agent, it can be divided into three modules, namely the calculation module, the communication module, and the consensus module.

[0045] Figure 2 Illustrates the overall framework of the decentralized multi-agent reinforcement learning algorithm of the present invention.

[0046] For the calculation module of each agent, it is used for network training.

[0047] For the communication module of each agent, the features are as follows: during the communication phase, each agent only transmits its own information at the current moment to its neighboring agents, including its own actions, states, rewards, and the update gradient of the Critic network parameters. This information is only transmitted to the agents that are topologically connected to itself and will not be transmitted to other agents.

[0048] For the consensus module of each agent, the features are as follows: each agent shares the update gradient of the Critic network parameters with the agents that are topologically connected to itself according to the current topological relationship. The update gradients of the Critic network parameters of the neighboring agents collected and received are used for the training of its own Critic network parameters to achieve the consistency among the Critic networks of each agent through the "consensus" of the gradients.

[0049] Regarding the topological relationship between agents, as Figure 2 shown on the left, where the bidirectional arrows represent the relationship between the network constructed by each agent and the agent number. In fact, the networks of these agents can be modeled as a graph with a node set and an edge set. That is to say, the set of neighbors of each agent can be obtained. For time-invariant topologies, the neighbors of an agent are fixed at the initial moment, and for time-varying topologies, the neighbors of an agent change over time.

[0050] Figure 2 The middle part shows the Actor-Critic network architecture of the agents. Each agent maintains a replay buffer that can only collect information from the agent itself and the agent's neighbors, which ensures that the training is decentralized.

[0051] Figure 2 The right part introduces two important mechanisms: the decentralized distributed implementation of information exchange between different buffers and the switching of the update gradients of decentralized distributed optimization. The bidirectional arrow above represents the decentralized distributed implementation of information exchange between different buffers, and the bidirectional arrow below represents the different decentralized distributed gradient exchanges.

[0052] Furthermore, for the computing module, for each individual agent, preferably, during the training process, the method of Deep Deterministic Policy Gradient (DDPG) is used to determine the Actor policy network and the Critic network. For the computing module of each agent, it uses the Deep Deterministic Policy Gradient (abbreviated as DDPG) to determine its own actions, specifically including two parts: learning a Q function and learning its own action policy.

[0053] The key technical points of the method in this embodiment lie in the application of decentralized distributed optimization, the connection of the inter-agent network topology, and the implementation of collaborative multi-agent reinforcement learning, which can solve the problems of the existing multi-agent reinforcement learning-based unmanned system cluster control scheme, such as unstable performance indicators, physical implementation, high hardware requirements, and insufficient system robustness.

[0054] As an optional implementation, the experience replay buffer adopts a first-in-first-out (FIFO) queue mode. When it is full, the oldest data is evicted and new data is allowed to enter.

[0055] Figure 3 The specific architecture of a single agent computing module is shown. Before explaining its specific details, the following instructions are given:

[0056] (1) Model abstraction: Decentralized multi-agent reinforcement learning can be regarded as a "decentralized partially observable Markov process" with a networked topological relationship, defined by the following tuple:

[0057]

[0058] where is the set of agents, is the communication topology among agents. For each agent i, its neighboring agents are marked as is the finite state of the environment, and its constituent element s is a specific state. is the joint action of the agents, A i represents the action space of the i-th agent, and each action in it is represented by a i And is the joint reward function of the agents, respectively represent the Markov state transition probability function, the finite joint observation set, and the finite joint observation set. The global reward at time t can be defined as:

[0059]

[0060] It should be noted that each agent cannot obtain this value. represents the reward obtained by the i-th agent at time t, and the value of R i (t) at time t is

[0061] Figure 4Provides a detailed description of the dynamics of a networked decentralized partially observable Markov process. At each state transition stage of the Markov process, the environment is in a specific state, and each agent obtains its individual reward given by the environment from its own observation model. Thereafter, each agent (through the Actor Net) uses the observation information (observation value o) to select an action a and thereby effect a state transition of the environment (i.e., s → s ′ ). Each cluster agent communicates only with the agents corresponding to the adjacent nodes defined by the network topology. Each agent completes the shared task based on its own and its neighbors' observations (observation information), costs (determined based on rewards), and learning (the learning gradient of the Critic).

[0062] Without loss of generality, the optimization objective for decentralized multi-agent reinforcement learning is to maximize the cumulative reward, which can be defined as follows:

[0063]

[0064] where, are the joint policy parameters of the agent cluster (parameters of n Actor neural networks), is the joint policy of the agent cluster (the action policies of the Actor networks in each agent combined), and γ is the discount factor. a ∼ π θ denotes the action, o ∼ ε denotes the observation value (observation information), and E denotes the mathematical expectation following the s0 ∼ p0, o ∼ ε, a ∼ π θ distribution.

[0065] For decentralized distributed optimization, multi-agent reinforcement learning can be trained in a decentralized manner, where there are arbitrarily network-topology-connected nodes. Each agent locally updates the model parameters based on its local dataset and then averages them with its neighbor agents. Finally, as long as the network topology is connected, the updates from this stage will eventually reach all other agents. The sum of the objective functions of these agents is:

[0066]

[0067] And the expected cumulative reward of all agents can be partitioned as:

[0068]

[0069] Therefore, the above decentralized distributed optimization problem can be reformulated into a more tractable form:

[0070]

[0071] Due to the above explanations, the distributed optimization problem form of fully decentralized collaborative networked multi-agent reinforcement learning can be naturally derived:

[0072]

[0073] where Ω is the joint parameter of the agent policies.

[0074] This concludes the model abstraction part of the decentralized multi-agent reinforcement learning problem.

[0075] (2) Model assumptions: The agents of "policy learning" can act independently, and the learning steps can be achieved in a decentralized manner. By maximizing the total number of decentralized functions, it can achieve the same effect as maximizing a virtual center. In fact, the decomposition of the global reward function means that the rewards from the environment can be obtained in a distributed manner. In fact, this assumption is isomorphic to the decomposition assumption:

[0076]

[0077] In addition, it should be noted that Ω is still a centralized parameter and can be expressed as:

[0078] Ω = {π(·|s; θ), Q total (s,a; θ joint )}

[0079] Therefore, it can be further considered that the problem can be decomposed into the following form:

[0080]

[0081] represents the loss function of the i-th agent, θ total represents the joint parameter of n Critic neural networks, represents the loss function, represents the total of the i-th agent and its neighbor agents number of joint parameters of the Critic neural networks; t represents the time, Ω i = {π i (·|s; θ i ), Q i (s,a; θ i )}, s represents the state, θ represents the parameter of the Critic neural network of the i-th agent, and a represents the action. γ t represents the t-th power of the discount factor, represents the reward obtained by the i-th agent at time t+1. represents the mathematical expectation subject to a ∼ π θ .

[0082] So far, the objective function of deep learning has been determined. Next, the specific details of the calculation module are introduced as follows:

[0083] To train the Actor (i.e., learn the action policy), the update gradient optimization objective of the Actor parameters is:

[0084]

[0085] In the formula, represents the deterministic policy, and Q φ represents the Q function with φ as the schematic parameter, represents the mathematical expectation following distribution.

[0086] Therefore, for the i-th agent in the decentralized multi-agent reinforcement learning problem, the input of its policy network (or Actor network) is the observation value which contains the observation information of itself and its neighboring agents, and the output is the action a. Preferably, the update gradient of the parameters of the i-th agent's policy network is expressed as:

[0087]

[0088] In the formula, represents the policy network of the i-th agent; represents the parameters in the policy network; represents the loss function of the policy network of the i-th agent; represents the mathematical expectation following distribution, represents the partial derivative with respect to θ i , represents the partial derivative with respect to ; a i represents the action of the i-th agent, and o i represents the observation of the i-th agent, represents the partial derivative with respect to a i , represents the Q function of the i-th agent jointly with its neighboring agents, represents the observation of the i-th agent jointly with its neighboring agents, represents the action of the i-th agent jointly with its neighboring agents.

[0089] For the gradient of the deterministic policy, the stochastic policy π is changed to the deterministic policy μ here.

[0090] refers to the experience replay buffer, which contains the data of the corresponding agent and its neighboring agents interacting with the environment respectively and is expressed as:

[0091]

[0092] In the formula, represents the observation reward of the i-th agent combined with its neighbor agents, represents the observation of the next state of the i-th agent combined with its neighbor agents.

[0093] In addition, the update mechanism of the policy network (Actor network) and the target network in the DDPG algorithm can be expressed as:

[0094]

[0095] where τ is a hyperparameter between 0 and 1, are the parameters of the target network.

[0096] In addition, The possibility of selection behavior is directly enhanced and weakened by the reward. That is to say, by this method, the probability that the agent selects the correct behavior next time will increase, while the probability of selecting the wrong behavior next time will decrease.

[0097] To train the Critic network, the Bellman equation for describing the optimal state-action value function is as follows:

[0098]

[0099] In the formula, Q * (s,a) represents the optimal Q function, represents the mathematical expectation following the s′~P distribution, and r(s,a) represents the reward function.

[0100] In fact, the state-action value function (i.e., Q function) corresponding to each agent can preferably be updated by minimizing the loss:

[0101]

[0102] where θ i represents the parameters of the Critic network corresponding to the i-th agent, represents θ i the loss function of the Critic network below, is the abbreviation of the collectible tuple for the i-th agent. represents the mathematical expectation following the distribution, represents the Q function of the i-th agent combined with its neighbor agents, is an intermediate variable.

[0103] And there is:

[0104]

[0105] In the formula, r i 、r j represent the reward values of agent i and agent j; α i,j represents the trust degree between agents and is a hyperparameter; represents the Q function of the i-th agent.

[0106] Therefore, for the Critic network of each independent agent, the calculation formula of the gradient of its loss function is:

[0107]

[0108] Similarly, the update mechanism of the Critic network and the target network can be expressed as:

[0109] θ′←ρθ+(1-ρ)θ′

[0110] In fact, the designed agent network can learn in a dynamic environment and interact with the dynamic environment in practice. When collecting data, the agent will use the latest state-action value function and conduct exploration in actions to obtain some transition data that can be put into the experience pool. In addition, the experience replay buffer is designed as a first-in-first-out (FIFO) queue mode. When the experience pool is full, the oldest data is evicted and new data enters. Then, the target parameters will be updated according to the current parameters at a set time interval.

[0111] For the communication module and the consensus module, first, the above local calculation is reduced to a kind of local calculation behavior: the Actor networks of each agent are initialized to the same network. At the t-th moment, each agent has its own parameter version. In fact, the joint policy parameter can be written as follows:

[0112]

[0113] Since each agent can only communicate with neighboring agents, this process has the property of decentralization. Then comes the consensus part, which is a weight consensus behavior. Preferably, each agent updates its parameters in the following way:

[0114]

[0115] Among them, is the weight of parameter transfer between agents, related to time t and the topological network graph Relatedly, this method performs weighted summation according to certain parameters to achieve "consensus" of gradients, thereby realizing the transmission of gradient information.

[0116] In the experiment, the matrix parameters are the mixing matrices determined by the initial topological relationships. In fact, when each node (agent) performs some local calculations and shares the results with neighboring agents, each neighboring agent can use the mixing weights to incorporate these results into the next round of local calculations.

[0117] Assume that the elements of the mixing matrix satisfy the following properties:

[0118] For \(i,j\in\{1,\ldots,n\}\), there is

[0119] If and then

[0120] is a symmetric matrix, that is

[0121] is a doubly stochastic matrix, that is, the sum of each row and each column of the matrix is 1.

[0122] For example, for a five-node undirected loop, assume that the learning rate of each adjacent edge is \(\zeta\), then the mixing matrix is:

[0123]

[0124] Note: For topological relationships such as a five-node undirected loop, see Figure 7 as shown, which includes schematic diagrams of common agent cluster topological relationships.

[0125] That is to say, the above weight consensus behavior can be rewritten as a joint parameter transformation formula based on time state transitions, that is:

[0126]

[0127] Where:

[0128]

[0129] This is the specific implementation scheme of decentralized distributed gradient descent optimization. Generally speaking, in consensus-based optimization algorithms, nodes will interleave the steps of local gradient descent with the steps of consensus iteration. The gradient descent steps will push the solution towards the minimum value, while the consensus iteration will synchronize the solution values. In this way, when the objective is convex and separable, all agents will converge to the optimal value of the entire decentralized distributed neural network.

[0130] To better illustrate the present invention, the following examples are given. In a real simulation environment, two different clusters of intelligent autonomous unmanned systems are considered, including a cluster of unmanned surface vehicles (USV) and a cluster of unmanned aerial vehicles (UAV). To be as close to the real situation as possible, in both the USV cluster and the UAV cluster, each agent in the cluster can only obtain partial rewards. These agents need to complete a task called "cooperative navigation", that is, the agents achieve a common goal through cooperation, such as moving to a specific destination, avoiding obstacles, or exploring an unknown environment and other specific cooperative tasks.

[0131] Figure 5 It is a schematic diagram of the application of the unmanned surface vehicle cluster environment. Figure 6 It is a schematic diagram of the application of the unmanned aerial vehicle cluster environment. Figure 7 The shown network topology under 8 time-invariant topological relationships is used as a concrete description of the communication topology among agents. Among the nine time-varying topologies, there are three sub-topologies, which are connected undirected graphs, connected directed graphs, and disconnected undirected graphs. In fact, generally speaking, the modes of communication topology can be divided into loop topology (Loop), half-net topology (HalfNet), star topology (Star), and dual-flat topology (DualFlat). In an undirected graph, two agents connected by an edge can exchange their observations, their actions, and their real-time rewards at the current moment, but in a directed graph, this information can only flow unidirectionally along the arrow direction.

[0132] In addition, this example also designs large-scale unmanned surface vehicle cluster experiments and large-scale unmanned aerial vehicle cluster experiments to prove the scalable performance of the proposed unmanned system cluster control scheme based on decentralized multi-agent reinforcement learning on large-scale systems.

[0133] In fact, this example selects three state-of-the-art MARL algorithms, MADDPG, ID2PG, and MAD3PG, to compare with the unmanned system cluster control scheme based on decentralized multi-agent reinforcement learning proposed in the present invention. Among them, MADDPG, ID2PG, and MAD3PG adopt the policy gradient method, and ID2PG and MAD3PG adopt a distributed training paradigm.

[0134] In addition, in terms of neural network parameters, the experience pool buffer size and batch size are set to 100000 and 1024 respectively. The training set update interval for each round of training is 100. In addition, the learning rates of the Actor network and the Critic network are 1e-2, the discount factor is 0.95, and the consensus weight for decentralized distributed optimization is 1e-2.

[0135] The experimental results are as Figure 8As shown, it can be seen that the convergence speed of the unmanned system cluster control scheme designed by the present invention based on decentralized multi-agent reinforcement learning is still the fastest among all algorithms, which is quite advantageous in the applications of actual distributed uncertain physical systems (such as drone clusters, unmanned boat clusters, autonomous robot clusters, distributed multi-node power grids, distributed logistics scheduling systems, distributed industrial networks, etc.). This proves the superiority and advancement of this patent.

[0136] Embodiment 2

[0137] This application also relates to an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0138] The electronic device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by calling the data stored in the memory.

[0139] The related technical solutions are the same as above and will not be elaborated here.

[0140] Embodiment 3

[0141] This application also relates to a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0142] Specifically, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash device, or other volatile solid-state storage devices.

[0143] The related technical solutions are the same as above and will not be elaborated here.

[0144] Embodiment 4

[0145] An embodiment of the present application provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of the method according to the above embodiments of the present application.

[0146] The related technical solutions are the same as above and will not be elaborated here.

[0147] It is easy for those skilled in the art to understand that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for controlling an unmanned system cluster based on decentralized multi-agent reinforcement learning, characterized in that, Including: Training stage: constructing a neural network Actor-Critic model including an Actor policy network and a Critic network for each agent in the target unmanned system cluster and performing decentralized multi-agent reinforcement learning training on it; Among them, during the model training process of each agent, an experience replay buffer is configured for each agent, and this replay buffer is used to store its own actions, states, and rewards at each moment; In each training of each agent, according to the current inter-agent topology, communicate with neighbor agents to transfer its own actions, states, and rewards at the current moment and the Critic network parameter update gradient. Then, the replay buffer of this agent is also used to store the actions, states, and rewards of the neighbor agents collected and received. The actions, states, and rewards of the neighbor agents collected are used for the training of its own Critic network parameters to achieve decentralized learning; the Critic network parameter update gradients of the neighbor agents collected and received are used for the training of its own Critic network parameters to achieve the consistency between the Critic networks of each agent; Execution stage: Each agent uses the trained Actor policy network to execute tasks.

2. The method for controlling an unmanned system cluster according to claim 1, wherein The experience replay buffer adopts a first-in-first-out (FIFO) queue mode. When it is full, the oldest data is evicted and new data enters.

3. The method for controlling an unmanned system cluster according to claim 1, wherein During the training process, the method of Deep Deterministic Policy Gradient (DDPG) is used to determine the Actor policy network and the Critic network.

4. The method for controlling an unmanned system cluster according to claim 3, characterized in that, The parameter update gradient of the Actor policy network of the i-th agent is expressed as: wherein, represents the policy network of the i-th agent; represents the parameters in the policy network; represents the loss function of the policy network of the i-th agent; represents the mathematical expectation that follows distribution, represents the abbreviation of the collectible tuple of the experience replay buffer of the i-th agent ; represents the partial derivative of θ i ; represents the partial derivative of ; a i represents the action of the i-th agent, o i represents the observation of the i-th agent, represents the partial derivative of a i ; represents the Q function of the i-th agent's joint neighbor agents ; respectively represent the current observation and the next observation of the i-th agent's joint neighbor agents ; represents the current action of the i-th agent's joint neighbor agents ; represents the action output by the i-th agent based on the observation.

5. The method for controlling an unmanned system cluster according to claim 4, characterized in that, The parameter update gradient of the Critic network of the i-th agent is expressed as: Among them, θ i represents the parameters of the Critic network corresponding to the i-th agent, represents θ i the loss function of the Critic network below, represents the expectation that follows the distribution, is an intermediate variable; where r i , r j represent the reward values of agent i and agent j; α i,j represents the trust level between agents i and j, which is a hyperparameter; γ represents the discount factor; represents the Q-function of the i-th agent, and μ′(θ) represents the corresponding policy.

6. The method for controlling an unmanned system cluster according to claim 5, wherein The Critic network of each agent updates its parameters in the following manner: Among them, represents the parameter transfer weight between the preset $i$-th agent and its $j$-th neighbor agent, which is related to time $t$ and the current agent-to-agent topology graph $\theta$ i,t represents the parameters of the Critic network corresponding to the $i$-th agent at time $t$; represents the partial derivative of $\theta$ i,t ; $\theta$ i,t represents the loss function of the Critic network under $\theta$; $\beta$ represents the hyperparameter of gradient descent.

7. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program. Among them, when the computer program is run by a processor, it controls the device where the storage medium is located to execute the steps of the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instruction is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images