A reinforcement learning cluster swarming control method based on graph neural network

Abstract

The present invention discloses a cluster swarming control method based on reinforcement learning of graph neural network, comprising the following steps: establishing a cluster swarming control model; determining a method for representing the topological structure characteristics of the cluster; determining a method for representing the observation information characteristics of the intelligent body; designing the state space, behavior space and reward function; designing the policy network and evaluation network model in the deep reinforcement learning algorithm; designing the algorithm framework and the network parameter updating method; and designing the training process of the cluster swarming control algorithm. The present invention realizes the cluster swarming control algorithm with the help of deep reinforcement learning technology, and uses the graph neural network to extract the topological structure characteristics and observation information characteristics of the cluster, effectively improving the convergence speed of the cluster swarming control algorithm and its adaptability to dynamic environments, while being able to ensure the stability of the algorithm under interference such as control noise.

Description

Technical Field

[0001] The patent of this invention belongs to the field of multi-agent clustering and reinforcement learning. It is a deep reinforcement learning cluster swarming control method based on graph neural network, involving a series of methods such as cluster topology structure feature extraction and state and action space construction in reinforcement learning, reward function and training process design. Background Art

[0002] Cluster swarming control has important application and scientific research value, such as adaptive control of drones and unmanned vehicle clusters. The current swarming control method lacks effective use of system topology information, and does not consider the equivalence between agents in the method design. The dynamic changes of the system topology and the random noise interference in the control system greatly weaken the performance of the control algorithm. Therefore, improving the efficiency of the swarming control algorithm and achieving topology convergence and speed consistency in the shortest time is an important research direction of multi-agent cluster swarming control.

[0003] Graph neural networks are a type of deep learning method based on graph structures. They can extract structural features based on the adjacency relationship between graph data and model the dependency relationship between graph data. Graph convolutional neural networks applied convolution operations to graph data for the first time, effectively making up for the inability of traditional convolutional neural networks to process non-Euclidean data. Currently, graph neural networks have been widely used in tasks such as regression, prediction, and classification of non-Euclidean data in social networks, knowledge graphs, and natural language processing.

[0004] Reinforcement learning is a machine learning method that optimizes decisions through a trial-and-error learning mechanism. It has the advantage of not relying on any prior knowledge or external environment model. Deep reinforcement learning uses deep neural networks to replace traditional action-value functions, greatly improving the feature representation and decision-making capabilities of reinforcement learning algorithms. It has been widely studied and applied in control fields such as robotics. In the process of swarm control in complex environments, multi-agents can use deep neural networks to learn the characteristics of states and behaviors. After the deep reinforcement learning algorithm is trained, the optimal multi-agent swarm control strategy can be obtained. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a deep reinforcement learning cluster swarming control method based on graph neural network, so as to realize fast and stable cluster swarming control, effectively improve the cluster convergence speed and adaptability to dynamic environment, and at the same time ensure the stability of the algorithm under interference such as control noise.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A reinforcement learning cluster swarming control method based on graph neural network, comprising the following steps:

[0008] Step S1, establishing a cluster swarming control model. After a limited number of swarming controls, the cluster should gradually form a stable topological structure and maintain a consistent speed. After stabilization, the system error should approach 0.

[0009] Step S2: The entire cluster system is regarded as an undirected graph G, whose adjacency matrix is ​​A. The crowding control error is introduced into the adjacency matrix A of the undirected graph G to obtain the weighted adjacency matrix Based on the weighted adjacency matrix The topological structure features of the cluster system are extracted by using a two-layer graph convolutional neural network (GCN) connected sequentially to obtain the final cluster topological structure feature representation H A ′;

[0010] Step S3: Determine the observation information feature representation method of the agent and construct the global and local observation information matrix X sg and X oi , using a two-layer graph convolutional neural network connected sequentially to respectively analyze the global and local observation information matrices X sg and X oi Perform feature extraction to obtain the global observation information matrix X sg The characteristic representation H sg ′ and the feature representation H of the local observation information matrix oi ′;

[0011] Step S4: Design state space: Based on the weighted adjacency matrix in step S2 Calculation method and the global and local observation information matrix X in step S3 sg and X oi , respectively define the global observation state s and the local observation state o of agent i i ;

[0012] Design action space: select continuous space as the action space of the agent and determine the speed update method of agent i;

[0013] Design reward function: The principle is to keep the speed of agents consistent and avoid collisions during movement;

[0014] Step S5: Based on the graph convolutional neural network structure in step S3, the global observation state s and the local observation state o in step S4 are transformed into i As network input, we design the policy network and evaluation network models in the deep reinforcement learning algorithm respectively;

[0015] Step S6: Based on the policy network and evaluation network models in step S5, a reinforcement learning cluster swarming control algorithm framework is designed, and a parameter update method of the policy network and the evaluation network is given;

[0016] Step S7, design a training process for the cluster swarming control algorithm, based on the experience information generated by the continuous interaction between the intelligent agent and the environment, use the parameter updating method in step S6 to learn the network parameters, and after the training is completed, obtain an intelligent agent strategy network that can be used for cluster swarming control.

[0017] In the above technical solution, step S1 comprises:

[0018] A swarm control model is established. The swarm system consists of N agents, where the i-th agent is defined as agent i. The agents move in a two-dimensional plane and satisfy the following particle motion model:

[0019]

[0020] Among them, p i , v i and u i ∈R 2 are the position, velocity, and acceleration control values ​​of agent i, respectively. Respectively represent p i , v i Derivatives with respect to time;

[0021] At a distance less than the communication distance r c There are communication connections between agents, and the neighbor set of agent i is defined according to the connection relationship:

[0022] N i ={j∈V:||p j -p i ||≤r c , j≠i}

[0023] Where V is the set of all agents, ||·|| is the Euclidean norm, and neighboring agents can exchange information such as position and speed through communication;

[0024] Designate an agent in the cluster as the leader, denoted as leader, and the remaining N-1 agents as followers, denoted as followers. The swarming control model can be described as follows: followers follow the leader and avoid collisions during the movement, and eventually the entire cluster reaches a consistent speed and forms a stable topological structure;

[0025] The error of the swarming control system at time k is defined as follows:

[0026] e(k)=elf (k)+e ff (k)

[0027] Among them, e lf (k) is the relative distance and speed error between the follower and the leader, defined as follows:

[0028]

[0029] in, is the expected distance between leader and follower, and are the position and velocity of follower i, p0 and v0 are the position and velocity of leader, φ l (·) is a convex function with a global minimum value of 0 and φ l (0) = 0;

[0030] e ff (k) is the error of the relative distance and speed between followers, which is defined as follows:

[0031]

[0032] in, is the expected distance between followers, N i is the set of neighboring agents of agent i, φ f (·) is a convex function with a global minimum value of 0 and φ f (0) = 0;

[0033] After a limited number of swarming controls, the cluster should gradually form a stable topology and maintain a consistent speed. After stabilization, the system error should approach 0:

[0034] lim k→∞ e(k)=0.

[0035] In the above technical solution, step S2 comprises:

[0036] Determine the topological structure feature representation method of the cluster: the entire cluster system can be regarded as an undirected graph G = (V, E), the set V of all agents constitutes the vertex set of the undirected graph G, and the connection relationship between agents constitutes the edge set E of the graph. The adjacency matrix of the undirected graph G is denoted by A, which is an N×N binary symmetric square matrix. The element value 1 represents the existence of communication connection between agents, and the value 0 represents no communication connection. In order to better reflect the degree of association between agents in the cluster, the crowding control error is introduced into the adjacency matrix A of the undirected graph G, and the weighted adjacency matrix is ​​obtained:

[0037]

[0038] Among them, c norm is the regularization coefficient, ⊙ represents the Hadamard product, and the error weight matrix W loss The elements in W are defined as loss (i, j) = L ij , i, j = 0, 1, ... N, error L ij The definition is as follows:

[0039]

[0040] in, is a constant coefficient, the convex function φ l (·) and φ f (·) is defined as follows:

[0041] φ l (z)=min{(c n z) 4 u(-z)+(c p z) 3 u(z),c rmax}

[0042] φ f (z)=(c n z) 4 u(-z)

[0043] Among them, c n , c p is the adjustment coefficient, c rmax is the maximum error threshold, and the unit step function u(z) is defined as follows:

[0044]

[0045] The element values ​​in the weighted adjacency matrix The larger the value, the greater the relative speed or position error between agent i and agent j, which means that the swarming control strategy should pay more attention to agent j when adjusting the position or speed of agent i. Therefore, Effectively represents the topological structure information of the cluster;

[0046] Based on the weighted adjacency matrix The topological structure features of the cluster system are extracted using a two-layer graph convolutional neural network (GCN) connected sequentially:

[0047]

[0048]

[0049] Where W, W′ are the parameter matrices of the graph convolutional neural network layer, b, b′ are the bias vectors, σ(·) is the activation function, and H A It is the cluster topology structure feature output by the first layer GCN; based on H A , and then use a layer of GCN to further extract the high-dimensional feature information of the cluster topology structure to obtain the final cluster topology feature representation H A ′.

[0050] In the above technical solution, step S3 comprises:

[0051] Determine the method of representing the observation information features of the agent, and use the speed and position information of neighbor agent j relative to the current agent i, as well as the category information of agent i (leader or follower in the cluster) to construct the observation information of agent i The definition is as follows:

[0052]

[0053] Among them, c j represents the category of agent j, 0 represents follower, and 1 represents leader. To improve the stability of the neural network training process, the relative distance and relative speed between agents are further normalized to the interval (-1, 1), and the local observation information matrix of agent i can be obtained:

[0054]

[0055] Among them, c v is the maximum amplitude of the relative velocity;

[0056] Considering that each follower agent in the cluster system is isomorphic and has an equivalent relationship, and X oi will change due to the change of the order of agent numbers, so in order to satisfy X oi With respect to the invariance of the agent number, the row order of the observation information matrix needs to be adjusted as follows:

[0057] X oi :=X oi [P order ]

[0058] Among them, P order is the new numbering order of the agents in the cluster, which is obtained by arranging the relative distances between agents in ascending order:

[0059] P order =argsort([||p i -p0||,||p i -p1||,…,||pi -p N ||])

[0060] In order to maintain the consistency of the order of agents, the weighted adjacency matrix in step S2 needs to be recalculated according to the new agent number.

[0061] Based on the order-adjusted local observation information matrix X of agent i oi , using a sequentially connected two-layer graph convolutional neural network to extract observation information features:

[0062] H oi =σ(X oi W+b)

[0063] H oi ′=σ(H oi W′+b′)

[0064] Among them, H oi is the observation information feature output by the first layer of GCN; in H oi On the basis of , another layer of GCN is used to further extract the high-dimensional features of local observation information to obtain the final local observation information feature representation H oi ′;

[0065] In addition, in order to distinguish the global observation information from the local observation information of the agent, the observation information with the leader as the observation center is regarded as the global observation information of the cluster system, that is, the global observation information matrix X sg It can be expressed as follows:

[0066]

[0067] Global observation information matrix X sg The characteristic representation H sg ′ is also combined with the local observation information feature H oi 'The same method obtains:

[0068] H sg =σ(X sg W+b)

[0069] H sg ′=σ(H sg W′+b′).

[0070] In the above technical solution, step S4 comprises:

[0071] Design state space: Based on the weighted adjacency matrix in step S2 And the global and local observation information matrix X in step S3 sg and X oi, respectively define the global observation state s and the local observation state o of agent i i :

[0072]

[0073]

[0074] in, and are the weighted adjacency matrices of leader and agent i respectively;

[0075] Design action space: Select continuous space as the action space of the agent, that is, directly define the continuous value output by the policy network of agent i as the acceleration of the motion control amount of the agent, that is,

[0076] u i =a i

[0077] Among them, u i is the acceleration control value of agent i, a i is the action output of agent i; thus the speed update method of agent i is defined as follows:

[0078] v i+1 =v i +a i

[0079] Design reward function: Based on the principle of keeping the speed of agents consistent and avoiding collisions during movement, design the following reward function:

[0080] r=max{-L sym , r min}

[0081] Among them, r min <0 is the minimum return value, L sym is the control error loss of the entire cluster system, which is defined as follows:

[0082]

[0083] The first two items describe the errors of the relative distance and relative speed between the follower and the leader, respectively. The third and fourth items describe the errors of the relative distance and relative speed between the followers, respectively. By minimizing L sym, which can make all followers maintain the same movement speed and constant distance with the leader, forming the effect of followers following the leader, while maintaining a stable and convergent swarming topology; in the swarming process, the closer the positions between followers and between followers and leaders are to the ideal stable cluster structure, the lower the system control loss value L sys The closer it is to 0, the greater the return value r is;

[0084] In the above technical solution, step S5 comprises:

[0085] Design strategy network model: The strategy network of agent i is based on the local observation state o in step S4. i As input, the two-layer GCN in step S2 is used to extract the cluster topology structure features of the cluster system The two-layer GCN in step S3 is used to extract the local observation information feature representation H of agent i oi ′, the two features are concatenated after vectorization, and finally the action decision a is output through two layers of fully connected layer networks using tanh(·) as the activation function i ∈(-1, 1);

[0086] Design evaluation network model: The evaluation network is based on the global observation state s in step S4 and the action matrix X of the intelligent agent. a As input, the two-layer GCN in step S2 is used to extract the topological structure features of the cluster system. The two-layer GCN in step S3 is used to extract the global observation information feature representation H sg ′, use two layers of GCN to extract the action matrix X a Finally, the three features are concatenated after vectorization, and the state value Q is output through two layers of fully connected network using relu (·) as the activation function; the action matrix X a The definition is as follows:

[0087] X a =(a0, a1, ..., a N-1 ) T

[0088] Among them, a i The meaning is the agent output action in step S4.

[0089] In the above technical solution, step S6 comprises:

[0090] Design algorithm framework and network parameter updating method: The algorithm uses the evaluation network and policy network in step S5 to build a reinforcement learning cluster swarming control algorithm framework. In order to avoid the overestimation of the output Q value of the evaluation network, two independent evaluation networks Q1 and Q2 are used, and the one with the smaller output Q value is taken as the result Q value; in order to improve the stability of the network training process, the algorithm adopts a current value and target value dual network structure for the evaluation network and the policy network, and the current value and target value policy network are respectively denoted as μ and The current value and target value evaluation networks are respectively denoted as and Where k = 1, 2;

[0091] In the process of cluster swarm control, all agents share the same policy network, evaluation network, and experience pool D. Agent i uses local observation information o i Make action decisions a through the policy network μ i , and calculate the reward r, enter the next state s′, and the sample in the experience pool D is defined as follows:

[0092] tuple=[s,(o0,o1,…,o N-1 ), X a ,s′,(o0′,o1′,…,o N-1 ′), r]

[0093] The parameters of the evaluation network are denoted as By minimizing the time difference error Update as shown below:

[0094]

[0095] Among them, s is the global observation state, s′ represents the global state at the next moment, and o j and o′ j Respectively represent the current and next moment observation states of agent j, a j and a′ j They represent the action outputs of the current value and target value policy network respectively. The data are all sampled from the experience pool D. The variable γ is defined as:

[0096] γ=min{δ·n eps , γ max}

[0097] Among them, γ max represents the maximum value of γ, δ is the growth rate of γ and its value is non-negative, n epsis the current number of trainings; since the optimization target of the evaluation network takes γ as a parameter, and γ will gradually increase as the training progresses, the contribution of the Q value at the future moment to the target Q value at the current moment will also gradually increase, which effectively alleviates the problem of unstable data in the early stage of training, which is conducive to accelerating the convergence speed of the network and improving the stability of the training process;

[0098] Let the parameters of the policy network be θ μ ,θ μ The update depends on the Q value output by the current value evaluation network. The parameter optimization target gradient of the current value policy network is defined as follows:

[0099]

[0100] The parameters of the target value strategy network and the target value evaluation network are respectively denoted as and And the soft update method is used to update the parameters, which is defined as follows:

[0101]

[0102] Among them, τ Q With τ μ are the parameter update ratios of the evaluation network and the policy network respectively;

[0103] In order to improve the network's adaptability to dynamic environments, Gaussian noise G is added to the actions output by the policy network. v ~N(0,σ v 2 ), so the speed update formula of agent i can be changed to:

[0104] v i =υ i +a i +G v

[0105] Among them, the noise variance σ v As the training progresses, it continues to decrease, and the update formula is as follows:

[0106]

[0107] Among them, λ v is v The initial value of γ v is the attenuation coefficient, is the minimum value of variance;

[0108] In the above technical solution, step S7 includes:

[0109] Design the training process of the swarm control algorithm, and the total number of training rounds is recorded as N T, the experience pool capacity is recorded as C max , the training process includes the following steps:

[0110] Step 7.1: Initialize τ, r min , γ max Equal algorithm parameters, initialization size is C max Experience pool D;

[0111] Step 7.2: Create and initialize all policy networks and evaluation networks;

[0112] Step 7.3: Traverse all episodes from episode = 1 to N T ,implement:

[0113] Step 7.3.1, randomly initialize the positions and velocities of all agents;

[0114] Step 7.3.2: Calculate the initial global observation state s and local observation state o i ;

[0115] Step 7.3.3: If the current round has not ended, execute:

[0116] Step 7.3.3.1, agent i selects action a based on the policy network i =μ(o i );

[0117] Step 7.3.3.2, Update the action matrix X a , position p i and speed v i ;

[0118] Step 7.3.3.3: Calculate the global observation state s′ and local observation state o′ at the next moment i ;

[0119] Step 7.3.3.4, calculate the return r;

[0120] Step 7.3.3.5, construct a sample tuple and store it in the experience pool D;

[0121] Step 7.3.3.6: If the number of samples in the experience pool D is greater than batch_size, randomly select batch_size sample tuples from the experience pool D and update the current value network parameters. and θ μ , update the target value network parameters and Otherwise proceed to step 7.3.3.

[0122] Because the present invention adopts the above technical solution, it has the following beneficial effects:

[0123] The present invention realizes the cluster swarming control algorithm with the help of deep reinforcement learning technology, and uses graph neural network to extract the topological structure characteristics and observation information characteristics of the cluster, which effectively improves the convergence speed of the cluster swarming control algorithm and its adaptability to dynamic environments, while ensuring the stability of the algorithm under interference such as control noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] Figure 1 is a flow chart of the method of the present invention;

[0125] Figure 2 This is a schematic diagram of the strategy network and evaluation network model structure; DETAILED DESCRIPTION

[0126] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0127] like Figure 1 As shown, the present invention provides a reinforcement learning cluster swarming control method based on graph neural network, comprising the following steps:

[0128] Step S1, establishing a cluster swarming control model. After a limited number of swarming controls, the cluster should gradually form a stable topological structure and maintain a consistent speed. After stabilization, the system error should approach 0.

[0129] Step S2: The entire cluster system is regarded as an undirected graph G, whose adjacency matrix is ​​A. The crowding control error is introduced into the adjacency matrix A of the undirected graph G to obtain the weighted adjacency matrix Based on the weighted adjacency matrix The topological structure features of the cluster system are extracted by using a two-layer graph convolutional neural network (GCN) connected sequentially to obtain the final cluster topological structure feature representation H A ′;

[0130] Step S3: Determine the observation information feature representation method of the agent and construct the global and local observation information matrix X sg and X oi , using a two-layer graph convolutional neural network connected sequentially to respectively analyze the global and local observation information matrices X sg and X oi Perform feature extraction to obtain the global observation information matrix X sg The characteristic representation H sg ′ and the feature representation H of the local observation information matrix oi ′;

[0131] Step S4, design state space: based on the weighted adjacency matrix in step S2 Calculation method and the global and local observation information matrix X in step S3sg and X oi , respectively define the global observation state s and the local observation state o of agent i i ;

[0132] Design action space: select continuous space as the action space of the agent and determine the speed update method of agent i;

[0133] Design reward function: The principle is to keep the speed of agents consistent and avoid collisions during movement;

[0134] Step S5: Based on the graph convolutional neural network structure in step S3, the global observation state s and the local observation state o in step S4 are transformed into i As network input, we design the policy network and evaluation network models in the deep reinforcement learning algorithm respectively;

[0135] Step S6: Based on the policy network and evaluation network models in step S5, a reinforcement learning cluster swarming control algorithm framework is designed, and a parameter update method of the policy network and the evaluation network is given;

[0136] Step S7, design a training process for the cluster swarming control algorithm, based on the experience information generated by the continuous interaction between the intelligent agent and the environment, use the parameter updating method in step S6 to learn the network parameters, and after the training is completed, obtain an intelligent agent strategy network that can be used for cluster swarming control.

[0137] In the above technical solution, step S1 comprises:

[0138] A swarm control model is established. The swarm system consists of N agents, where the i-th agent is defined as agent i. The agents move in a two-dimensional plane and satisfy the following particle motion model:

[0139]

[0140] Among them, p i , i and u i ∈R 2 are the position, velocity, and acceleration control values ​​of agent i, respectively. Respectively represent p i , i Derivatives with respect to time;

[0141] At a distance less than the communication distance r c There are communication connections between agents, and the neighbor set of agent i is defined according to the connection relationship:

[0142] N i ={j∈V:||pj -p i ||≤r c , j≠i}

[0143] Where V is the set of all agents, ||·|| is the Euclidean norm, and neighboring agents can exchange information such as position and speed through communication;

[0144] Designate an agent in the cluster as the leader, denoted as leader, and the remaining N-1 agents as followers, denoted as followers. The swarming control model can be described as follows: followers follow the leader and avoid collisions during the movement, and eventually the entire cluster reaches a consistent speed and forms a stable topological structure;

[0145] The error of the swarming control system at time k is defined as follows:

[0146] e(k)=e lf (k)+e ff (k)

[0147] Among them, e lf (k) is the relative distance and speed error between the follower and the leader, defined as follows:

[0148]

[0149] in, is the expected distance between leader and follower, and are the position and velocity of follower i, p0 and v0 are the position and velocity of leader, φ l (·) is a convex function with a global minimum value of 0 and φ l (0) = 0;

[0150] e ff (k) is the error of the relative distance and speed between followers, which is defined as follows:

[0151]

[0152] in, is the expected distance between followers, N i is the set of neighboring agents of agent i, φ f (·) is a convex function with a global minimum value of 0 and φ f (0) = 0;

[0153] After a limited number of swarming controls, the cluster should gradually form a stable topology and maintain a consistent speed. After stabilization, the system error should approach 0:

[0154] lim k→∞ e(k)=0.

[0155] In the above technical solution, step S2 comprises:

[0156] Determine the method of expressing the topological structure characteristics of the cluster: the entire cluster system can be regarded as an undirected graph G = (V, E), the set y of all agents constitutes the vertex set of the undirected graph G, and the connection relationship between agents constitutes the edge set E of the graph. The adjacency matrix of the undirected graph G is denoted by A, which is an N×N binary symmetric square matrix. The element value 1 represents the existence of communication connection between agents, and the value 0 represents no communication connection. In order to better reflect the degree of association between agents in the cluster, the crowding control error is introduced into the adjacency matrix A of the undirected graph G, and the weighted adjacency matrix is ​​obtained:

[0157]

[0158] Among them, c norm is the regularization coefficient, ⊙ represents the Hadamard product, and the error weight matrix W loss The elements in W are defined as loss (i, j) = L ij , i, j = 0, 1, ... N, error L ij The definition is as follows:

[0159]

[0160] in, is a constant coefficient, the convex function φ l (·) and φ f (·) is defined as follows:

[0161] φ l (z)=min{(c n z) 4 u(-z)+(c p z) 3 u(z),c rmax}

[0162] φ f (z)=(c n z) 4 u(-z)

[0163] Among them, c n , c p is the adjustment coefficient, c rmax is the maximum error threshold, and the unit step function u(z) is defined as follows:

[0164]

[0165] The element values ​​in the weighted adjacency matrix The larger the value, the greater the relative speed or position error between agent i and agent j, which means that the swarming control strategy should pay more attention to agent j when adjusting the position or speed of agent i. Therefore, Effectively represents the topological structure information of the cluster;

[0166] Based on the weighted adjacency matrix The topological structure features of the cluster system are extracted using a two-layer graph convolutional neural network (GCN) connected sequentially:

[0167]

[0168]

[0169] Where W, W′ are the parameter matrices of the graph convolutional neural network layer, b, b′ are the bias vectors, σ(·) is the activation function, and H A It is the cluster topology structure feature output by the first layer GCN; based on H A , and then use a layer of GCN to further extract the high-dimensional feature information of the cluster topology structure to obtain the final cluster topology feature representation H A ′.

[0170] In the above technical solution, step S3 includes:

[0171] Determine the feature representation method of the agent's observation information, and use the speed and position information of neighbor agent i relative to the current agent i, as well as the category information of agent i (leader or follower in the cluster) to construct the observation information of agent i The definition is as follows:

[0172]

[0173] Among them, c represents the category of agent j, 0 represents follower, and 1 represents leader. In order to improve the stability of the neural network training process, the relative distance and relative speed between agents are further normalized to the interval (-1, 1), and the local observation information matrix of agent i can be obtained:

[0174]

[0175] Among them, c v is the maximum amplitude of the relative velocity;

[0176] Considering that each follower agent in the cluster system is isomorphic and has an equivalent relationship, and X oi will change due to the change of the order of agent numbers, so in order to satisfy X oi With respect to the invariance of the agent number, the row order of the observation information matrix needs to be adjusted as follows:

[0177] X oi :=X oi [P order ]

[0178] Among them, P order is the new numbering order of the agents in the cluster, which is obtained by arranging the relative distances between agents in ascending order:

[0179] P order =arg sort([||p i -p0||,||p i -p1||,…,||p i -p N ||])

[0180] In order to maintain the consistency of the order of agents, the weighted adjacency matrix in step S2 needs to be recalculated according to the new agent number.

[0181] Based on the sequentially adjusted local observation information matrix X of agent i oi , using a sequentially connected two-layer graph convolutional neural network to extract observation information features:

[0182] H oi =σ(X oi W+b)

[0183] H oi ′=σ(H oi W′+b′)

[0184] Among them, H oi is the observation information feature output by the first layer of GCN; in H oi On the basis of , another layer of GCN is used to further extract the high-dimensional features of local observation information to obtain the final local observation information feature representation H oi ′;

[0185] In addition, in order to distinguish the global observation information from the local observation information of the agent, the observation information with the leader as the observation center is regarded as the global observation information of the cluster system, that is, the global observation information matrix X sg It can be expressed as follows:

[0186]

[0187] Global observation information matrix X sg The characteristic representation H sg ′ is also combined with the local observation information feature H oi 'The same method obtains:

[0188] H sg =σ(X sg W+b)

[0189] H sg ′=σ(H sg W′+b′).

[0190] In the above technical solution, step S4 comprises:

[0191] Design state space: Based on the weighted adjacency matrix in step S2 And the global and local observation information matrix X in step S3 sg and X oi , respectively define the global observation state s and the local observation state o of agent i i :

[0192]

[0193]

[0194] in, and are the weighted adjacency matrices of leader and agent i respectively;

[0195] Design action space: Select continuous space as the action space of the agent, that is, directly define the continuous value output by the policy network of agent i as the acceleration of the motion control amount of the agent, that is,

[0196] u i =a i

[0197] Among them, u i is the acceleration control value of agent i, a i is the action output of agent i; thus the speed update method of agent i is defined as follows:

[0198] v i+1 =v i +d i

[0199] Design reward function: Based on the principle of keeping the speed of agents consistent and avoiding collisions during movement, design the following reward function:

[0200] r=max{-Lsym , r min}

[0201] Among them, r min <0 is the minimum return value, L sym is the control error loss of the entire cluster system, which is defined as follows:

[0202]

[0203] The first two items describe the errors of the relative distance and relative speed between the follower and the leader, respectively. The third and fourth items describe the errors of the relative distance and relative speed between the followers, respectively. By minimizing L sym , which can make all followers maintain the same movement speed and constant distance with the leader, forming the effect of followers following the leader, while maintaining a stable and convergent swarming topology; in the swarming process, the closer the positions between followers and between followers and leaders are to the ideal stable cluster structure, the lower the system control loss value L sys The closer it is to 0, the greater the return value r is;

[0204] In the above technical solution, step S5 comprises:

[0205] Design strategy network model: The strategy network of agent i is based on the local observation state o in step S4. i As input, the two-layer GCN in step S2 is used to extract the cluster topology structure features of the cluster system The two-layer GCN in step S3 is used to extract the local observation information feature representation H of agent i oi ′, the two features are concatenated after vectorization, and finally the action decision a is output through two layers of fully connected layer networks using tanh(·) as the activation function i ∈(-1, 1);

[0206] Design evaluation network model: The evaluation network is based on the global observation state s in step S4 and the action matrix X of the intelligent agent. a As input, the two-layer GCN in step S2 is used to extract the topological structure features of the cluster system. The two-layer GCN in step S3 is used to extract the global observation information feature representation H sg ′, use two layers of GCN to extract the action matrix X a Finally, the three features are concatenated after vectorization, and the state value Q is output through two layers of fully connected network using relu (·) as the activation function; the action matrix X a The definition is as follows:

[0207] X a =(a0, a1, ..., a N-1 ) T

[0208] Among them, a i The meaning is the agent output action in step S4.

[0209] In the above technical solution, step S6 comprises:

[0210] Design algorithm framework and network parameter updating method: The algorithm uses the evaluation network and policy network in step S5 to build a reinforcement learning cluster swarming control algorithm framework. In order to avoid the overestimation of the output Q value of the evaluation network, two independent evaluation networks Q1 and Q2 are used, and the one with the smaller output Q value is taken as the result Q value; in order to improve the stability of the network training process, the algorithm adopts a current value and target value dual network structure for the evaluation network and the policy network, and the current value and target value policy network are respectively denoted as μ and The current value and target value evaluation networks are respectively denoted as and Where k = 1, 2;

[0211] In the process of cluster swarm control, all agents share the same policy network, evaluation network, and experience pool D. Agent i uses local observation information o i Make action decisions a through the policy network μ i , and calculate the reward r, enter the next state s′, and the sample in the experience pool D is defined as follows:

[0212] tuple=[s,(o0,o1,…,o N-1 ), X a ,s′,(o0′,o1′,…,o N-1 ′), r]

[0213] The parameters of the evaluation network are denoted as By minimizing the time difference error Update as shown below:

[0214]

[0215] Among them, s is the global observation state, s′ represents the global state at the next moment, and o j and o′ j Respectively represent the current and next moment observation states of agent j, a j and a′ j They represent the action outputs of the current value and target value policy network respectively. The data are all sampled from the experience pool D. The variable γ is defined as:

[0216] γ=min{δ·n eps , γ max}

[0217] Among them, γ max represents the maximum value of γ, δ is the growth rate of γ and its value is non-negative, n eps is the current number of trainings; since the optimization target of the evaluation network takes γ as a parameter, and γ will gradually increase as the training progresses, the contribution of the Q value at the future moment to the target Q value at the current moment will also gradually increase, which effectively alleviates the problem of unstable data in the early stage of training, which is conducive to accelerating the convergence speed of the network and improving the stability of the training process;

[0218] Let the parameters of the policy network be θ μ ,θ μ The update depends on the Q value output by the current value evaluation network. The parameter optimization target gradient of the current value policy network is defined as follows:

[0219]

[0220] The parameters of the target value strategy network and the target value evaluation network are respectively denoted as and And the soft update method is used to update the parameters, which is defined as follows:

[0221]

[0222] Among them, τ Q With τ μ are the parameter update ratios of the evaluation network and the policy network respectively;

[0223] In order to improve the network's adaptability to dynamic environments, Gaussian noise G is added to the actions output by the policy network. v ~N(0,σ v 2 ), so the speed update formula of agent i can be changed to:

[0224] υ i =υ i +a i +G υ

[0225] Among them, the noise variance σ υ As the training progresses, it continues to decrease, and the update formula is as follows:

[0226]

[0227] Among them, λ υ is υ The initial value of γ υis the attenuation coefficient, is the minimum value of variance;

[0228] In the above technical solution, step S7 includes:

[0229] Design the training process of the swarm control algorithm, and the total number of training rounds is recorded as N T , the experience pool capacity is recorded as C max , the training process includes the following steps:

[0230] Step 7.1: Initialize τ, r min , γ max Equal algorithm parameters, initialization size is C max Experience pool D;

[0231] Step 7.2: Create and initialize all policy networks and evaluation networks;

[0232] Step 7.3: Traverse all episodes from episode = 1 to N T ,implement:

[0233] Step 7.3.1, randomly initialize the positions and velocities of all agents;

[0234] Step 7.3.2: Calculate the initial global observation state s and local observation state o i ;

[0235] Step 7.3.3: If the current round has not ended, execute:

[0236] Step 7.3.3.1, agent i selects action a based on the policy network i =μ( xi );

[0237] Step 7.3.3.2, Update the action matrix X a , position p i and speed i ;

[0238] Step 7.3.3.3: Calculate the global observation state s′ and local observation state o′ at the next moment i ;

[0239] Step 7.3.3.4, calculate the return r;

[0240] Step 7.3.3.5, construct a sample tuple and store it in the experience pool D;

[0241] Step 7.3.3.6: If the number of samples in the experience pool D is greater than batch_size, randomly select batch_size sample tuples from the experience pool D and update the current value network parameters. and θ μ , update the target value network parameters and Otherwise proceed to step 7.3.3.

[0242] The present invention realizes the cluster swarming control algorithm with the help of deep reinforcement learning technology, and uses graph neural network to extract the topological structure characteristics and observation information characteristics of the cluster, which effectively improves the convergence speed of the cluster swarming control algorithm and its adaptability to dynamic environments, while ensuring the stability of the algorithm under interference such as control noise.

[0243] The above is a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art do not deviate from the spirit and scope of the present invention, and should be within the scope of protection of the claims attached to the present invention.

Claims

1. A reinforcement learning cluster swarming control method based on graph neural network, characterized in that: The following steps are involved: Step S1, establishing a cluster swarming control model. After a limited number of swarming controls, the cluster should gradually form a stable topological structure and maintain a consistent speed. After stabilization, the system error should approach 0. Step S2: The entire cluster system is regarded as an undirected graph G, whose adjacency matrix is ​​A. The crowding control error is introduced into the adjacency matrix A of the undirected graph G to obtain the weighted adjacency matrix Based on the weighted adjacency matrix The topological structure features of the cluster system are extracted by using a two-layer graph convolutional neural network (GCN) connected sequentially to obtain the final cluster topological structure feature representation H A ′; Step S3: Determine the observation information feature representation method of the agent and construct the global and local observation information matrix X sg and X oi , using a two-layer graph convolutional neural network connected sequentially to respectively analyze the global and local observation information matrices X sg and X oi Perform feature extraction to obtain the global observation information matrix X sg The characteristic representation H sg ′ and the feature representation H of the local observation information matrix oi ′; Step S4, design state space: based on the weighted adjacency matrix in step S2 Calculation method and the global and local observation information matrix X in step S3 sg and X oi , respectively define the global observation state s and the local observation state o of agent i i ; Design action space: select continuous space as the action space of the agent and determine the speed update method of agent i; Design reward function: The principle is to keep the speed of agents consistent and avoid collisions during movement; Step S5: Based on the graph convolutional neural network structure in step S3, the global observation state s and the local observation state o in step S4 are transformed into i As network input, we design the policy network and evaluation network models in the deep reinforcement learning algorithm respectively; Step S6: Based on the policy network and evaluation network models in step S5, a reinforcement learning cluster swarming control algorithm framework is designed, and a parameter update method of the policy network and the evaluation network is given; Step S7, designing a training process for the cluster swarming control algorithm, based on the experience information generated by the continuous interaction between the intelligent agent and the environment, using the parameter updating method in step S6 to learn the network parameters, and after the training is completed, obtaining an intelligent agent strategy network that can be used for cluster swarming control; The step S6 comprises: Design algorithm framework and network parameter updating method: The algorithm uses the evaluation network and policy network in step S5 to build a reinforcement learning cluster swarming control algorithm framework. In order to avoid the overestimation of the output Q value of the evaluation network, two independent evaluation networks Q1 and Q2 are used, and the one with the smaller output Q value is taken as the result Q value; in order to improve the stability of the network training process, the algorithm adopts a current value and target value dual network structure for the evaluation network and the policy network, and the current value and target value policy network are respectively denoted as μ and The current value and target value evaluation networks are respectively denoted as and Where k = 1, 2; In the process of cluster swarming control, all agents share the same policy network, evaluation network, experience pool D, and agent based on local observation information o i Make action decisions a through the policy network μ i , and calculate the reward r, enter the next state s′, and the sample in the experience pool D is defined as follows: tuple=[s,(o0,o1,…,o N-1 ),X a ,s′,(o0′,o1′,…,o N-1 ′),r] The parameters of the evaluation network are denoted as By minimizing the time difference error Update as shown below: Among them, s is the global observation state, s′ represents the global state at the next moment, and o j and o′ j Respectively represent the current and next moment observation states of agent j, a j and a′ j They represent the action outputs of the current value and target value policy network respectively. The data are all sampled from the experience pool D. The variable γ is defined as: γ=min{δ·n eps ,c max } Among them, γ max represents the maximum value of γ, δ is the growth rate of γ and its value is non-negative, n eps is the current number of trainings; since the optimization target of the evaluation network takes γ as a parameter, and γ will gradually increase as the training progresses, the contribution of the Q value at the future moment to the target Q value at the current moment will also gradually increase, which effectively alleviates the problem of unstable data in the early stage of training, which is conducive to accelerating the convergence speed of the network and improving the stability of the training process; Let the parameters of the policy network be θ μ ,θ μ The update depends on the Q value output by the current value evaluation network. The parameter optimization target gradient of the current value policy network is defined as follows: The parameters of the target value strategy network and the target value evaluation network are respectively denoted as and And the soft update method is used to update the parameters, which is defined as follows: Among them, τ Q With τ μ are the parameter update ratios of the evaluation network and the policy network respectively; In order to improve the network's adaptability to dynamic environments, Gaussian noise G is added to the actions output by the policy network. v ~N(0,σ v 2 ), so the speed update formula of agent i can be changed to: v i =v i +a i +G v Among them, the noise variance σ v As the training progresses, it continues to decrease, and the update formula is as follows: Among them, λ υ is υ The initial value of γ v is the attenuation coefficient, is the minimum value of the variance.

2. The method for cluster swarming control based on reinforcement learning of graph neural network according to claim 1, characterized in that: The step S1 comprises: A swarm control model is established. The swarm system consists of N agents, where the i-th agent is defined as agenti. The agents move in a two-dimensional plane and satisfy the following particle motion model: Among them, p i , v i and u i ∈R 2 are the position, velocity, and acceleration control values ​​of agent i, respectively. Respectively represent p i , v i Derivatives with respect to time; At a distance less than the communication distance r c There are communication connections between agents, and the neighbor set of agent i is defined according to the connection relationship: N i ={j∈V:||p j -p i ||≤r c ,j≠i} Where V is the set of all agents, ||·|| is the Euclidean norm, and neighboring agents can exchange information such as position and speed through communication; Designate an agent in the cluster as the leader, denoted as leader, and the remaining N-1 agents as followers, denoted as followers. The swarming control model can be described as follows: followers follow the leader and avoid collisions during the movement, and eventually the entire cluster reaches a consistent speed and forms a stable topological structure; The error of the swarming control system at time k is defined as follows: e(k)=e lf (k)+e ff (k) Among them, e lf (k) is the relative distance and speed error between the follower and the leader, defined as follows: in, is the expected distance between leader and follower, and are the position and velocity of followeri, p0 and v0 are the position and velocity of leader, φ l (·) is a convex function with a global minimum value of 0 and φ l (0) = 0; e ff (k) is the error of the relative distance and speed between followers, which is defined as follows: in, is the expected distance between followers, N i is the set of neighboring agents of agent i, φ f (·) is a convex function with a global minimum value of 0 and φ f (0) = 0; After a limited number of swarming controls, the cluster should gradually form a stable topology and maintain a consistent speed. After stabilization, the system error should approach 0: glue k→∞ e(k)=0.

3. The method for cluster swarming control based on reinforcement learning of graph neural network according to claim 2, characterized in that: The step S2 comprises: Determine the topological structure feature representation method of the cluster: the entire cluster system can be regarded as an undirected graph G = (V, E), the set V of all agents constitutes the vertex set of the undirected graph G, and the connection relationship between agents constitutes the edge set E of the graph. The adjacency matrix of the undirected graph G is denoted by A, which is an N×N binary symmetric square matrix. The element value 1 represents the existence of communication connection between agents, and the value 0 represents no communication connection. In order to better reflect the degree of association between agents in the cluster, the crowding control error is introduced into the adjacency matrix A of the undirected graph G, and the weighted adjacency matrix is ​​obtained: Among them, c norm is the regularization coefficient, ⊙ represents the Hadamard product, and the error weight matrix W loss The elements in W are defined as loss (i, j) = L ij , i, j = 0, 1, ... N, error L ij The definition is as follows: in, is a constant coefficient, the convex function φ l (·) and φ f (·) is defined as follows: φ l (z)=min{(c n With) 4 u(-z)+(c p With) 3 u(z),c rmax } φ f (z)=(c n from) 4 in (-z) Among them, c n , c p is the adjustment coefficient, c rmax is the maximum error threshold, and the unit step function u(z) is defined as follows: The element values ​​in the weighted adjacency matrix The larger the value, the greater the relative speed or position error between agent i and agent j, which means that the swarming control strategy should pay more attention to agent j when adjusting the position or speed of agent i. Therefore, Effectively represents the topological structure information of the cluster; Based on the weighted adjacency matrix The topological structure features of the cluster system are extracted using a two-layer graph convolutional neural network (GCN) connected sequentially: Where W, W′ are the parameter matrices of the graph convolutional neural network layer, b, b′ are the bias vectors, σ(·) is the activation function, and H A It is the cluster topology structure feature output by the first layer GCN; based on H A , and then use a layer of GCN to further extract the high-dimensional feature information of the cluster topology structure to obtain the final cluster topology feature representation H A ′.

4. The method for cluster swarming control based on reinforcement learning of graph neural network according to claim 3 is characterized by: The step S3 comprises: Determine the method of representing the observation information features of the agent, and use the speed and position information of neighbor agent j relative to the current agent i, as well as the category information of agent j (leader or follower in the cluster) to construct the observation information of agent i The definition is as follows: Among them, c j represents the category of agent j, 0 represents follower, and 1 represents leader. To improve the stability of the neural network training process, the relative distance and relative speed between agents are further normalized to the interval (-1, 1), and the local observation information matrix of agent i can be obtained: Among them, c v is the maximum amplitude of the relative velocity; Considering that each follower agent in the cluster system is isomorphic and has an equivalent relationship, and X oi will change due to the change of the order of agent numbers, so in order to satisfy X oi With respect to the invariance of the agent number, the row order of the observation information matrix needs to be adjusted as follows: X oi :=X oi [P order ] Among them, P order is the new numbering order of the agents in the cluster, which is obtained by arranging the relative distances between agents in ascending order: P order =argsort([||p i -p0||,||p i -p1||,…,||p i -p N ||]) In order to maintain the consistency of the order of agents, the weighted adjacency matrix in step S2 needs to be recalculated according to the new agent number. Based on the order-adjusted local observation information matrix X of agent i oi , using a sequentially connected two-layer graph convolutional neural network to extract observation information features: H oi =σ(X oi W+b) H oi ′=σ(H oi W′+b′) Among them, H oi is the observation information feature output by the first layer of GCN; in H oi On the basis of , another layer of GCN is used to further extract the high-dimensional features of local observation information to obtain the final local observation information feature representation H oi ′; In addition, in order to distinguish the global observation information from the local observation information of the agent, the observation information with the leader as the observation center is regarded as the global observation information of the cluster system, that is, the global observation information matrix X sg It can be expressed as follows: Global observation information matrix X sg The characteristic representation H sg ′ is also combined with the local observation information feature H oi 'The same method obtains: H sg =σ(X sg W+b) H sg ′=σ(H sg W′+b′)。 5. The method for cluster swarming control based on reinforcement learning of graph neural network according to claim 4, characterized in that: The step S4 comprises: Design state space: Based on the weighted adjacency matrix in step S2 And the global and local observation information matrix X in step S3 sg and X oi , respectively define the global observation state s and the local observation state o of agent i i : in, and are the weighted adjacency matrices of leader and agent i respectively; Design action space: Select continuous space as the action space of the agent, that is, directly define the continuous value output by the policy network of agent i as the acceleration of the motion control amount of the agent, that is, u i =a i Among them, u i is the acceleration control value of agent i, a i is the action output of agent i; thus the speed update method of agent i is defined as follows: in i+1 =in i +a i Design reward function: Based on the principle of keeping the speed of agents consistent and avoiding collisions during movement, design the following reward function: r=max{-L sym ,r min } Among them, r min <0 is the minimum return value, L sym is the control error loss of the entire cluster system, which is defined as follows: The first two items describe the errors of the relative distance and relative speed between the follower and the leader, respectively. The third and fourth items describe the errors of the relative distance and relative speed between the followers, respectively. By minimizing L sym , which can make all followers maintain the same movement speed and constant distance with the leader, forming the effect of followers following the leader, while maintaining a stable and convergent swarming topology; in the swarming process, the closer the positions between followers and between followers and leaders are to the ideal stable cluster structure, the lower the system control loss value L sys The closer it is to 0, the greater the reward value r is.

6. The method for cluster swarming control based on reinforcement learning of graph neural network according to claim 1, characterized in that: The step S5 comprises: Design strategy network model: The strategy network of agent i is based on the local observation state o in step S4. i As input, the two-layer GCN in step S2 is used to extract the cluster topology structure features of the cluster system The two-layer GCN in step S3 is used to extract the local observation information feature representation H of agent i oi ′, the two features are concatenated after vectorization, and finally the action decision a is output through two layers of fully connected layer networks using tanh(·) as the activation function i ∈(-1, 1); Design evaluation network model: The evaluation network is based on the global observation state s in step S4 and the action matrix X of the intelligent agent. a As input, the two-layer GCN in step S2 is used to extract the topological structure features of the cluster system. The two-layer GCN in step S3 is used to extract the global observation information feature representation H sg ′, use two layers of GCN to extract the action matrix X a Finally, the three features are concatenated after vectorization, and the state value Q is output through two layers of fully connected network using relu (·) as the activation function; the action matrix X a The definition is as follows: X a =(a0,a1,…,a N-1 ) T Among them, a i The meaning is the agent output action in step S4.

7. The method for cluster swarming control based on reinforcement learning of graph neural network according to claim 1, characterized in that: The step S7 comprises: Design the training process of the swarm control algorithm, and the total number of training rounds is recorded as N T , the experience pool capacity is recorded as C max , the training process includes the following steps: Step 7.1: Initialize τ, r min , γ max Equal algorithm parameters, initialization size is C max Experience pool D; Step 7.2: Create and initialize all policy networks and evaluation networks; Step 7.3: Traverse all episodes from episode = 1 to N T ,implement: Step 7.3.1, randomly initialize the positions and velocities of all agents; Step 7.3.2: Calculate the initial global observation state s and local observation state o i ; Step 7.3.3: If the current round has not ended, execute: Step 7.3.3.1, agent i selects action a based on the policy network i =μ(o i ); Step 7.3.3.2, Update the action matrix X a , position p i and speed v i ; Step 7.3.3.3: Calculate the global observation state s′ and local observation state o′ at the next moment i ; Step 7.3.3.4, calculate the return r; Step 7.3.3.5, construct a sample tuple and store it in the experience pool D; Step 7.3.3.6: If the number of samples in the experience pool D is greater than batch_size, randomly select batch_size sample tuples from the experience pool D and update the current value network parameters. and θ μ , update the target value network parameters and Otherwise proceed to step 7.3.3.

Images