Complex motion body system cooperative control method based on decentration networked multi-agent reinforcement learning

Abstract

The invention designs a complex motion body system cooperative control method based on decentration networked multi-agent reinforcement learning, and motion body cooperation is promoted in a decentration scene. Firstly, an entity graph is adopted to model a complex moving body system for representing spatial correlation between entities, then an information aggregation strategy is designed, a graph neural network and an attention mechanism are utilized to aggregate messages of neighbors and pay attention to important neighbor information, and in addition, a graph information bottleneck is utilized to reduce the influence of redundant information on optimal action selection. According to the method, the problem that in a scene in which a global state cannot be obtained, traditional multi-agent reinforcement learning cannot promote collaboration among the moving bodies due to limited observation information can be solved, the collaboration performance of the system is improved, and an effective method is provided for the field of collaboration control of various unmanned moving body systems.

Description

Technical Field

[0001] Based on the multi-agent particle environment, the present invention establishes a complex motion body collaborative control method based on decentralized networked multi-agent reinforcement learning, uses graph neural networks to aggregate node features with spatial correlation of agent neighbors, and uses graph information bottlenecks to reduce the impact of redundant information on optimal action selection, and finally completes the collaborative control of complex motion body systems. This complex motion body system collaborative control method based on decentralized networked multi-agent reinforcement learning can solve the problem that traditional multi-agent reinforcement learning cannot promote collaboration between motion bodies due to limited observation information in scenarios where the global state cannot be obtained in the actual unmanned motion body collaborative control process, improves the system's collaborative performance, and provides technical support for various unmanned motion body system collaborative control fields. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, the application scope of unmanned moving body systems is expanding. These systems not only play a key role in the military and aviation fields, but are also widely used in civilian fields such as autonomous driving, intelligent warehousing and medical services. Traditional cooperative control methods usually rely on accurate dynamic models, which limits their application in complex unmanned moving body systems. Therefore, the development of model-free adaptive methods has become the key to achieving cooperative control of complex moving body systems. The moving body cooperative control method based on multi-agent reinforcement learning effectively promotes the cooperation between moving bodies and significantly improves the cooperative control performance. However, in many actual moving body scenarios, the goal of each moving body is usually to complete its own task by sharing as little information as possible. In a decentralized environment, communication limitations or the problem of no central controller make it difficult to promote cooperation between agents even if there is a global goal. Therefore, in practical applications, achieving efficient cooperative control of complex unmanned moving body systems in decentralized scenarios has significant economic and social benefits and shows broad application prospects.

[0003] It is difficult to obtain dynamic models based on traditional methods, and the accuracy is low. With the development of artificial intelligence technology in my country, the application of unmanned moving bodies is becoming more and more extensive. Traditional methods have been unable to achieve accurate cooperative control in complex environments. In order to solve the problem of cooperative control in complex scenarios, some moving body control methods based on reinforcement learning have been proposed. However, when applied to solve practical problems, especially in complex moving body systems, reinforcement learning still faces many challenges. The basic idea of ​​​​cooperation is to coordinate and communicate between moving bodies to effectively complete tasks. There are non-stationarity problems in the actual environment, that is, the actions of each moving body affect the decisions of adjacent moving bodies, which limits the direct applicability of single-agent reinforcement learning technology in the field of complex multi-unmanned moving bodies. In contrast, multi-agent reinforcement learning enables moving bodies to interact and effectively solves the non-stationarity of the environment. The effectiveness of traditional multi-agent reinforcement learning algorithms depends on the amount of information contained in the input state of the neural network. For most multi-agent reinforcement learning algorithms, access to the global state of the system is usually crucial, but in many actual complex moving body systems, there are problems such as no central controller or network communication, which makes sharing the global state usually infeasible. Therefore, how to use limited local environmental information to promote cooperation between moving bodies and effectively perform collision-free collaborative control in a decentralized scenario has become an important research topic in the field of unmanned moving body systems and has important practical significance.

[0004] The present invention designs a complex motion body collaborative control method based on decentralized networked multi-agent reinforcement learning to promote motion body cooperation in a decentralized scenario. First, an entity graph is used to model a complex motion body system to represent the spatial correlation between entities, and then an information aggregation strategy is designed to use graph neural networks and attention mechanisms to aggregate neighbor messages and focus on important neighbor information. In addition, the graph information bottleneck is used to reduce the impact of redundant information on the optimal action selection. This method can solve the problem that traditional multi-agent reinforcement learning cannot promote collaboration between motion bodies due to limited observation information in scenarios where the global state cannot be obtained, improves the system's collaborative performance, and provides an effective method for the field of collaborative control of various unmanned motion body systems. Summary of the invention

[0005] The present invention proposes a complex moving body collaborative control method based on decentralized networked multi-agent reinforcement learning. The method first designs a moving body collaborative control task based on a multi-agent particle environment, and obtains observation information data by interacting with the task environment; secondly, the data is transmitted to a server and converted into a graph structure according to the position of each entity in the global coordinate system to characterize the data features; then, a graph neural network is used to aggregate the node features of each moving body and its neighboring entities, thereby training a decentralized networked multi-agent reinforcement learning algorithm; finally, each moving body uses the trained model to independently make decisions through odometer positioning and data collected by an inertial measurement unit to quickly reach the desired position and avoid collision.

[0006] Odometer is a method that uses mobile sensor data to estimate the change in the position of an object over time. It is widely used in many moving body systems to calculate the distance the moving body has moved relative to its initial position. Inertial measurement units are mainly used to detect the acceleration of moving bodies. They are usually equipped with accelerometers and gyroscopes to measure the attitude angle and acceleration of objects on one, two or three axes.

[0007] The present invention adopts the following technical solutions and implementation steps:

[0008] A method for coordinated control of a complex motion system based on decentralized networked multi-agent reinforcement learning, characterized by designing a task environment and a reward function, characterizing node features, performing information aggregation, training algorithms, and using decentralized networked multi-agent reinforcement learning to achieve coordinated motion control, including the following steps:

[0009] (1) Designing the task environment and reward function

[0010] A 2D multi-agent particle environment is used as the basis. The maximum number of time steps per round in this environment is T = 25. The area of ​​the environment adopts the default value of the multi-agent particle environment, which is 4×4. The number of moving bodies is N, the number of landmarks is D, and the number of obstacles is O. i represents the number of any moving body, i∈{1,...,N}; t represents the current time of the environment, t∈{1,...,T}; the i-th moving body A i The reward function r at time t during the movement i,t Designed for:

[0011]

[0012] Among them, pos i,t and Respectively represent A i The position at time t and the position of the target;

[0013] (2) Characterizing node features

[0014] The moving body, obstacles and landmarks are collectively referred to as entities e m , where m is the number of any entity, m∈{1,...,N+D+O}; the set {1,...,i-1,i+1,...,N+D+O} is used to represent all entities except A i The ID of any entity other than mn,t ∈R represents any two entities e m and e n The Euclidean distance between them, where m≠n, n∈{1,...,N+D+O} is the number of any entity except m; the definition is located in A i The set of entities within the perception radius ρ is N i,t , call these entities A i Neighbors, use e j Represents any neighbor, where j is the neighbor's number, and V i,t = {N i,t ∪A i} means A i and the set of its neighbors; using graph G t (V,E t ) represents a complex moving body system, where V = {e 1 ,...,e N+D+O} represents the set of all entities, Represents the set of edges at time t, using A i and e j The Euclidean distance d between ij,t As the feature of the edge; A i At time t, a graph network g is formed. i,t ∈G t , define A i At time t, the position, velocity and position of the target relative to itself in the two-dimensional coordinate system are pos i,t ∈R 2 ,vel i,t ∈R 2 as well as Where R represents a real number;

[0015] Use POS i,t ,vel i,t ,and Indicates observation information definition and e j Relative to A i The position, velocity and e j Target relative to A i position; if e j is an obstacle or target, set use Indicates g i,t The node features of each neighbor entity on the above, where et(j) is the entity type, defined as et(j)∈{"agent":0,"landmark":1,"obstacle":2}, the string in {·} represents the text, the dictionaries agent, landmark and obstacle represent the moving body, landmark and obstacle respectively, and the numbers 0, 1 and 2 represent the index of the dictionary;

[0016] (3) Information aggregation

[0017] Encode et(j) using an embedding layer with a dictionary size of 3 and an output dimension of 2 to obtain the encoded entity type et ij,t , splicing et ij,t and d ij,t Get vector in Represents a vector concatenation operation; Input to a linear layer with an input dimension of 9 and an output dimension of 16 for feature extraction, thereby obtaining the node features input to the graph neural network A three-layer graph neural network is used as the information aggregation module. Each layer is numbered l∈{1,...,L}, the maximum number of layers is L=3, and the input layer dimension of the graph neural network is set to 16;

[0018] ① On the first layer of the graph neural network, use

[0019]

[0020] Computing the Attention Map Where u and v are g respectively. i,t The numbers of any moving body and its neighbors on the u,t , N u,t Representative A u A collection of neighbors; is the node feature of the moving body with arbitrary number u in the l-1 layer, is the node feature of any neighbor numbered v in the l-1th layer; c is the dimension of the output of each layer of the graph neural network, when l = 1 or 3, c = 16, when l = 2, c = 32; W Q ∈R 16×c·h is the learnable weight matrix of the query, A learnable weight matrix with key ; using

[0021]

[0022] Calculate the moving body A with any number u uAny neighbor entity e with the number v v The attention coefficient between The number of attention heads is set to 3; using

[0023]

[0024] Calculate the l=1th layer e v Pass to A u News in, and The l=1th layer A u and e v The node characteristics, W V ∈R 16×c·h A learnable weight matrix with values ​​of

[0025]

[0026] Calculate A u Aggregated information, where W x ∈R 16×c·h ;

[0027] ② On the second layer of the graph neural network, use equations (2) and (3) to calculate the attention coefficient between entities, and then use

[0028]

[0029] Structural sampling in the information bottleneck of computational graphs

[0030] A u The joint expression of the attention coefficients is Among them, |N u,t | stands for N u,t The number of neighbors contained in ; for each attention head, generate |N u,t | independent samples from a (0,1) uniform distribution use

[0031]

[0032] Compute the Gumbel distribution use

[0033]

[0034] Calculate the reparameterized attention coefficient Among them, tem is the temperature parameter, and its value is 0.1; Finally, use formula (4) to calculate

[0035] Calculate the mean of a Gaussian distribution and variance

[0036]

[0037] Among them, [·,·] represents the message vector Extract the specific fragments in the brackets; generate a Gaussian distribution using the mean and variance, and obtain the actual transmitted message by sampling from the Gaussian distribution; Add 1 dimension to the end of

[0038]

[0039] Calculate Gaussian distribution The log probability of use

[0040]

[0041] Calculate the scaling parameter list after the Softplus activation function Among them, σ u,z ∈R c×z is the list of scaling parameters for the z-th Gaussian distribution in the Gaussian mixture model, initialized at the beginning of training, is u,z The learnable parameters after the activation function, z is the number of any Gaussian distribution in the Gaussian mixture model, z∈{1,...,Z}, Z is the maximum number of Gaussian distributions, set to 100; using

[0042]

[0043] Calculate the value of any Gaussian distribution z The log probability of Among them, the weighted logarithmic probability μ u,z ∈R c×Z is the learnable parameter matrix, which is also initialized at the beginning of training; using

[0044]

[0045] Calculate the Gaussian mixture model The log probability of use

[0046]

[0047] Node feature sampling in computational graph information bottleneck Finally, use formula (5) to calculate the output of the second layer;

[0048] ③ On the third layer of the graph neural network, first, use equations (2) and (3) to calculate the attention coefficient between entities, and use equations (4) and (6) to obtain and Secondly, use formula (5) to obtain the output of the graph neural network module; then, from the node data Select A i Node data Finally, get As input to the actor network, and compute the input to the critic network:

[0049]

[0050] ④ For all moving bodies A 1 ,...,A N Use operations ① to ③ to obtain the actor and critic network inputs of each moving body, save the input and output data of the graph neural network and the actor-critic network into training batches, and prepare for algorithm training;

[0051] (4) Training decentralized networked multi-agent reinforcement learning algorithms

[0052] ① Set algorithm training parameters: the maximum number of training steps is 2×10 6 , the number of training times of the proximal strategy optimization algorithm is 10, the number of parallel environments is 128, the size of the training batch is B = 25 × 128 = 3200, ρ = 1, N = D = O = 3;

[0053] ② Set the hyperparameters of the actor-critic network: 1 layer of multilayer perceptron, with an input layer dimension of 22, a hidden layer dimension of 64, and an output layer dimension of 64; 1 layer of gated recurrent unit, with an input layer, hidden layer, and output layer dimension of 64; the fully connected layer input dimension of the actor network is 64, and the output dimension is 5; the fully connected layer input dimension of the critic network is 64, and the output dimension is 1; the activation function is the ReLU function, and the learning rate is 7×10 -4 ;

[0054] ③ Adoption

[0055]

[0056] A i The actor network is trained; where the clip(·,·,·) function limits the data of the first item to the range of the last two items, τ is the number of each group of data in the batch, and θ i are the parameters of the actor network, is the policy update range limit parameter, κ=0.01 is the policy entropy coefficient, β 1 =0.001 is the graph information bottleneck optimization coefficient of the actor network, is the action conditional probability distribution output by the actor network, θ i Actor network output P before update i,τ for The probability of each item in and are the structure sampling and node feature sampling data stored in the τth group, A i,τ is the advantage function, and the calculation formula is

[0057]

[0058] Among them, λ = 0.95 is the generalized advantage estimator coefficient, γ = 0.99 is the discount factor, are the parameters of the critic network, Store the output of the critic network for the τth group, is the output of the τ+1th group of critic networks, r i,τ is the reward for storing the τth group of data in the training batch;

[0059]

[0060] A i The critic network is trained;

[0061]

[0062] A i The expected reward of the round starting from time t, β 2 =0.001 is the graph information bottleneck optimization coefficient of the critic network, for The output of the critic network before updating, ξ = 0.2 is the parameter limiting the update range of the value function;

[0063] ④ Use the training parameters of ① and ② and the optimization target of ③ to train the algorithm. After the training reaches the maximum number of steps, terminate the training, save the weight parameters of the last training, and complete the model training;

[0064] (5) Using decentralized networked multi-agent reinforcement learning to achieve collaborative control

[0065] ① Use the hyperparameters and loss function of (4) to train the decentralized networked multi-agent reinforcement learning algorithm, save the round reward, and output the reward function curve;

[0066] ② After completing the algorithm training, save the trained model and apply the model to the collaborative control task environment to complete the collaborative control of complex moving body systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1This is a reward curve diagram for the coordinated control of complex motion body systems according to the present invention.

[0068] Figure 2 This is a control effect diagram of the present invention for the coordinated control of complex motion body systems DETAILED DESCRIPTION

[0069] A method for coordinated control of a complex motion system based on decentralized networked multi-agent reinforcement learning, characterized by designing a task environment and a reward function, characterizing node features, performing information aggregation, training algorithms, and using decentralized networked multi-agent reinforcement learning to achieve coordinated motion control, including the following steps:

[0070] (1) Designing the task environment and reward function

[0071] A 2D multi-agent particle environment is used as the basis. The maximum number of time steps per round in this environment is T = 25. The area of ​​the environment adopts the default value of the multi-agent particle environment, which is 4×4. The number of moving bodies is N, the number of landmarks is D, and the number of obstacles is O. i represents the number of any moving body, i∈{1,...,N}; t represents the current time of the environment, t∈{1,...,T}; the i-th moving body A i The reward function r at time t during the movement i,t Designed for:

[0072]

[0073] Among them, pos i,t and Respectively represent A i The position at time t and the position of the target;

[0074] (2) Characterizing node features

[0075] The moving body, obstacles and landmarks are collectively referred to as entities e m , where m is the number of any entity, m∈{1,...,N+D+O}; the set {1,...,i-1,i+1,...,N+D+O} is used to represent all entities except A i The ID of any entity other than mn,t ∈R represents any two entities e m and e n The Euclidean distance between them, where m≠n, n∈{1,...,N+D+O} is the number of any entity except m; the definition is located in A i The set of entities within the perception radius ρ is N i,t , call these entities A i Neighbors, use e j Represents any neighbor, where j is the neighbor's number, and Vi,t = {N i,t ∪A i} means A i and the set of its neighbors; using graph G t (V,E t ) represents a complex moving body system, where V = {e 1 ,...,e N+D+O} represents the set of all entities, Represents the set of edges at time t, using A i and e j The Euclidean distance d between ij,t As the feature of the edge; A i At time t, a graph network g is formed. i,t ∈G t , define A i At time t, the position, velocity and position of the target relative to itself in the two-dimensional coordinate system are pos i,t ∈R 2 ,vel i,t ∈R 2 as well as Where R represents a real number;

[0076] Use POS i,t ,vel i,t ,and Indicates observation information definition and e j Relative to A i The position, velocity and e j Target relative to A i position; if e j is an obstacle or target, set use Indicates g i,t The node features of each neighbor entity on the above, where et(j) is the entity type, defined as et(j)∈{"agent":0,"landmark":1,"obstacle":2}, the string in {·} represents the text, the dictionaries agent, landmark and obstacle represent the moving body, landmark and obstacle respectively, and the numbers 0, 1 and 2 represent the index of the dictionary;

[0077] (3) Information aggregation

[0078] Encode et(j) using an embedding layer with a dictionary size of 3 and an output dimension of 2 to obtain the encoded entity type et ij,t , splicing et ij,t and dij,t Get vector in Represents a vector concatenation operation; Input to a linear layer with an input dimension of 9 and an output dimension of 16 for feature extraction, thereby obtaining the node features input to the graph neural network A three-layer graph neural network is used as the information aggregation module. Each layer is numbered l∈{1,...,L}, the maximum number of layers is L=3, and the input layer dimension of the graph neural network is set to 16;

[0079] ① On the first layer of the graph neural network, use

[0080]

[0081] Computing the Attention Map Where u and v are g respectively. i,t The numbers of any moving body and its neighbors on the u,t , N u,t Representative A u A collection of neighbors; is the node feature of the moving body with arbitrary number u in the l-1 layer, is the node feature of any neighbor numbered v in the l-1th layer; c is the dimension of the output of each layer of the graph neural network, when l = 1 or 3, c = 16, when l = 2, c = 32; W Q ∈R 16×c·h is the learnable weight matrix of the query, A learnable weight matrix with key ; using

[0082]

[0083] Calculate the moving body A with any number u u Any neighbor entity e with the number v v The attention coefficient between The number of attention heads is set to 3; using

[0084]

[0085] Calculate the l=1th layer e v Pass to A u News in, and The l=1th layer A u and e v The node characteristics, W V ∈R 16×c·h A learnable weight matrix with values ​​of

[0086]

[0087] Calculate A u Aggregated information, where W x ∈R 16×c·h ;

[0088] ② On the second layer of the graph neural network, use equations (2) and (3) to calculate the attention coefficient between entities, and then use

[0089]

[0090] Structural sampling in the information bottleneck of computational graphs

[0091] A u The joint expression of the attention coefficients is Among them, |N u,t | stands for N u,t The number of neighbors contained in ; for each attention head, generate |N u,t | independent samples from a (0,1) uniform distribution use

[0092]

[0093] Compute the Gumbel distribution use

[0094]

[0095] Calculate the reparameterized attention coefficient Among them, tem is the temperature parameter, and its value is 0.1; Finally, use formula (4) to calculate

[0096] Calculate the mean of a Gaussian distribution and variance

[0097]

[0098] Among them, [·,·] represents the message vector Extract the specific fragments in the brackets; generate a Gaussian distribution using the mean and variance, and obtain the actual transmitted message by sampling from the Gaussian distribution; Add 1 dimension to the end of

[0099]

[0100] Calculate Gaussian distribution The log probability of use

[0101]

[0102] Calculate the scaling parameter list after the Softplus activation function Among them, σ u,z ∈R c×z is the list of scaling parameters for the z-th Gaussian distribution in the Gaussian mixture model, initialized at the beginning of training, is u,z The learnable parameters after the activation function, z is the number of any Gaussian distribution in the Gaussian mixture model, z∈{1,...,Z}, Z is the maximum number of Gaussian distributions, set to 100; using

[0103]

[0104] Calculate the value of any Gaussian distribution z The log probability of Among them, the weighted logarithmic probability μ u,z ∈R c×Z is the learnable parameter matrix, which is also initialized at the beginning of training; using

[0105]

[0106] Calculate the Gaussian mixture model The log probability of use

[0107]

[0108] Node feature sampling in computational graph information bottleneck Finally, use formula (5) to calculate the output of the second layer;

[0109] ③ On the third layer of the graph neural network, first, use equations (2) and (3) to calculate the attention coefficient between entities, and use equations (4) and (6) to obtain and Secondly, use formula (5) to obtain the output of the graph neural network module; then, from the node data Select A i Node data Finally, get As input to the actor network, and compute the input to the critic network:

[0110]

[0111] ④ For all moving bodies A 1 ,...,A NUse operations ① to ③ to obtain the actor and critic network inputs of each moving body, save the input and output data of the graph neural network and the actor-critic network into training batches, and prepare for algorithm training;

[0112] (4) Training decentralized networked multi-agent reinforcement learning algorithms

[0113] ① Set algorithm training parameters: the maximum number of training steps is 2×10 6 , the number of training times of the proximal strategy optimization algorithm is 10, the number of parallel environments is 128, the size of the training batch is B = 25 × 128 = 3200, ρ = 1, N = D = O = 3;

[0114] ② Set the hyperparameters of the actor-critic network: 1 layer of multilayer perceptron, with an input layer dimension of 22, a hidden layer dimension of 64, and an output layer dimension of 64; 1 layer of gated recurrent unit, with an input layer, hidden layer, and output layer dimension of 64; the fully connected layer input dimension of the actor network is 64, and the output dimension is 5; the fully connected layer input dimension of the critic network is 64, and the output dimension is 1; the activation function is the ReLU function, and the learning rate is 7×10 -4 ;

[0115] ③ Adoption

[0116]

[0117] A i The actor network is trained; where the clip(·,·,·) function limits the data of the first item to the range of the last two items, τ is the number of each group of data in the batch, and θ i are the parameters of the actor network, is the policy update range limit parameter, κ=0.01 is the policy entropy coefficient, β 1 =0.001 is the graph information bottleneck optimization coefficient of the actor network, is the action conditional probability distribution output by the actor network, θ i Actor network output P before update i,τ for The probability of each item in and are the structure sampling and node feature sampling data stored in the τth group, A i,τ is the advantage function, and the calculation formula is

[0118]

[0119] Among them, λ = 0.95 is the generalized advantage estimator coefficient, γ = 0.99 is the discount factor, are the parameters of the critic network, Store the output of the critic network for the τth group, is the output of the τ+1th group of critic networks, r i,τ is the reward for storing the τth group of data in the training batch;

[0120]

[0121] A i The critic network is trained;

[0122]

[0123] A i The expected reward of the round starting from time t, β 2 =0.001 is the graph information bottleneck optimization coefficient of the critic network, for The output of the critic network before updating, ξ = 0.2 is the parameter limiting the update range of the value function;

[0124] ④ Use the training parameters of ① and ② and the optimization target of ③ to train the algorithm. After the training reaches the maximum number of steps, terminate the training, save the weight parameters of the last training, and complete the model training;

[0125] (5) Using decentralized networked multi-agent reinforcement learning to achieve collaborative control

[0126] ① Use the hyperparameters and loss function of (4) to train the decentralized networked multi-agent reinforcement learning algorithm, save the round reward, and output the reward function curve;

[0127] ② After completing the algorithm training, save the trained model and apply the model to the collaborative control task environment to complete the collaborative control of complex moving body systems.

Claims

1. A complex motion system collaborative control method based on decentralized networked multi-agent reinforcement learning, characterized in that: Design the task environment and reward function, characterize node features, aggregate information, train algorithms, and use decentralized networked multi-agent reinforcement learning to achieve collaborative motion control, including the following steps: (1) Designing the task environment and reward function A 2D multi-agent particle environment is used as the basis. The maximum number of time steps per round in this environment is T = 25. The area of ​​the environment adopts the default value of the multi-agent particle environment, which is 4×4. The number of moving bodies is N, the number of landmarks is D, and the number of obstacles is O. i represents the number of any moving body, i∈{1,...,N}; t represents the current time of the environment, t∈{1,...,T}; the i-th moving body A i The reward function r at time t during the movement i,t Designed for: Among them, pos i,t and Respectively represent A i The position at time t and the position of the target; (2) Characterizing node features The moving body, obstacles and landmarks are collectively referred to as entities e m , where m is the number of any entity, m∈{1,...,N+D+O}; the set {1,...,i-1,i+1,...,N+D+O} is used to represent all entities except A i The ID of any entity other than mn ,t∈R represents any two entities e m and e n The Euclidean distance between them, where m≠n, n∈{1,...,N+D+O} is the number of any entity except m; the definition is located in A i The set of entities within the perception radius ρ is N i,t , call these entities A i Neighbors, use e j Represents any neighbor, where j is the neighbor's number, and V i,t = {N i,t ∪A i } means A i and the set of its neighbors; using graph G t (V,E t ) represents a complex moving body system, where V = {e1,...,e N+D+O } represents the set of all entities, Represents the set of edges at time t, using A i and e j The Euclidean distance d between ij,t As the feature of the edge; A i At time t, a graph network g is formed. i,t ∈G t , define A i At time t, the position, velocity and position of the target relative to itself in the two-dimensional coordinate system are pos i,t ∈R 2 ,vel i,t ∈R 2 as well as Where R represents a real number; Use POS i,t ,vel i,t ,and Indicates observation information definition and e j Relative to A i The position, velocity and e j Target relative to A i position; if e j is an obstacle or target, set use Indicates g i,t The node features of each neighbor entity on the above, where et(j) is the entity type, defined as et(j)∈{"agent":0,"landmark":1,"obstacle":2}, the string in {·} represents the text, the dictionaries agent, landmark and obstacle represent the moving body, landmark and obstacle respectively, and the numbers 0, 1 and 2 represent the index of the dictionary; (3) Information aggregation Use an embedding layer with a dictionary size of 3 and an output dimension of 2 to encode et(j) to obtain the encoded entity type et ij,t , splicing et ij,t and d ij,t Get vector in Represents a vector concatenation operation; Input to a linear layer with an input dimension of 9 and an output dimension of 16 for feature extraction, thereby obtaining the node features input to the graph neural network A three-layer graph neural network is used as the information aggregation module. Each layer is numbered l∈{1,...,L}, the maximum number of layers is L=3, and the input layer dimension of the graph neural network is set to 16; ① On the first layer of the graph neural network, use Computing the Attention Map Where u and v are g respectively. i,t The numbers of any moving body and its neighbors on the u,t , N u,t Representative A u A collection of neighbors; is the node feature of the moving body with arbitrary number u in the l-1 layer, is the node feature of any neighbor numbered v in the l-1th layer; c is the dimension of the output of each layer of the graph neural network, when l = 1 or 3, c = 16, when l = 2, c = 32; W Q ∈R 16×c·h is the learnable weight matrix of the query, A learnable weight matrix with key ; using Calculate the moving body A with any number u u Any neighbor entity e with the number v v The attention coefficient between The number of attention heads is set to 3; using Calculate the l=1th layer e v Pass to A u News in, and The l=1th layer A u and e v The node characteristics, W V ∈R 16×c·h A learnable weight matrix with values ​​of Calculate A u Aggregated information, where W x ∈R 16×c·h ; ② On the second layer of the graph neural network, use equations (2) and (3) to calculate the attention coefficient between entities, and then use Structural sampling in the information bottleneck of computational graphs A u The joint expression of the attention coefficients is Among them, |N u,t | stands for N u,t The number of neighbors contained in ; for each attention head, generate |N u,t | independent samples from a (0,1) uniform distribution use Compute the Gumbel distribution use Calculate the reparameterized attention coefficient Among them, tem is the temperature parameter, and its value is 0.1; Finally, use formula (4) to calculate Calculate the mean of a Gaussian distribution and variance Among them, [·,·] represents the message vector Extract the specific fragments in the brackets; generate a Gaussian distribution using the mean and variance, and obtain the actual transmitted message by sampling from the Gaussian distribution; Add 1 dimension to the end of Calculate Gaussian distribution The log probability of use Calculate the scaling parameter list after the Softplus activation function Among them, σ u,z ∈R c×z is the list of scaling parameters for the z-th Gaussian distribution in the Gaussian mixture model, initialized at the beginning of training, is u,z The learnable parameters after the activation function, z is the number of any Gaussian distribution in the Gaussian mixture model, z∈{1,...,Z}, Z is the maximum number of Gaussian distributions, set to 100; using Calculate the value of any Gaussian distribution z The log probability of Among them, the weighted logarithmic probability μ u,z ∈R c×Z is the learnable parameter matrix, which is also initialized at the beginning of training; using Calculate the Gaussian mixture model The log probability of use Node feature sampling in computational graph information bottleneck Finally, use formula (5) to calculate the output of the second layer; ③ On the third layer of the graph neural network, first, use equations (2) and (3) to calculate the attention coefficient between entities, and use equations (4) and (6) to obtain and Secondly, use formula (5) to obtain the output of the graph neural network module; then, from the node data Select A i Node data Finally, get As input to the actor network, and compute the input to the critic network: ④For all moving bodies A1,...,A N Use operations ① to ③ to obtain the actor and critic network inputs of each moving body, save the input and output data of the graph neural network and the actor-critic network into training batches, and prepare for algorithm training; (4) Training decentralized networked multi-agent reinforcement learning algorithms ① Set algorithm training parameters: the maximum number of training steps is 2×10 6 , the number of training times of the proximal strategy optimization algorithm is 10, the number of parallel environments is 128, the size of the training batch is B = 25 × 128 = 3200, ρ = 1, N = D = O = 3; ② Set the hyperparameters of the actor-critic network: 1 layer of multilayer perceptron, with an input layer dimension of 22, a hidden layer dimension of 64, and an output layer dimension of 64; 1 layer of gated recurrent unit, with an input layer, hidden layer, and output layer dimension of 64; the fully connected layer input dimension of the actor network is 64, and the output dimension is 5; the fully connected layer input dimension of the critic network is 64, and the output dimension is 1; the activation function is the ReLU function, and the learning rate is 7×10 -4 ; ③ Adoption A i The actor network is trained; where the clip(·,·,·) function limits the data of the first item to the range of the last two items, τ is the number of each group of data in the batch, and θ i are the parameters of the actor network, is the policy update range limit parameter, κ=0.01 is the policy entropy coefficient, β1=0.001 is the graph information bottleneck optimization term coefficient of the actor network, is the action conditional probability distribution output by the actor network, θ i Actor network output P before update i,τ for The probability of each item in and are the structure sampling and node feature sampling data stored in the τth group, A i,τ is the advantage function, and the calculation formula is Among them, λ = 0.95 is the generalized advantage estimator coefficient, γ = 0.99 is the discount factor, are the parameters of the critic network, Store the output of the critic network for the τth group, is the output of the τ+1th group of critic networks, r i,τ is the reward for storing the τth group of data in the training batch; A i The critic network is trained; A i The expected round reward starting from time t, β2=0.001 is the graph information bottleneck optimization coefficient of the critic network, for The output of the critic network before updating, ξ = 0.2 is the parameter limiting the update range of the value function; ④ Use the training parameters of ① and ② and the optimization target of ③ to train the algorithm. After the training reaches the maximum number of steps, terminate the training, save the weight parameters of the last training, and complete the model training; (5) Using decentralized networked multi-agent reinforcement learning to achieve collaborative control ① Use the hyperparameters and loss function of (4) to train the decentralized networked multi-agent reinforcement learning algorithm, save the round reward, and output the reward function curve; ② After completing the algorithm training, save the trained model and apply the model to the collaborative control task environment to complete the collaborative control of complex moving body systems.

2. The method according to claim 1, characterized in that: The method first designs a moving body cooperative control task based on a multi-agent particle environment and obtains observation information data by interacting with the task environment; secondly, the data is transmitted to the server and converted into a graph structure according to the position of each entity in the global coordinate system to represent the data features; Then, a graph neural network is used to aggregate the node features of each moving body and its neighboring entities, thereby training a decentralized networked multi-agent reinforcement learning algorithm; finally, each moving body uses the trained model to make independent decisions based on the data collected by the odometer positioning and inertial measurement unit to quickly reach the desired position and avoid collision.

Images