A multi-agent obstacle avoidance navigation control method based on deep reinforcement learning

Abstract

The present invention discloses a multi-agent obstacle avoidance navigation control method based on deep reinforcement learning, comprising the following steps: establishing a random obstacle environment model; designing the state space, action space, and reward function of the agent; designing an agent strategy network and evaluation network model based on an attention mechanism; designing an obstacle avoidance navigation control algorithm and a network parameter update method; and designing a training process for the obstacle avoidance navigation control algorithm. The present invention utilizes deep reinforcement learning technology to realize multi-agent obstacle avoidance navigation control in an unknown obstacle environment. By designing an attention mechanism, the important feature information of neighboring agents and obstacles is adaptively extracted, and the dynamic changes in obstacle scenes and the number of agents can be coped with, and the flexibility advantage of distributed execution is achieved.

Description

Technical Field

[0001] The patent of this invention belongs to the field of multi-agent systems and reinforcement learning. It is a multi-agent obstacle avoidance navigation control method based on deep reinforcement learning, involving a series of methods such as random obstacle environment modeling, reinforcement learning element design, network model design based on attention mechanism, and algorithm training process design. Background Art

[0002] The problem of multi-agent collaborative obstacle avoidance navigation is an important research branch of multi-agent systems such as drones and unmanned vehicles. The current obstacle avoidance navigation control algorithm lacks the ability to make collaborative decisions among agents, which easily leads to conflicts in obstacle avoidance paths among agents, greatly reducing the overall obstacle avoidance efficiency of the system. In addition, the robustness and flexibility of the control method in dealing with random obstacle environments and dynamic changes in the number of agents needs to be improved.

[0003] Reinforcement learning is a machine learning method that learns through trial and error and delayed reward mechanisms. The agent evaluates the pros and cons of behavioral decisions based on the reward feedback information from the environment, and learns and optimizes the agent strategy with the goal of maximizing the cumulative reward. The deep reinforcement learning algorithm has demonstrated strong learning capabilities and has achieved remarkable results on various complex problems. The multi-agent collaborative obstacle avoidance navigation control task in a random obstacle environment can be modeled as a distributed partially observable Markov decision process, and the multi-agent deep reinforcement learning algorithm is used to solve the agent strategy. By loading the optimal obstacle avoidance motion control strategy trained by the reinforcement learning algorithm, the agent can make collaborative action decisions based on the limited communication interaction between the agents and the observation information of the environment, successfully move to their respective target points and avoid collisions.

[0004] The attention mechanism is a method in the field of deep learning that can adaptively select different parts of the input data and weight them, so that the network model can focus on the part of the data that contributes most to the result to improve network performance. Compared with traditional fully connected neural networks, the attention mechanism has the characteristics of localization, flexibility and strong interpretability, which helps to reduce model parameters and computational complexity and improve model efficiency. At present, the attention mechanism has been widely used in natural language processing, computer vision, recommendation systems and other fields and has achieved remarkable results. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-agent obstacle avoidance navigation control method based on deep reinforcement learning, to achieve safe and robust multi-agent collaborative obstacle avoidance navigation control, to effectively improve the adaptability of multi-agent clusters to obstacle scenes and dynamic changes in the number of agents, and the algorithm has the flexibility advantage of distributed execution.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A multi-agent obstacle avoidance navigation control method based on deep reinforcement learning, comprising the following steps:

[0008] Step S1, establishing a random obstacle environment model, including agent motion, random communication probability and obstacle detection model, to provide a simulation environment for offline training of multi-agent obstacle avoidance navigation control algorithm;

[0009] Step S2, state space design: Based on the random obstacle environment model in step S1, the local observation o of agent i is constructed using the agent motion state and obstacle position information i , construct the global state s of the environment by splicing the local observations of all agents;

[0010] Action space design: The acceleration that controls the movement of the agent is defined as the action of the agent, and the agent has a continuous action space;

[0011] Reward function design: The goal is for the agent to move to the target position, and rewards are given for reaching the target point. The obstacle avoidance between the agent and obstacles is used as a constraint, and collisions are penalized. The sparsity problem of rewards is solved by constructing auxiliary rewards.

[0012] Step S3: respectively convert the global state s and the observed state o in step S2 into i As the input of the evaluation network and policy network, the policy network and evaluation network model structure of the intelligent agent are designed based on the attention mechanism;

[0013] Step S4, designing a multi-agent obstacle avoidance navigation algorithm framework based on deep reinforcement learning, and providing a parameter update method for the strategy network and the evaluation network in step S3;

[0014] Step S5, design a training process for the obstacle avoidance navigation control algorithm, the intelligent agent continuously interacts with the obstacle environment in step S1 to generate experience information, and the parameter updating method in step S4 is used to optimize the network parameters in step S3. The intelligent agent can realize distributed collaborative obstacle avoidance navigation by loading the strategy network model obtained after training.

[0015] In the above technical solution, step S1 comprises:

[0016] Establish an agent motion model and regard each agent as having a radius r a The circular entity uses a second-order differential system to describe the motion state of the center of the circle of agent i:

[0017]

[0018] st||v i ||≤v max

[0019] ||u i ||≤u max

[0020] Among them, p i 、v i 、u i denote the position, velocity and acceleration vector of agent i respectively, and Respectively represent p i With v i The derivative of , n represents the number of agents in the cluster. Since the agents need to satisfy the actual kinematic constraints, the maximum speed and acceleration are limited to v max and u max ;

[0021] A random communication probability model is established, and the probability of successful communication between agents decreases exponentially as the distance increases:

[0022]

[0023] in, represents the probability that agent i and agent j can successfully communicate, r c represents the deterministic communication radius, within which agents can communicate stably, r max represents the maximum communication radius of the agent, c a is a non-negative constant used to adjust the attenuation degree of the probability of successful communication. The set of neighboring agents that successfully communicate with agent i is recorded as

[0024] Establish an obstacle detection model based on a single-line laser radar and define the radius as r tr The circular credible area is denoted as In order to exclude obstacles that are far away and have little impact on the current agent, the obstacle detection model only considers the obstacle data within the trust area. The obstacle point set obtained by agent i according to the obstacle detection model is expressed as follows:

[0025]

[0026] in, Represents the position vector of obstacle point cloud j relative to agent i.

[0027] In the above technical solution, step S2 comprises:

[0028] Design state space: In the navigation and obstacle avoidance task, the agents need to move to their respective target positions. The observation of agent i at time t should include the relative position of the target point and its own speed information:

[0029]

[0030] in, represents the target point position of agent i, is the relationship between agent i and target point γ i The normalized relative position between ;

[0031] is the normalized speed of agent i; the superscript t in the above formula represents the physical quantity at the tth moment. Similarly, the subsequent superscript t also has the same meaning.

[0032] To avoid collisions with other agents and obstacles during movement, agent observations should include the relative position information of neighboring agents and detected obstacles:

[0033]

[0034]

[0035] in, represents the normalized position of neighbor agent j relative to agent i, represents the position of the obstacle point cloud J detected by agent i, represents the normalized relative position between obstacle point cloud j and agent i;

[0036] In order to better assist the coordinated decision-making between agents to form early avoidance behavior, the speed information of neighboring agents is added to the agent observation:

[0037]

[0038] in, represents the normalized relative speed between agent i and its neighbor agent j;

[0039] Define the observation of agent i at time t as It is expressed as follows:

[0040]

[0041] Where k represents the number of neighboring agents of agent i, satisfying m represents the number of obstacle point clouds detected by agent i, satisfying

[0042] Construct the global environment state s at time t based on the observations of all agentst :

[0043]

[0044] Design action space: According to the motion model of the agent in step S1, continuous motion control of the agent can be achieved by providing only acceleration, so the action of agent i is Defined as the acceleration control quantity:

[0045] u i =a i

[0046] Design reward function: The reward function should reward the agent when it reaches the target point. Before the agent reaches the target point, the change in distance between the agent and the target point is used as the intermediate reward value. Design the target point reward item in the reward function:

[0047]

[0048] Among them, c1 is a positive proportional constant, ε represents the allowable target distance error, within which the agent is considered to have successfully reached the target point, and R goal Represents the reward value obtained by the agent after reaching the target point, Respectively represent the distance between agent i and the target planning point γ at the previous time t-1 and the current time t, defined as

[0049] In order to enable the agent strategy to learn the autonomous obstacle avoidance behavior between agents and obstacles, a penalty for collision is added to the reward function. The radius of the agent's obstacle avoidance buffer is d o , the obstacle avoidance penalty term is designed as follows:

[0050]

[0051] Where c2 is a positive proportional constant, d min，i It represents the minimum distance between agent i and other agents and obstacles. The calculation method is as follows:

[0052]

[0053] Among them, when the collection and When both are empty sets, there are no other agents or obstacles within the agent’s obstacle avoidance range, and the obstacle avoidance penalty term is directly taken as r β (s t )=0, min() means taking the minimum value operation;

[0054] In order to make the agent move to the target point as quickly as possible, a penalty for the agent's frequent changes in movement direction is added to the reward function, that is, the acceleration change penalty term: Where c3 is a positive proportional coefficient. This penalty term suppresses the acceleration mutation of the agent and is more in line with the kinematic constraints of the robot.

[0055] The reward r obtained by agent i at time t i (s t ) can be composed of a target point reward item, an obstacle avoidance penalty item, and an acceleration change penalty item:

[0056] r i (st) = r γ (s t )+r β (s t )+r u (s t ).

[0057] In the above technical solution, step S3 comprises:

[0058] Design feature extraction network model: The strategy network transforms the observed state o in step S2 into i As input, the attention module is used to adaptively extract important feature information of neighboring agents and obstacles. The attention module that aggregates the feature information of obstacle positions is denoted as Atten β , which contains three parameter matrices, The attention module that extracts the motion feature information of neighboring agents is denoted as Atten α ,Include The three parameter matrices are used to map input information into query and key-value pairs, as shown below:

[0059]

[0060]

[0061] Among them, d h , d v is the feature vector dimension, the speed of agent i and the relative position of the target point splicing vector x γ , the relative position splicing vector x of m obstacles β , the relative position and velocity concatenation vector x of k neighboring agents α , respectively defined as follows:

[0062]

[0063] The attention module uses scaled dot product attention to calculate the attention coefficient and defines the aggregated obstacle position feature as hβ , the neighbor agent motion feature vector is h α , the feature aggregation process is as follows:

[0064]

[0065] Design a policy network model based on the attention mechanism: Before the attention module, first use a multi-layer perceptron (MLP) to map the input agent observation to a high-dimensional space, and select LeakyReLU as the activation function of the MLP. Then, the feature vector h output by the attention module is β 、h α It is concatenated with the feature vector representing the agent's speed and target position information as the output of the feature extraction network, and finally the MLP is further used to obtain the agent's strategy output; during the training process, a Dropout layer is added to the attention module to set the weight coefficients of some nodes to 0 with a certain probability;

[0066] Design an evaluation network model based on distance attention: The evaluation network is based on the global environment state s in step S2 t and the joint action a consisting of the actions of all agents t As input, the feature extraction network is first used to extract the environmental state features. Secondly, considering that the distant agents generally have less influence on the current agent, in order to reduce the number of parameters of the neural network and reduce the difficulty of training, a distance attention module is designed to further aggregate and reduce the dimension of the environmental state feature vector:

[0067]

[0068] Among them, h o，i represents the observed feature vector of agent i output by the feature extraction network, α i，j It represents the distance attention coefficient of agent i to agent j, and the calculation method is as follows:

[0069]

[0070]

[0071] Among them, c α is a non-negative coefficient used to adjust the influence of neighboring agents at different distances on the current agent;

[0072] The evaluation network model first uses MLP to map the agent action combination vector in the input to a high-dimensional space, then vectorizes and concatenates the feature vector aggregated by the feature extraction network with the action feature vector, and finally uses MLP to output the global evaluation Q value, and selects LeakyReLU as the activation function of MLP in the evaluation network model.

[0073] In the above technical solution, step S4 includes:

[0074] Design obstacle avoidance navigation control algorithm and network parameter update method: In the multi-agent collaborative obstacle avoidance navigation task, the state space and behavior space attributes of the agents are exactly the same. Therefore, in the obstacle avoidance navigation control algorithm based on deep reinforcement learning, all agents are equivalent and share the same evaluation network, policy network and experience pool data. The policy network is denoted as π φ , the two independent evaluation networks are denoted as Q θ，1 , Q θ，2 , and the corresponding target evaluation network is recorded as

[0075] Evaluate network parameters θ i Update: The evaluation network of the agent takes the combination of observations and actions of all agents as input and optimizes parameters by minimizing the temporal difference error. The loss function of the evaluation network is defined as follows:

[0076]

[0077] Among them, the tuple (s t , a t , r t ,s t+1 , d) Sampling from the experience replay pool is the action combination of n agents, For the return portfolio, the target fitted value y is defined as follows:

[0078]

[0079] Among them, λ is the discount factor, d is the termination flag Boolean value, and when the value is 1, it means that the current environment task has been completed. At this time, s t+1 is the end state of the environment. When the value is 0, it is the opposite. The action combination at time t+1 The latest strategy π of all current agents θ According to the observations t+1 Re-decision is obtained, α represents the temperature coefficient, which is used to adjust the exploration and utilization of the agent strategy;

[0080] The soft update method based on exponential moving average is used to evaluate the network parameters of the target To update:

[0081]

[0082] Among them, τ is the soft update proportional coefficient of the target evaluation network parameters;

[0083] Policy network parameter φ update: The policy network of agent i will observe state o i As input, and with the optimization goal of maximizing the sum of expected return and policy entropy, the loss function of the policy network is defined as It is expressed as follows:

[0084]

[0085] in, represents the action samples sampled from the policy output distribution of the agent using the reparameterization technique. The sampling process is as follows:

[0086]

[0087] in, Denote the random strategy π φ The output of is the mean and standard deviation of the normal distribution. ξ is a multidimensional random variable that obeys the standard normal distribution. The tanh function limits the action to a limited range. Therefore, the probability distribution of the random strategy output has changed. The entropy of the strategy is recalculated as follows:

[0088]

[0089] Among them, dim(A) represents the dimension of the action space A; Represents the action vector The kth element of ;

[0090] In order to better balance the exploration and utilization process of the intelligent agent, an adaptive learning method is used to dynamically adjust the temperature coefficient α, and the loss function of α is defined as follows:

[0091]

[0092] in, Represents the lower limit of the expected value of the policy entropy, that is,

[0093] In the above technical solution, step S5 comprises:

[0094] Design the training process of the obstacle avoidance navigation control algorithm, record the total number of training rounds as M, and the maximum step length of the round as T max , the parameter update interval is recorded as T update , the training process includes the following steps:

[0095] Step 7.1, Initialize temperature coefficient α, experience pool Policy network parameters φ, evaluation network parameters θ 1,2 , head

[0096] Evaluate network parameters Evaluation network, policy network, learning rate λ of temperature coefficient θ , φ , α , soft update ratio τ and other parameters;

[0097] Step 7.2: Traverse all episodes from episode = 1 to M and execute:

[0098] Step 7.2.1: Initialize all agents and their target positions to obtain the initial state ;

[0099] Step 7.2.2: traverse the round step length t = 1 → T max ,implement:

[0100] Step 7.2.2.1: Agent strategy selects actions based on observations

[0101] Step 7.2.2.2, execute a t , the state is transferred to s t+1 , termination flag d, return

[0102] Step 7.2.2.3, add the interaction data to the experience pool,

[0103] Step 7.2.2.4, if t is T update Integer multiples, execute:

[0104] Step 7.2.2.4.1. From Sample a batch of data

[0105] Step 7.2.2.4.2, Update the evaluation network parameters,

[0106] Step 7.2.2.4.3, Update the policy network parameters,

[0107] Step 7.2.2.4.4, Update the temperature coefficient,

[0108] Step 7.2.2.4.5, Update the target network parameters,

[0109] Step 7.2.2.5: If all agents have moved to the target point, end the current round and execute step 7.2.1; otherwise, execute step 7.2.2.1.

[0110] Because the present invention adopts the above technical solution, it has the following beneficial effects:

[0111] The present invention utilizes deep reinforcement learning technology to realize multi-agent obstacle avoidance navigation control in an environment with unknown obstacles. By designing an attention mechanism, it adaptively extracts important feature information of neighboring agents and obstacles. It can cope with dynamic changes in obstacle scenes and the number of agents, and has the flexibility advantage of distributed execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 is a flow chart of the method of the present invention;

[0113] Figure 2 This is a schematic diagram of the obstacle detection model;

[0114] Figure 3 It is a schematic diagram of the strategy network structure;

[0115] Figure 4 Schematic diagram of the evaluation network structure; DETAILED DESCRIPTION

[0116] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0117] like Figure 1 As shown, the present invention provides a multi-agent obstacle avoidance navigation control method based on deep reinforcement learning, comprising the following steps:

[0118] Step S1, establishing a random obstacle environment model, including agent motion, random communication probability and obstacle detection model, to provide a simulation environment for offline training of multi-agent obstacle avoidance navigation control algorithm;

[0119] Step S2, state space design: Based on the random obstacle environment model in step S1, the local observation o of agent i is constructed using the agent motion state and obstacle position information i , construct the global state s of the environment by splicing the local observations of all agents;

[0120] Action space design: The acceleration that controls the movement of the agent is defined as the action of the agent, and the agent has a continuous action space;

[0121] Reward function design: The goal is for the agent to move to the target position, and rewards are given for reaching the target point. The obstacle avoidance between the agent and obstacles is used as a constraint, and collisions are penalized. The sparsity problem of rewards is solved by constructing auxiliary rewards.

[0122] Step S3: respectively convert the global state s and the observed state o in step S2 into i As the input of the evaluation network and policy network, the policy network and evaluation network model structure of the intelligent agent are designed based on the attention mechanism;

[0123] Step S4, designing a multi-agent obstacle avoidance navigation algorithm framework based on deep reinforcement learning, and providing a parameter update method for the strategy network and the evaluation network in step S3;

[0124] Step S5, design a training process for the obstacle avoidance navigation control algorithm, the intelligent agent continuously interacts with the obstacle environment in step S1 to generate experience information, and the parameter updating method in step S4 is used to optimize the network parameters in step S3. The intelligent agent can realize distributed collaborative obstacle avoidance navigation by loading the strategy network model obtained after training.

[0125] In the above technical solution, step S1 comprises:

[0126] Establish an agent motion model and regard each agent as having a radius r a The circular entity uses a second-order differential system to describe the motion state of the center of the circle of agent i:

[0127]

[0128] st||v i ||≤v max

[0129] ||u i ||≤u max

[0130] Among them, p i 、v i 、u i denote the position, velocity and acceleration vector of agent i respectively, and Respectively represent p i With v i The derivative of , n represents the number of agents in the cluster. Since the agents need to satisfy the actual kinematic constraints, the maximum speed and acceleration are limited to v max and u max ;

[0131] A random communication probability model is established, and the probability of successful communication between agents decreases exponentially as the distance increases:

[0132]

[0133] in, represents the probability that agent i and agent j can successfully communicate, r c represents the deterministic communication radius, within which agents can communicate stably, r max represents the maximum communication radius of the agent, c ais a non-negative constant used to adjust the attenuation degree of the probability of successful communication. The set of neighboring agents that successfully communicate with agent i is recorded as

[0134] like Figure 2 As shown, an obstacle detection model based on a single-line laser radar is established, and the radius is defined as r tr The circular credible area is denoted as In order to exclude obstacles that are far away and have little impact on the current agent, the obstacle detection model only considers the obstacle data within the trust area. The obstacle point set obtained by agent i according to the obstacle detection model is expressed as follows:

[0135]

[0136] in, Represents the position vector of obstacle point cloud j relative to agent i.

[0137] In the above technical solution, step S2 comprises:

[0138] Design state space: In the navigation and obstacle avoidance task, the agents need to move to their respective target positions. The observation of agent i at time t should include the relative position of the target point and its own speed information:

[0139]

[0140] in, represents the target point position of agent i, is the relationship between agent i and target point γ i The normalized relative position between ; is the normalized speed of agent i;

[0141] To avoid collisions with other agents and obstacles during movement, agent observations should include the relative position information of neighboring agents and detected obstacles:

[0142]

[0143]

[0144] in, represents the target point position of agent i, is the relationship between agent i and target point γ i The normalized relative position between ;

[0145] is the normalized speed of agent i; the superscript t in the above formula represents the physical quantity at the tth moment. Similarly, the subsequent superscript t also has the same meaning.

[0146] In order to better assist the coordinated decision-making between agents to form early avoidance behavior, the speed information of neighboring agents is added to the agent observation:

[0147]

[0148] in, represents the normalized relative speed between agent i and its neighbor agent j;

[0149] Define the observation of agent i at time t as It is expressed as follows:

[0150]

[0151] Where k represents the number of neighboring agents of agent i, satisfying m represents the number of obstacle point clouds detected by agent i, satisfying

[0152] Construct the global environment state s at time t based on the observations of all agents t :

[0153]

[0154] Design action space: According to the motion model of the agent in step S1, continuous motion control of the agent can be achieved by providing only acceleration, so the action of agent i is Defined as the acceleration control quantity:

[0155] u i =a i

[0156] Design reward function: The reward function should reward the agent when it reaches the target point. Before the agent reaches the target point, the change in distance between the agent and the target point is used as the intermediate reward value. Design the target point reward item in the reward function:

[0157]

[0158] Among them, c1 is a positive proportional constant, ε represents the allowable target distance error, within which the agent is considered to have successfully reached the target point, and R goal Represents the reward value obtained by the agent after reaching the target point, Respectively represent the distance between agent i and the target planning point γ at the previous time t-1 and the current time t, defined as

[0159] In order to enable the agent strategy to learn the autonomous obstacle avoidance behavior between agents and obstacles, a penalty for collision is added to the reward function. The radius of the agent's obstacle avoidance buffer is d o , the obstacle avoidance penalty term is designed as follows:

[0160]

[0161] Where c2 is a positive proportional constant, d min，i It represents the minimum distance between agent i and other agents and obstacles. The calculation method is as follows:

[0162]

[0163] Among them, when the collection and When both are empty sets, there are no other agents or obstacles within the agent’s obstacle avoidance range, and the obstacle avoidance penalty term is directly taken as r β (s t )=0, min() means taking the minimum value operation;

[0164] In order to make the agent move to the target point as quickly as possible, a penalty for the agent's frequent changes in movement direction is added to the reward function, that is, the acceleration change penalty term: Where c3 is a positive proportional coefficient. This penalty term suppresses the acceleration mutation of the agent and is more in line with the kinematic constraints of the robot.

[0165] The reward r obtained by agent i at time t i (s t ) can be composed of a target point reward item, an obstacle avoidance penalty item, and an acceleration change penalty item:

[0166] r i (s t )=r γ (s t )+r β (s t )+r u (s t ).

[0167] In the above technical solution, step S3 comprises:

[0168] Design feature extraction network model: The strategy network transforms the observed state o in step S2 into i As input, the attention module is used to adaptively extract important feature information of neighboring agents and obstacles. The attention module that aggregates the feature information of obstacle positions is denoted as Atten β , which contains three parameter matrices, The attention module that extracts the motion feature information of neighboring agents is denoted as Atten α ,Include The three parameter matrices are used to map input information into query and key-value pairs, as shown below:

[0169]

[0170]

[0171] Among them, d h , d v is the feature vector dimension, the speed of agent i and the relative position of the target point splicing vector x γ , the relative position splicing vector x of m obstacles β , the relative position and velocity concatenation vector x of k neighboring agents α , respectively defined as follows:

[0172]

[0173] The attention module uses scaled dot product attention to calculate the attention coefficient and defines the aggregated obstacle position feature as h β , the neighbor agent motion feature vector is h α , the feature aggregation process is as follows:

[0174]

[0175] like Figure 3 As shown in Figure 2, a policy network model based on the attention mechanism is designed: before the attention module, a multi-layer perceptron (MLP) is first used to map the input agent observation to a high-dimensional space, and LeakyReLU is selected as the activation function of the MLP. β 、h α It is concatenated with the feature vector representing the agent's speed and target position information as the output of the feature extraction network, and finally the MLP is further used to obtain the agent's strategy output; during the training process, a Dropout layer is added to the attention module to set the weight coefficients of some nodes to 0 with a certain probability;

[0176] like Figure 4 As shown, a distance attention-based evaluation network model is designed: the evaluation network is based on the global environment state s in step S2 t and the joint action a consisting of the actions of all agents tAs input, the feature extraction network is first used to extract the environmental state features. Secondly, considering that the distant agents generally have less influence on the current agent, in order to reduce the number of parameters of the neural network and reduce the difficulty of training, a distance attention module is designed to further aggregate and reduce the dimension of the environmental state feature vector:

[0177]

[0178] Among them, h o，i represents the observed feature vector of agent i output by the feature extraction network, α i，j It represents the distance attention coefficient of agent i to agent j, and the calculation method is as follows:

[0179]

[0180]

[0181] Among them, c α is a non-negative coefficient used to adjust the influence of neighboring agents at different distances on the current agent;

[0182] The evaluation network model first uses MLP to map the agent action combination vector in the input to a high-dimensional space, then vectorizes and concatenates the feature vector aggregated by the feature extraction network with the action feature vector, and finally uses MLP to output the global evaluation Q value, and selects LeakyReLU as the activation function of MLP in the evaluation network model.

[0183] In the above technical solution, step S4 comprises:

[0184] Design obstacle avoidance navigation control algorithm and network parameter update method: In the multi-agent collaborative obstacle avoidance navigation task, the state space and behavior space attributes of the agents are exactly the same. Therefore, in the obstacle avoidance navigation control algorithm based on deep reinforcement learning, all agents are equivalent and share the same evaluation network, policy network and experience pool data. The policy network is denoted as π φ , the two independent evaluation networks are denoted as Q θ，1 , Q θ，2 , and the corresponding target evaluation network is recorded as

[0185] Evaluate network parameters θ i Update: The evaluation network of the agent takes the observation and action combination of all agents as input and optimizes the parameters by minimizing the temporal difference error. The loss function of the evaluation network is defined as follows:

[0186]

[0187] Among them, the tuple (s t , at , r t ,s t+1 , d) Sampling from the experience replay pool is the action combination of n agents, For the return portfolio, the target fitted value y is defined as follows:

[0188]

[0189] Among them, λ is the discount factor, d is the termination flag Boolean value, and when the value is 1, it means that the current environment task has been completed. At this time, s t+1 is the end state of the environment. When the value is 0, it is the opposite. The action combination at time t+1 The latest strategy π of all current agents θ According to the observations t+1 Re-decision is obtained, α represents the temperature coefficient, which is used to adjust the exploration and utilization of the agent strategy;

[0190] The soft update method based on exponential moving average is used to evaluate the network parameters of the target To update:

[0191]

[0192] Among them, τ is the soft update proportional coefficient of the target evaluation network parameters;

[0193] Policy network parameter φ update: The policy network of agent i will observe state o i As input, and with the optimization goal of maximizing the sum of expected return and policy entropy, the loss function of the policy network is defined as It is expressed as follows:

[0194]

[0195] in, represents the action samples sampled from the policy output distribution of the agent using the reparameterization technique. The sampling process is as follows:

[0196]

[0197] in, Denote the random strategy π φ The output of is the mean and standard deviation of the normal distribution. ξ is a multidimensional random variable that obeys the standard normal distribution. The tanh function limits the action to a limited range. Therefore, the probability distribution of the random strategy output has changed. The entropy of the strategy is recalculated as follows:

[0198]

[0199] Among them, dim(A) represents the dimension of the action space A; Represents the action vector The kth element of ;

[0200] In order to better balance the exploration and utilization process of the intelligent agent, an adaptive learning method is used to dynamically adjust the temperature coefficient α, and the loss function of α is defined as follows:

[0201]

[0202] in, Represents the lower limit of the expected value of the policy entropy, that is,

[0203] In the above technical solution, step S5 comprises:

[0204] Design the training process of the obstacle avoidance navigation control algorithm, record the total number of training rounds as M, and the maximum step length of the round as T max , the parameter update interval is recorded as T update , the training process includes the following steps:

[0205] Step 7.1, Initialize temperature coefficient α, experience pool Policy network parameters φ, evaluation network parameters θ 1，2 , target evaluation network parameters Evaluation network, policy network, learning rate λ of temperature coefficient θ , φ , α , soft update ratio τ and other parameters;

[0206] Step 7.2: Traverse all episodes from episode = 1 to M and execute:

[0207] Step 7.2.1: Initialize all agents and their target positions to obtain the initial state

[0208] Step 7.2.2: traverse the round step length t = 1 → T max ,implement:

[0209] Step 7.2.2.1: Agent strategy selects actions based on observations

[0210] Step 7.2.2.2, execute a t , the state is transferred to s t+1 , termination flag d, return

[0211] Step 7.2.2.3, add the interaction data to the experience pool,

[0212] Step 7.2.2.4, if t is T update Integer multiples, execute:

[0213] Step 7.2.2.4.1. From Sample a batch of data

[0214] Step 7.2.2.4.2, Update the evaluation network parameters,

[0215] Step 7.2.2.4.3, Update the policy network parameters,

[0216] Step 7.2.2.4.4, Update the temperature coefficient,

[0217] Step 7.2.2.4.5, Update the target network parameters,

[0218] Step 7.2.2.5: If all agents have moved to the target point, end the current round and execute step 7.2.1; otherwise, execute step 7.2.2.1.

[0219] The present invention uses deep reinforcement learning technology to realize multi-agent obstacle avoidance navigation control in an unknown obstacle environment. By designing an attention mechanism, it adaptively extracts important feature information of neighboring agents and obstacles. It can cope with dynamic changes in obstacle scenes and the number of agents, and has the flexibility advantage of distributed execution.

[0220] The above is a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art do not depart from the spirit and scope of the present invention, and should be within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-agent obstacle avoidance navigation control method based on deep reinforcement learning, characterized in that: The following steps are involved: Step S1, establishing a random obstacle environment model, including agent motion, random communication probability and obstacle detection model, to provide a simulation environment for offline training of multi-agent obstacle avoidance navigation control algorithm; Step S2, state space design: Based on the random obstacle environment model in step S1, the local observation o of agent i is constructed using the agent motion state and obstacle position information i , construct the global state s of the environment by splicing the local observations of all agents; Action space design: The acceleration that controls the movement of the agent is defined as the action of the agent, and the agent has a continuous action space; Reward function design: The goal is for the agent to move to the target position, and the agent that reaches the target point is rewarded. The obstacle avoidance between the agent and the obstacle is used as a constraint, and the agent that collides is punished. The sparsity problem of the reward is solved by constructing auxiliary rewards. Step S3: respectively convert the global state s and the observed state o in step S2 into i As the input of the evaluation network and policy network, the policy network and evaluation network model structure of the intelligent agent are designed based on the attention mechanism; Step S4, designing a multi-agent obstacle avoidance navigation algorithm framework based on deep reinforcement learning, and providing a parameter update method for the strategy network and the evaluation network in step S3; Step S5, designing a training process for the obstacle avoidance navigation control algorithm, wherein the agent continuously interacts with the obstacle environment in step S1 to generate experience information, and the parameter updating method in step S4 is used to optimize the network parameters in step S3, and the agent can realize distributed collaborative obstacle avoidance navigation by loading the strategy network model obtained after training; The step S1 comprises: Establish an agent motion model and regard each agent as having a radius r a The circular entity uses a second-order differential system to describe the motion state of the center of the circle of agent i: s.t.||v i ||≤v max ||in i ||≤u max Among them, p i ,υ i 、u i denote the position, velocity and acceleration vector of agent i respectively, and Respectively represent p i With i The derivative of n represents the number of agents in the cluster. Since the agents need to satisfy the actual kinematic constraints, the maximum speed and acceleration are limited to υ max and u max ; A random communication probability model is established, and the probability of successful communication between agents decreases exponentially as the distance increases: in, represents the probability that agent i and agent j can successfully communicate, r c represents the deterministic communication radius, within which agents can communicate stably, r max represents the maximum communication radius of the agent, c a is a non-negative constant used to adjust the attenuation degree of the probability of successful communication. The set of neighboring agents that successfully communicate with agent i is recorded as Establish an obstacle detection model based on a single-line laser radar and define the radius as r tr The circular credible area is denoted as In order to exclude obstacles that are far away and have little impact on the current agent, the obstacle detection model only considers the obstacle data within the trust area. The obstacle point set obtained by agent i according to the obstacle detection model is expressed as follows: in, Represents the position vector of obstacle point cloud j relative to agent i; The step S3 comprises: Design feature extraction network model: The strategy network transforms the observed state o in step S2 into i As input, the attention module is used to adaptively extract important feature information of neighboring agents and obstacles. The attention module that aggregates the feature information of obstacle positions is denoted as Atten β , which contains three parameter matrices, The attention module that extracts the motion feature information of neighboring agents is denoted as Atten α ,Include The three parameter matrices are used to map input information into query and key-value pairs, as shown below: Among them, d h , d υ is the feature vector dimension, the speed of agent i and the relative position of the target point splicing vector x γ , the relative position splicing vector x of m obstacles β , the relative position and velocity concatenation vector x of k neighboring agents α , respectively defined as follows: The attention module uses scaled dot product attention to calculate the attention coefficient and defines the aggregated obstacle position feature as h β , the neighbor agent motion feature vector is h α , the feature aggregation process is as follows: Design a policy network model based on the attention mechanism: Before the attention module, first use a multi-layer perceptron to map the input agent observation to a high-dimensional space, and select LeakyReLU as the activation function of the MLP. β 、h α It is concatenated with the feature vector representing the agent's speed and target position information as the output of the feature extraction network, and finally the MLP is further used to obtain the agent's strategy output; during the training process, a Dropout layer is added to the attention module to set the weight coefficients of some nodes to 0 with a certain probability; Design an evaluation network model based on distance attention: The evaluation network is based on the global environment state s in step S2 t and the joint action a consisting of the actions of all agents t As input, the feature extraction network is first used to extract the environmental state features. Secondly, considering that the distant agents generally have less influence on the current agent, in order to reduce the number of parameters of the neural network and reduce the difficulty of training, a distance attention module is designed to further aggregate and reduce the dimension of the environmental state feature vector: Among them, h o,i represents the observed feature vector of agent i output by the feature extraction network, α i,j It represents the distance attention coefficient of agent i to agent j, and the calculation method is as follows: Among them, c α is a non-negative coefficient used to adjust the influence of neighboring agents at different distances on the current agent; The evaluation network model first uses MLP to map the agent action combination vector in the input to a high-dimensional space, then vectorizes and concatenates the feature vector aggregated by the feature extraction network with the action feature vector, and finally uses MLP to output the global evaluation Q value, and selects LeakyReLU as the activation function of MLP in the evaluation network model.

2. According to the multi-agent obstacle avoidance navigation control method based on deep reinforcement learning in claim 1, it is characterized by: The step S2 comprises: Design state space: In the navigation and obstacle avoidance task, the agents need to move to their respective target positions. The observation of agent i at time t should include the relative position of the target point and its own speed information: in, represents the target point position of agent i, is the relationship between agent i and target point γ i The normalized relative position between ; is the normalized speed of agent i; the superscript t in the above formula represents the physical quantity at the tth moment, and similarly, the subsequent superscript t also has the same meaning; To avoid collisions with other agents and obstacles during movement, the agent's observation state should include the relative position information of it, its neighboring agents, and the detected obstacles: in, represents the normalized position of neighbor agent j relative to agent i, represents the position of obstacle point cloud j detected by agent i, represents the normalized relative position between obstacle point cloud j and agent i; In order to better assist the collaborative decision-making between agents to form early avoidance behavior, the speed information of neighboring agents is added to the agent's observation state: in, represents the normalized relative speed between agent i and its neighbor agent j; Define the observation information of agent i at time t as It is expressed as follows: Where k represents the number of neighboring agents of agent i, satisfying m represents the number of obstacle point clouds detected by agent i, satisfying Construct the global environment state s at time t based on the observed states of all agents t : Design action space: According to the motion model of the agent in step S1, continuous motion control of the agent can be achieved by providing only acceleration, so the action of agent i is Defined as the acceleration control quantity: u i =a i Design reward function: The reward function should reward the agent when it reaches the target point. Before the agent reaches the target point, the change in distance between the agent and the target point is used as the intermediate reward value. Design the target point reward item in the reward function: Among them, c1 is a positive proportional constant, ε represents the allowable target distance error, within which the agent is considered to have successfully reached the target point, and R goal Represents the reward value obtained by the agent after reaching the target point, Respectively represent the distance between agent i and the target planning point γ at the previous time t-1 and the current time t, defined as In order to enable the agent strategy to learn the autonomous obstacle avoidance behavior between agents and obstacles, a penalty for collision is added to the reward function. The radius of the agent's obstacle avoidance buffer is d o , the obstacle avoidance penalty term is designed as follows: Where c2 is a positive proportional constant, d min，i It represents the minimum distance between agent i and other agents and obstacles. The calculation method is as follows: Among them, when the collection and When both are empty sets, there are no other agents or obstacles within the obstacle avoidance range of the agent, and the obstacle avoidance penalty term is directly taken as r β (s t )=0, min() means taking the minimum value operation; In order to make the agent move to the target point as quickly as possible, a penalty for the agent's frequent changes in movement direction is added to the reward function, that is, the acceleration change penalty term: Where c3 is a positive proportional coefficient. This penalty term suppresses the acceleration mutation of the agent and is more in line with the kinematic constraints of the robot. The reward r obtained by agent i at time t i (s t ) can be composed of a target point reward item, an obstacle avoidance penalty item, and an acceleration change penalty item: r i (s t )=r γ (s t )+r β (s t )+r u (s t )。 3. According to claim 2, a multi-agent obstacle avoidance navigation control method based on deep reinforcement learning is characterized in that: The step S4 comprises: Design obstacle avoidance navigation control algorithm and network parameter update method: In the multi-agent collaborative obstacle avoidance navigation task, the state space and behavior space attributes of the agents are exactly the same. Therefore, in the obstacle avoidance navigation control algorithm based on deep reinforcement learning, all agents are equivalent and share the same evaluation network, policy network and experience pool data. The policy network is denoted as π φ , the two independent evaluation networks are denoted as Q θ，1 , Q θ，2 , and the corresponding target evaluation network is recorded as Evaluate network parameters θ i Update: The evaluation network of the agent takes the observation and action combination of all agents as input and optimizes the parameters by minimizing the temporal difference error. The loss function of the evaluation network is defined as follows: Among them, the tuple (s t , a t , r t ,s t+1 , d) Sampling from the experience replay pool is the action combination of n agents, For the return portfolio, the target fitted value y is defined as follows: Among them, λ is the discount factor, d is the termination flag Boolean value, and when the value is 1, it means that the current environment task has been completed. At this time, s t+1 is the end state of the environment. When the value is 0, it is the opposite. The action combination at time t+1 The latest strategy π of all current agents θ According to the observations t+1 Re-decision is obtained, α represents the temperature coefficient, which is used to adjust the exploration and utilization of the agent strategy; The soft update method based on exponential moving average is used to evaluate the network parameters of the target To update: Among them, τ is the soft update proportional coefficient of the target evaluation network parameters; Policy network parameter φ update: The policy network of agent i will observe state o i As input, and with the optimization goal of maximizing the sum of expected return and policy entropy, the loss function of the policy network is defined as It is expressed as follows: in, represents the action samples sampled from the policy output distribution of the agent using the reparameterization technique. The sampling process is as follows: in, Denote the random strategy π φ The output of is the mean and standard deviation of the normal distribution. ξ is a multidimensional random variable that obeys the standard normal distribution. The tanh function limits the action to a limited range. Therefore, the probability distribution of the random strategy output has changed. The entropy of the strategy is recalculated as follows: Among them, dim(A) represents the dimension of the action space A; Represents the action vector The kth element of ; In order to better balance the exploration and utilization process of the intelligent agent, an adaptive learning method is used to dynamically adjust the temperature coefficient α, and the loss function of α is defined as follows: in, Represents the lower limit of the expected value of the policy entropy, that is, 4. The multi-agent obstacle avoidance navigation control method based on deep reinforcement learning according to claim 3 is characterized in that: The step S5 comprises: Design the training process of the obstacle avoidance navigation control algorithm, record the total number of training rounds as M, and the maximum step length of the round as T max , the parameter update interval is recorded as T update , the training process includes the following steps: Step 7.1, Initialize temperature coefficient α, experience pool Policy network parameters φ, evaluation network parameters θ 1，2 , target evaluation network parameters Evaluation network, policy network, learning rate λ of temperature coefficient θ , φ , α , soft update ratio τ and other parameters; Step 7.2: Traverse all episodes from episode = 1 to M and execute: Step 7.2.1: Initialize all agents and their target positions to obtain the initial state Step 7.2.2: traverse the round step length t = 1 → T max ,implement: Step 7.2.2.1: Agent strategy selects actions based on observations Step 7.2.2.2, execute a t , the state is transferred to s t+1 , termination flag d, return Step 7.2.2.3, add the interaction data to the experience pool, Step 7.2.2.4, if t is T update Integer multiples, execute: Step 7.2.2.4.

1. From Sampling a batch of data Step 7.2.2.4.2, Update the evaluation network parameters, Step 7.2.2.4.3, Update the policy network parameters, Step 7.2.2.4.4, Update the temperature coefficient, Step 7.2.2.4.5, Update the target network parameters, Step 7.2.2.5: If all agents have moved to the target point, end the current round and execute step 7.2.1; otherwise, execute step 7.2.2.1.

Images