Unmanned vehicle formation obstacle avoidance control method and device based on deep reinforcement learning

Abstract

The present application relates to an obstacle avoidance control method and device for unmanned vehicle formation based on deep reinforcement learning. The method does not use SLAM to explicitly map the environment. The unmanned vehicle can perceive, make decisions and act in real time in an unknown environment without obtaining an environmental map in advance, which can achieve higher real-time and adaptability. The deep reinforcement learning method learns the dynamic characteristics of the environment and the shape characteristics of obstacles by interacting with the environment, so as to achieve accurate decision-making for unmanned vehicles in the face of different similar environments. The unmanned vehicle can make decisions autonomously based on the learned strategy and can adapt to different environments and scenarios. The method has certain autonomy and robustness; the decision-making ability of the unmanned vehicle is optimized through continuous learning, so as to achieve rapid deployment and continuous progress; the formation control method based on graph theory realizes dynamic adjustment and transformation of the formation through a distributed control protocol, so that the formation can quickly adapt to environmental changes and task requirements.

Description

Technical Field

[0001] The present application relates to the field of obstacle avoidance control technology, and in particular to a method and device for obstacle avoidance control of an unmanned vehicle formation based on deep reinforcement learning. Background Art

[0002] Existing unmanned vehicles rely on SLAM navigation. SLAM technology achieves unmanned vehicle navigation by simultaneously performing self-localization and map construction. Typical SLAM algorithms include EKF-SLAM (Extended Kalman Filter SLAM), FastSLAM, and Graph-Based SLAM. The advantage of SLAM technology is that it can provide high-precision maps and positioning information, but its disadvantages are high computational complexity, strong dependence on sensors, and limited performance in dynamic and complex environments.

[0003] In recent years, more and more research has focused on exploratory navigation methods without SLAM mapping. Reinforcement Learning (RL), as a highly promising technology, is gradually becoming an important direction in this field. RL enables intelligent agents to autonomously navigate and avoid obstacles in dynamic and complex environments through interactive learning strategies with the environment, overcoming some limitations of traditional SLAM methods.

[0004] Deep reinforcement learning is a machine learning method in which an agent learns the optimal strategy through interaction with the environment to maximize the cumulative reward. In the RL framework, the agent perceives the state of the environment, takes action, and obtains reward signals from the environment to adjust its strategy. Deep reinforcement learning (DRL) combines deep learning technology with RL, using neural networks to process complex high-dimensional state and action spaces, making RL methods more effective in practical applications.

[0005] Patent publication CN118349003A proposes a robot navigation and obstacle avoidance method based on reinforcement learning. This method determines the current position of the robot at the current moment on a pre-modeled target map, and uses the current position to obtain the target point position information of the robot relative to the specified position on the target map. The environmental information collected by the robot at the current moment is obtained, including static obstacle and dynamic obstacle information map information, and the target point position information, obstacle map information and surrounding environment information are input into the pre-trained PPO-YSH algorithm model to obtain the robot action parameters output by the model. The robot action parameters are used to control the navigation and obstacle avoidance operation of the robot at the current moment. This method requires obtaining the environmental map in advance, and is not effective in terms of real-time and adaptability. Summary of the invention

[0006] Based on this, it is necessary to provide a method and device for obstacle avoidance control of unmanned vehicle formation based on deep reinforcement learning to address the above technical issues. This method solves the problem of existing unmanned vehicles' dependence on mapping (SLAM) technology, improves the unmanned vehicle's autonomous navigation and obstacle avoidance capabilities in unknown environments, and its ability to transform formations in dynamic environments.

[0007] A method for obstacle avoidance control of unmanned vehicle formation based on deep reinforcement learning, the method comprising:

[0008] A graphical model is constructed to describe the structure of a multi-agent system; each unmanned vehicle is an agent in the multi-agent system, a node in the graphical model corresponds to an agent, and the edges in the graphical model refer to the interactions and communication relationships between agents.

[0009] According to the graph model, a formation model based on graph theory is constructed, and according to the formation model, the formation potential field force of each intelligent agent is calculated.

[0010] Construct the initial state space of the multi-agent system.

[0011] Construct a reward function for an unmanned vehicle when using reinforcement learning to navigate an obstacle potential field.

[0012] According to the initial state space and reward function, the deep reinforcement learning method is used to interact with the environment to learn the dynamic characteristics of the environment and the shape characteristics of obstacles, continuously optimize the movement strategy of the unmanned vehicle, realize the obstacle avoidance navigation of moving objects in complex environments and realize the avoidance of moving obstacles.

[0013] In one embodiment, a formation model based on graph theory is constructed according to the graph model, and the formation potential field force of each intelligent agent is calculated according to the formation model, including:

[0014] Each unmanned vehicle senses the position of other unmanned vehicles in the formation relative to the reference coordinate system through the odometer; each unmanned vehicle is a system in n-dimensional space, and the dynamic characteristics of each unmanned vehicle are:

[0015] ;

[0016] in, Indicates unmanned vehicle Current location, , Indicates unmanned vehicle The potential force it is experiencing now.

[0017] There are N unmanned vehicles in the system, and the multi-unmanned vehicle formation problem is defined as:

[0018] ,i j ;

[0019] in, , Respectively represent The first driverless car and The location of the unmanned vehicle, Indicates The first driverless car and The expected position between the unmanned vehicles, N is an integer greater than 0.

[0020] The formation potential force on each unmanned vehicle is calculated based on the relative position controller of the window:

[0021] ;

[0022] in, For formation control gain, For driverless cars The formation potential field force it is subjected to.

[0023] According to the formation potential field force on the unmanned vehicle, using the characteristics of the Laplace matrix, the formation potential field function of the unmanned vehicle is determined as:

[0024] ;

[0025] in, is the potential field function of the unmanned vehicle formation, is the bias parameter of the unmanned vehicle, is the Laplace matrix of the group interaction topology, is an N-dimensional vector consisting of 1s, is a real-valued constant.

[0026] In one embodiment, constructing an initial state space of a multi-agent system includes:

[0027] Obtain observation data of the external environment received by the unmanned vehicle perception system, including forward lidar data, unmanned vehicle odometer information, and target point location.

[0028] According to the observation data, determine the distance between the current unmanned vehicle and the destination, the current direction angle of the unmanned vehicle and the angle between the line connecting the unmanned vehicle and the destination, the current linear speed of the car, and the current angular speed of the car.

[0029] The 180° data extracted in front of the unmanned vehicle is divided into intervals of every 9°; the maximum value in the interval is selected as a 20-dimensional data, and the initial state space is obtained as follows:

[0030] [distance,theta,action[0],action[1],laserdat[0],…laserdat

[19] ];

[0031] Where Distance is the distance between the unmanned vehicle and the target point, Theta is the angle between the unmanned vehicle and the line connecting the target point, action[0] and action[1] are the linear velocity and angular velocity of the unmanned vehicle respectively, and laserdat[0], ..., laserdat

[19] are radar data.

[0032] In one embodiment, the reward function for constructing the unmanned vehicle when strengthening the learning obstacle potential field is:

[0033] ;

[0034] in, They are the current robot formation state and the current robot action respectively. As a reward for reaching your destination, For collision rewards, is the speed of a single robot, is the angular velocity, is the distance between the robot and the destination, For a safe distance.

[0035] In one embodiment, according to the initial state space and reward function, a deep reinforcement learning method is used to interact with the environment to learn the dynamic characteristics of the environment and the shape characteristics of obstacles, continuously optimize the movement strategy of the unmanned vehicle, realize the obstacle avoidance navigation of the moving object in a complex environment and realize the obstacle avoidance of the moving obstacles, including:

[0036] Build an intelligent agent. Each unmanned vehicle deploys an intelligent agent. The intelligent agent includes an Actor network and a Critic network. The Actor network is used to learn a strategy in the initial state space, and the Critic network is used to estimate the value of taking different actions under different states.

[0037] In the training environment, the initial state space is input into the Actor network to generate an action value. Experience and rewards are collected by interacting with the environment. A better policy network is learned using policy gradients under the guidance of the Critic network value function. The experience data is stored in the playback buffer, and batch data is regularly collected from it for learning. During the training process, by constantly interacting with the environment and adjusting the estimate of the value function based on feedback, the agent gradually improves its decision-making ability, realizes obstacle avoidance navigation of moving objects in complex environments, and avoids moving obstacles.

[0038] A device for unmanned vehicle formation obstacle avoidance control based on deep reinforcement learning, the device is used to implement unmanned vehicle formation obstacle avoidance control by adopting any of the above-mentioned unmanned vehicle formation obstacle avoidance control based on deep reinforcement learning; the device comprises:

[0039] A graph theory formation module is used to construct a graph model; the graph model is used to describe the structure of a multi-agent system; a graph theory-based formation model is constructed according to the graph model, and the formation potential field force of each agent is calculated according to the formation model; wherein each unmanned vehicle is an agent in the multi-agent system, a node in the graph model corresponds to an agent, and the edges in the graph model refer to the interactions and communication relationships between agents.

[0040] The initial state space construction module is used to construct the initial state space of the multi-agent system.

[0041] The reward function building module is used to build the reward function of the unmanned vehicle when strengthening the learning obstacle potential field.

[0042] The unmanned vehicle formation obstacle avoidance control module is used to learn the dynamic characteristics of the environment and the shape characteristics of obstacles through the interaction with the environment using deep reinforcement learning methods based on the initial state space and reward function, continuously optimize the movement strategy of the unmanned vehicle, realize obstacle avoidance navigation of moving objects in complex environments, and achieve obstacle avoidance of moving obstacles.

[0043] The above-mentioned unmanned vehicle formation obstacle avoidance control method and device based on deep reinforcement learning does not use SLAM to explicitly map the environment. The unmanned vehicle can perceive, make decisions and act in real time in an unknown environment without obtaining an environmental map in advance, which can achieve higher real-time and adaptability. The deep reinforcement learning method learns the dynamic characteristics of the environment and the shape characteristics of obstacles through interaction with the environment, so that the unmanned vehicle can make accurate decisions in the face of different similar environments. The unmanned vehicle can make decisions autonomously based on the learned strategy and can adapt to different environments and scenarios. The method has certain autonomy and robustness; the decision-making ability of the unmanned vehicle is optimized through continuous learning, so as to achieve rapid deployment and continuous progress; the formation control method based on graph theory realizes dynamic adjustment and transformation of the formation through a distributed control protocol, so that the formation can quickly adapt to environmental changes and task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 1 is a flow chart of an unmanned vehicle formation obstacle avoidance control method based on deep reinforcement learning in one embodiment;

[0045] Figure 2 A schematic diagram of a deep reinforcement learning training framework in another embodiment;

[0046] Figure 3A structural diagram of a TD3 algorithm system in another embodiment;

[0047] Figure 4 Schematic diagram of formation transformation in another embodiment, wherein (a) is a schematic diagram of the initial state of the formation, and (b) is a schematic diagram of the completed state of the formation;

[0048] Figure 5 It is a structural block diagram of an unmanned vehicle formation obstacle avoidance control device based on deep reinforcement learning in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] In one embodiment, Figure 1 As shown, a method for obstacle avoidance control of an unmanned vehicle formation based on deep reinforcement learning is provided, and the method comprises the following steps:

[0051] Step 100: Construct a graphical model for describing the structure of a multi-agent system; wherein each unmanned vehicle is an agent in the multi-agent system, a node in the graphical model corresponds to an agent, and the edges in the graphical model refer to the interactions and communication relationships between agents.

[0052] Specifically, each agent is regarded as a node in the graph, and the interactions and communication relationships between agents are regarded as edges in the graph. In this way, a graph model can be constructed to describe the structure of the multi-agent system. In formation control, graph theory can be used to analyze and design the positional relationship between agents, as well as to achieve the generation, maintenance and transformation of formations.

[0053] Step 102: Construct a formation model based on graph theory according to the graph model, and calculate the formation potential field force of each intelligent agent according to the formation model.

[0054] Specifically, the formation model based on graph theory can describe the topological relationship between multiple agents and define the interaction between agents. Based on the formation model, the formation potential field force of each agent is calculated, and the potential field force includes attraction and repulsion, which is used to guide the agent to move to the target position while avoiding collision with other agents or obstacles.

[0055] Step 104: Construct the initial state space of the multi-agent system.

[0056] Specifically, the system uses deep learning technology to process and analyze sensor data to obtain environmental information. Sensor data can include camera images, lidar scanning data, etc. Through the deep learning model, the unmanned vehicle can more accurately understand key information such as objects in the environment and road conditions.

[0057] The unmanned vehicle receives observation data of the external environment through the perception system and passes this data to the deep reinforcement learning algorithm. These observation data come from lidar data, robot odometer information and target point location, and these data are defined as state. The state space of deep reinforcement learning is [distance, theta, action[0], action[1], laserdat[0], ... laserdat

[19] ]; where Distance is the distance between the car and the target point, Theta is the angle between the car's direction and the line connecting the target point, action[0] and action[1] are the car's linear velocity and angular velocity respectively, and laserdat[0], ... laserdat

[19] are 19 sets of radar data.

[0058] Step 106: Construct a reward function for the unmanned vehicle when reinforcing the obstacle potential field.

[0059] Step 108: Based on the initial state space and reward function, a deep reinforcement learning method is used to interact with the environment to learn the dynamic characteristics of the environment and the shape characteristics of obstacles, continuously optimize the movement strategy of the unmanned vehicle, realize obstacle avoidance navigation of moving objects in complex environments, and realize obstacle avoidance of moving obstacles.

[0060] Specifically, the unmanned vehicle uses deep reinforcement learning algorithms to learn the optimal movement strategy. In an unknown environment, the unmanned vehicle needs to make decisions based on real-time environmental information, including choosing a suitable path and taking obstacle avoidance actions. The deep reinforcement learning algorithm continuously optimizes the unmanned vehicle's movement strategy through interactive learning with the environment, enabling it to safely and efficiently traverse complex environments.

[0061] In traditional methods, sensor data processing is usually time-consuming and easily interfered by environmental noise. However, the perception system based on deep learning can understand and interpret sensor data more accurately by training models, thereby improving the perception ability of unmanned vehicles to the environment.

[0062] Traditional SLAM (Simultaneous Localization and Mapping) technology requires pre-construction of maps, while non-SLAM technology can build environmental maps in real time during movement. This technology enables unmanned vehicles to locate and build maps in unknown environments, providing necessary information support for path planning and obstacle avoidance. In traditional methods, path planning is usually based on pre-defined map information, which performs poorly in dynamic environments. This application uses deep reinforcement learning algorithms to implement path planning and action decisions, which can learn the optimal action strategy in the process of continuous interaction with the environment, allowing unmanned vehicles to flexibly respond to various complex road conditions and obstacles.

[0063] In the above-mentioned unmanned vehicle formation obstacle avoidance control method based on deep reinforcement learning, the method does not use SLAM to explicitly map the environment. The unmanned vehicle can perceive, make decisions and act in real time in an unknown environment without obtaining an environmental map in advance, which can achieve higher real-time and adaptability. The deep reinforcement learning method learns the dynamic characteristics of the environment and the shape characteristics of obstacles through interaction with the environment, so that the unmanned vehicle can make accurate decisions in the face of different similar environments. The unmanned vehicle can make decisions autonomously based on the learned strategy and can adapt to different environments and scenarios. The method has certain autonomy and robustness; through continuous learning and optimization of the decision-making ability of the unmanned vehicle, rapid deployment and continuous progress can be achieved; the formation control method based on graph theory realizes dynamic adjustment and transformation of the formation through a distributed control protocol, so that the formation can quickly adapt to environmental changes and task requirements.

[0064] In one embodiment, step 102 includes: each unmanned vehicle senses the position of other unmanned vehicles in the formation relative to the reference coordinate system through an odometer; each unmanned vehicle is a system in an n-dimensional space, and the dynamic characteristics of each unmanned vehicle are:

[0065] (1)

[0066] in, Indicates unmanned vehicle Current location, , Indicates unmanned vehicle The potential force it is experiencing now.

[0067] There are N unmanned vehicles in the system, and the multi-unmanned vehicle formation problem is defined as:

[0068] ,i j (2)

[0069] in, , Respectively represent The first driverless car and The location of the unmanned vehicle, Indicates The first driverless car and The expected position between the unmanned vehicles, N is an integer greater than 0.

[0070] The formation potential force on each unmanned vehicle is calculated based on the relative position controller of the window:

[0071] (3)

[0072] in, For formation control gain, For driverless cars The formation potential field force it is subjected to.

[0073] According to the formation potential field force on the unmanned vehicle, using the characteristics of the Laplace matrix, the formation potential field function of the unmanned vehicle is determined as:

[0074] (4)

[0075] in, is the potential field function of the unmanned vehicle formation, is the bias parameter of the unmanned vehicle, is the Laplace matrix of the group interaction topology, is an N-dimensional vector consisting of 1s, is a real-valued constant.

[0076] Specifically, each unmanned vehicle can sense the position of other group members relative to the reference coordinate system through the odometer. In the unmanned vehicle formation system, the internal control loop is used to track the reference path. Assume that each unmanned vehicle is a system in n-dimensional space, and its dynamic characteristics are shown in formula (1).

[0077] Assume that the system has a total of N unmanned vehicles, and the formation potential force on each unmanned vehicle is , Indicates i The first driverless car and j The desired position between the unmanned vehicles. Must meet , Then the formation problem is defined as follows:

[0078] Assuming that When , the position p of each robot is as shown in formula (2).

[0079] Then the potential force of each formation can be calculated by the relative position controller of the window, and the calculation formula is shown in formula (3). The strategy in formula (3) can also be considered as a consensus problem.

[0080] Formula (3) can also be written as:

[0081] (5)

[0082] in, is the input bias.

[0083] As pointed out in 1 (JA Fax and RM Murray, “Information flow and cooperativecon:IEEE Tansactions on AutomaticConiof vehicle formations.vol.49,no.9,pp.1465-1476,Sept 2004.), the dynamics along each axis are decoupled, which shows that the one-dimensional analysis is valid for the n-dimensional case. In other words, for the rest of this example, it is assumed that The results are applicable to general limited . In order to exploit the properties of the Laplace matrix, the dynamics of the group system with control signals are re-expressed in a more compact matrix form;

[0084] (6)

[0085] in, , Car position information and offset parameters, is the Laplace matrix of the group interaction topology, which is defined as ,in is a diagonal matrix, D No. i The diagonal elements are i The degree of the elements, A is the adjacency matrix, and since the control matrix is a completely connected graph, so L No. i Line j Elements of a column Defined as:

[0086] (7)

[0087] For the Laplacian matrix L For these values ​​of the elements of , the Laplacian matrix has only one zero eigenvalue, and the rest of the eigenvalues ​​are all identically positive (i.e., it is positive semidefinite). Note that the vectorb is the eigenvector of the Laplace matrix L, and its corresponding eigenvalue is N , then:

[0088] (8)

[0089] Next, define the Lyapunov-like function of the group system as:

[0090] (9)

[0091] right along Taking the derivative, we get:

[0092] (10)

[0093] Defining a Collection ,when , p Converge to The maximum invariant subset of . , ;in, is an N-dimensional vector consisting of 1s, is a real-valued constant. Here we know All elements of The constant translation of Has invariance. That is, when When tending to be stable, the cars are in formation.

[0094] The potential force of each car at time t As shown in formula (4), this application uses formula (4) to approximate the potential field function of the unmanned vehicle formation.

[0095] In one embodiment, step 104 includes: obtaining observation data of the external environment received by the unmanned vehicle perception system, the observation data including forward laser radar data, unmanned vehicle odometer information and target point position; determining the distance between the current unmanned vehicle and the destination, the current unmanned vehicle direction angle and the angle between the line connecting the unmanned vehicle and the destination, the current vehicle linear speed, and the current vehicle angular speed based on the observation data; dividing the 180° data extracted in front of the unmanned vehicle into an interval of every 9°; selecting the maximum value in the interval as a 20-dimensional data, and obtaining the initial state space as follows:

[0096] [distance,theta,action[0],action[1],laserdat[0],…laserdat

[19] ]; (11)

[0097] Where Distance is the distance between the unmanned vehicle and the target point, Theta is the angle between the unmanned vehicle and the line connecting the target point, action[0] and action[1] are the linear velocity and angular velocity of the unmanned vehicle respectively, and laserdat[0], ..., laserdat

[19] are radar data.

[0098] In one embodiment, step 106 includes: constructing a reward function for the unmanned vehicle when strengthening learning the obstacle potential field as:

[0099] (12)

[0100] in, They are the current robot formation state and the current robot action respectively. As a reward for reaching your destination, For collision rewards, is the speed of a single robot, is the angular velocity, is the distance between the robot and the destination, For a safe distance.

[0101] Specifically, as a preferred embodiment, set is +100, is -100.

[0102] The first case in the reward function indicates that the robot gives a reward of +100 after reaching the goal point.

[0103] The second case in the reward function indicates that a reward of -100 is given when the robot hits an obstacle.

[0104] The third case in the reward function means that when the robot is moving, a reward is given when the robot moves towards the target point, and the greater the speed, the greater the reward. This training can ensure that the robot runs as fast as possible.

[0105] In one embodiment, step 108 includes: constructing an intelligent agent, deploying one intelligent agent for each unmanned vehicle, the intelligent agent includes an Actor network and a Critic network, the Actor network is used to learn a strategy in the initial state space, and the Critic network is used to estimate the value of taking different actions in different states; in a training environment, the initial state space is input into the Actor network to generate an action value, experience and rewards are collected by interacting with the environment, and a better policy network is learned using a policy gradient under the guidance of the Critic network value function, the experience data is stored in a playback buffer, and batch data is regularly collected from it for learning. During the training process, by continuously interacting with the environment and adjusting the estimate of the value function based on feedback, the intelligent agent gradually improves its decision-making ability, realizes obstacle avoidance navigation of moving objects in complex environments and realizes obstacle avoidance of moving obstacles.

[0106] Specifically, the Critic network is responsible for estimating the value of taking different actions under different states, that is, estimating the value function. Collect a batch of data from the playback buffer, calculate the minimum value function estimate of each sample taking different actions under different states, update the value function to the target point, and then use the loss function to update the Critic network parameters. The action selection here is obtained by the Actor network.

[0107] The Actor network is responsible for learning a strategy, that is, outputting an action that is most likely to bring high value through the Actor network in a given state. Collect a batch of data from the playback buffer, calculate the policy gradient of each batch, and then perform gradient clipping and update the Actor network parameters.

[0108] During training:

[0109] (1) Actor Network:

[0110] Actor networks are often used to learn policy functions, which are probability distributions for selecting actions given a state. The output of an actor network is a probability distribution, which represents the probability of each action being selected. In discrete action space, the softmax function can be used to convert the network output into a probability distribution; in continuous action space, the mean of the action can be directly output, and then appropriate constraints can be added as needed to ensure the legality of the action.

[0111] The Actor network interacts with the environment and uses policy gradients to learn a better policy network under the guidance of the Critic network value function.

[0112] (2) Critic Network:

[0113] Critic networks are usually used to estimate value functions or action value functions. The value function represents the long-term cumulative reward that can be obtained by taking an action in the current state. The action value function represents the expected cumulative reward starting from the next state after taking an action in the current state.

[0114] The Critic network learns a value function through the data collected by the Actor network from the interaction with the environment. This function is used to judge the quality of an action and thus help the Actor network update its strategy.

[0115] In the training environment, after receiving the state information, the deep reinforcement learning method generates an action value through a random or deterministic strategy. The action will act on the unmanned vehicle and interact with the surrounding environment, mainly completing path planning and obstacle avoidance functions, and collecting experience and rewards through interaction. The reinforcement learning algorithm will store the experience data in the playback buffer and regularly collect batches of data from it for learning.

[0116] In deep reinforcement learning training, learning is done through path planning and obstacle avoidance, and interaction with the environment, which includes randomly selecting a series of actions and observing the feedback from the environment. The selection of these actions is based on the value function method, which aims to evaluate the potential value of each action to guide decision-making. The value function can help evaluate the long-term cumulative reward of taking an action in the current state, thereby helping the agent make the best action decision. During the training process, by constantly interacting with the environment and adjusting the estimate of the value function based on feedback, the agent gradually improves its decision-making ability, allowing it to act more intelligently and efficiently in complex environments.

[0117] like Figure 3 As shown in the figure, the TD3 algorithm also uses two Actor networks, one for generating actions under the current strategy, and the other as the target network for generating actions in the target Q value calculation. The TD3 algorithm uses two Critic networks (Q1 and Q2), both of which estimate the value function of the state-action pair, i.e., the Q value. The main function of these two networks is to reduce the deviation of the Q value estimation.

[0118] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0119] In a verification example, the hardware used in the experiment is a four-wheel drive unmanned vehicle equipped with a laser radar and an odometer, and an Nvidia NX computing board; the trained model file is deployed on the Nvidia NX computing board. During the experiment, it is ensured that the sensor equipment can normally obtain the observation data of the environment and pass this data to the deployed model.

[0120] During the real-time control of the unmanned vehicle, the reachable target point is published and the unmanned vehicle is started.

[0121] At the beginning of training: the unmanned vehicle will start at a fixed position and move towards the target point. Random actions are selected to calculate rewards and stored in the critic network. The training environment includes tables, random obstacles, various special-shaped obstacles, and various corner simulations.

[0122] When encountering an obstacle, the car will choose the direction of movement based on the current radar information and the distance between the target point and the currently unknown information. By designing random boxes to increase the redundancy of the environment, the training car will not overfit. Figure 4 As shown, (a) is a schematic diagram of the initial state of the formation, and (b) is a schematic diagram of the completed state of the formation.

[0123] In one embodiment, Figure 5 As shown, a device for unmanned vehicle formation obstacle avoidance control based on deep reinforcement learning is provided, which is used to implement unmanned vehicle formation obstacle avoidance control by adopting any of the above-mentioned unmanned vehicle formation obstacle avoidance control based on deep reinforcement learning. The device includes: a graph theory formation module, an initial state space construction module, a reward function construction module and an unmanned vehicle formation obstacle avoidance control module, wherein:

[0124] A graph theory formation module is used to construct a graph model; the graph model is used to describe the structure of a multi-agent system; a graph theory-based formation model is constructed according to the graph model, and the formation potential field force of each agent is calculated according to the formation model; wherein each unmanned vehicle is an agent in the multi-agent system, a node in the graph model corresponds to an agent, and the edges in the graph model refer to the interactions and communication relationships between agents.

[0125] The initial state space construction module is used to construct the initial state space of the multi-agent system.

[0126] The reward function building module is used to build the reward function of the unmanned vehicle when strengthening the learning obstacle potential field.

[0127] The unmanned vehicle formation obstacle avoidance control module is used to learn the dynamic characteristics of the environment and the shape characteristics of obstacles through the interaction with the environment using deep reinforcement learning methods based on the initial state space and reward function, continuously optimize the movement strategy of the unmanned vehicle, realize obstacle avoidance navigation of moving objects in complex environments, and achieve obstacle avoidance of moving obstacles.

[0128] In one embodiment, the graph theory formation module is also used for each unmanned vehicle to perceive the position of other unmanned vehicles in the formation relative to the reference coordinate system through the odometer; each unmanned vehicle is a system in n-dimensional space, and the dynamic characteristics of each unmanned vehicle are shown in formula (1). There are N unmanned vehicles in the system, and the multi-unmanned vehicle formation problem is defined as shown in formula (2). The formation potential field force on each unmanned vehicle is calculated according to the relative position controller of the window, and the formation potential field force is shown in formula (3). According to the formation potential field force on the unmanned vehicle, the characteristics of the Laplace matrix are used to determine the formation potential field function of the unmanned vehicle as shown in formula (4).

[0129] In one of the embodiments, the initial state space construction module is also used to obtain observation data of the external environment received by the unmanned vehicle perception system, the observation data including forward laser radar data, unmanned vehicle odometer information and target point position; based on the observation data, determine the current distance between the unmanned vehicle and the destination, the current direction angle of the unmanned vehicle and the angle between the line connecting the unmanned vehicle and the destination, the current linear speed of the vehicle, and the current angular speed of the vehicle; divide the 180° data extracted from the front of the unmanned vehicle into an interval of every 9°; select the largest value in the interval as a 20-dimensional data, and obtain the initial state space shown in formula (11).

[0130] In one embodiment, the reward function construction module is also used to construct a reward function for the unmanned vehicle when reinforcing the obstacle potential field; the reward function is shown in formula (12).

[0131] In one of the embodiments, the unmanned vehicle formation obstacle avoidance control module is also used to build an intelligent agent. Each unmanned vehicle is deployed with an intelligent agent. The intelligent agent includes an Actor network and a Critic network. The Actor network is used to learn a strategy in the initial state space, and the Critic network is used to estimate the value of taking different actions in different states. In the training environment, the initial state space is input into the Actor network to generate an action value, experience and rewards are collected by interacting with the environment, and a better policy network is learned using policy gradients under the guidance of the Critic network value function. The experience data is stored in a playback buffer, and batch data is regularly collected from it for learning. During the training process, by constantly interacting with the environment and adjusting the estimate of the value function based on feedback, the intelligent agent gradually improves its decision-making ability, realizes obstacle avoidance navigation of moving objects in complex environments, and realizes obstacle avoidance of moving obstacles.

[0132] For the specific limitations of the unmanned vehicle formation obstacle avoidance control device based on deep reinforcement learning, please refer to the limitations of the unmanned vehicle formation obstacle avoidance control method based on deep reinforcement learning above, which will not be repeated here. Each module in the above-mentioned unmanned vehicle formation obstacle avoidance control device based on deep reinforcement learning can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An unmanned vehicle formation obstacle avoidance control method based on deep reinforcement learning, characterized in that: The method comprises: Construct a graph model for describing the structure of a multi-agent system; each unmanned vehicle is an agent in the multi-agent system, a node in the graph model corresponds to an agent, and the edges in the graph model refer to the interactions and communication relationships between agents; Constructing a formation model based on graph theory according to the graph model, and calculating the formation potential field force of each intelligent agent according to the formation model; Construct the initial state space of the multi-agent system; The reward function for building an unmanned vehicle in the reinforcement learning obstacle potential field is: ； in, They are the current robot formation state and the current robot action respectively. As a reward for reaching your destination, For collision rewards, is the speed of a single robot, is the angular velocity, is the distance between the robot and the destination, For safe distance; According to the initial state space and the reward function, a deep reinforcement learning method is used to interact with the environment to learn the dynamic characteristics of the environment and the shape characteristics of obstacles, and the movement strategy of the unmanned vehicle is continuously optimized to achieve obstacle avoidance navigation of the moving object in a complex environment and avoidance of moving obstacles; Among them, according to the initial state space and the reward function, a deep reinforcement learning method is used to interact with the environment to learn the dynamic characteristics of the environment and the shape characteristics of obstacles, continuously optimize the movement strategy of the unmanned vehicle, realize the obstacle avoidance navigation of the moving object in a complex environment and realize the obstacle avoidance of the moving obstacles, including: Constructing an intelligent agent, where each unmanned vehicle deploys an intelligent agent, wherein the intelligent agent includes an Actor network and a Critic network, wherein the Actor network is used to learn a strategy in the initial state space, and the Critic network is used to estimate the value of taking different actions in different states; In the training environment, the initial state space is input into the Actor network to generate an action value. Experience and rewards are collected by interacting with the environment. A better policy network is learned using policy gradients under the guidance of the Critic network value function. The experience data is stored in the playback buffer, and batch data is regularly collected from it for learning. During the training process, by continuously interacting with the environment and adjusting the estimate of the value function based on feedback, the agent gradually improves its decision-making ability, realizes obstacle avoidance navigation of mobile objects in complex environments, and realizes obstacle avoidance of moving obstacles. A formation model based on graph theory is constructed according to the graph model, and the formation potential field force of each intelligent agent is calculated according to the formation model, including: Each unmanned vehicle senses the position of other unmanned vehicles in the formation relative to the reference coordinate system through the odometer; each unmanned vehicle is a system in n-dimensional space, and the dynamic characteristics of each unmanned vehicle are: ； in, Indicates unmanned vehicle Current location, , Indicates unmanned vehicle The potential force currently being experienced; There are N unmanned vehicles in the system, and the multi-unmanned vehicle formation problem is defined as: ； in, , Respectively represent The first driverless car and The location of the unmanned vehicle, Indicates The first driverless car and The expected position between the unmanned vehicles, N is an integer greater than 0; The formation potential force on each unmanned vehicle is calculated based on the relative position controller of the window: ； in, For formation control gain, For driverless cars The potential field force of the formation; According to the formation potential field force on the unmanned vehicle, using the characteristics of the Laplace matrix, the formation potential field function of the unmanned vehicle is determined as: ； in, is the potential field function of the unmanned vehicle formation, is the bias parameter of the unmanned vehicle, is the Laplace matrix of the group interaction topology, is an N-dimensional vector consisting of 1s, is a real-valued constant; Construct the initial state space of the multi-agent system, including: Obtaining observation data of the external environment received by the unmanned vehicle perception system, wherein the observation data includes forward laser radar data, unmanned vehicle odometer information, and target point location; According to the observation data, determine the distance between the current unmanned vehicle and the destination, the current direction angle of the unmanned vehicle and the angle between the line connecting the unmanned vehicle and the destination, the current linear speed of the vehicle, and the current angular speed of the vehicle; The 180° data extracted in front of the unmanned vehicle is divided into intervals of every 9°; the maximum value in the interval is selected as a 20-dimensional data, and the initial state space is obtained as follows: [distance,theta,action[0],action[1],laserdat[0],…laserdat[19]] Where Distance is the distance between the unmanned vehicle and the target point, Theta is the angle between the unmanned vehicle and the line connecting the target point, action[0] and action[1] are the linear velocity and angular velocity of the unmanned vehicle respectively, and laserdat[0], ..., laserdat[19] are 19 radar data.

2. An unmanned vehicle formation obstacle avoidance control device based on deep reinforcement learning, characterized in that: The device is used to implement the unmanned vehicle formation obstacle avoidance control by using the unmanned vehicle formation obstacle avoidance control method based on deep reinforcement learning according to claim 1; the device comprises: A graph theory formation module is used to construct a graph model; the graph model is used to describe the structure of a multi-agent system; a graph theory-based formation model is constructed according to the graph model, and the formation potential field force of each agent is calculated according to the formation model; wherein each unmanned vehicle is an agent in the multi-agent system, a node in the graph model corresponds to an agent, and an edge in the graph model refers to the interaction and communication relationship between agents; Initial state space construction module, used to construct the initial state space of the multi-agent system; The reward function building module is used to build the reward function of the unmanned vehicle when strengthening the learning obstacle potential field. ; in, They are the current robot formation state and the current robot action respectively. As a reward for reaching your destination, For collision rewards, is the speed of a single robot, is the angular velocity, is the distance between the robot and the destination, For safe distance; The unmanned vehicle formation obstacle avoidance control module is used to learn the dynamic characteristics of the environment and the shape characteristics of obstacles by using a deep reinforcement learning method to interact with the environment according to the initial state space and the reward function, continuously optimize the movement strategy of the unmanned vehicle, realize the obstacle avoidance navigation of the moving object in a complex environment and realize the obstacle avoidance of the moving obstacles; Among them, the unmanned vehicle formation obstacle avoidance control module is also used to build an intelligent agent. Each unmanned vehicle deploys an intelligent agent. The intelligent agent includes an Actor network and a Critic network. The Actor network is used to learn a strategy under the initial state space, and the Critic network is used to estimate the value of taking different actions under different states. In the training environment, the initial state space is input into the Actor network to generate an action value, and experience and rewards are collected by interacting with the environment. Under the guidance of the Critic network value function, a better policy network is learned using the policy gradient, and the experience data is stored in the playback buffer, and batch data is regularly collected from it for learning. In the training process, by constantly interacting with the environment and adjusting the estimation of the value function according to feedback, the intelligent agent gradually improves its decision-making ability, realizes obstacle avoidance navigation of mobile objects in complex environments, and realizes obstacle avoidance of moving obstacles. The graph theory formation module is also used for each unmanned vehicle to perceive the position of other unmanned vehicles in the formation relative to the reference coordinate system through the odometer; each unmanned vehicle is a system in n-dimensional space, and the dynamic characteristics of each unmanned vehicle are: ； in, Indicates unmanned vehicle Current location, , Indicates unmanned vehicle The potential force currently being experienced; There are N unmanned vehicles in the system, and the multi-unmanned vehicle formation problem is defined as: ； in, , Respectively represent The first driverless car and The location of the unmanned vehicle, Indicates The first driverless car and The expected position between the unmanned vehicles, N is an integer greater than 0; The formation potential force on each unmanned vehicle is calculated based on the relative position controller of the window: ； in, For formation control gain, For driverless cars The potential field force of the formation; According to the formation potential field force on the unmanned vehicle, using the characteristics of the Laplace matrix, the formation potential field function of the unmanned vehicle is determined as: ； in, is the potential field function of the unmanned vehicle formation, is the bias parameter of the unmanned vehicle, is the Laplace matrix of the group interaction topology, is an N-dimensional vector consisting of 1s, is a real-valued constant; The initial state space construction module is also used to obtain the observation data of the external environment received by the unmanned vehicle perception system, the observation data includes forward laser radar data, unmanned vehicle odometer information and target point position; according to the observation data, determine the distance between the current unmanned vehicle and the destination, the current unmanned vehicle direction angle and the angle between the line connecting the unmanned vehicle and the destination, the current car's linear speed, and the current car's angular speed; divide the 180° data extracted in front of the unmanned vehicle into an interval of every 9°; select the largest value in the interval as a 20-dimensional data, and the initial state space is obtained as follows: [distance,theta,action[0],action[1],laserdat[0],…laserdat[19]] Where Distance is the distance between the unmanned vehicle and the target point, Theta is the angle between the unmanned vehicle and the line connecting the target point, action[0] and action[1] are the linear velocity and angular velocity of the unmanned vehicle respectively, and laserdat[0], ..., laserdat[19] are radar data.

Images