Regional full-coverage path planning method for unmanned underwater vehicle cluster

Abstract

The invention discloses a regional full-coverage path planning method for an unmanned underwater vehicle cluster based on multi-agent reinforcement learning, and the method comprises the steps: setting constraint conditions in consideration of the cooperative regional coverage task characteristics of unmanned underwater vehicles, and meeting various constraint requirements of a user; the method comprises the following steps: standardizing a cluster collaborative area coverage problem as a partially observable Markov decision process, and establishing an observation state set considering obstacle information, dynamic map coverage rate information and aircraft information; considering a real three-dimensional dynamic underwater task scene, and designing a discrete action space containing six directions of a three-dimensional coordinate system; a feature extractor combining a graph neural network and a convolutional neural network is designed, and multi-dimensional features of detection information are extracted more efficiently; according to the designed reward function considering the individual and collaboration levels, the efficient collaboration task effect of the unmanned underwater vehicle cluster is achieved; the designed additional auxiliary loss function accelerates the training convergence speed.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of unmanned underwater vehicle path planning, and particularly relates to a regional full-coverage path planning method for an unmanned underwater vehicle cluster based on multi-agent deep reinforcement learning. BACKGROUND

[0002] With the deepening of ocean development and utilization, unmanned underwater vehicles play a key role in resource exploration, environmental monitoring, search and rescue, and military reconnaissance. In particular, in wide-area tasks, unmanned underwater vehicle clusters can significantly improve task efficiency and coverage range through collaborative work. However, in real three-dimensional underwater environments, dynamic obstacles, the complexity of action space, and collaborative tasks of the cluster significantly increase the complexity of the task, requiring rapid real-time path planning for a large number of position obstacles to safely and efficiently complete the collaborative regional full-coverage task.

[0003] Early traditional path planning algorithms such as A* algorithm, Rapidly-exploring Random Tree (RRT), artificial potential field method, and Breadth-First Search (BFS) have shown certain practicality in simple and static environments. However, path planning is essentially an NP optimization problem, and in complex and variable environments, the computational complexity of traditional algorithms grows exponentially, resulting in low efficiency or even failure to obtain effective results.

[0004] To cope with complex environments, heuristic algorithms have alleviated this problem to some extent. Algorithms such as Ant Colony Optimization, Genetic Algorithm, and Particle Swarm Optimization have strong global search ability and robustness by simulating the behavior of biological groups to plan optimal paths. However, the convergence speed of heuristic algorithms is usually slow, and their performance is highly sensitive to parameter selection.

[0005] In recent years, with the development of deep reinforcement learning technology, a new solution has been provided for regional full-coverage dynamic path planning problems. Any planning system is a process of perception and decision-making, and deep learning has become a very promising method due to its strong non-linear modeling ability. Existing research has used Double Delayed Deep Deterministic Policy Gradient (TD3), Proximal Policy Optimization (PPO), and Deep Q Network (DQN) to implement path planning tasks for unmanned underwater vehicles. However, most existing research focuses on single-agent deep reinforcement learning algorithms, and when dealing with collaborative tasks of unmanned underwater vehicle clusters, they cannot fully consider the influence and mutual cooperation within the cluster, resulting in low efficiency and difficulty in adapting to dynamic and complex actual application requirements.

[0006] To solve the problem of multi-agent cooperative path planning, the multi-agent proximal policy optimization (MAPPO) algorithm emerges as the times require. It adopts the classical Actor-Critic architecture and uses a centralized training, decentralized execution framework. That is, a unified value function network and an independent policy network are used for all agents, which improves the cooperative task capability of multi-agent in complex environment to a certain extent. However, the algorithm still has some defects: first, its ability to extract complex environmental features is limited, and it is difficult to accurately capture key information when facing complex scenes such as random distribution of obstacles, dynamic changes and three-dimensional, resulting in limited path planning efficiency and accuracy. Secondly, when the environment is complex, the neural network layer is deepened and the size is increased, which will lead to slow training convergence, and even difficult to converge. SUMMARY

[0007] To solve the above technical problems, the present application provides a regional full coverage path planning method for an unmanned underwater vehicle cluster, which uses multi-agent reinforcement learning to perform full coverage path planning in an underwater environment with dynamic characteristics, three-dimensional characteristics and random characteristics, and maximizes the dynamic coverage rate of the task area within a specified time.

[0008] The purpose of the present application is achieved by the following technical scheme, a regional full coverage path planning method for an unmanned underwater vehicle cluster, comprising the following steps:

[0009] S1, each vehicle in the unmanned underwater vehicle cluster is regarded as an agent, and an unmanned underwater vehicle cluster cooperative regional coverage model based on a partially observable Markov decision process is established;

[0010] S2, initialize the policy network parameters, feature extraction network parameters, value function network parameters and experience replay pool of each agent in the cluster;

[0011] S3, initialize a three-dimensional seabed map, randomly initialize the distribution position of obstacles, and the number of obstacles depends on the obstacle generation probability;

[0012] S4, randomly initialize the initial position of each vehicle;

[0013] S5, obtain the current observation state of the ith vehicle, input the observation state of the ith vehicle into the feature extraction network to extract features, and then input the spliced observation state into the policy network to obtain the action of the ith vehicle;

[0014] S6, determine whether all vehicles have made action selection, if yes, go to S7, otherwise go to S5 for action selection of the next vehicle;

[0015] S7, after all the actions selected by the vehicles in the cluster are executed in the simulation environment, the simulation environment is updated;

[0016] S8, the reward function of each vehicle is calculated according to the action made by each vehicle and the update of the simulation environment after the action is executed;

[0017] S9, the round end flag of all vehicles and the maximum step truncation flag of the simulation environment are obtained;

[0018] S10, after the global observation state is input into the feature extraction network to extract features, the features are spliced with all the actions selected by the vehicles as the input of the value function network to obtain the value function estimation of the current state;

[0019] S11, the current observation state, action, reward, probability of selecting action, value function estimation of the current state, round end flag and maximum step truncation flag of all vehicles are combined into a tuple and stored in the experience replay pool;

[0020] S12, it is judged whether the maximum number of training rounds is reached; if yes, the training of the unmanned underwater vehicle cluster cooperation regional coverage model is completed, and the trained unmanned underwater vehicle cluster cooperation regional coverage model is obtained, otherwise, go to S13;

[0021] S13, it is judged whether all the round flags of the vehicles are equal to True or the maximum step truncation flag is equal to True; if yes, go to S14, otherwise, go to S5 to continue sampling the training data;

[0022] S14, the advantage function of each round is calculated in the experience replay pool, GAE is used as the advantage estimation, and the calculated advantage estimation is stored back in the experience replay pool;

[0023] S15, a number of experience sequences are randomly sampled from the experience replay pool, and the loss function of the policy network update and the loss function of the value function network update are calculated according to each experience sequence;

[0024] S16, the label of the convolutional neural network in the feature extraction network is generated according to prior knowledge, and the label is used to supervise the calculation of the loss function of the convolutional neural network update;

[0025] S17, the loss is back propagated to update the network parameters;

[0026] S18, it is judged whether the maximum number of network update rounds is reached; if yes, go to S19, otherwise, go to S15 to enter the next round of network update;

[0027] S19, the experience replay pool is emptied, and the training data sampling of the next round is performed in S4.

[0028] Preferably, in step S1, the experience replay pool is established based on the observation set, the action set, the reward, the probability of selecting the action, the value function estimation of the current state, the round end flag and the maximum step truncation flag of each vehicle, and the establishment of the unmanned underwater vehicle cluster cooperative regional coverage model based on the partially observable Markov decision process is completed.

[0029] Preferably, in step S1, the reward of each vehicle includes:

[0030] The exploration reward, the collision out-of-bounds reward, the cooperation reward, the cluster distance reward and the round end reward represent the reward of the vehicle by establishing a reward function.

[0031] Preferably, in step S3, the three-dimensional seabed map environment is modeled in the form of a grid map.

[0032] Preferably, in step S5, the method for obtaining the action of the i-th vehicle includes the following steps:

[0033] Step S51, feature extraction of the graph neural network, modeling all vehicles in the cluster and obstacles within the detection range of the i-th vehicle as different types of nodes in the graph neural network, obtaining the input of the graph neural network of the current state of the i-th vehicle;

[0034] Step S52, convolutional neural network feature extraction, obtaining the convolutional neural network input of the current state of the i-th vehicle;

[0035] Step S53, combining the feature extraction of the graph neural network and the convolutional neural network feature extraction, obtaining the observation state of the i-th vehicle by the following formula:

[0036]

[0037] where self_position i is the current position of the i-th vehicle, is the output of the graph neural network of the i-th vehicle, is the convolutional neural network output of the i-th vehicle.

[0038] Preferably, in step S9, the round end flag and the maximum step truncation flag of the simulation environment are obtained by the following formula:

[0039]

[0040] where done i is the round end flag of the i-th vehicle, and if the vehicle is out of bounds or collision occurs, the vehicle is considered dead.

[0041]

[0042] where truncation is the maximum step size truncation bit of the simulation environment, step is the number of steps performed in the current episode, and episode_maxstep is the maximum step limit of the current episode.

[0043] Preferably, in step S10, the input step of the value function network of the state according to the global observation comprises:

[0044] Step S101, feature extraction of the graph neural network, modeling all the aircraft in the cluster and all the obstacle nodes in the map as different types of nodes in the graph neural network, the position of the obstacle node being an absolute position, obtaining the input of the current state graph neural network of the i-th aircraft;

[0045] Step S102, feature extraction of the convolutional neural network, obtaining the convolutional neural network input of the current state of all aircraft;

[0046] Step S103, combining the feature extraction of the graph neural network and the feature extraction of the convolutional neural network, obtaining the global observation of the i-th state by the following formula:

[0047] s i =[actions,GNN output ,CNN output ]

[0048] Where actions is a set of action vectors taken by all aircraft at the i-th time step, GNN output is the output of the graph neural network, and CNN output is the output of the convolutional neural network.

[0049] Preferably, in step S14, the generalized advantage estimate of the i-th step in one episode of any aircraft is calculated by the following formula:

[0050] advantage gae,i =δ+λγ×advantage gae,i+1

[0051] δ=reward i +(1-done i )×γ×vaule i+1 -value i

[0052]

[0053] Where advantage gae,iis the generalized advantage estimate for the i-th step of the episode for the agent, δ is the temporal difference error that reflects the difference between the current estimated state value and the target state value, λ and γ are the discount factors for the generalized advantage estimate and advantage, respectively, reward i is the reward obtained by the agent at the i-th step of the episode, done i is the end flag for the i-th step of the episode for the agent, value i is the state value for the i-th step of the episode, is the discounted reward for the i-th step of the episode, which is used to calculate the loss function of the value function network in the subsequent calculation.

[0054] Preferably, in step S15, the loss function of the policy network update is calculated by the following formula:

[0055]

[0056]

[0057] where B n is the batch size, n is the number of agents, is the new-old probability ratio of the selected action of the policy network of the k-th agent, is the advantage estimate for the step of the k-th agent, represents that the ratio is clipped, limiting the range of the ratio to 1-ε to 1+ε; S represents the entropy of the policy, and σ is a hyperparameter controlling the entropy coefficient.

[0058] Preferably, in step S15, the loss function of the value network update is calculated by the following formula:

[0059]

[0060] where B n is the batch size, n is the number of agents, is the output of the value function network, is the discounted reward.

[0061] Compared with the prior art, the present application has the following advantages:

[0062] The application provides a regional full coverage path planning method for an unmanned underwater vehicle cluster based on multi-agent reinforcement learning, a dynamic three-dimensional underwater reinforcement learning environment is built to simulate a real scene, a reliable foundation is provided for algorithm training, and the practicability and adaptability of the algorithm are improved; a designed feature extractor combines a graph neural network and a convolutional neural network, the expansibility of the algorithm is enhanced, the number of vehicles in the cluster and the number of obstacles are variable, and the actual scene is more in line with the actual scene; a multi-dimensional reward function designed in the embodiment considers the effects of individuals and cooperation, and optimizes the cooperation strategy of the multi-agent in the learning process; and a designed additional convolutional neural network supervision label accelerates the convergence speed of training.

[0063] The application combines a graph neural network and a convolutional neural network as a feature extractor of observation information, simultaneously designs reward functions at individual and cooperative levels, and improves the existing algorithm to accelerate the convergence of the training process by autonomously generating a training supervision label of a convolutional neural network. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A regional full coverage path planning method for an unmanned underwater vehicle cluster based on multi-agent reinforcement learning in the embodiment of the application is a flowchart;

[0065] Figure 2 A whole architecture diagram of multi-agent deep reinforcement learning in the embodiment of the application is shown;

[0066] Figure 3 A graph neural network structure diagram in the embodiment of the application is shown;

[0067] Figure 4 A convolutional neural network structure diagram in the embodiment of the application is shown;

[0068] Figure 5 A total reward curve diagram of one training in the embodiment of the application is shown. DETAILED DESCRIPTION

[0069] The application will be further described in detail below in combination with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the application, and not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings, not all structures.

[0070] As shown in Figure 1 The technical scheme of the application provides a regional full coverage path planning method for an unmanned underwater vehicle cluster, including the following steps:

[0071] S1, each vehicle in the unmanned underwater vehicle cluster is regarded as an agent, and an unmanned underwater vehicle cluster cooperative regional coverage model based on a partially observable Markov decision process is established;

[0072] S2, initializing the strategy network parameter θ of each agent in the cluster, the feature extraction network (graph neural network α, convolutional neural network β) parameter, the value function network parameter φ, and the experience replay pool Buffer;

[0073] S3, initializing a three-dimensional seabed map with a size of 10x10x10, and initializing the obstacle distribution position, wherein the positions of the obstacles are random positions, and the number of obstacles depends on the obstacle generation probability obstacle_ratio;

[0074] S4, initializing the initial position of each vehicle, wherein the position of each vehicle is a random position;

[0075] S5, obtaining the current observation state s i of the i-th vehicle, inputting the observation state of the i-th vehicle into the feature extraction network to extract features, and the features of the graph neural network are denoted as gnn_feat actor , and the features extracted by the convolutional neural network are denoted as cnn_feat actor , the position pos i of the current vehicle is concatenated with the feature vector and input into the strategy network to obtain the action a i →π θ (.|[pos i ,gnn_feat actor ,cnn_feat actor ]) of the i-th vehicle;

[0076] S6, determining whether all vehicles have made action selection, if yes, going to S7, otherwise, going to S5 for action selection of the next vehicle;

[0077] S7, after all vehicles in the cluster make action selection, uniformly executing the action selection made by all vehicles in the simulation environment, and updating the simulation environment;

[0078] S8, calculating the reward function of each vehicle according to the action made by each vehicle and the update of the simulation environment after executing the action;

[0079] S9, obtaining the round end flag done of all vehicles and the maximum step truncation flag truncation of the simulation environment;

[0080] S10, input the global observation state into the feature extraction network to extract features, the feature of the graph neural network is recorded as gnn_feat critic , the feature extracted by the convolutional neural network is recorded as cnn_feat critic , and all the selected actions of the aircraft are concatenated as the input of the value function network to obtain the value function estimate value→V([actions,gnn_feat critic ,cnn_feat critic ]) of the current state;

[0081] S11, the current observation state, action, reward, probability of selected action, value function estimate of the current state, round end flag, maximum step truncation flag of all aircrafts are combined into a tuple (s i ,action i ,reward i ,prob i ,value i ,done i ,truncation) and stored in the experience replay pool;

[0082] S12, determine whether the maximum number of training rounds is reached. If yes, the training of the unmanned underwater vehicle cluster cooperative regional coverage model is completed, and the trained unmanned underwater vehicle cluster cooperative regional coverage model is obtained, otherwise, go to S13;

[0083] S13, determine whether all round flags of the aircrafts are equal to True or the maximum step truncation flag is equal to True. If yes, go to S14, otherwise, go to S5 to continue sampling the training data;

[0084] S14, calculate the advantage function of each round in the experience replay pool, use GAE as the advantage estimate, and store the calculated advantage estimate back into the experience replay pool;

[0085] S15, randomly sample several groups of experience sequences from the experience replay pool, and calculate the loss function of the policy network update and the loss function of the value function network update according to each group of experience sequences;

[0086] S16, generate the label of the convolutional neural network in the feature extraction network according to the prior knowledge, and use the label to supervise the calculation of the loss function of the convolutional neural network update;

[0087] S17, back-propagate the loss to update the network parameters;

[0088] S18, determine whether the maximum number of network update rounds is reached. If yes, go to S19, otherwise, go to S15 to enter the next round of network update;

[0089] S19, clear the experience replay pool and proceed to S4 for the next round of training data sampling.

[0090] This invention models the problem of full regional coverage in swarm collaboration as a partially observable Markov decision process. It defines an observation state set S considering obstacle information, dynamic map coverage, and vehicle information. Considering a real underwater 3D environment, it designs an action space with six directions (up, down, forward, backward, left, and right) in 3D coordinates. A policy network and value function network based on a fusion graph neural network and convolutional neural network feature extractor are designed to extract local and partially global features from the detection information. A reward function including individual and collaborative levels is designed to reflect the efficiency of swarm collaboration tasks. An additional loss function is designed for the convolutional neural network to accelerate the convergence of the training process. Based on the proposed deep reinforcement learning-based algorithm for full regional coverage path planning and obstacle avoidance of unmanned underwater vehicle swarms, autonomous obstacle avoidance and trajectory planning of the swarm are achieved. This invention enables unmanned underwater vehicles to quickly and efficiently complete dynamic regional coverage tasks within a specified time.

[0091] like Figure 2 As shown, this is the overall architecture of multi-agent deep reinforcement learning used in this invention. First, local and global observations are obtained through interaction with the environment. Then, the local and global observations are input into graph neural networks and convolutional neural networks, respectively, to extract features. Next, the features are fused with the vehicle's position and the actions selected by all vehicles. The results are input into the policy network and value function network, respectively, to obtain the actions taken by the vehicle, the probability of taking the actions, and the state-action evaluation value.

[0092] In one embodiment of the present invention, in step S1, an experience replay pool is established based on the observation state set, action set, reward, probability of action selection, value function estimate of the current state, round end flag, and maximum step size truncation flag of each unmanned underwater vehicle, thereby completing the establishment of a cooperative area coverage model for an unmanned underwater vehicle swarm based on a partially observable Markov decision process. Specifically, the following steps are included:

[0093] Step S101: Establish an observation state set based on obstacle information, map coverage information and other vehicle information, wherein obstacle information and other vehicle information are extracted by the graph neural network in the feature extraction network based on the original observation information, and map coverage is extracted by the convolutional neural network in the feature extraction network based on the original observation information;

[0094] Step S102: Establish a discrete set of actions containing six directions in a three-dimensional coordinate system;

[0095] Step S103: Design the reward function in the model;

[0096] Step S104, based on the observation state set of each vehicle, the action set, the reward, the probability of selecting the action, the value function estimation of the current state, the round end flag and the maximum step truncation flag, an experience replay pool is established, so that the establishment of the unmanned underwater vehicle cluster cooperative regional coverage model based on the partially observable Markov decision process is completed.

[0097] In this embodiment, according to an embodiment of the application, the detection range of the vehicle is set as a 3x3x3 region centered on the current position of the vehicle, and the specific positions of the obstacles in the detection range can be obtained during the navigation process. obstacle .

[0098] Preferably, in step S1, the reward of each vehicle includes:

[0099] The exploration reward, the collision out-of-bound reward, the cooperation reward, the cluster distance reward and the round end reward are used to represent the reward of the vehicle by establishing a reward function.

[0100] Specifically, in this embodiment, the model reward function is constructed as follows:

[0101] The exploration reward r c When the vehicle performs an action to access the next position, the vehicle is given a reward according to the control rate of the position, if the control rate of the region is low, the maximum reward of +5 is given, if the control rate of the region is high, the minimum penalty of -3 is given:

[0102]

[0103] Wherein, GridControlRate represents the control rate of the position accessed by the vehicle, Threshold represents the threshold for determining whether the position is out of control, which is embodied as that when the control rate of the position is greater than or equal to the threshold, it indicates that the position is in a controlled state, and when the control rate of the position is less than the threshold, it indicates that the position is in an out-of-control state.

[0104] The collision out-of-bound reward r o If the vehicle collides with an obstacle or another vehicle, a penalty of -50 is given, and if the vehicle travels out of the task designated region, a penalty of -20 is given:

[0105]

[0106] The cooperation reward r co If the action of the cluster makes the coverage rate of the global map increase, all vehicles in the cluster are given a reward:

[0107] r co= (current map coverage - last step map coverage) x number of grids in the map

[0108] Cluster distance reward r dist If the distance between the vehicles in the cluster is maintained outside the preset safe distance, no reward or punishment is performed; otherwise, once the distance is less than the safe distance, a penalty is designed according to the distance deviation between the vehicles in the cluster, the closer the distance, the greater the penalty, so as to prompt the vehicle cluster to maintain a proper distance when performing a task.

[0109]

[0110] Wherein MinDistance represents the preset safe distance; d i represents the minimum distance between the ith vehicle and other vehicles in the cluster; ∈ represents a very small value to prevent the denominator from being equal to 0;

[0111] End of round reward r end When each game round ends, a reward is given to all vehicles in the cluster according to the global map coverage rate reached at the end:

[0112] r end = global map coverage rate x 50

[0113] Wherein the global map coverage rate is the proportion of the area position under control to all area positions.

[0114] The reward obtained by the vehicle i at this step can be obtained by the following formula:

[0115] r i = w1 x r c + w2 x r o + w3 x r co + w3 x r dist + w4 x done i x r end

[0116] In the formula, w i is the weight of each reward, and done i is a flag indicating whether the round is over

[0117] In an embodiment of the present application, in step S3, the three-dimensional seabed map environment is initialized to be modeled in the form of a grid map. The initialized environment is modeled in the form of a grid map, including the following characteristics:

[0118] Dynamic characteristic: the dynamic characteristic of the modeled grid map environment is reflected in having a dynamic coverage rate, that is, the controlled area will lose control over time, and the rate of losing control is controlled by a specific decay rate;

[0119] Random characteristics: the number and distribution of obstacles in each round of the simulation environment are random, controlled by the obstacle generation probability;

[0120] Three-dimensional characteristics: the simulation environment is a three-dimensional grid map, which is more consistent with the real task scene and more complex.

[0121] In an embodiment of the present application, in step S5, the action of the ith vehicle according to the observation state comprises the following steps:

[0122] S51, the feature extraction step of the graph neural network, model all vehicles in the cluster and obstacles within the detection range of the ith vehicle as different types of nodes in the graph neural network, and obtain the input of the current state graph neural network of the ith vehicle through the following formula:

[0123] GNN Inputi = [[rel_pos x1 , rel_pos y1 , rel_pos z1 , node_type1]... [rel_pos xn , rel_pos yn , rel_pos zn , node_type n ]]

[0124] Where, rel_pos represents the relative position of the node to the ith vehicle, node_type represents the number of nodes, and equal to 0 indicates that the node is an agent, and equal to 1 indicates that the node is an obstacle. Wherein, abs_pos represents the absolute position of the node, that is, the three-dimensional coordinates in the map.

[0125] The graph neural network designed in this embodiment uses a graph attention mechanism to calculate the attention between input nodes and add the distance between nodes as a bias term to the attention weight, so that the current vehicle pays more attention to nodes with closer distances. The specific calculation details are as follows: first, the features of each node are extracted through a fully connected layer; second, the attention weight between nodes is calculated and the distance between nodes is added as a bias to the weight to obtain a distance-related attention weight; then the distance-related attention weight calculated is multiplied and added to the node feature, and the feature containing the information of the obstacles and other vehicles within the detection range of the ith vehicle is obtained.

[0126] Step S52, convolutional neural network feature extraction, obtain the convolutional neural network input of the current state of the ith vehicle, and obtain the convolutional neural network input of the current state of the ith vehicle through the following formula:

[0127]

[0128] where GridMap self and the simulation environment map have the same size, where the value corresponding to the position where the vehicle i is located is equal to 1, and the rest of the positions are 0; CoverageMap and the simulation environment map have the same size, where the value indicates the control rate of the corresponding position in the map; ΔCoverage is the difference between the CoverageMap of the current step and the CoverageMap of the last time step; RefreshTime and the simulation environment map have the same size, where the value records the number of time steps that each position has been away from the last visit; DelaySteps and the simulation environment map have the same size, where the value records the number of time steps that each position needs to be away from the loss of control.

[0129] The convolutional neural network designed in this embodiment contains multiple dimensions of data, which can enable the network to learn the dynamic characteristics of the map and guide the vehicle to the area with lower control rate. The convolutional neural network of the present application contains several convolutional layers, pooling layers, and a fully connected layer. The output of the convolutional neural network designed in this embodiment includes a vector with the same size as the map and a feature vector with a specified length; the former is used as a score map to guide the vehicle to the position with high score in the map, and the latter is used as part of the input of the policy network or value function network.

[0130] In step S53, the observation state of the i-th vehicle is obtained by combining the feature extraction of the graph neural network and the feature extraction of the convolutional neural network through the following formula:

[0131]

[0132] where self_position i is the current position of the i-th vehicle, is the output of the graph neural network of the i-th vehicle, is the output of the convolutional neural network of the i-th vehicle. Specifically, CNN iutput is the result of splicing the highest score position in the map score vector output by the convolutional neural network and the global feature vector output by the convolutional neural network.

[0133] As shown in Figure 3 , it is the graph neural network structure used in this embodiment. The nodes in the input are composed of vehicles and obstacles in the simulation environment map; the graph neural network in the invention is based on the graph attention mechanism, and considers distance as a bias term for attention coefficient, so as to pay more attention to nodes with shorter distance, and finally obtain a feature vector that integrates all node features according to different attention weights.

[0134] AsFigure 4 As shown, the structure of the convolutional neural network used in the embodiment, wherein the input containing multiple dimensional map information is subjected to two layers of three-dimensional convolutional network and corresponding activation function to extract the state features of the current map, and then divided into two branches: the first branch generates a vector with the same size as the map through a three-dimensional convolutional network as the map score vector; the second branch reduces the feature dimension through an adaptive pooling layer, and finally captures the features through a fully connected network as a global feature vector.

[0135] In an embodiment of the application, in step S9, the episode end flag and the maximum step truncation flag of the simulation environment are obtained by the following formula:

[0136]

[0137] In the formula, donc i The episode end flag of the i-th vehicle, if the vehicle is out of bounds or collision occurs, the vehicle is considered dead.

[0138]

[0139] In the formula, truncation is the maximum step truncation of the simulation environment, step is the number of steps in the current episode, and episode_maxstep is the maximum step limit of the current episode. In the embodiment, the graph neural network realizes the graph attention mechanism, which is composed of a learnable attention weight matrix and a fully connected network, and the input is the agent node and obstacle node in the map, and the output is a feature vector; the convolutional neural network is composed of a three-dimensional convolutional network, a three-dimensional pooling layer and a fully connected network, the input is a multi-channel state generated according to the current and previous state of the map, and the output is global map score information and a feature vector; the policy network is composed of a fully connected network, the input is a feature vector and a vehicle's own position, and the output is an action; the value function network is composed of a fully connected network, the input is a feature vector and all the actions taken by the vehicles, and the output is a state-action evaluation value.

[0140] In an embodiment of the application, in step S10, the input step of the value function network of the state according to the global observation includes:

[0141] Step S101, feature extraction of the graph neural network, modeling all vehicles in the cluster and all obstacle nodes in the map as different types of nodes in the graph neural network, the obstacle node position is an absolute position, the input of the i-th vehicle current state graph neural network can be obtained by the following formula:

[0142]

[0143] The above formula includes all the aircraft nodes and all the obstacle nodes in the map, pos is the absolute position of each node, and node_type is the type of each node.

[0144] In this embodiment, the feature extraction step of the graph neural network is roughly similar to the extraction step in S51, and the difference is that the obstacle nodes modeled by the graph neural network in S51 only include obstacles within the detection range, while all obstacles should be included in the global observation, and the node position here is an absolute position.

[0145] Step S102, feature extraction of the convolutional neural network, obtains the convolutional neural network input of the current state of all aircrafts, and obtains the following formula:

[0146]

[0147] where GridMap all is a vector including the positions of all aircrafts, and the meanings of the remaining vectors are the same as in S52. In this embodiment, the feature extraction step of the convolutional neural network is roughly similar to the extraction step in S52, and the difference is that the first channel of the convolutional neural network input here includes the positions of all aircrafts, not just the position of one aircraft.

[0148] Step S103, combining the feature extraction of the graph neural network and the feature extraction of the convolutional neural network, obtaining the global observation of the state of the i-th step by the following formula:

[0149] s i =[actions,GNN output ,CNN output ]

[0150] where actions is a set of action vectors of all aircrafts at the i-th time step, GNN output is the output of the graph neural network, and CNN output is the output of the convolutional neural network. Compared with S53, the output of the graph neural network here needs to take the average of the graph neural network output on dimension 0 before being input into the value function network to keep the size of the features matched.

[0151] In one embodiment of the application, in step S14, the generalized advantage estimation of the i-th step in one round of any aircraft is calculated by the following formula:

[0152] advantage gae,i =δ+λγ×advantage gae,i+1

[0153] δ=reward i +(1-donei ) x y x value i+1 -value i

[0154]

[0155] where advantage gae,i is the generalized advantage estimate for the i-th step in an episode of the agent, δ is the time-difference error (TD Error) that reflects the difference between the current estimate of the state value and the target state value; λ and γ are the discount factors for the generalized advantage estimate and advantage, respectively, reward i is the reward obtained by the agent at the i-th step in the episode, done i is the end flag for the i-th step in the episode of the agent, value i is the state value (output of the value function) at the i-th step in the episode, is the discounted reward at the i-th step in an episode, which is used to calculate the loss function of the value function network in the subsequent calculation.

[0156] In an embodiment of the present application, in step S15, the loss function of the policy network update and the loss function of the value function network update corresponding to each set of experience sequences need to be calculated. Specifically, the loss function of the policy network update is calculated by the following formula:

[0157]

[0158] where B n is the batch size, n is the number of agents, is the new-old probability ratio of the selected action of the policy network of the k-th agent, is the advantage estimate of the k-th agent at this step, indicates that the ratio is clipped to limit the range of the ratio to 1-ε to 1+ε; S represents the entropy of the policy, and σ is a hyperparameter that controls the entropy coefficient.

[0159] The loss function of the value network update is calculated by the following formula:

[0160]

[0161] where B n is the batch size, n is the number of agents, is the output of the value function network, is the discounted reward.

[0162] In an embodiment of the present application, in S16, the loss function of the convolutional neural network update corresponding to each set of experience sequences needs to be calculated. The specific steps include:

[0163] S161, obtain the supervision label of the convolutional neural network from the following formula:

[0164]

[0165] Wherein, the size of label is equal to the size of the simulation environment map; CoverageMap and the simulation environment map have the same size, and the value therein indicates the control rate of the corresponding position in the map; Threshold represents the threshold for defining whether the position is out of control, which is embodied as that when the control rate of the position is greater than or equal to the threshold, the position is in a controlled state, and when the control rate is less than the threshold, the position is in an out-of-control state.

[0166] S162, obtain the loss function of the convolutional neural network update from the following formula:

[0167]

[0168] Wherein, B n is the batch size, global_score_map is an output of the convolutional neural network, and label cnn is the supervision label obtained in S161.

[0169] As Figure 5 shown, it is the total reward curve of one training of the present application, which is stabilized at about 4200 in the later period, indicating that the multi-agent system can fully utilize the environmental information and realize efficient collaboration; at the same time, it has a faster convergence process, proving the effect of the additional auxiliary loss function designed in the application.

[0170] The regional full-coverage path planning method for the unmanned underwater vehicle cluster based on multi-agent reinforcement learning provided in the embodiment of the present application sets a constraint condition considering the characteristics of the cooperative regional coverage task of the underwater unmanned vehicle, meets various constraint requirements of the user, regulates the cluster cooperative regional coverage problem as a partially observable Markov decision process, establishes an observation state set considering obstacle information, dynamic map coverage rate information and vehicle information, designs a discrete action space containing six directions of a three-dimensional coordinate system considering a real three-dimensional dynamic underwater task scene, designs a feature extractor combining a graph neural network and a convolutional neural network to more efficiently extract multi-dimensional features of the detection information, realizes efficient cooperative task effect of the unmanned underwater vehicle cluster according to the designed reward function considering individual and cooperative levels, accelerates the training convergence speed by the designed additional auxiliary loss function, and enables the unmanned underwater vehicle cluster to efficiently complete cooperative tasks such as search and rescue salvage, military reconnaissance and regional control in a dynamic unknown three-dimensional environment only relying on the sensor information of the unmanned underwater vehicle cluster within a specified time.

[0171] The above is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can also be made several improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for regional full-coverage path planning of a cluster of unmanned underwater vehicles, characterized in that: The method comprises the following steps: S1, each vehicle in the unmanned underwater vehicle cluster is regarded as an agent, and an unmanned underwater vehicle cluster cooperative regional coverage model based on a partially observable Markov decision process is established; S2, the strategy network parameters, feature extraction network parameters, value function network parameters, and experience replay pool of each agent in the cluster are initialized; S3, a three-dimensional seabed map is initialized, and the positions of obstacles are randomly initialized, and the number of obstacles depends on the obstacle generation probability; S4, the initial position of each vehicle is randomly initialized; S5, the current observation state of the ith vehicle is obtained, the observation state of the ith vehicle is input into the feature extraction network to extract features, and then the observation state of the ith vehicle is spliced with the partially observed state and input into the strategy network to obtain the action of the ith vehicle; S6, it is judged whether all vehicles have made action selection, if yes, step S7 is turned to, otherwise, step S5 is turned to for action selection of the next vehicle; S7, after all vehicles in the cluster make action selection, the action selection made by all vehicles is uniformly executed in the simulation environment, and the simulation environment is updated; S8, the reward function of each vehicle is calculated according to the action made by each vehicle and the update of the simulation environment after the action is executed; S9, the round end flag of all vehicles and the maximum step truncation flag of the simulation environment are obtained; S10, the global observation state is input into the feature extraction network to extract features, and then the observation state is spliced with all the selected actions of the vehicles as the input of the value function network to obtain the value function estimation of the current state; S11, the current observation state, action, reward, probability of selecting action, value function estimation of the current state, round end flag, and maximum step truncation flag of all vehicles are combined into a tuple and stored in the experience replay pool; S12, it is judged whether the maximum number of training rounds is reached; if yes, the training of the unmanned underwater vehicle cluster cooperative regional coverage model is completed, a trained unmanned underwater vehicle cluster cooperative regional coverage model is obtained, otherwise, step S13 is turned to; S13, it is judged whether all round flags of the vehicles are equal to True or the maximum step truncation flag is equal to True, if yes, step S14 is turned to, otherwise, step S5 is turned to for continuing the sampling of training data; S14, the advantage function of each round is calculated in the experience replay pool, GAE is used as the advantage estimation, and the calculated advantage estimation is stored back in the experience replay pool; S15, a plurality of experience sequences are randomly sampled from the experience replay pool, and the loss function of the strategy network update and the loss function of the value function network update are calculated according to each experience sequence; S16, the label of the convolutional neural network in the feature extraction network is generated according to prior knowledge, and the label is used to supervise the calculation of the loss function of the convolutional neural network update; S17, the loss is back propagated, and the network parameters are updated; S18, it is judged whether the maximum number of network update rounds is reached, if yes, step S19 is turned to, otherwise, step S15 is turned to for entering the next round of network update; S19, the experience replay pool is emptied, and step S4 is turned to for sampling training data in the next round.

2. The regional full coverage path planning method for a UUV swarm according to claim 1, wherein: In the step S1, the experience replay pool is established based on the observation set, the action set, the reward, the probability of selecting the action, the value function estimation of the current state, the episode end flag and the maximum step truncation flag of each vehicle, and the establishment of the unmanned underwater vehicle cluster cooperative regional coverage model based on the partially observable Markov decision process is completed.

3. The regional full coverage path planning method for a UUV swarm according to claim 2, wherein: In the step S1, the reward of each vehicle includes: The exploration reward, the collision out-of-bound reward, the cooperation reward, the cluster distance reward and the episode end reward are used to represent the reward of the vehicle by establishing a reward function.

4. The regional full coverage path planning method for a UUV swarm according to claim 3, wherein: In the step S3, the three-dimensional seabed map environment is initialized to be modeled in the form of a grid map.

5. The regional full coverage path planning method for a UUV swarm according to claim 4, wherein: In the step S5, the action acquisition method of the i-th vehicle includes the following steps: In step S51, the feature extraction of the graph neural network models all the vehicles in the cluster and the obstacles within the detection range of the i-th vehicle as different types of nodes in the graph neural network, and the input of the current state graph neural network of the i-th vehicle is obtained. In step S52, the convolutional neural network feature extraction obtains the convolutional neural network input of the current state of the i-th vehicle. In step S53, the feature extraction of the graph neural network and the convolutional neural network feature extraction are combined, and the observation state of the i-th vehicle is obtained by the following formula: where self_position i is the current position of the ith vehicle, is the output of the graph neural network of the ith vehicle, is the convolutional neural network output of the ith vehicle.

6. The regional full coverage path planning method for a UUV swarm of claim 5, wherein: In the step S9, the episode end flag and the maximum step truncation flag of the simulation environment are obtained by the following formula: where done i is the end-of-turn flag for the i-th vehicle. A vehicle is considered dead if it goes out of bounds or collides with another vehicle. Where truncation is the maximum step truncation flag of the simulation environment, step is the number of steps in the current episode, and episode_maxstep is the maximum step limit of the current episode.

7. The regional full coverage path planning method for a UUV swarm of claim 6, wherein: In the step S10, the input step of the value function network according to the global observation includes: In step S101, the feature extraction of the graph neural network models all the vehicles in the cluster and all the obstacle nodes in the map as different types of nodes in the graph neural network, and the input of the current state graph neural network of the i-th vehicle is obtained. In step S102, the convolutional neural network feature extraction obtains the convolutional neural network input of the current state of all vehicles. In step S103, the feature extraction of the graph neural network and the convolutional neural network feature extraction are combined, and the global observation of the i-th step is obtained by the following formula: s i = [actions, GNN output , CNN output ] where actions is a vector of action sets taken by all agents at the i-th time step, GNN output is the output of the graph neural network, CNN output is the output of the convolutional neural network.

8. The regional full coverage path planning method for a UUV swarm of claim 7, wherein: In the step S14, the generalized advantage estimation of the i-th step in an episode of any vehicle is calculated by the following formula: advantage gae,i = δ + λγ x advantage gae,i+1 delta = reward i + (1 - done i ) * gamma * value i+1 - value i where advantage gae,i is the generalized advantage estimate for the i-th step of an episode for the agent, δ is the temporal difference error that reflects the difference between the current estimate of the state value and the target state value; λ and γ are the discount factors for the generalized advantage estimate and advantage, respectively, reward i is the reward obtained by the agent at the i-th step of the episode, done i is the end flag for the i-th step of the episode for the agent, value i is the state value for the i-th step of the episode, is the discounted reward for the i-th step of an episode, which is used in the loss function for subsequent computation of the value function network.

9. The regional full coverage path planning method for a UUV swarm of claim 8, wherein: In the step S15, the loss function of the policy network update is calculated by the following formula: where B n is the batch size, n is the number of agents, is the new-old probability ratio of the selected action for the policy network of the kth vehicle, is the advantage estimate for this step of the kth vehicle, denotes clipping the ratio, limiting the range of the ratio to 1 - e to 1 + e; S denotes the entropy of the policy, and s is a hyperparameter controlling the entropy coefficient.

10. The regional full-coverage path planning method for a UUV swarm of claim 9, wherein: In the step S15, the loss function of the value network update is calculated by the following formula: where B n is the batch size, n is the number of agents, is the output of the value function network, is the discounted reward.

Images