A multi-UAV autonomous collaborative navigation method based on graph neural network

Abstract

The present invention discloses a multi-UAV autonomous collaborative navigation method based on graph neural network, which models the multi-UAV navigation problem by using Markov decision process and combines graph neural network to process the complex interactive relationship between UAVs and obstacles. The present invention can effectively improve the navigation efficiency and communication stability of UAV clusters during mission execution.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) navigation technology, and in particular to a multi-UAV autonomous collaborative navigation method based on a graph neural network. Background Art

[0002] In the past few years, drone technology has rapidly emerged, bringing a wide range of application prospects, especially in disaster relief, agricultural monitoring, wildlife protection and other fields. At the same time, due to their excellent flexibility and maneuverability, drones are expected to play a key role in future wireless networks. With the acceleration of urbanization, the application of drones in urban environments faces many challenges. The urban environment is complex and changeable, with a large number of buildings, vehicles and people, all of which increase the navigation difficulty and safety risks of drones. At the same time, factors such as signal interference and building occlusion in the city will also affect the communication connection of drones, thereby affecting the efficiency and success rate of mission execution.

[0003] In the field of multi-UAV collaborative navigation, it is crucial to effectively solve the communication coverage and navigation problems in urban environments. Traditional UAV navigation methods often use pre-planned paths or simple obstacle avoidance strategies, but these methods are difficult to cope with the complexity and variability of urban environments. Therefore, a new method is needed to achieve autonomous collaborative navigation of multiple UAVs and maintain a stable communication connection during navigation to meet the challenges brought by urban environments. Summary of the invention

[0004] In order to solve the problem of drone navigation in urban environments, the present invention proposes a multi-drone autonomous collaborative navigation method based on graph neural network, which can effectively improve the navigation efficiency and communication stability of drone clusters during mission execution. This method can not only be applied to various scenarios such as disaster rescue, urban monitoring and logistics distribution, but also provides new technical support and development direction for the future development of drone technology.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] A multi-UAV autonomous collaborative navigation method based on graph neural network includes the following steps:

[0007] Model the communication coverage scenario in the urban environment and obtain a simulation environment for multi-UAV collaborative navigation;

[0008] The drones, target mission points and obstacles in the simulation environment are taken as nodes of the graph, and the communication links between the drones and the perception relationship of the drones to the target mission points and obstacles are taken as edges of the graph, which are converted into a graph representation; the drones interact with the simulation environment according to the current action data to obtain an interaction result, and the interaction result is the next state data and node data information corresponding to the action data and the reward data corresponding to the drone under the action data; the state, action, reward and node data information are stored in the experience replay pool;

[0009] Randomly extract samples from the experience replay pool and input them into the embedding layer of the graph neural network to process the complex interaction information between drones and between drones and obstacles, and obtain the node data information of the drones; perform aggregation operations on the node data information of the drones based on the unified message passing model in the graph neural network with the introduction of the attention mechanism, and obtain the local node feature vector of the drone;

[0010] A reinforcement learning model consisting of an Actor network and a Critic network is constructed. The state data of the UAV and the feature vector of the local node are concatenated and input into the Actor network and the Critic network respectively for training and updating the parameters of the Actor network and the Critic network until the model converges. The optimal movement action of the UAV is output by the updated reinforcement learning model, so as to perform autonomous collaborative navigation of multiple UAVs on the optimal path.

[0011] Preferably, the base station antenna model in the simulation environment complies with 3GPP specifications.

[0012] Preferably, the state data includes the position, speed, signal-to-interference-noise ratio of the drone at the location, the distance from the target mission point and obstacles, and observation values ​​associated with the graph data.

[0013] Preferably, the signal to interference noise ratio at the location of the drone is calculated by the following steps:

[0014] The baseband equivalent channel between the UAV and the associated communication base station k at time t is represented by H k (t) represents, where 1≤k≤K, K represents the total number of communication base stations associated with the drone during flight, and the baseband equivalent channel H k (t) Antenna gain G of the ground base station k , large-scale path loss β k and small-scale fading h k Three factors are involved, among which G k and β k The size of is related to the position p(t) of the UAV at time t, and h k is a random variable. The signal power received by the drone from the communication base station k at time t can be expressed as:

[0015] p k (t) = P k |H k (t)| 2 =P k β k (p(t))G k (p(t))h k

[0016] Where P k is the transmission power of base station k, which is a constant; at the same time, the signal power p of the non-associated communication base station m at time t is considered m (t) and the influence of noise σ, the signal-to-interference-noise ratio index can be expressed as:

[0017]

[0018] Preferably, the reward function includes a target reward R dist , collision penalty R coll , Out of Bounds Penalty R out and signal-to-interference-to-noise ratio reward R SINRt The composition is as follows:

[0019]

[0020]

[0021] Among them, x u ,y u is the coordinate of the drone; x target ,y target is the coordinate of the target task point; D is the preset target threshold, L u is the distance between the drone and obstacles or other drones; L is the minimum safe distance; r coll is the preset collision penalty constant.

[0022] Preferably, each layer in the unified message delivery model is updated as follows:

[0023] x i ′=W 1 x i +∑ j∈N(i) α i，j W 2 x j

[0024] Where x i 'Indicates that node x i The calculated node features; N(i) represents the set of nodes connected to node i; x j represents the nodes connected to node i; W 1and W 2 is the learnable weight matrix; α i，j Attention coefficient calculated for multi-head dot product attention:

[0025]

[0026] where e ij represents the edge feature connecting nodes i and j, c represents the dimension of the output layer, and W 3 , W 4 , W 5 is the learnable weight matrix.

[0027] Based on the above technical scheme, the beneficial effect of the present invention is as follows: the present invention proposes a multi-UAV cooperative navigation method based on graph neural network, which aims to solve the problem of multi-UAV cooperative navigation under communication coverage in urban environment. By using Markov decision process (MDP) to model the multi-UAV navigation problem and combining graph neural network to process the complex interaction relationship between UAVs and obstacles, the present method can effectively improve the navigation efficiency and communication stability of UAV clusters in the process of executing tasks. The core of the present invention is to introduce a two-stream multi-agent reinforcement learning strategy optimization method (Two-stream GMAPPO) based on graph neural network. The algorithm realizes the dynamic optimization of the navigation strategy of UAV clusters by learning the dynamic interaction relationship between UAVs, including path planning, obstacle avoidance strategy and communication strategy. Experimental results show that the present invention can significantly improve the cooperative navigation performance and communication coverage effect of multiple UAVs in complex urban environments, and shows better strategy convergence performance, safety and robustness compared with the prior art. The present invention is suitable for various application scenarios of UAVs in urban environments, such as disaster relief, urban monitoring and logistics distribution, and provides a new solution for the development of UAV technology in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of a multi-UAV navigation task under communication coverage in an urban environment in an embodiment;

[0029] Figure 2 is a schematic diagram of communication coverage signal strength in an embodiment;

[0030] Figure 3 In one embodiment, a multi-agent proximal strategy optimization algorithm framework based on a graph neural network in a multi-UAV autonomous collaborative navigation method based on a graph neural network is provided;

[0031] Figure 4 is the average reward curve of different reinforcement learning algorithms;

[0032] Figure 5 is a box plot of the average reward distribution of different reinforcement learning algorithms;

[0033] Figure 6 The figure is a schematic diagram of the flight path of a UAV in one embodiment. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0035] As a deep learning model, graph neural network (GNN) has unique advantages in processing graph structured data. By designing a suitable graph neural network model, the dynamic interaction information between drones and obstacles can be effectively captured, and intelligent decision-making can be achieved during navigation. Compared with traditional methods, the graph neural network-based method can perceive environmental information more accurately and adjust navigation strategies more flexibly, thereby improving the navigation efficiency and communication coverage performance of multiple drones in urban environments.

[0036] like Figures 1 to 3 As shown, this embodiment provides a multi-UAV autonomous collaborative navigation method based on graph neural network, which provides a new idea and solution to solve the problem of UAV navigation in urban environments. It introduces a two-stream multi-agent reinforcement learning strategy optimization method based on graph neural network (Two-stream GMAPPO algorithm, including a front-end network and a back-end network, wherein the front-end network is completed by the graph neural network through an embedding layer and a unified message passing model respectively, and the back-end network is completed by a reinforcement learning model composed of an Actor network and a Critic network), and the detailed processing process is as follows.

[0037] Step 1, such as Figure 1 As shown in the figure, the communication coverage scenario in the urban environment is modeled. The base station antenna model follows the 3GPP specification, that is, a uniform linear array consisting of 8 units is placed vertically. Standard sectorization technology is used, and each base station site contains 3 sectors. There are 5 base stations in the experimental area, and the base station antenna height is 25 meters.

[0038] The baseband equivalent channel between the UAV and the associated communication base station k at time t is represented by H k (t) represents, where 1≤k≤K, K represents the total number of communication base stations associated with the drone during flight. Baseband equivalent channel H k (t) Antenna gain G of the ground base station k , large-scale path loss β k and small-scale fading h k Three factors are involved, among which G k and β k The size of is related to the position p(t) of the UAV at time t, and h kis a random variable. The signal power received by the drone from the communication base station k at time t can be expressed as:

[0039] p k (t) = P k |H k (t)| 2 =P k β k (p(t))G k (p(t))h k

[0040] Where P k is the transmission power of base station k, is a constant. The signal-to-interference-plus-noise ratio (SINR) is used as an important criterion for evaluating the communication coverage performance of drones, and the signal power p of the non-associated communication base station m at time t is also considered. m (t) and the influence of noise σ. The SINR index can be expressed as:

[0041]

[0042] In order to meet the communication coverage requirements of drones in the air, if the signal to interference and noise ratio (SINR) received by the drone is lower than the preset signal to interference and noise ratio threshold γ th , that is, SINR(t)<γ th , then the drone is said to be not under communication coverage at time t. The communication coverage signal strength is as follows Figure 2 As shown. Green areas indicate that the area has good communication coverage. The depth of green reflects the strength of communication coverage, and the darker the green, the better the coverage. Conversely, light areas indicate poor communication coverage, and the possibility of drones entering the area should be minimized.

[0043] Step 2: Perform environmental modeling and use Markov decision process methods to model the navigation environment of the drone, including state space, action space, reward function, etc.

[0044] In deep reinforcement learning, the decision-making process of the drone depends heavily on the state information perceived from the environment, such as the need to accurately contain its current state information and information about other important objects in the environment (e.g., obstacles and target mission points). The state space of the drone is composed of the drone's position, speed, signal-to-interference-and-noise ratio (SINR) at its location, the distance from the target mission point and obstacles, and the observations associated with the graph data. The observations associated with the graph are in turn composed of each node and the adjacency matrix associated with the node. The safe flight distance range of the drone is set to a constant L. If the distance between the drone and another entity is less than the constant L, it means that the drone has collided and is recorded. The movement of the drone is simplified, and the flight speed of the drone is constant.

[0045] The reward function of the drone consists of four main parts: target reward R dist , collision penalty R coll , Out of Bounds Penalty R out and signal-to-interference-to-noise ratio reward R SINRt First, in R dist In the part, the reward function is related to the drone coordinates (x u ,y u ) and the target task point coordinates (x target ,y target ) to motivate the drone to fly toward the target mission point; when the distance between the drone and the target mission point is less than the target threshold D, the drone is judged to have reached the target mission point, and the drone will receive a constant reward r goal . R coll In order to ensure the safe and reliable flight of the drone, a constant L is set as the minimum safe distance when the drone is flying. If the distance L between the drone and obstacles or other drones is u If the distance is less than the minimum safety distance L, the drone is considered to have collided and a constant collision penalty r is given. coll In addition, if the drone flies out of the target mission area, it will receive a constant value R out Finally, the reward function is also related to the signal to interference and noise ratio (SINR) obtained by the drone at time t, and this reward is recorded as R SINRt , to encourage drones to fly in areas with better communication coverage. The reward function of the drone is as follows:

[0046]

[0047]

[0048] The SINR of the drone’s location at time t directly affects the value of the reward function. Areas with better communication coverage have better signal-to-interference-and-noise ratio (SINR). Therefore, drones are encouraged to fly preferentially in areas with better communication coverage, which prompts them to move extensively to these areas.

[0049] Step 3: Design a two-stream multi-agent reinforcement learning strategy optimization algorithm based on graph neural network to process complex interactive information between drones and between drones and the environment.

[0050] Each target in the communication coverage navigation environment, such as drones, target mission points, obstacles, etc., is defined as an entity. At each time t, an agent-entity graph about drone i is defined, expressed as G = (V, E), where each node V represents an entity in the environment. Assume that the drone can only observe the existence of obstacles or other drones within a specific limited radius r. If the distance between the drone and the entity is within this range, there is an edge E between the two. If the entity is a drone, the edge E is bidirectional and messages can be passed between the two. If the entity is an obstacle or a target mission point, the edge E is unidirectional and messages can only be passed from the entity to the drone.

[0051] The collected drone i-node features and edge features are transformed and updated through the embedding layer of feature conversion. This process retains the topology and node information of the network and effectively captures the underlying structure of the network and the relationship between nodes.

[0052] Subsequently, the node data is aggregated, and the node features and edge features are processed using the unified message passing model (UniMP) in the graph neural network. An attention mechanism is introduced to allow nodes to selectively prioritize messages from neighboring nodes based on importance. This ensures that each drone pays more attention to its local neighborhood information and aggregates observations from other drones. The perception range of the drone is broadened, the possibility of collision is reduced, and the overall safety of drone flight is ultimately improved. A two-layer network is constructed because deep network structures generally do not produce better results in GNNs, and feature representations may diverge as the number of layers increases. Among them, the update of each UniMP layer is defined as:

[0053] x i ′=W 1 x i +∑ j∈N(i) α i，j W 2 x j

[0054] Where x i 'Indicates that node x i The calculated node features; N(i) represents the set of nodes connected to node i; x j represents the nodes connected to node i; W 1 and W 2 is the learnable weight matrix; α i，j Attention coefficient calculated for multi-head dot product attention:

[0055]

[0056] where e ijrepresents the edge feature connecting nodes i and j, c represents the dimension of the output layer, and W 3 , W 4 , W 5 is the learnable weight matrix. UAV i performs deep feature extraction processing on the node feature data through the UniMP module, aggregates the observation information of other UAVs, and obtains the corresponding local feature vector Sg.

[0057] exist Figure 3 In the dual-stream network structure shown in the figure, a drone strategy is first initialized. The drone interacts with the environment based on the strategy to obtain the drone's state information, action information, reward value and node data information, and store these values ​​in the experience replay pool. Set a certain update step size requirement, and when a certain round of training is reached, the Actor and Critic networks are updated. A batch of samples are randomly selected from the experience replay pool, and these previous samples are used to optimize the drone's new strategy.

[0058] Step 4: Pass the randomly sampled node data information Gi and the drone's own state information Su to the backend network (a reinforcement learning model composed of an Actor network and a Critic network). The front-end graph neural network completes the aggregation operation of the node data information Gi through the embedding layer and the unified message passing model, respectively, and obtains the local node feature vector Sg of the drone. In the Actor network part, the drone's own state information Su is spliced ​​with the local node feature vector Sg to obtain a new network input. Compared with traditional input data, this method is more flexible and can adapt to changes in the drone's position, while being more sensitive to the movement trends of other drones. This enables the actor network to make more informed decisions, generate safer maneuvers, and effectively prevent collisions in advance. In the Critic network stage, the local node feature vector Sg of each drone is spliced ​​with the global state information Su as the input of the Critic network. Such an operation more comprehensively integrates the local features and overall information of the drone to more accurately evaluate the value of the actions performed by the drone. By splicing the local node feature vector Sg, the ability of the Critic network to perceive and understand each drone is effectively enhanced, and the accuracy of evaluating the value of the strategy is improved.

[0059] Next, the Actor and Critic network parameters θ and ω are updated. This completes the update of the drone strategy. The drone continues to interact with the environment based on the new strategy, continues to store the obtained information in the experience replay pool, and then samples to complete the training and update of the Actor and Critic network and generate a new strategy until the final strategy converges.

[0060] experiment

[0061] A simulation experiment was conducted in a simulated urban scene to compare the Two-stream GMAPPO algorithm with the other five multi-agent reinforcement learning benchmark algorithms (MADDPG algorithm, MASAC algorithm, MAPPO algorithm, MATD3 algorithm, and RMAPPO algorithm). Figure 4 It can be clearly seen that compared with the Two-stream GMAPPO algorithm, the average rewards of the remaining algorithms in each round are almost all lower, and they are also inferior to the Two-stream GMAPPO algorithm in terms of convergence speed and final convergence results. This strongly proves the effectiveness of the work done in this experiment. The use of two-stream network structure and GNN can enhance the ability of drones to perceive environmental changes, thereby improving their flight safety. In order to more clearly compare the performance of these algorithms, box plots of the average rewards per episode of different algorithms are also provided, such as Figure 5 As shown in the figure, it is obvious that the distribution and size of the average reward obtained by the Two-stream GMAPPO algorithm are better than those of other algorithms. Figure 6 This is the route chosen by the drone during the experiment, which shows that the routes chosen by the drone during flight have relatively good communication coverage and will choose routes that avoid obstacles.

[0062] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0063] The above is only a preferred implementation of a multi-UAV autonomous collaborative navigation method based on a graph neural network disclosed in the present invention, and is not intended to limit the protection scope of the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included in the protection scope of the embodiments of this specification.

Claims

1. A multi-UAV autonomous collaborative navigation method based on graph neural network, characterized in that: The steps include: Model the communication coverage scenario in the urban environment and obtain a simulation environment for multi-UAV collaborative navigation; The drones, target mission points, and obstacles in the simulation environment are taken as nodes of the graph, and the communication links between the drones and the perception relationship of the drones to the target mission points and obstacles are taken as edges of the graph, which are converted into a graph representation; the drones interact with the simulation environment according to the current action data to obtain an interaction result, and the interaction result is the next state data and node data information corresponding to the action data and the reward data corresponding to the drone under the action data; Store status, actions, rewards and node data information into the experience replay pool; Randomly extract samples from the experience replay pool and input them into the embedding layer of the graph neural network to process the complex interaction information between drones and between drones and obstacles, and obtain the node data information of the drones; Based on the unified message passing model in the graph neural network with the introduction of the attention mechanism, the node data information of the drone is aggregated to obtain the local node feature vector of the drone; A reinforcement learning model consisting of an Actor network and a Critic network is constructed. The state data of the UAV and the feature vector of the local node are concatenated and input into the Actor network and the Critic network respectively for training and updating the parameters of the Actor network and the Critic network until the model converges. The optimal movement action of the UAV is output by the updated reinforcement learning model, so as to perform autonomous collaborative navigation of multiple UAVs on the optimal path.

2. According to claim 1, a multi-UAV autonomous collaborative navigation method based on graph neural network is characterized in that: The base station antenna model in the simulation environment complies with the 3GPP specification.

3. The multi-UAV autonomous collaborative navigation method based on graph neural network according to claim 1 is characterized in that: The state data includes the position, speed, signal-to-interference-noise ratio of the drone, the distance to the target mission point and obstacles, and observation values ​​associated with the graph data.

4. The multi-UAV autonomous collaborative navigation method based on graph neural network according to claim 3 is characterized in that: The signal-to-interference-to-noise ratio at the drone’s location is calculated by the following steps: The baseband equivalent channel between the UAV and the associated communication base station k at time t is represented by H k (t) represents, where 1≤k≤K, K represents the total number of communication base stations associated with the drone during flight, and the baseband equivalent channel H k (t) Antenna gain G of the ground base station k , large-scale path loss β k and small-scale fading h k Three factors are involved, among which G k and β k The size of is related to the position p(t) of the UAV at time t, and h k is a random variable. The signal power received by the drone from the communication base station k at time t can be expressed as: p k (t)=P k |H k (t)| 2 =P k β k (p(t))G k (p(t))h k Where P k is the transmission power of base station k, is a constant; considering the influence of the signal power pm(t) and noise σ of the non-associated communication base station m at time t, the signal-to-interference-noise ratio index can be expressed as:

5. The multi-UAV autonomous collaborative navigation method based on graph neural network according to claim 1 is characterized in that: The reward function of the drone includes the target reward R dist , collision penalty R coll , Out of Bounds Penalty R out and signal-to-interference-to-noise ratio reward R SINRt The composition is as follows: Among them, x u ,y u is the coordinate of the drone; x target ,y target is the coordinate of the target task point; D is the preset target threshold, L u is the distance between the drone and obstacles or other drones; L is the minimum safe distance; r coll is the preset collision penalty constant.

6. The multi-UAV autonomous collaborative navigation method based on graph neural network according to claim 5 is characterized in that: The update definitions of each layer in the unified messaging model are as follows: x i ′=W1x i +∑ j∈N(i) a i，j W2x j Where x i 'Indicates that node x i Calculated node features; N(i) represents the set of nodes connected to node i; x j represents the nodes connected to node i; W1 and W2 are learnable weight matrices; α i，j Attention coefficient calculated for multi-head dot product attention: where e ij represents the edge feature connecting nodes i and j, c represents the dimension of the output layer, and W3, W4, and W5 are learnable weight matrices.

Images