An unmanned aerial vehicle obstacle avoidance method based on multi-agent graph reinforcement learning

Abstract

The application discloses a kind of unmanned aerial vehicle obstacle avoidance methods based on multi-agent graph reinforcement learning, comprising: constructing the heterogeneous graph capable of reflecting the association between UAV and network entity in real time, so that the system can accurately capture the dynamic change of network topology and environmental state, provide more comprehensive state information for subsequent decision-making;Select the system communication rate as the main performance index, and model the communication rate and maximum problem as a partially observable Markov decision process;Construct a heterogeneous graph that can depict the interaction of multiple agents, and use a graph attention network to realize state coding;Propose a collaborative training mechanism based on the multi-agent soft actor-critic framework, which realizes the joint optimization of Actor network and Critic network.The method of the application combines multi-agent reinforcement learning and heterogeneous graph neural network technology, which can maximize the system communication rate and improve the obstacle avoidance and communication decision-making ability of UAV in high dynamic environment.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication, specifically relating to an obstacle avoidance method for unmanned aerial vehicles (UAVs) based on multi-agent graph reinforcement learning. Background Technology

[0002] With the rapid development of wireless communication technology, unmanned aerial vehicles (UAVs) have been widely used in various fields such as intelligent driving, environmental protection, and disaster relief due to their high mobility and rapid deployment capabilities. However, most existing UAV trajectory optimization methods rely on static or simple network topologies, which have significant shortcomings in practical applications.

[0003] UAV communication systems often operate in dynamically changing environments, such as user location movement, fluctuations in communication link status, and real-time changes in obstacle distribution. Faced with these complex factors, traditional decentralized learning methods struggle to adapt to changes in network topology in a timely manner, often leading to degraded system performance, task planning failures, and even increased security risks, lacking sufficient adaptability and robustness.

[0004] In contrast, multi-agent graph reinforcement learning can model the dynamic interactions between agents using graph structures, maintaining good self-learning and adaptive capabilities even when network topology changes. However, current technologies have not yet effectively applied multi-agent graph reinforcement learning to the field of UAV-assisted wireless communication.

[0005] Therefore, there is an urgent need for a new solution that can cope with network topology changes in complex and dynamic environments and further improve the adaptability and robustness of UAV communication systems. Summary of the Invention

[0006] The problem to be solved by this invention is to provide an obstacle avoidance method for unmanned aerial vehicles (UAVs) based on multi-agent graph reinforcement learning, which integrates multi-agent reinforcement learning and heterogeneous graph neural network technology, aiming to maximize the system communication rate and improve the obstacle avoidance and communication decision-making capabilities of UAVs in highly dynamic environments.

[0007] This invention adopts the following technical solution: a drone obstacle avoidance method based on multi-agent graph reinforcement learning, comprising the following steps:

[0008] Step 1: System Initialization and Offline Training

[0009] Construct a UAV-assisted wireless communication system, including multiple UAVs, mobile terminals, and dynamic obstacles, and model the channel link between UAVs and mobile terminals as a probabilistic model;

[0010] UAVs carry base stations and fly within the target area to provide communication services to end users in the area. At the same time, they collect environmental information in real time, including terminal location, link status and obstacle distribution, and dynamically adjust their own three-dimensional flight trajectory and communication strategy based on the acquired data.

[0011] Step 2: Model the UAV obstacle avoidance and communication optimization task as a Partially Observable Markov Decision Process (POMDP), and jointly model and optimize the interaction relationships between the UAV, mobile terminal, and obstacles, including:

[0012] A multi-agent graph reinforcement learning method is introduced, which constructs a heterogeneous graph based on POMDP to represent the interaction relationship between UAV, obstacles and mobile terminal, and uses a graph attention network (GAT) for state encoding.

[0013] A multi-agent soft actor-critic (MASAC) algorithm framework is introduced for collaborative training. Through a mechanism of centralized training and distributed execution, the heterogeneous graph is learned to dynamically optimize the communication strategy and flight trajectory of the UAV.

[0014] Step 3: In each decision time slot, the UAV determines the current optimal strategy based on its own state, the mobile terminal's state, and the obstacle's state. This includes: adjusting the UAV's flight trajectory, transmission power, and the relationship between the UAV and the mobile terminal based on the current environmental state to maximize the sum of system communication rates and ensure secure communication; and executing the next action according to the optimal strategy to complete obstacle avoidance.

[0015] As a preferred option, in step 1, the environment adopts a three-dimensional Cartesian coordinate system, the UAV performs three-dimensional trajectory planning, and the mobile terminal and obstacles are randomly distributed and move on the ground; considering that each time slot interval is very short, it is assumed that the UAV, mobile terminal and obstacles are static in each time slot.

[0016] The entire mission lasted for 10 hours. , equally divided into In a single time slot, each UAV serves multiple end users simultaneously, and each end user is associated with only one UAV.

[0017] As a preferred approach, in step 2, to address the issue that frequent changes in network topology in highly dynamic environments may lead to system performance degradation, the UAV obstacle avoidance and communication optimization task is modeled as a POMDP (Problem-Oriented Multi-Agent Graph Reinforcement Learning) model. Based on this, a multi-agent graph reinforcement learning method is introduced to jointly model and optimize the interaction relationships between UAVs, mobile terminals, and obstacles.

[0018] Specifically, the communication rate maximization problem is modeled as a POMDP model, as follows:

[0019] Each UAV is treated as an intelligent agent, and its state space is represented as follows: The local observation space of each agent is The action space is , Represents a set of time slots;

[0020] In the observation space, UAV The total number of observations is:

[0021] ;

[0022] in, For UAV Local observations, including UAV location Flight speed and remaining energy ; For UAV Observations of other UAVs within the sensing area; For observation of mobile terminals, this includes the location of the mobile terminal, whether the terminal is associated with the current UAV, and whether it is associated with other UAVs; For the observation of obstacles, including the obstacle's coordinates, side length vectors, and height;

[0023] The action space is defined by continuously controlling the actions. Adjust the UAV trajectory in the time slot UAV Actions include acceleration in three-dimensional space and power distribution , represented as ; This represents a set of users.

[0024] As a preferred approach, in each decision-making time slot, the UAV determines the current optimal strategy based on its own state, the state of the mobile terminal, and the state of the obstacle, and then executes the next action according to the strategy.

[0025] In the POMDP model, each UAV agent calculates the final reward function. And a penalty mechanism is introduced in the final reward function:

[0026] ;

[0027] in, Indicates in time slot The sum of communication rates, UAV In the time slot The number of errors made by the agent is recorded, and errors are divided into four types: exceeding boundaries, running out of battery power, violating safe distances, and colliding with obstacles. Each time the agent violates the penalty mechanism, it is penalized with a fixed amount. .

[0028] As a preferred approach, the multi-agent graph reinforcement learning in step 2 constructs a heterogeneous graph based on POMDP to represent the interaction relationships between UAVs, obstacles, and mobile terminals. MASAC efficiently learns from the heterogeneous graph through a mechanism of centralized training and distributed execution, dynamically optimizing the UAV's communication strategy and flight trajectory.

[0029] Specifically, constructing heterogeneous graphs Indicates network environment information;

[0030] in, Represents the set of edges; Represents a set of vertices, including: the UAV vertex set. Mobile terminal vertex set and the set of obstacle vertices , recorded as ;

[0031] Based on the perception relationships between UAVs and network entities, three meta-paths are defined in the heterogeneous graph: UAV to UAV, mobile terminal to UAV, and obstacle to UAV; each path is represented as follows. ,in, It is the source vertex. It is the target vertex, and , ;

[0032] side UAV Located in UAV Within the range of perception; edge Indicates mobile terminal Located in UAV Within the range of perception; edge Indicates obstacles Located in UAV Within the range of perception;

[0033] Each UAV agent in a time slot Observations Convert to heterogeneous graph representation It can be decomposed into a one-way bipartite graph containing three elementary paths, denoted as subgraphs. , and Input image feature encoding module.

[0034] Furthermore, the graph feature encoding module, by introducing an attention mechanism, adaptively captures potential relationships between UAVs and other UAVs, mobile terminals, or obstacles;

[0035] Subgraph Output characteristics Represented as:

[0036] ;

[0037] in, It is the attention value between the UAV vertices and the mobile terminal vertices. This is the softmax function, used to normalize the attention coefficients; It is a learnable weight matrix. It is the feature vector of the mobile terminal; Represents the dot product;

[0038] Subgraph and The output features are respectively and Through multilayer perceptron , and Perform a linear transformation to generate the final output features of the graph feature encoding module. .

[0039] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0040] 1. This invention proposes a joint optimization scheme for UAV obstacle avoidance and communication based on multi-agent graph reinforcement learning, which integrates multi-agent reinforcement learning and heterogeneous graph neural network technology, aiming to maximize the system communication rate and improve the UAV's obstacle avoidance and communication decision-making capabilities in highly dynamic environments.

[0041] 2. This invention introduces heterogeneous graphs to represent the relationships between entities in the network. The state features of the heterogeneous graph are encoded using GAT (Generative Attribution-Based Graph) and then passed as input to the Actor and Critic networks of MASAC. When the network topology dynamically changes due to terminal movement or obstacle changes, the graph structure and policy learning mechanism employed in this invention remain effective, thereby improving the overall communication efficiency and decision-making stability of the system.

[0042] 3. The multi-agent graph reinforcement learning method proposed in this invention fully utilizes the comprehensive representation capabilities of heterogeneous graphs of various entities and their interactions in the network, and combines them with the MASAC algorithm for policy optimization. MASAC uses cumulative reward and policy entropy as value functions in its optimization objective. By introducing policy randomness, it improves the agent's exploration ability during action selection, thereby effectively avoiding getting trapped in local optima and enhancing learning efficiency and decision-making performance in complex dynamic environments. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the UAV-assisted wireless communication system of the present invention.

[0044] Figure 2 This is a flowchart of the MASAC-GAT algorithm of the present invention;

[0045] Figure 3 This is a flowchart of the UAV obstacle avoidance method based on multi-agent graph reinforcement learning according to the present invention;

[0046] Figure 4 The figure shows a comparison of the convergence performance of the MASAC-GAT algorithm of this invention with other multi-agent reinforcement learning algorithms under the system model of this invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0048] In one embodiment of the present invention, a drone obstacle avoidance method based on multi-agent graph reinforcement learning is proposed, the process of which is as follows: Figure 3As shown, firstly, by constructing a heterogeneous graph that reflects the real-time relationships between UAVs and network entities (including other UAVs, mobile terminals, and obstacles), the system can accurately capture the dynamic changes in network topology and environmental state, thus providing more comprehensive state information for subsequent decision-making. Then, the system communication rate is selected as the main performance indicator, and the communication rate maximization problem is modeled as a partially observable Markov decision process (POMDP). Further, a heterogeneous graph characterizing the multi-agent interaction relationships is constructed based on POMDP, and state encoding is implemented using a graph attention network (GAT). On this basis, a collaborative training mechanism based on the multi-agent soft actor-critic (MASAC) framework is proposed to achieve joint optimization of the Actor network and the Critic network.

[0049] (a) System initialization and offline training.

[0050] Constructing a UAV-assisted wireless communication system, such as Figure 1 As shown, including One UAV, mobile terminals and A sparsely distributed and non-overlapping rectangular obstacle, with the terminal and the obstacle moving randomly on the ground.

[0051] In this system environment, each UAV is equipped with a base station and a communication module to provide communication services to end users. The mobile terminal set includes user devices requiring communication support, such as smartphones, tablets, and autonomous vehicles. The dynamic obstacle set encompasses factors that may interfere with the communication link, such as other UAVs, low-altitude aircraft, sudden crowd gatherings, factors causing signal deviations (such as positioning errors due to tall buildings), and temporary no-fly zones (such as areas under large-scale event control). In the same time slot, each UAV can serve multiple end users simultaneously, but each end user can only be associated with one UAV.

[0052] UAVs, equipped with base stations, fly within a target area and provide communication services to end users in that area. During flight, the UAVs collect environmental information in real time, including terminal location, link status, and obstacle distribution, and dynamically adjust their three-dimensional flight trajectory and communication strategies based on the acquired data.

[0053] Let the total duration of the task be... Divide it into equal parts There are 1 time slot, and the length of each time slot is 1. .

[0054] UAV represented by time discretization method and mobile terminals In the time slot The following positions are respectively and .

[0055] in, For position coordinates For position The coordinates; , and These are the minimum and maximum altitudes at which UAVs are allowed to fly, respectively.

[0056] UAV location updates follow the formula below:

[0057] ;

[0058] in, For UAV In the time slot The lower position; UAV In the time slot The speed at that time; UAV In the time slot acceleration at time; This indicates the length of each time slot.

[0059] To ensure the feasibility of the trajectory and the accuracy of control, the flight speed of each UAV is... and acceleration The following conditions must be met:

[0060] ;

[0061] ;

[0062] in, and These are maximum speed and acceleration, respectively.

[0063] In addition, to ensure that UAVs avoid collisions during communication missions, any two UAVs must meet the following requirements:

[0064] ;

[0065] in, For drones Location, A pre-set minimum safe distance.

[0066] Considering that the obstacles in the scene are dynamic, let's assume the obstacles... The center position and the distance vector from the center to the side length are respectively and ;in, The central position is coordinates These are the length and width of the rectangular obstacle, respectively.

[0067] The height of the obstacle; to ensure a safe distance between the UAV and the obstacle, when... When the UAV's horizontal position is not considered, the constraint is ignored; otherwise, the UAV must satisfy the following constraints:

[0068] ;

[0069] in, Represents the horizontal coordinates of the UAV, symbol This indicates a comparison of the magnitudes of each component in the vector.

[0070] These initializations and calculations enable a precise description of the UAV's positional changes in space, providing fundamental data for subsequent trajectory optimization.

[0071] (II) Constructing a communication model

[0072] Considering the impact of obstacles on the communication link, this embodiment models the channel link between the UAV and the mobile terminal as a probabilistic model that combines line-of-sight and non-line-of-sight links.

[0073] UAV To mobile terminals Channel gain is expressed as:

[0074] ;

[0075] in, Indicates channel power gain. It is the path loss index. Indicates time slot UAV and mobile terminals The distance between them This represents the Rician factor.

[0076] UAV and mobile terminals The non-line-of-sight channel coefficients are modeled as a random vector containing independent and identically distributed complex Gaussian variables with a mean of zero and a variance of units. , UAV and mobile terminals Non-line-of-sight channel coefficients between; This indicates the line-of-sight component.

[0077] UAV To mobile terminals The data transmission rate is calculated using the following formula:

[0078] ;

[0079] In the formula, UAV Power distribution, Indicates related decision-making. Indicates channel bandwidth. This represents noise. Therefore, the sum of the system communication rates is:

[0080] ;

[0081] in, Indicates in time slot The sum of communication rates, For UAV To mobile terminals The data transmission rate.

[0082] (III) POMDP Modeling

[0083] In multi-agent reinforcement learning, each UAV is an agent. The state space is represented as follows: The local observation space of each agent is denoted as . The action space is denoted as .

[0084] The specific modeling is as follows:

[0085] (1) Observation space

[0086] UAV Local observation Including: UAV location Flight speed and remaining energy , represented as:

[0087] ;

[0088] In addition to local observations, UAV It can also acquire observations of other UAVs within its sensing area. Observation of mobile terminals and the observation of obstacles Specifically, it can be expressed as:

[0089] ;

[0090] ;

[0091] ;

[0092] Among them, set , , UAV In the time slot The set of all other perceived UAVs, mobile devices, and obstacles;

[0093] Composition and Same, including: other UAV locations Flight speed and remaining energy ;

[0094] Includes: the location of the mobile terminal Is the terminal associated with the current UAV? And whether it is related to other UAVs, It is an indicator function;

[0095] Includes: coordinates of obstacles Side length vector and height .

[0096] UAV The total number of observations is: This is achieved through collaboration between the UAV, mobile terminal, and obstacles. Furthermore, the UAV's sensing area varies with flight altitude, resulting in different observation dimensions for each time slot.

[0097] (2) Action space

[0098] Because the target of optimization is particularly sensitive to the flight trajectory, it is necessary to design continuous control actions to finely adjust the UAV's trajectory.

[0099] In the time slot UAV The action includes its acceleration and power distribution in three-dimensional space, represented as:

[0100] ;

[0101] Among them, set This represents a set of users.

[0102] (3) Reward function

[0103] To further address the limitations of local observation information, the reward is divided into a global term and an individual term. First, the global reward term is calculated. To further promote cooperation among agents:

[0104] ;

[0105] After receiving a round of global rewards, each UAV agent calculates its final reward. .

[0106] To prevent catastrophic events, this embodiment introduces a penalty mechanism in the final reward function, in time slots. Its definition is:

[0107] ;

[0108] in, UAV In the time slot The number of errors committed. Errors are categorized into four types: exceeding boundaries, running out of battery power, violating safe distances, and colliding with obstacles. Each time an agent violates any penalty mechanism, it is penalized with a fixed amount. .

[0109] (iv) MASAC-GAT Algorithm

[0110] The MASAC-GAT algorithm flow in this embodiment includes: constructing a heterogeneous graph, graph feature encoding, and MASAC algorithm processing. The MASAC algorithm processing includes: Actor network action generation, reward calculation for specified actions, and network updating.

[0111] 1. Heterogeneous graph representation of UAVs and network entities

[0112] This embodiment uses heterogeneous graphs. Let V represent the network environment information; where V represents the set of vertices and E represents the set of edges.

[0113] The vertex set contains three categories: UAVs, mobile terminals, and obstacles, denoted as . .

[0114] Based on the perception relationships between UAVs and other network entities, three meta-paths are defined in the heterogeneous graph: 1) UAV to UAV; 2) Mobile terminal to UAV; 3) Obstacle to UAV. Each path is represented as... ,in, It is the source vertex. It is the target vertex, and , It is important to note that since UAVs are intelligent agents, all target vertices are of type UAV.

[0115] Remember the edge UAV Located in UAV Within the range of perception; edge Indicates mobile terminal Located in UAV Within the range of perception; edge Indicates obstacles Located in UAV Within the perception range. Since each UAV agent can only acquire partial environmental information, its observation results are heterogeneous graphs. A local subgraph. The agent in the time slot. Observations Converted into a heterogeneous graph representation This representation can be further decomposed into a one-way bipartite graph containing three meta-paths, denoted as follows: , and This serves as the input to the framework graph feature encoding module.

[0116] 2. Graph Feature Coding

[0117] In the graph feature encoding module, an attention mechanism is introduced to adaptively capture the potential relationships between UAVs and network entities (other UAVs, mobile terminals, or obstacles), helping them allocate more attention to more valuable entities, thereby enhancing their flight strategy.

[0118] UAV Nodes communicate with their neighbors through connections and update their representations based on the states of their neighbors. (Subgraph) For example, its output features Represented as:

[0119] ;

[0120] in, It is the attention value between the UAV vertices and the mobile terminal vertices. The softmax function is used to normalize the attention coefficients, making them comparable across different vertices. It is a learnable weight matrix. This is the feature vector of the mobile terminal. Other subgraphs output features using the same method, resulting in the final subgraph. and The outputs are respectively and ; This represents the dot product.

[0121] Finally, through a multilayer perceptron... , and A linear transformation is performed to finally generate the output of the graph feature encoding module. .

[0122] 3. MASAC Algorithm

[0123] like Figure 2 As shown, the agent in each time slot Interacting with the environment: UAV Based on the current environmental conditions Obtain its local observation And represent it as a graph Input to the Actor network.

[0124] Actor networks are based on current graph representations. Output Action UAV This allows the agent to interact with its environment. The environment then returns rewards based on the agent's actions. and local observations at the next moment Then represent it as a graph. .final, It is stored in the experience buffer pool for use in subsequent training.

[0125] A batch of empirical data is randomly sampled from the empirical replay pool, and then this data is input into the Critic network and the Actor network. The Critic network is used to evaluate the value function of the state-action pairs and outputs... , indicating that during observation Next action The expected cumulative return.

[0126] The goal of the Critic network is to update its parameters by minimizing the temporal difference loss error, the loss function of which is defined as:

[0127] ;

[0128] in, It is a discount factor. Target Critic network, The next action is generated by the current Actor network. Simultaneously, the Actor network updates its parameters by maximizing the Q-value and the log-likelihood function of the action distribution, ensuring that the action chosen by the agent in a given state yields a higher expected reward.

[0129] Its objective function is as follows:

[0130] ;

[0131] in, It is the temperature coefficient, used to balance exploration and utilization; This represents the strategy of the Actor network.

[0132] Furthermore, to maintain training stability, MASAC performs soft updates on both the target Critic network and the target Actor network. The update formula for the target network is:

[0133] ;

[0134] in, It's the rate of soft updates. and This represents the target network before and after the update. This soft update strategy helps mitigate fluctuations during training and maintain network stability.

[0135] (v) Maximize system speed and

[0136] In each decision-making time slot, the UAV determines the current optimal strategy based on its own state, the state of the mobile terminal, and the state of the obstacle. This includes: adjusting the UAV's flight trajectory, transmission power, and the relationship between the UAV and the mobile terminal based on the current environmental state to maximize the sum of system communication rates and ensure secure communication; and executing the next action according to the optimal strategy to complete obstacle avoidance.

[0137] Furthermore, under the high-dynamic hybrid obstacle UAV communication model proposed in this invention, the proposed MASAC-GAT algorithm exhibits excellent performance. For example... Figure 4 As shown, compared with traditional reinforcement learning algorithms MAPPO and MADDPG, the algorithm proposed in this invention has stronger robustness when facing changes in network topology, and at the same time, it has faster convergence speed and higher overall communication rate during training.

[0138] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for obstacle avoidance in unmanned aerial vehicles (UAVs) based on multi-agent graph reinforcement learning, characterized in that, Includes the following steps: Step 1, System Initialization and Offline Training: Construct a UAV-assisted wireless communication system, including multiple UAVs, mobile terminals, and dynamic obstacles, and model the channel link between UAVs and mobile terminals as a probabilistic model; UAVs carry base stations to fly within the target area, providing communication services to end users, collecting environmental information in real time, including terminal location, link status and obstacle distribution, and dynamically adjusting their own three-dimensional flight trajectory and communication strategy based on the acquired data; The probabilistic model combines line-of-sight and non-line-of-sight links, UAV To mobile terminals Channel gain Represented as: ； in, Indicates channel power gain. It is the path loss index. Indicates time slot UAV and mobile terminals The distance between them Represents the Rician factor; Indicates the line-of-sight component. UAV and mobile terminals The non-line-of-sight channel coefficients between them For UAV Time slot The lower position, For mobile terminals In the time slot The lower position; Step 2: Model the UAV obstacle avoidance and communication optimization task as a POMDP, and jointly model and optimize the interaction relationship between UAV, mobile terminal and obstacle, including: A multi-agent graph reinforcement learning method is introduced, and a heterogeneous graph is constructed based on POMDP to represent the interaction relationship between UAV, obstacles and mobile terminal. A graph attention network is used for state encoding. The MASAC algorithm framework is introduced for collaborative training. The heterogeneous graph is learned through a mechanism of centralized training and distributed execution, which dynamically optimizes the communication strategy and flight trajectory of the UAV. The communication rate maximization problem is modeled as a POMDP model, with each UAV treated as an agent, and the state space represented as follows: The local observation space of each agent is The action space is , Represents a set of time slots; In the observation space, UAV The total number of observations is: ； in, For UAV Local observations, including UAV location Flight speed and remaining energy ; For UAV Observations of other UAVs within the sensing area were acquired; For observation of mobile terminals, this includes the location of the mobile terminal, whether the terminal is associated with the current UAV, and whether it is associated with other UAVs; For the observation of obstacles, including the obstacle's coordinates, side length vectors, and height; The action space is defined by continuously controlling the actions. Adjust the UAV trajectory in the time slot UAV Actions include acceleration in three-dimensional space and power distribution , represented as ; Represents a set of users; The MASAC algorithm framework processes the following: The agent in each time slot UAVs interact with the environment Based on the current environmental conditions Obtain local observations , representing the graph Input to the Actor network; The Actor network is based on the current graph representation. Output Action UAV According to the action The agent interacts with its environment, which then returns rewards based on the agent's actions. and local observations at the next moment , representing the graph ,Will Store in the experience replay pool; Experience data is randomly sampled from the experience replay pool and input into the Critic and Actor networks. The Critic network evaluates the value function of the state-action pair and outputs the result. , indicating that during observation Next action Expected cumulative return; The Actor network updates its parameters by maximizing the Q-value and the log-likelihood function of the action distribution, so that the action chosen by the agent in a given state will obtain a higher expected reward. Step 3: In each decision time slot, the UAV determines the current optimal strategy based on its own state, the mobile terminal's state, and the obstacle's state. This includes: adjusting the UAV's flight trajectory, transmission power, and the relationship between the UAV and the mobile terminal based on the current environmental state to maximize the sum of system communication rates and ensure secure communication; and executing the next action according to the optimal strategy to complete obstacle avoidance.

2. The obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that, In step 1, the UAV-assisted wireless communication system includes: One UAV, mobile terminals and A sparsely distributed and non-overlapping set of obstacles; The mobile terminal and obstacles move randomly on the ground, and the entire mission lasts for [duration missing]. , equally divided into There are 1 time slot, and the length of each time slot is 1. In the same time slot, each UAV serves multiple terminal users simultaneously, and each terminal user is associated with only one UAV; within each time slot, the UAV, mobile terminal, and obstacle are static. UAV Time slot The position below is The coordinates are Mobile terminal In the time slot The position below is The coordinates are ; UAV location updates follow: ;in, For UAV In the time slot The lower position; UAV In the time slot The speed at that time; UAV In the time slot acceleration at time; Indicates the length of each time slot; The following conditions must be met between any two UAV positions: ;in, For drones Location, A pre-set minimum safe distance; obstacle The central location is The height is ;obstacle The distance vector from the center to the side length is ; when When, the horizontal position constraint of the UAV is not considered; when At that time, the UAV satisfies the following constraints: ;in, Indicates the horizontal coordinate position of the UAV, symbol This indicates a comparison of the magnitudes of each component in the vector.

3. The obstacle avoidance method for unmanned aerial vehicles according to claim 2, characterized in that, In step 1, the UAV-assisted wireless communication system has a total communication rate of: : ； in, Indicates in time slot The sum of communication rates, Indicates related decision-making; For UAV To mobile terminals The data transmission rate is expressed as: ； in, UAV Power distribution, Indicates channel bandwidth Indicates noise.

4. The obstacle avoidance method for unmanned aerial vehicles according to claim 3, characterized in that, In step 2, in the POMDP model, each UAV agent calculates the final reward function. And a penalty mechanism is introduced in the final reward function: ； in, UAV In the time slot The number of errors made by the agent is recorded, and errors are divided into four types: exceeding boundaries, running out of battery power, violating safe distances, and colliding with obstacles. Each time the agent violates the penalty mechanism, it is penalized with a fixed amount. .

5. The obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that, In step 2, a heterogeneous graph is constructed. Indicates network environment information; in, Represents the set of edges; Represents a set of vertices, including: the UAV vertex set. Mobile terminal vertex set and the set of obstacle vertices , recorded as ; Based on the perception relationships between UAVs and network entities, three meta-paths are defined in the heterogeneous graph: UAV to UAV, mobile terminal to UAV, and obstacle to UAV; each path is represented as follows. ,in, It is the source vertex. It is the target vertex, and , ; side UAV Located in UAV Within the range of perception; edge Indicates mobile terminal Located in UAV Within the range of perception; edge Indicates obstacles Located in UAV Within the range of perception; Each UAV agent in a time slot Observations Convert to heterogeneous graph representation It can be decomposed into a one-way bipartite graph containing three elementary paths, denoted as subgraphs. , and Input image feature encoding module.

6. The obstacle avoidance method for unmanned aerial vehicles according to claim 5, characterized in that, In step 2, the graph feature encoding module adaptively captures potential relationships between UAVs and other UAVs, mobile terminals, or obstacles by introducing an attention mechanism. Subgraph Output characteristics Represented as: ； in, It is the attention value between the UAV vertices and the mobile terminal vertices. This is the softmax function, used to normalize the attention coefficients; It is a learnable weight matrix. It is the feature vector of the mobile terminal; Represents the dot product; Subgraph and The output features are respectively and Through multilayer perceptron , and Perform a linear transformation to generate the final output features of the graph feature encoding module. .

7. The obstacle avoidance method for unmanned aerial vehicles according to claim 6, characterized in that, In step 2, within the MASAC algorithm framework, the goal of the Critic network is to update parameters by minimizing the temporal difference loss error. The loss function is defined as: ； in, Expressing expectations, It is a discount factor. It is the target Critic network. It is the next action generated by the current Actor network; The objective function of the Actor network is: ； in, It is the temperature coefficient, used to balance exploration and utilization; Represents the strategy of the Actor network; Soft updates are performed on both the target Critic network and the target Actor network to maintain training stability. The update formula for the target network is: ； in, It's the rate of soft updates. and This indicates the target network before and after the update.

Images