Unmanned aerial vehicle conflict resolution method based on reinforcement learning and self-attention mechanism

Abstract

The invention provides an unmanned aerial vehicle conflict resolution method based on reinforcement learning and a self-attention mechanism, and the method comprises the steps: firstly, modeling a dynamic relation between an unmanned aerial vehicle and an airspace as an interaction process between an intelligent agent and an environment in reinforcement learning; then, designing an action space, an observation space and a reward function of the intelligent agent, and building a conflict resolution training framework based on reinforcement learning; and finally, designing an unmanned aerial vehicle attention model based on a self-attention mechanism, which comprises an encoder for extracting unmanned aerial vehicle positions and conflict characteristics in a dynamic environment and a decoder for generating unmanned aerial vehicle conflict resolution actions by using an autoregression mode. According to the novel unmanned aerial vehicle free flight conflict resolution method, reasonable combined action distribution can be carried out on multiple unmanned aerial vehicles flying in the airspace, conflict resolution is completed, and flight safety is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent air traffic management, and in particular relates to a method for resolving conflicts between unmanned aerial vehicles (UAVs) based on reinforcement learning and self-attention mechanisms. Background Art

[0002] With the rapid development of the low-altitude economy, the frequency and density of urban airspace use are increasing, and the number and density of drones are rising, which brings more potential conflicts and makes the flight safety and control of drones in low-altitude airspace face greater challenges. In this environment, traditional air traffic management methods are difficult to cope with the complex flight patterns of drones in low-altitude and high-density environments. Therefore, it is particularly important to develop conflict resolution technology for drones, which requires drones to quickly identify potential collision risks in complex and changing environments and make autonomous decisions to perform obstacle avoidance actions.

[0003] The conflict detection and resolution of drones has the following characteristics and difficulties: (1) In the low-altitude airspace structure, drones can autonomously choose resolution strategies when facing conflicts, which leads to complex and diverse drone conflict scenarios and dynamic and changeable resolution strategies. It is necessary to design a resolution strategy generation method for the global situation to reduce the complexity of conflict control. (2) The high density of drone operations and the non-fixed flight routes will make the multi-drone conflict control complex and highly coupled. There may be a combination of multiple scenarios such as intersection, crossing, and convergence at the same time. After clarifying the resolution strategy, it is necessary to determine the conflict control method under multi-drone conflicts to improve the efficiency and scenario adaptability of conflict control. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for resolving UAV conflicts based on reinforcement learning and self-attention mechanism in view of the shortcomings of the prior art. On the basis of completing the UAV motion model and the definition of the operating environment, the relationship between UAVs and UAVs, and UAVs and airspace are modeled as the interactive relationship between the intelligent agent and the environment in reinforcement learning. The self-attention mechanism is used to construct the UAV attention model, and the reinforcement learning training framework is used to train the UAV.

[0005] The method of the present invention comprises the following steps:

[0006] Step 1: Design the conflict resolution problem as a finite Markov decision process, apply reinforcement learning to build and solve the interaction model between the UAV and the airspace environment;

[0007] Step 2: Design the characteristics of the intelligent agent in the interaction model between the drone and the airspace environment;

[0008] Step 3: Design a UAV attention model based on the self-attention mechanism. The encoder captures the positions and conflict characteristics of aircraft in a dynamic environment, as well as the intrinsic connections between UAVs, and the decoder outputs the joint actions of UAVs in an autoregressive manner.

[0009] Step 1 includes: the finite Markov decision process includes elements<S,A,P,R,γ> , where S is the state set, representing all states that the agent may encounter; A is the action set, representing all actions that the agent may take in a certain state; P is the state transition probability, representing the probability distribution of the agent entering the next state after taking a certain action in a certain state; the reward function R represents the immediate reward obtained by the agent when taking a certain action in a certain state; the discount factor γ represents the discount rate of future rewards.

[0010] In step 1, when the state transition probability P is known, a model-based reinforcement learning method is used to solve the finite Markov decision process: at the beginning of the run, the environment is initialized and an initial state is provided for the agent, which is a drone. The drone is regarded as an agent and actions are assigned, that is, different types of maneuvers are performed in the airspace; in the next time step, all drones will reach a new state by executing or completing their maneuvers at the previous moment; and in the new state, the agent will receive a set of rewards, each of which is related to the corresponding drone.

[0011] Step 2 includes: calculating the height change value Δh when the drone performs the height adjustment strategy by the following formula:

[0012] Δh=v h *(t keep -t change )

[0013] Among them, v h Indicates the vertical speed of the drone, t change Indicates the moment when the drone starts to climb or descend; t keep Indicates the moment when the drone levels off;

[0014] The action space is designed as a 5-dimensional discrete action space A vert :

[0015] A vert =[noaction,climb,decsent,keep vert ,return vert ]

[0016] Among them, noaction means that the agent does not take any maneuvers at the current moment, climb means that the drone starts to climb, descent means that the drone starts to descend, and keep means that the drone starts to climb.vert Indicates that the drone maintains the current altitude, return vert Indicates the altitude the drone returned to.

[0017] Step 2 also includes: the heading maneuvering space of the UAV is designed as a four-dimensional discrete space A hori :

[0018] A hori =[noaction,deviate,keep hori ,return hori ]

[0019] Among them, deviate means that the drone starts to deviate from the current route, corresponding to the time of rerouting keep hori Indicates that the drone returns to the original heading angle. hori Indicates that the drone returns to its original route.

[0020] Step 2 also includes: the action space of speed adjustment is defined as A spd :

[0021] A spd =[Noaction,spd,return spd ]

[0022] Among them, spd means that the drone starts to adjust the flight speed, return spd Returns to original speed.

[0023] Step 2 also includes: setting the local drone as i, the surrounding drones as j, and the observation of drone i on j includes the following parameters:

[0024] Relative situation information R ij : Describes the position and speed information of UAV j relative to UAV i. Taking the body coordinate axis of UAV i as the reference coordinate system, the position and speed of UAV j are projected onto the body axis of UAV i to obtain the corresponding radial distance Horizontal distance and vertical distance At the same time, in order to express dynamic information, the relative velocity is projected onto the body axis of UAV i to obtain the relative radial velocity Relative lateral speed and relative vertical velocity The relative situation information is expressed as:

[0025]

[0026] Conflict Information C ij: The conflict information describes the conflict relationship between UAV j and UAV i, including the conflict probability Conflict Corner and moments of conflict The conflict information is expressed as:

[0027]

[0028] When there is no conflict between UAV j and UAV i, record C ij =(0,0,0);

[0029] The observation of UAV i on UAV j is denoted as o ij =(R ij ,C ij ), using the UAV maneuvering state M i Indicates the current state of the drone, represented by a one-dimensional code. The observation space of drone i is O i :

[0030] O i ={o i1 ,o i2 ,…,o ij ,…o iN ,M i},j≠i,

[0031] Where N is the number of drones in the airspace, o iN Represents the observation of UAV i on UAV N.

[0032] Step 2 also includes: The terminal value condition of the UAV during operation is defined as: if the horizontal interval between any two UAVs in the airspace is less than the minimum allowable interval d sp and the vertical distance is less than d h If both of them occur at the same time, it is determined that the conflict resolution has failed and the terminal condition has been reached; if the flight time t of the drone is greater than or equal to the maximum time T max , then it is determined that the conflict is resolved successfully and the final value condition is met. The other cases are considered to be failure to meet the final value condition. The expression is:

[0033]

[0034] Among them, d ij With h ij Represent the horizontal distance and height difference between drones i and j, respectively, and determine the value of the final state Done. The values ​​of the final state Done include: collision D crash , escape successfully D success Resolve ongoing conflict execute; When the environment reaches the terminal value state, only the conflicting drone will be rewarded, and the surrounding drones without conflict will not be rewarded. The failure rate of drone release is defined as r los The ratio of the number of unsuccessful drones to the number of starting drones when the termination condition is reached:

[0035]

[0036] in is the number of conflicting drones when the termination condition is reached, n if is the number of conflicting drones under initial conditions;

[0037] For the altitude-adjusted drone, the reward function R vertical Designed for:

[0038]

[0039] Among them, Δh is the height change of the UAV during maneuvering, h norm is a hyperparameter, e is a natural constant;

[0040] For heading adjustment of the drone, the reward function R horizontal for:

[0041]

[0042] Among them, Δγ is the maximum heading angle change during the UAV maneuver, γ norm is a hyperparameter;

[0043] For the speed-adjusted drone, the reward function R speed for:

[0044]

[0045] Among them, Δv is the maximum speed change of the UAV during maneuvering, v norm is a hyperparameter.

[0046] Step 3 includes: the drone attention model includes an observation encoder and an autoregressive drone maneuver decoder, the observation encoder is used to learn the intrinsic expression of the drone joint observation, and the autoregressive drone maneuver decoder outputs the optimal action of each drone in an autoregressive manner;

[0047] The local situation fusion module is used to capture the relevant information of the local UAV i on all surrounding UAVs j, and the relative observation sequence of the surrounding UAVs, that is, the observation of UAV i on UAV j, is recorded as o ij , and are input into the local situation fusion module in sequence. The final output is passed through a one-dimensional convolution layer to obtain a fixed-dimensional expression, which is then combined with the drone i’s own state Mi The embedded expression is vectorized and the local observation situation expression of the UAV is output

[0048] Step 3 includes: the drone attention model uses a self-attention mechanism and a reinforcement learning algorithm to output results, including the following steps:

[0049] Step 3-1, input: batch data size B, number of agents n, number of random conflict scenarios K, number of maneuvers per scenario T;

[0050] Initialization: Encoder {φ 0}, decoder {θ 0}, playback buffer;

[0051] Step 3-2, for each conflict scenario k = 0, 1, ..., K-1, for time step t = 0, 1, ..., T-1, obtain the joint local observation sequence from the environment Represents the observation value of UAV n at time t.

[0052] Step 3-3, sequence Input encoder to get the intrinsic representation of the joint observation space is the observation value of UAV n at time t after passing through the encoder.

[0053] Step 3-4: Express the inner Input to decoder;

[0054] Step 3-5, for drone m = 0, 1, ..., n-1, place drone i 0 action Drone m action Input to the decoder and infer the drone i m+1 Action

[0055] Steps 3-6, perform joint actions And get reward R(o t ,a t ), where o t with a t They represent the set of observation values ​​and joint actions of all UAVs at time t respectively;

[0056] Step 3-7, (o t ,a t ,R(o t ,a t ))Save to playback buffer;

[0057] Step 3-8, randomly sampling a training sample with a data size of B from the playback buffer;

[0058] Step 3-9, input the training sample into the encoder to obtain drone i 1 The state estimation function value Drone n The state estimation function value

[0059] Step 3-10, calculate the encoder loss function L encoder (φ);

[0060] Step 3-11, use the Generalized Advantage Estimator (GAE) method according to Calculate the joint advantage function

[0061] Step 3-12, place the drone II 1 The intrinsic expression observed value Drone n The intrinsic expression observed value and dronei 1 Precursor action Drone n Precursor action Input into the decoder to generate drone i 1 Strategy distribution Drone n Strategy distribution

[0062] Step 3-13, calculate the decoder loss function L Decoder (θ);

[0063] Step 3-14, update the encoder and decoder parameters by minimizing the following loss function using the gradient descent method:

[0064] The total loss function L = L encoder (φ)+L decoder (θ)+L entropy , where L entropy is the cross entropy loss in the network model.

[0065] The principle of the present invention is to apply the self-attention mechanism to extract the conflict characteristics and internal connections between drones, complete the conflict resolution action allocation, and use the reinforcement learning training framework to optimize the effect of drone conflict resolution.

[0066] Compared with the prior art, the present invention has the following beneficial effects: the present invention realizes the generation of a release strategy for the global situation in dynamic and diverse scenarios, and determines a conflict control method under multi-machine conflicts to improve the efficiency and scenario adaptability of conflict control. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0068] Figure 1 It is a schematic diagram of the technical route of the present invention.

[0069] Figure 2 It is a schematic diagram of the interaction between drones and airspace environment under the reinforcement learning framework.

[0070] Figure 3 It is a diagram of the overall structure of the UAV conflict resolution operating environment and model under the reinforcement learning framework.

[0071] Figure 4 It is a schematic diagram of the trajectories corresponding to route expansion and route deviation in the UAV horizontal adjustment strategy.

[0072] Figure 5 It is the flight trajectory diagram of the UAV within the set flight range.

[0073] Figure 6 It is a schematic diagram of the speed changes of each UAV during operation. DETAILED DESCRIPTION

[0074] like Figure 1 As shown, an embodiment of the present invention provides a method for resolving conflicts between drones based on reinforcement learning and self-attention mechanism, including:

[0075] Establish a model training framework based on reinforcement learning and UAV features, including conflict resolution training environment and intelligent agent control, such as Figure 2As shown. The conflict resolution training environment part builds a virtual simulation training environment to simulate the behavior and reaction of the UAV in various complex scenarios. The training process starts with initialization. The system sets all necessary parameters, including the environment state and the initial configuration of the UAV, to ensure that the training starts from a consistent state. Then, the scene recognition module obtains various information in the environment after initialization, including the current position of the UAV, flight status, and potential sources of conflict. This data is crucial for training how the UAV reacts in a real environment. After obtaining the scene information, the system generates corresponding maneuvering instructions, which are conveyed to the UAV through the UAV maneuvering module, commanding it to perform specific flight actions, such as avoiding obstacles or adjusting the flight path. The entire process is displayed in real time through the visualization module, allowing researchers to intuitively observe the performance and behavior of the UAV in different scenarios.

[0076] The control agent is the core of the entire system. It is responsible for making decisions based on environmental information and commanding the drone to perform specific operations. After the scene recognition module provides the state information of the current environment, the control agent first processes and understands this information through the state perception module to form a comprehensive understanding of the current scene. Then, based on these perceptions, the agent makes the next action decision through the action generation module. This decision-making process is completely dependent on the strategies previously obtained through reinforcement learning. These strategies are the result of repeated adjustments and optimizations of the agent during the training process. Through this series of perception and decision-making steps, the agent directly affects the behavior of the drone, which is the key to ensuring that the drone can successfully avoid conflicts and complete the task.

[0077] Each time the drone performs an action, the system enters the learning and updating phase, further optimizing the agent's decision-making ability by evaluating the effect of the action. Specifically, the system uses a reward function to evaluate the drone's performance, which is based on indicators such as whether the drone successfully avoids conflicts and whether it maintains a safe flight distance. Through these feedbacks, the control agent will adjust its internal strategy to perform better in subsequent training. This continuous learning and updating process is the key to the gradual improvement of the entire system and ultimately reaching an efficient and stable state. Through continuous learning and optimization, the agent enables the drone to autonomously fly safely and avoid conflicts in more diverse scenarios.

[0078] Then follow these steps:

[0079] Step 1: Design the conflict resolution problem as a finite Markov decision process and apply reinforcement learning to model and solve it;

[0080] Reinforcement learning is often used to solve Markov decision processes (MDPs). Markov processes consist of five elements:<S,A,P,R,γ> It consists of: S is the state set, which represents all the states that the agent may encounter; A is the action set, which represents all the actions that the agent may take in a certain state; P is the state transition probability, which represents the probability distribution of the agent entering the next state after taking a certain action in a certain state; the reward function R represents the immediate reward that the agent can obtain by taking a certain action in a certain state; the discount factor represents the discount rate of future rewards.

[0081] When the state transition probability P is known, the problem can be solved by a model-based reinforcement learning method: at the beginning of the operation, the environment is initialized and the initial state is provided for the agent (drone). The drone is regarded as an agent and assigned actions, that is, different types of maneuvers are performed in the airspace. In the next time step, all drones will reach a new state by executing or completing their maneuvers at the previous moment. And in the new state, the agent will receive a set of rewards, each of which is related to the corresponding drone. The interactive operation process of the drone and the airspace environment is as follows: Figure 3 shown.

[0082] Step 2: Design the characteristics of the agent in the model, including the local observation value, state value, action space and reward function of the drone;

[0083] (1) Agent action space A

[0084] Altitude release control parameters: In free flight, the altitude of the drone is generally relatively fixed. The altitude release strategy is divided into temporary climbing altitude and temporary descending altitude. This strategy instructs the cruise drone to climb / descend to a certain altitude, and then return to the cruise altitude after a period of time. The parameters that need to be controlled are the start climbing / descending time t change ; Start leveling time t keep and the moment t at which the vehicle starts to return to its original height return The descent rate of the drone is relatively stable when climbing / descending, which is v h The altitude change when the drone performs altitude adjustment strategy is calculated by the following formula:

[0085] Δh=v h *(t keep -t change )

[0086] For a drone in the cruising phase, its altitude adjustment action space is designed to determine at each decision moment whether to adjust the altitude, whether to level, and whether to return to altitude. Therefore, the action space in the present invention is designed as a 5-dimensional discrete action space:

[0087] A vert =[noaction,climb,decsent,keep,return]

[0088] Among them, noaction means that the agent does not take maneuvers at the current moment, and climb / descent means that the drone starts to climb / descent. It can be seen that there is a temporal sequence relationship between actions, that is, keep can only be selected after climb / descent.

[0089] (2) Heading release control parameters

[0090] In the context of free flight, drones often decide on an optimal trajectory during the cruising phase. In the time dimension considered in this chapter's experiments, the drone's cruising trajectory can be considered as flying in a straight line toward the target point. At this time, the drone's heading release strategies are:

[0091] PS: The UAV adjusts its heading angle and maintains it for a period of time, then returns to the original waypoint. The decision parameter is the time of rerouting t change and return time t return , and there is also the angle γ for the change of course.

[0092] Route deviation (RO): Similar to path extension, but involves maintaining flight in a trajectory parallel to the original flight plan, so three decision moments are required. They are the start time of the diversion, t change , the moment t at which the heading angle is converted to the original keep and the time t to return to the original route return .

[0093] The schematic diagram of route expansion (PS) and route deviation (RO) is as follows Figure 4 As shown. Therefore, the heading maneuvering space of the UAV is designed as a four-dimensional discrete space A hori :

[0094] A hori =[noaction,deviate,keep,return]

[0095] Among them, deviate means starting to deviate from the current route, corresponding to the time of rerouting t change , keep means returning to the original heading angle and continuing to fly, while return means returning to the previous route.

[0096] (3) Speed ​​release control parameters

[0097] Temporary speed change: refers to the drone changing its speed for a period of time and then adjusting back to the set speed. Including acceleration\deceleration

[0098] Permanent speed change: After the drone changes speed, it will not adjust back to the set speed.

[0099] For the speed adjustment strategy, it is necessary to determine when to start adjusting the speed and the size of the speed adjustment Δv. Its action space design also follows the design patterns of the first two action spaces. The action space of speed adjustment is defined as A spd :

[0100] A spd =[Noaction,spd,return]

[0101] Among them, spd means that the drone starts to adjust its flight speed, return means returning to the original speed, and the size of the speed adjustment Δv is also determined by searching the conflict probability.

[0102] (4) Agent Observation Space O

[0103] In multi-UAV conflict resolution, the agent's observation space is the relative situation information of other UAVs in the airspace that the conflicting UAV can observe, which provides important information support for the UAV's maneuvering decision. Its observation perspective is from the UAV. Let the local UAV be i and the surrounding UAV be j. The observation of UAV i on UAV j includes the following contents:

[0104] Relative situation information R ij : Describes the position and speed information of UAV j relative to UAV i. Taking the body coordinate axis of UAV i as the reference coordinate system, the position and speed of UAV j are projected onto the body axis of UAV i to obtain the corresponding radial distance Horizontal distance and vertical distance At the same time, in order to express dynamic information, the relative velocity is also projected onto the body axis of UAV i to obtain the relative radial velocity Relative lateral speed and relative vertical velocity Therefore, the relative situation information can be expressed as:

[0105]

[0106] Conflict Information C ij : The conflict information describes the conflict relationship between UAV j and UAV i, including the conflict probability Conflict Corner and moments of conflict Then the conflict information can be expressed as:

[0107]

[0108] When there is no conflict between UAV j and UAV i, record C ij =(0,0,0);

[0109] The observation of UAV i on UAV j is denoted as o ij =(R uj ,C ij ), in addition, as described in the action space analysis, the UAV maneuver state M u Indicates the current state of the drone, represented by a one-dimensional code. The overall observation space of drone i consists of its observations of all drones in the airspace. The observation space of drone i is O u :

[0110] O u ={o u1 ,o u2 ,…,o ij ,…o iN ,M i},j≠i,

[0111] Where N is the number of drones in the airspace.

[0112] (5) Reward function R

[0113] The reward function is used to characterize the UAV’s i Take an action a i The design of the reward function is crucial for the learning and training of the agent, because both the state value function and the action value function are essentially updated based on the value of the reward r(o,a). In the present invention, only after the conflict is resolved, the corresponding terminal value reward is given according to the actual result. This design belongs to sparse rewards in reinforcement learning.

[0114] To this end, the terminal value condition of the UAV during operation is defined as: if the horizontal interval between any two UAVs in the airspace is less than the minimum allowable interval d sp and the vertical distance is less than d h If both of them occur at the same time, it is considered that the conflict resolution has failed and the terminal condition has been reached; if the flight time of the drone exceeds the maximum time T max , then the conflict is considered to be resolved successfully and the final value condition is met. The remaining cases are considered to be failure to meet the final value condition, and the expression is:

[0115]

[0116] When the environment reaches the final value state, only the conflicting drones will be rewarded, and the surrounding drones without conflicts will not be rewarded. Since the reward function is a sparse reward, the scene of failed release also needs to give a reward value in the end to indicate how "failed" the release is, or how close it is to "success". The "failure ratio" of drone release is defined as the ratio of the number of drones that failed to be released to the number of starting drones when the termination condition is reached:

[0117]

[0118] in is the number of conflicting drones when the termination condition is reached, n if is the number of conflicting drones under the initial conditions.

[0119] For the altitude adjustment drone, its reward function R vertical Designed for:

[0120]

[0121] Among them, Δh is the maximum height change of the UAV during maneuvering, h norm It is a hyperparameter, and its purpose is to balance the variable dimension problems of altitude release, heading release, and speed release. For the drone that completes the entire altitude adjustment maneuver (i.e. returns to the original route), an additional reward of 1 is given.

[0122] For heading adjustment of the drone, the reward function R horizontal for:

[0123]

[0124] Among them, Δγ is the maximum heading angle change during the UAV maneuver, γ norm is a hyperparameter with the same purpose as above. For the drone that completes the entire heading adjustment maneuver (i.e. returns to the original route), an additional reward of 1 is given.

[0125] For the speed-adjusted drone, the reward function R speed for:

[0126]

[0127] Among them, Δv is the maximum speed change of the UAV during maneuvering, v norm is a hyperparameter with the same purpose as above. For the drone that completes the entire speed adjustment maneuver (i.e. returns to the original speed), an additional reward of 1 is given.

[0128] Step 3: Design a UAV attention model based on the self-attention mechanism. The encoder captures the positions and conflict characteristics of aircraft in a dynamic environment, as well as the intrinsic connections between UAVs. The decoder outputs the joint actions of UAVs in an autoregressive manner.

[0129] The drone attention model consists of an observation encoder and an autoregressive drone maneuver decoder, such as Figure 3 As shown in Figure 2, the former learns the intrinsic expression of joint observations of UAVs, and the latter outputs the optimal action of each UAV in an autoregressive manner. (1) Local Situation Fusion Module

[0130] Before the encoder uses the attention mechanism to capture the relationship between the observation sequence and action sequence of multiple agents, the relative relationship between the local drone and the surrounding drones in the observation sequence of the agent (drone) is captured by the local situation fusion module. The main function of this module is to capture the relevant information of the local drone i on all surrounding drones j∈n,j≠i. The local observation space of agent i on surrounding drone j is defined as o ij =(R ij ,C ij ), where R ij Represents relative situation information, C ij Represents conflict information. The observation space of agent i is:

[0131] O i ={o i1 ,o i2 ,…,o ij ,…o iN ,M i},j≠i

[0132] Since the number of surrounding drones j is changing in real time, that is, the dimension of the observation space of drone i is also changing, a simple method is to assume that the number of drones has a maximum value. Surrounding drones exceeding the maximum value are not considered, but this will cause various problems in implementation. The relative observation sequence of surrounding drones is input into the local situation fusion module in sequence, and the final output is passed through a one-dimensional convolution layer to obtain a fixed-dimensional expression, which is then combined with the drone's own state M i The embedded expression is vectorized and the local observation situation expression of the UAV is output For any variable number of surrounding drones, the dimension of expression is also fixed and unique. Through this network structure, information capture of all drones in the airspace can be achieved.

[0133] (2) Encoder

[0134] The encoder of the multi-agent attention mechanism is parameterized by φ, which takes a sequence of observations (O 1,…,o n ) and pass them through several encoding layers. Each encoding layer consists of a self-attention mechanism and a multi-layer perceptron with residual connections. Residual connections can prevent the problem of gradient disappearance and network performance degradation as the depth increases. The output of the multi-agent observation space encoder is expressed as It not only affects the intelligent agent (i 1 ,…,i n ) and also captures the high-level connections of agent interactions. In order to improve the learning ability of the encoder. During the training phase, the encoder aims to approximate the state value function, and its loss function is to minimize the Bellman error:

[0135]

[0136] in It is the parameter of the target network. After some iterations, it will be updated to the main network parameter in order to alleviate the problem of dynamic update.

[0137] (3) Decoder

[0138] The decoder parameters are denoted by θ, and its input is the joint action sequence embedded in (where i 0 is an arbitrary symbol indicating the start of decoding) to the decoding layer. Each decoding block has a masked self-attention mechanism to prevent the current decoded sequence from seeing the subsequent sequence information during training, thereby ensuring sequential updates. Then there is a second masked attention module, which calculates the attention between the joint action head and the joint observation encoding output. Finally, the output layer result is connected through full connection and residual connection. The decoding layer output of the last layer is the joint action embedding of the agent Finally, the agent i is output through a fully connected structure m The action distribution is:

[0139]

[0140] The training of the decoder is similar to that of the Actor in the Actor-Critic structure, where the pruned Proximal Policy Gradient (PPO) method is used.

[0141]

[0142] in is the importance weight of the new and old strategies, To estimate the joint advantage function, we can use the Generalized Advantage Estimation (GAE) method. In the inference stage, each action is generated in an autoregressive manner, that is, will be fed into the decoder again to generate In the training phase, the output of all actions can be computed in parallel because have been collected and stored in the replay buffer.

[0143] In order to ensure the exploratory nature of the strategy, cross entropy loss is added to the optimization process of the model to prevent the strategy from falling into an overly deterministic state, thereby improving the robustness and generalization ability of the algorithm. The expression of cross entropy loss is as follows:

[0144] L entropy =-W entropy ∑ a π(a|s)logπ(a|s).

[0145] The designed self-attention mechanism-reinforcement learning algorithm is as follows:

[0146] Step 3-1, input: batch data size B, number of agents n, number of random conflict scenarios K, number of maneuvers per scenario T;

[0147] Initialization: Encoder {φ 0}, decoder {θ 0}, playback buffer;

[0148] Step 3-2, for each conflict scenario k = 0, 1, ..., K-1, for time step t = 0, 1, ..., T-1, obtain the joint local observation sequence from the environment Represents the observation value of UAV n at time t.

[0149] Step 3-3, sequence Input encoder to get the intrinsic representation of the joint observation space is the observation value of UAV n at time t after passing through the encoder.

[0150] Step 3-4: Express the inner Input to decoder;

[0151] Step 3-5, for drone m = 0, 1, ..., n-1, place drone i 0 action Drone m action Input to the decoder and infer the drone im+1 Action

[0152] Steps 3-6, perform joint actions And get reward R(o t ,a t ), where o t with a t Represents the set of observations and joint actions of all drones at time t;

[0153] Step 3-7, (o t ,a t ,R(o t ,a t ))Save to playback buffer;

[0154] Step 3-8, randomly sampling a training sample with a data size of B from the playback buffer;

[0155] Step 3-9, input the training sample into the encoder to obtain drone i 1 The state estimation function value Drone n The state estimation function value

[0156] Step 3-10, calculate the encoder loss function L encoder (φ);

[0157] Step 3-11, use the Generalized Advantage Estimator (GAE) method according to Calculate the joint advantage function

[0158] Step 3-12: Place the drone i 1 The intrinsic expression observed value Drone n The intrinsic expression observed value and dronei 1 Precursor action Drone n Precursor action Input into the decoder to generate drone i 1 Strategy distribution Drone n Strategy distribution

[0159] Step 3-13, calculate the decoder loss function L Decoder (θ);

[0160] Step 3-14, update the encoder and decoder parameters by minimizing the following loss function using the gradient descent method:

[0161] The total loss function L = L encoder (φ)+L decoder (θ)+L entropy , where L entropy is the cross entropy loss in the network model.

[0162] Step 4: Based on the above environment construction and model building, complete the drone conflict resolution example:

[0163] like Figure 5 The figure shows the flight trajectory of the drone within the set flight range. In the figure, X, Y and H are the horizontal, vertical and height coordinates of the drone in the simulation environment. This simulation scene conforms to the real situation of free flight of low-altitude drones in cities. Figure 6 The horizontal axis is the simulation time during the test, and the vertical axis is the speed of the drone. In the initial stage of the simulation, there are conflicting pairs / groups between drones. To resolve the initial conflicts, the model assigns actions to each agent based on the conflict characteristics and internal connections of the drones:

[0164] Drone K2 speed adjustment: 10.2m / s→19.7m / s;

[0165] Drone K3 speed adjustment: 8.7m / s→17.6m / s;

[0166] Drone K5 heading adjustment: 45° to the left;

[0167] Drone K8 altitude adjustment: 105m→85m→105m;

[0168] Drone K9 altitude adjustment: 85m → 140m → 85m.

[0169] The present invention provides a method for resolving conflicts between drones based on reinforcement learning and self-attention mechanisms. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A UAV conflict resolution method based on reinforcement learning and self-attention mechanism, characterized in that: The following steps are involved: Step 1: Design the conflict resolution problem as a finite Markov decision process, apply reinforcement learning to build and solve the interaction model between the UAV and the airspace environment; Step 2: Design the characteristics of the intelligent agent in the interaction model between the drone and the airspace environment; Step 3: Design a UAV attention model based on the self-attention mechanism. The encoder captures the positions and conflict characteristics of aircraft in a dynamic environment, as well as the intrinsic connections between UAVs, and the decoder outputs the joint actions of UAVs in an autoregressive manner.

2. The method according to claim 1, characterized in that: Step 1 includes: the finite Markov decision process includes elements <S, A, P, R, γ>, where S is a state set, representing all states that the agent may encounter; A is an action set, representing all actions that the agent may take in a certain state; P is a state transition probability; the reward function R represents the immediate reward obtained by the agent for taking a certain action in a state; and the discount factor γ represents the discount rate of future rewards.

3. The method according to claim 2, characterized in that In step 1, when the state transition probability P is known, a model-based reinforcement learning method is used to solve the finite Markov decision process: at the beginning of the run, the environment is initialized and an initial state is provided for the agent, which is a drone. The drone is regarded as an agent and actions are assigned, that is, different types of maneuvers are performed in the airspace; in the next time step, all drones will reach a new state by executing or completing their maneuvers at the previous moment; and in the new state, the agent will receive a set of rewards, each of which is related to the corresponding drone.

4. The method according to claim 3, characterized in that Step 2 includes: calculating the height change value Δh when the drone performs the height adjustment strategy by the following formula: Δh=v h *(t keep -t change ) Among them, v h Indicates the vertical speed of the drone, t change Indicates the moment when the drone starts to climb or descend; t keep Indicates the moment when the drone levels off; The action space is designed as a 5-dimensional discrete action space A vert : A vert =[noaction,climb,decsent,keep vert ,return vert ] Among them, noaction means that the agent does not take any maneuvers at the current moment, climb means that the drone starts to climb, descent means that the drone starts to descend, and keep means that the drone starts to descend. vert Indicates that the drone maintains the current altitude, return vert Indicates the altitude the drone returned to.

5. The method according to claim 4, characterized in that Step 2 also includes: the heading maneuvering space of the UAV is designed as a four-dimensional discrete space A hori : A hori =[noaction,deviate,keep hori ,return hori ] Among them, deviate means that the drone starts to deviate from the current route, corresponding to the time of rerouting keep hori Indicates that the drone returns to the original heading angle. hori Indicates that the drone returns to its original route.

6. The method according to claim 5, characterized in that Step 2 also includes: the action space of speed adjustment is defined as A spd : A spd =[Noaction,spd,return spd ] Among them, spd means that the drone starts to adjust the flight speed, return spd Returns to original speed.

7. The method according to claim 6, characterized in that Step 2 also includes: setting the local drone as i, the surrounding drones as j, and the observation of drone i on j includes the following parameters: Relative situation information R ij : Describes the position and speed information of UAV j relative to UAV i. Taking the body coordinate axis of UAV i as the reference coordinate system, the position and speed of UAV j are projected onto the body axis of UAV i to obtain the corresponding radial distance Horizontal distance and vertical distance At the same time, in order to express dynamic information, the relative velocity is projected onto the body axis of UAV i to obtain the relative radial velocity Relative lateral speed and relative vertical velocity The relative situation information is expressed as: Conflict Information C ij : The conflict information describes the conflict relationship between UAV j and UAV i, including the conflict probability Conflict Corner and moments of conflict The conflict information is expressed as: When there is no conflict between UAV j and UAV i, record C ij =(0,0,0); The observation of UAV i on UAV j is denoted as o ij =(R ij , C ij ), using the UAV maneuvering state M i Indicates the current state of the drone, represented by a one-dimensional code. The observation space of drone i is O i : THE i ={the i1 ,the i2 ,...,the ij ,...the iN ,M i },j≠i, Where N is the number of drones in the airspace, o iN Represents the observation of UAV i on UAV N.

8. The method according to claim 7, characterized in that Step 2 also includes: The terminal value condition of the UAV during operation is defined as: if the horizontal interval between any two UAVs in the airspace is less than the minimum allowable interval d sp and the vertical distance is less than d h If both of them occur at the same time, it is determined that the conflict resolution has failed and the terminal condition has been reached; if the flight time t of the drone is greater than or equal to the maximum time T max , then it is determined that the conflict is resolved successfully and the final value condition is met. The other cases are considered to be failure to meet the final value condition. The expression is: Among them, d ij With h ij Represent the horizontal distance and height difference between drones i and j, respectively, and determine the value of the final state Done. The values ​​of the final state Done include: collision D crash , escape successfully D success Resolve ongoing conflict execute ; When the environment reaches the terminal value state, only the conflicting drone will be rewarded, and the surrounding drones without conflict will not be rewarded. The failure rate of drone release is defined as r los The ratio of the number of unsuccessful drones to the number of starting drones when the termination condition is reached: in is the number of conflicting drones when the termination condition is reached, n if is the number of conflicting drones under initial conditions; For the altitude-adjusted drone, the reward function R vertical Designed for: Among them, Δh is the height change of the UAV during maneuvering, h norm is a hyperparameter, e is a natural constant; For heading adjustment of the drone, the reward function R horizontal for: Among them, Δγ is the maximum heading angle change during the UAV maneuver, γnor m is a hyperparameter; For the speed-adjusted drone, the reward function R speed for: Among them, Δv is the maximum speed change of the UAV during maneuvering, v horm is a hyperparameter.

9. The method according to claim 8, characterized in that Step 3 includes: the drone attention model includes an observation encoder and an autoregressive drone maneuver decoder, the observation encoder is used to learn the intrinsic expression of the drone joint observation, and the autoregressive drone maneuver decoder outputs the optimal action of each drone in an autoregressive manner; The local situation fusion module is used to capture the relevant information of the local UAV i on all surrounding UAVs j, and the relative observation sequence of the surrounding UAVs, that is, the observation of UAV i on UAV j, is recorded as o ij , and are input into the local situation fusion module in sequence. The final output is passed through a one-dimensional convolution layer to obtain a fixed-dimensional expression, which is then combined with the drone i’s own state M i The embedded expression is vectorized and the local observation situation expression of the UAV is output 10. The method according to claim 9, characterized in that Step 3 includes: the drone attention model uses a self-attention mechanism and a reinforcement learning algorithm to output results, including the following steps: Step 3-1, input: batch data size B, number of agents n, number of random conflict scenarios K, number of maneuvers per scenario T; Initialization: encoder {φ0), decoder {θ0), playback buffer; Step 3-2, for each conflict scenario k = 0, 1, ..., K-1, for time step t = 0, 1, ..., T-1, obtain the joint local observation sequence from the environment represents the observation value of UAV n at time t; Step 3-3, sequence Input encoder to get the intrinsic representation of the joint observation space is the observation value of UAV n at time t after passing through the encoder; Step 3-4: Express the inner Input to decoder; Step 3-5, for drone m=0, 1, ..., n-1, move drone i0 ..., dronei m action Input to the decoder and infer the drone i m+1 Action Steps 3-6, perform joint actions And get reward R(o t , a t ), where o t with a t They represent the set of observation values ​​and joint actions of all UAVs at time t respectively; Step 3-7, (o t , a t , R(o t , a t ))Save to playback buffer; Step 3-8, randomly sampling a training sample with a data size of B from the playback buffer; Step 3-9, input the training sample into the encoder to obtain the state estimation function value of drone i1 ..., dronei n The state estimation function value Step 3-10, calculate the encoder loss function L encoder (φ); Step 3-11, using the generalized advantage estimation GAE method according to Calculate the joint advantage function Step 3-12: The intrinsic expression observation value of drone i1 ..., dronei n The intrinsic expression observed value And the previous action of drone i1 ..., dronei n Precursor action Input into the decoder to generate the strategy distribution of drone i1 ..., dronei n Strategy distribution Step 3-13, calculate the decoder loss function L Decoder (θ); Step 3-14, update the encoder and decoder parameters by minimizing the following loss function using the gradient descent method: The total loss function L = L encoder (φ)+L decoder (θ)+L entropy , where L entropy is the cross entropy loss in the network model.

Images