A kind of anti-collision flight control method for distributed cluster of multi-fixed-wing unmanned aerial vehicle

Abstract

The application discloses a kind of multi fixed wing unmanned aerial vehicle distributed cluster's anti-collision flight control method, first the entity in the observable range of each unmanned aerial vehicle is divided into different groups according to its characteristics and attributes. Design a group-invariant network architecture based on space-time attention, for processing environmental observation. The multi-agent deep deterministic policy gradient algorithm is improved using parameter sharing and truncated double Q learning techniques to effectively optimize the control policy. The application models the distributed cluster and collision avoidance control problem of scalable fixed-wing multi-unmanned aerial vehicle as a distributed partially observable Markov decision process model, and proposes a space-time attention multi-agent actor-critic. In this algorithm, a group-invariant network architecture based on space-time attention is designed, consisting of local spatial attention and global temporal attention modules. Through this architecture, STAAC can adapt to changes in the number of unmanned aerial vehicles and intruders.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and unmanned aerial vehicle (UAV) systems, and in particular to a collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs. Background Technology

[0002] Compared to rotary-wing UAVs, fixed-wing UAVs have significant advantages in range, endurance, and speed. Therefore, fixed-wing UAV swarms have broad application prospects, especially in industrial and military activities. However, due to the complex dynamics and non-holonomic constraints of fixed-wing UAVs, achieving swarm flight is quite difficult. To date, controlling the swarm flight of multiple fixed-wing UAVs remains a significant challenge.

[0003] In scenarios involving multiple drones, collision avoidance becomes more challenging because collisions between drones and obstacles, as well as collisions between drones and intruding drones, must be considered simultaneously. Currently, deep reinforcement learning is increasingly being applied in this field. However, existing research only considers static obstacles in simple scenarios, and the effectiveness of current methods in complex scenarios with dynamic obstacles requires further investigation. Summary of the Invention

[0004] Objective: This invention aims to provide a collision avoidance flight control method for distributed swarms of multiple fixed-wing UAVs. The invention considers the distributed swarming and collision avoidance flight control problem of multiple fixed-wing UAVs in a dynamic environment with varying numbers of intruders. Compared to previous work, this problem is more challenging. First, it involves multiple optimization objectives, making it difficult for UAVs to learn a swarming strategy that simultaneously satisfies UAV-UAV collision avoidance and UAV-intruder collision avoidance constraints. Furthermore, the size of the UAV swarm and the number of intruders are dynamically changing, requiring the swarm controller to be scalable and adaptable to variations in the number of intruders.

[0005] Technical solution: The present invention provides a collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs, comprising the following steps:

[0006] Step 1) Construct a multi-agent deep reinforcement learning framework based on an actuator network and an evaluator network, and set up a group invariant network based on a spatiotemporal attention mechanism in the multi-agent deep reinforcement learning framework. The group invariant network based on the spatiotemporal attention mechanism is used to aggregate the UAV state vectors of various dimensions input to the actuator network and the evaluator network.

[0007] Step 2): Multiple UAVs continuously interact with the environment to obtain observation vectors. The actuator network outputs the actions of each UAV based on the observation vectors. The agent executes the action and obtains the reward value. The above interaction data is saved in the experience pool, and the actuator network parameters and evaluator network parameters are updated and saved.

[0008] Step 3) Load the trained actuator network parameters, map the observation vectors obtained by the UAV from the environment to the action space, and the UAV performs the action to avoid dynamic obstacles and complete the distributed swarm and autonomous collision avoidance task.

[0009] Furthermore, in step 1), the actuator network adopts a group-invariant network architecture based on a spatiotemporal attention mechanism, including:

[0010] Step 1.1.1): Based on the characteristics and attributes of the entities, cluster the entities within the observable range of the agent into different groups; specifically, from the perspective of each wingman i, the entities within the observable range are divided into four groups: itself, the lead aircraft, neighboring wingmen, and neighboring intruders.

[0011] Step 1.1.2): Compute the spatial embedding of each group separately using the Local Spatial Attention (LSA) module;

[0012] Step 1.1.3) Use the Global Temporal Attention (GTA) module to selectively aggregate temporal features from different groups;

[0013] Step 1.1.4) embeds global temporal attention into the drone's actions.

[0014] Further, step 1.1.2) specifically involves: using the Local Spatial Attention (LSA) module to calculate the spatial embedding of each group individually; the spatial embedding of neighboring wingman i at time t can be defined as the aggregated state of all neighboring wingmen:

[0015]

[0016] Spatial attention weights Determined by the following formula;

[0017]

[0018] in, Indicates similarity. Let represent the state vector of wingman i at time t. Let represent the set of state vectors of neighboring wingmen i, (·). T It's a transpose operation; FC(·) is a fully connected FC layer; W fol The parameter to be learned, d fol yes The dimension; similarly, calculate the aggregate spatial embedding of the neighboring intruder at time t:

[0019]

[0020] Spatial attention weights Determined by the following formula:

[0021]

[0022] in, Indicates similarity. W represents the state vector of the neighboring intruder k. int The parameter to be learned, d int yes The dimension;

[0023] The spatial embeddings of the lead aircraft and wingman i at time step t are respectively represented as follows:

[0024]

[0025] in, This represents the state vector of the primary aircraft. Let i represent the state vector of wingman i.

[0026] Further, step 1.1.3) specifically involves: using the Global Temporal Attention (GTA) module to selectively aggregate temporal features from different groups; specifically, firstly, using a Long Short-Term Memory (LSTM) network to extract temporal features individually from the historical observations of each group:

[0027]

[0028] Where LSTM(·) represents the LSTM layer.

[0029] Global temporal embedding of agent i at time τ Defined as:

[0030]

[0031] Where || represents a concatenation operation; then, a time attention mechanism is used to distinguish the importance of different time slots, and the time attention weights are... The calculation is as follows:

[0032]

[0033] Global temporal attention embedding of agent i at time step t Defined as:

[0034]

[0035] Furthermore, step 1.1.4) specifically involves:

[0036] Global temporal attention is embedded into v using the following formula. i t Actions mapped to drones

[0037]

[0038] MLP(·) is a two-layer multilayer perceptron MLP.

[0039] Furthermore, in step 1), the evaluator network adopts a group-invariant network architecture based on a spatiotemporal attention mechanism, including:

[0040] Step 1.2.1): Use a spatiotemporal attention network to represent the environmental state. The specific calculation method is the same as in steps 1.1.1 to 1.1.3.

[0041]

[0042] Where s t Let STAN(·) represent the environment state vector of agent i at time t, and let STAN(·) represent the spatiotemporal attention network. This represents the embedded environmental state features after encoding;

[0043] Step 1.2.2): Use the FC layer as an action encoder to process joint actions. The motion encoder is defined as:

[0044]

[0045] Step 1.2.3) embeds the global temporal attention of the environment state. With motion embedding Connect them together and pass them to the MLP to approximate the Q value of agent i:

[0046]

[0047] Furthermore, step 2) specifically includes the following steps:

[0048] Step 2.1), Obtaining Observation and Environment State Vectors: The observations for each agent are defined as follows:

[0049]

[0050] in It is the state of the lead aircraft at time step τ; It is the state of wingman i at time step τ; It is the set of states of neighboring wingmen of wingman i; It is the set of states of neighboring intruders of wingman i; the system state includes the states of all entities:

[0051]

[0052] in It is the state of the lead aircraft at time step τ; It is a set of states of wingmen; It is the set of states of the intruding machine;

[0053] Step 2.2), Define the action space: All wingmen have the same action space, and each wingman i defines its action in the continuous space as follows:

[0054]

[0055] Among them, angular velocity of heading Forward acceleration

[0056] Step 2.3), calculate the reward function: the reward function for each wingman i is defined as:

[0057]

[0058] in and These represent the wingman following reward, the drone-drone collision avoidance reward, and the drone-intruder collision avoidance reward, respectively; specifically, to encourage wingmen to group with the wingman, Defined as:

[0059]

[0060] in This is the distance from the lead aircraft to the wingman i at time step t; P1 >> 0 is a large positive constant; w1 > 0 is an adjustment parameter; R a and R s These are the warning radius and minimum safe radius of the wingman, respectively;

[0061] To punish wingman i for colliding with the j-th neighboring wingman, Defined as;

[0062]

[0063] Where P2 >> 0 is a large positive constant;

[0064] w2 > 0 is an adjustment parameter; to prevent wingman i from colliding with the kth neighboring intruding aircraft. Defined as:

[0065]

[0066] Step 2.4), collect experience data: The agent continuously interacts with the environment, and the historical trajectories generated during this process are stored in the experience replay pool; a shared experience pool is used to store the experiences collected simultaneously by all agents, and the experiences are stored in the form of tuples (s t ,o t ,a t ,r t ,o t+1 ,s t+1 );in This represents the observations of all agents at time step t; This represents the actions of all agents at time step t. This represents the reward for all agents at time step t; the interaction process is as follows: the agent obtains the observation vector from the environment, the actuator network outputs the actions of each wingman according to the observation vector, the agent executes the action to obtain the state vector at the next moment, and then calculates the reward value according to the reward function;

[0067] Step 2.5), Network parameter update: Randomly sample from the experience pool, extracting mini-batches to update the evaluator and executor based on the spatiotemporal attention network architecture; the parameters θ of the policy network. μ Update based on policy gradient:

[0068]

[0069] in Let μ represent the experience pool, μ represent the policy network, and Q1 represent the executor network.

[0070] Parameters of the evaluator Optimize by minimizing the following loss:

[0071]

[0072] in The target Q value is represented by [value]. To reduce overestimation bias, a truncated double Q learning technique is introduced into the STAAC algorithm; specifically, the parameter [value] is learned in the same way as Q1. An additional executor Q2 forms the target Q value based on the minimum of the two Q values ​​estimated by Q1 and Q2;

[0073]

[0074] Smooth target motion Defined as;

[0075]

[0076] in And c represents Gaussian noise ∈ aThe variance and boundary conditions; it is important to note that μ - and These represent the policy and the target network of the two executors, respectively, with corresponding parameters. and Updated according to the following formula:

[0077]

[0078] Step 2.6): When the number of training rounds reaches the preset condition, the update process ends and the executor network parameters are saved.

[0079] Furthermore, step 3) specifically includes the following steps:

[0080] Step 3.1), observation vector acquisition: The state vector consists of four parts, including the lead aircraft state, the wingman's own state, the neighboring wingman's state, and the neighboring intruding aircraft's state;

[0081] Step 3.2) Construct an actuator network based on a spatiotemporal attention network architecture, load the trained network parameters, use the agent's current time step state vector as the network input, and output the agent's heading angular velocity and forward acceleration in the current state;

[0082] Step 3.3): The agent performs the action and obtains the state vector for the next time step;

[0083] Step 3.4), repeat steps 3.1-3.4 until the lead aircraft reaches the preset target point, completing the distributed cluster and autonomous collision avoidance mission.

[0084] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0085] (1) This invention employs a multi-agent deep reinforcement learning framework based on a spatiotemporal attention mechanism to achieve distributed collision avoidance control of UAV swarms through an actuator network and an evaluator network. This framework can not only achieve collision avoidance between UAVs in dynamic and complex environments, but also effectively address the collision risk between UAVs and intruding aircraft, thus improving system security.

[0086] (2) A local spatial attention module and a global temporal attention module are adopted, enabling the system to automatically aggregate key spatial and temporal features and flexibly adjust strategies according to changes in the actual environment. By aggregating dynamic obstacle and environmental information through the spatiotemporal attention mechanism, the UAV can better adapt to different numbers and distributions of intruders, thus possessing strong scalability and robustness.

[0087] (3) By introducing optimization methods such as experience pooling and truncated double-Q learning, this invention can efficiently utilize historical interaction data to improve learning efficiency and reduce overestimation bias. At the same time, based on the experience pooling sharing mechanism, multiple agents can acquire, share and reuse data in parallel, thereby effectively reducing data requirements and training costs. Attached Figure Description

[0088] Figure 1 This is a block diagram of the spatiotemporal attention multi-agent actuator-evaluator algorithm;

[0089] Figure 2 It is a population-invariant network architecture based on spatiotemporal attention mechanism;

[0090] Figure 3 This is the trajectory of the wingman using the learned STAAC strategy to avoid collisions with 10 intruding aircraft while maintaining contact with the lead aircraft cluster. The final position and direction of movement of the entities are marked on the trajectory with dots and arrows.

[0091] Figure 4 This is the performance curve of the learned STAAC strategy in the test. Detailed Implementation

[0092] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0093] A distributed swarming and collision avoidance flight control method for multi-fixed-wing UAVs based on spatiotemporal attention multi-agent reinforcement learning, the specific contents of which include:

[0094] 1. The swarming and collision avoidance problem of scalable fixed-wing UAV formations in dynamic environments is comprehensively described as a distributed partially observable Markov decision process (Dec-POMDP). The observations, states, actions, and rewards of each wingman are defined as follows:

[0095] 1.1 Observation Space:

[0096] Given the limited perception capabilities of wingmen, the environment is only partially observable. To mitigate this problem, this invention stacks the latest four frames of historical information for decision-making. Each wingman... Stacked observations at time step t can be defined as follows:

[0097]

[0098] 1.2 State Representation: System state includes the state of all entities. System state can be represented as...

[0099]

[0100] 1.3 Action Space: All followers have the same action space. Each wingman i defines its action in the continuous space as follows:

[0101]

[0102] 1.4 Reward Function: To ensure that wingmen and lead aircraft avoid collisions with neighboring wingmen and intruding aircraft when flying in a group, the reward function for each wingman i is defined as:

[0103]

[0104] 2. Since drone swarms are scalable and the number of intruders varies, the observation dimension of each wingman is dynamic. To represent dynamic dimension observations as fixed-size embeddings, a novel group-invariant network architecture based on spatiotemporal attention is designed.

[0105] 2.1.1 Spatiotemporal Attention Network: The spatiotemporal attention network first clusters entities in the environment into different groups. Then, a Local Spatial Attention (LSA) module is used to compute the spatial embedding of each group individually. Finally, a Global Temporal Attention (GTA) module is designed to extract the temporal embeddings of different groups.

[0106] 2.1.2 Entity Clustering: Multi-UAV swarm flight scenarios in dynamic environments involve different types of entities. Considering the heterogeneity and differences among these entities, it is more effective for the agent to process them individually. Therefore, firstly, based on the characteristics and attributes of the entities, the entities within the agent's observable range are clustered into different groups. Specifically, from the perspective of each wingman i, the entities within the observation range can be divided into four groups: itself, the lead aircraft, neighboring wingmen, and neighboring intruding aircraft. Next, each group is processed individually using the LSA module.

[0107] 2.1.3 Local Spatial Attention (LSA): Entities within the same group have different spatial features. Spatial features can be better extracted by distinguishing the importance of different entities. Taking a group of neighboring wingmen as an example, the spatial embedding of a neighboring wingman at time t can be defined as the aggregated state of all neighboring wingmen:

[0108]

[0109] Spatial attention weights Determined by the following formula:

[0110]

[0111] in(·) T It's a transpose operation; FC(·) is a fully connected (FC) layer; W fol The parameter to be learned, d fol yes The dimension of the space. Similarly, the aggregate spatial embedding of the neighboring intruders at time t can be calculated:

[0112]

[0113] It is noteworthy that the dimension of the spatial embedding of each group is independent of the number of entities in the group. When there is only one entity in a group, the spatial attention weight becomes a scalar of 1, indicating that the previously mentioned local spatial attention operation LSA(·) can degenerate into an FC(·) function for a group with a single entity. Therefore, the spatial embeddings of the lead aircraft and wingman i at time step t can be expressed as follows:

[0114]

[0115] 2.1.4 Global Temporal Attention (GTA): Following the LSA module, a GTA module is used to selectively aggregate temporal features from different groups. To improve some observability issues, four frames of past observation data are stacked for each wingman in swarm flight and collision avoidance control tasks. Since the role and contribution of observation data at different times are different, clearly distinguishing the importance of data at different times is beneficial for wingmen to achieve their control objectives. Therefore, a recurrent neural network with a temporal attention mechanism is integrated into the GTA module. Specifically, a Long Short-Term Memory (LSTM) network is first used to extract temporal features separately from the historical observations of each group:

[0116]

[0117] Where LSTM(·) represents the LSTM layer.

[0118] The global temporal embedding of agent i at time τ can be defined as:

[0119]

[0120] Here, || represents a concatenation operation. A temporal attention mechanism is then used to distinguish the importance of different time slots. Temporal attention weights. The following can be calculated:

[0121]

[0122] The global temporal attention embedding of agent i at time step t can be defined as:

[0123]

[0124] 2.2 Policy Network: Now we define the structure of the policy network, first using a spatiotemporal attention network to represent the agent's observations:

[0125]

[0126] Where STAN(·) represents the spatiotemporal attention network defined in 2.1.1. Then, the embeddings are... Global temporal attention is mapped to the action space:

[0127]

[0128] MLP(·) is a two-layer multilayer perceptron (MLP).

[0129] 2.3 Value Network: The structure of the value network is similar to that of the policy network, except that the value network adds an FC layer as an action encoder to process joint actions. Specifically, a spatiotemporal attention network is first used to represent the environmental state:

[0130]

[0131] The motion encoder is defined as:

[0132]

[0133] Then, embed global temporal attention. With motion embedding Connect them together and pass them to the MLP to approximate the Q value of the executor i:

[0134]

[0135] 3. STAAC Algorithm Update Process: To improve learning efficiency and scalability, STAAC introduces a parameter sharing mechanism, where different agents share the same control policy. Therefore, STAAC only needs to learn one shared policy and one centralized evaluator for all agents. Simultaneously, a shared experience pool is used to store the experiences collected simultaneously by all agents. Specifically, experiences are stored in the form of tuples (s...). t ,o t ,a t ,r t ,o t+1 ,s t+1 ).in This represents the observations of all agents at time step t. Similarly,

[0136] The policy and evaluator are updated using experience from the experience pool through random sampling of mini-batches. Assume... Let μ represent the experience pool, μ represent the policy network, and θ represent the parameters of the policy network. μ Update based on policy gradient:

[0137]

[0138] Where Q1 represents the actuator network. The parameters of the evaluator... Optimize by minimizing the following loss:

[0139]

[0140] in Let Q represent the target Q value. To reduce overestimation bias, a truncated double Q learning technique is introduced into the STAAC algorithm. Specifically, the parameterized double Q is learned in the same way as Q1. An additional executor Q2 forms the target Q value based on the minimum of the two Q values ​​estimated by Q1 and Q2;

[0141]

[0142] Smooth target motion Defined as;

[0143]

[0144] in And c represents Gaussian noise ∈ a The variance and boundary conditions. It is important to note that μ... - and These represent the policy and the target network of the two executors, respectively, with corresponding parameters. and Updated according to the following formula:

[0145]

[0146] Specific implementation steps:

[0147] Step 1: Set the maximum capacity N of the experience pool and the maximum time step N for each learning stage. s Batch size N b Update frequency C;

[0148] Step 2: Randomly initialize the network parameters θ of the strategy and the two evaluators. μ and

[0149] Step 3: Initialize the target network:

[0150] Step 4: Initialize the experience pool with the maximum capacity N.

[0151] Step 5 proceeds to the following loop;

[0152] Step 5-1-1: Randomly initialize the state of all entities;

[0153] Step 5-1-2 Receive initial state s 1 and initial observation value o 1 ;

[0154] Environmental state at time t t The definition is as follows;

[0155]

[0156] in It is a set of states of wingmen; It is the set of states of the intruding machine.

[0157] The definition is as follows:

[0158]

[0159] in It is the state of the lead aircraft at time step τ; It is the state of wingman i at time step τ; It is the set of states of neighboring wingmen of wingman i; It is the set of states of the neighboring intruders of wingman i;

[0160] Step 5-2-1 From time step t=1 to N s Perform the following loop;

[0161] Step 5-2-1-1 Select an action based on the current strategy and the noise exploration:

[0162] action The definition is as follows;

[0163]

[0164] in

[0165] Step 5-2-1-2: Perform the combined action and update the system state s t+1 ;

[0166] Step 5-2-1-3 Calculate the instant reward r i t according to:

[0167]

[0168] in and These represent the wingman following reward, the drone-drone collision avoidance reward, and the drone-intruder collision avoidance reward, respectively. Specifically, to encourage wingmen to group with the wingman, Defined as;

[0169]

[0170] in This is the distance from the lead aircraft to the wingman i at time step t; P1 >> 0 is a large positive constant; w1 > 0 is an adjustment parameter; R a and R s These are the warning radius and minimum safe radius of the wingman, respectively.

[0171] To punish wingman i for colliding with the j-th neighboring wingman, Defined as;

[0172]

[0173] Where P2 >> 0 is a large positive constant, and w2 > 0 is an adjustment parameter. Similarly, to prevent wingman i from colliding with the kth neighboring intruding aircraft, Defined as;

[0174]

[0175] Step 5-2-1-4 Receive new observation results t+1

[0176] Step 5-2-1-6 will (s) t ,o t ,a t ,r t ,o t+1 ,s t+1 Deposit in This represents the observations of all agents at time step t. Similarly,

[0177] Step 5-2-1-7 Update the state space and observation space s t ←s t+1 ;o t ←o t+1 ;

[0178] Step 5-2-1-8: Randomly sample from the experience pool of size N. b The sample;

[0179] Step 5-2-1-9 updates by minimizing the loss function.

[0180] Specifically, the parameters of the evaluator Optimize by minimizing the loss;

[0181]

[0182] in Let Q represent the target Q value. To reduce overestimation bias, truncated double Q learning is introduced into the STAAC algorithm. Specifically, first, the parameterized double Q is learned in the same way as Q1. An additional evaluator Q2 is used, and then the minimum of the two Q values ​​estimated by Q1 and Q2 is used to form the target Q value;

[0183]

[0184] Step 5-2-2 Perform the following steps every C steps:

[0185] Step 5-2-2-1 Update the policy parameter θ using the following gradient function. μ ;

[0186]

[0187] Q1 represents the evaluator network.

[0188] Step 5-2-2-2 Update the target network parameters according to the following formula and

[0189]

[0190] Figure 2 This is a schematic diagram of the network structure used in an embodiment of the present invention. In the LSA module, the FC layers of the lead machine, self-wingman, neighboring wingman, and neighboring intruder have 64, 64, 128, and 128 neurons, respectively. All FC layers are activated by the Modified Linear Unit (ReLU) function. Learnable parameters and In the GTA module, the LSTM layers for the lead machine, self-wingman, neighboring wingman, and neighboring intruder have 64, 64, 128, and 128 hidden units, respectively. For the actuator and evaluator, the MLP network contains two fully connected (FC) layers. Their hidden layers have 256 neurons, but their output layers have 2 and 1 neurons, activated by hyperbolic tangent (tanh) and linear functions, respectively.

[0191] Figure 3 The flight trajectory of the fixed-wing UAV swarm obtained in this embodiment of the invention is shown. In the test scenario of this embodiment, 10 intrusion aircraft are distributed within a rectangular mission area of ​​1200m × 400m. The lead aircraft and the intrusion aircraft fly along a predetermined trajectory, while 10 wingmen follow the lead aircraft using the learned STAAC strategy. The test mission lasts for a total of 60 seconds. As shown in the figure, regardless of the cluttered space in the early stage of the test or the free space in the later stage, the wingmen can successfully keep up with the lead aircraft, stay away from the intrusion aircraft, and successfully avoid collisions with each other.

[0192] The cluster performance curves of the STAAC strategy in the test are as follows: Figure 4 As shown, the distance between the lead aircraft and the wingman remained within 50 to 100 meters for most of the time. Furthermore, the minimum distance between the wingmen and the minimum distance between the intruding aircraft and the wingman at each time step were both greater than the wingman's minimum safe radius R. s =15m. This means that no collisions occurred between wingmen or between wingmen and intruding aircraft during the entire test. The above results demonstrate that the method proposed in this invention enables drone swarms to fly safely in dynamic obstacle environments.

[0193] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0194] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs, characterized in that, Includes the following steps: Step 1) Construct a multi-agent deep reinforcement learning framework based on an actuator network and an evaluator network, and set up a group invariant network based on a spatiotemporal attention mechanism in the multi-agent deep reinforcement learning framework. The group invariant network based on the spatiotemporal attention mechanism is used to aggregate the UAV state vectors of various dimensions input to the actuator network and the evaluator network. In step 1), the actuator network adopts a group-invariant network architecture based on a spatiotemporal attention mechanism, including: Step 1.1.1): Based on the characteristics and attributes of the entities, cluster the entities within the observable range of the agent into different groups; from the perspective of each wingman i, the entities within the observable range are divided into four groups: itself, the lead aircraft, neighboring wingmen, and neighboring intruders. Step 1.1.2): Compute the spatial embedding of each group separately using the Local Spatial Attention (LSA) module; Step 1.1.3) Use the Global Temporal Attention (GTA) module to selectively aggregate temporal features from different groups; Step 1.1.4) embeds the global temporal attention into the drone's actions; Step 2): Multiple UAVs continuously interact with the environment to obtain observation vectors. The actuator network outputs the actions of each UAV based on the observation vectors. The agent executes the action and obtains the reward value. The above interaction data is saved in the experience pool, and the actuator network parameters and evaluator network parameters are updated and saved. Step 3) Load the trained actuator network parameters, map the observation vectors obtained by the UAV from the environment to the action space, and the UAV performs the action to avoid dynamic obstacles and complete the distributed swarm and autonomous collision avoidance task.

2. The collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs according to claim 1, characterized in that, Step 1.1.2) specifically involves: using the Local Spatial Attention (LSA) module to calculate the spatial embedding of each group individually; the spatial embedding of neighboring wingman i at time t can be defined as the aggregated state of all neighboring wingmen: Spatial attention weights Determined by the following formula: in, Indicates similarity. Let represent the state vector of wingman i at time t. Let represent the set of state vectors of neighboring wingmen i, (·). T It's a transpose operation; FC(·) is a fully connected FC layer; W fol The parameter to be learned, d fol yes The dimension; similarly, calculate the aggregate spatial embedding of the neighboring intruder at time t: Spatial attention weights Determined by the following formula: in, Indicates similarity. W represents the state vector of the neighboring intruder k. int The parameter to be learned, d int yes The dimension; The spatial embeddings of the lead aircraft and wingman i at time step t are respectively represented as: in, This represents the state vector of the primary aircraft. Let i represent the state vector of wingman i.

3. The collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs according to claim 1, characterized in that, Step 1.1.3) specifically involves: using the Global Temporal Attention (GTA) module to selectively aggregate temporal features from different groups; specifically, firstly, using a Long Short-Term Memory (LSTM) network to extract temporal features individually from the historical observations of each group: Where LSTM(·) represents the LSTM layer. Global temporal embedding of agent i at time τ Defined as: Where || represents a concatenation operation; then, a time attention mechanism is used to distinguish the importance of different time slots, and the time attention weights are... The calculation is as follows: Global temporal attention embedding of agent i at time step t Defined as:

4. The collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs according to claim 1, characterized in that, Step 1.1.4) specifically refers to: Global temporal attention is embedded into v using the following formula. i t Actions mapped to drones MLP(·) is a two-layer multilayer perceptron MLP.

5. The collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs according to claim 1, characterized in that, In step 1), the evaluator network adopts a group-invariant network architecture based on a spatiotemporal attention mechanism, including: Step 1.2.1): Use a spatiotemporal attention network to represent the environmental state. The specific calculation method is the same as in steps 1.1.1 to 1.1.

3. Where s t Let STAN(·) represent the environment state vector of agent i at time t, and let STAN(·) represent the spatiotemporal attention network. This represents the embedded environmental state features after encoding; Step 1.2.2): Use the FC layer as an action encoder to process joint actions. The motion encoder is defined as: Step 1.2.3) embeds the global temporal attention of the environment state. With motion embedding Connect them together and pass them to the MLP to approximate the Q value of agent i:

6. The collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs according to claim 1, characterized in that, Step 2) specifically includes the following steps: Step 2.1), Obtaining Observation and Environment State Vectors: The observations for each agent are defined as follows: in It is the state of the lead aircraft at time step τ; It is the state of wingman i at time step τ; It is the set of states of neighboring wingmen of wingman i; It is the set of states of neighboring intruders of wingman i; the system state includes the states of all entities: in It is the state of the lead aircraft at time step τ; It is a set of states of wingmen; It is the set of states of the intruding machine; Step 2.2), Define the action space: All wingmen have the same action space, and each wingman i defines its action in the continuous space as follows: Among them, angular velocity of heading Forward acceleration Step 2.3), calculate the reward function: the reward function for each wingman i is defined as: in and These represent the wingman following reward, the drone-drone collision avoidance reward, and the drone-intruder collision avoidance reward, respectively; specifically, to encourage wingmen to group with the wingman, Defined as: in This is the distance from the lead aircraft to the wingman i at time step t; P1 >> 0 is a large positive constant; w1 > 0 is an adjustment parameter; R a and R s These are the warning radius and minimum safe radius of the wingman, respectively; To punish wingman i for colliding with the j-th neighboring wingman, Defined as: Where P2>>0 is a large positive number; w2 > 0 is an adjustment parameter; To prevent wingman i from colliding with the kth neighboring intruding aircraft, Defined as Step 2.4), collect experience data: the agent continuously interacts with the environment, and the historical trajectories generated in this process will be stored in the experience replay pool; A shared experience pool is used to store the experiences collected simultaneously by all agents; experiences are stored in the form of tuples (s t ,o t ,a t ,r t ,o t+1 ,s t+1 );in This represents the observations of all agents at time step t; This represents the actions of all agents at time step t. This represents the reward for all agents at time step t; the interaction process is as follows: the agent obtains the observation vector from the environment, the actuator network outputs the actions of each wingman according to the observation vector, the agent executes the action to obtain the state vector at the next moment, and then calculates the reward value according to the reward function; Step 2.5), Network parameter update: Randomly sample from the experience pool, extracting mini-batches to update the evaluator and executor based on the spatiotemporal attention network architecture; the parameters θ of the policy network. μ Update based on policy gradient: in Let μ represent the experience pool, μ represent the policy network, and Q1 represent the executor network. Parameters of the evaluator Optimize by minimizing the following loss: in Indicates the target Q value; To reduce overestimation bias, a truncated double-Q learning technique is introduced into the STAAC algorithm; specifically, the parameterized double-Q learning technique is learned in the same way as Q1. An additional executor Q2 forms the target Q value based on the minimum of the two Q values ​​estimated by Q1 and Q2; Smooth target motion Defined as; in And c represents Gaussian noise ∈ a variance and boundary conditions; μ - and These represent the policy and the target network of the two executors, respectively, with corresponding parameters. and Updated according to the following formula: Step 2.6): When the number of training rounds reaches the preset condition, the update process ends and the executor network parameters are saved.

7. The collision avoidance flight control method for a distributed swarm of multiple fixed-wing UAVs according to claim 1, characterized in that, Step 3) specifically includes the following steps: Step 3.1), observation vector acquisition: The state vector consists of four parts, including the lead aircraft state, the wingman's own state, the neighboring wingman's state, and the neighboring intruding aircraft's state; Step 3.2) Construct an actuator network based on a spatiotemporal attention network architecture, load the trained network parameters, use the agent's current time step state vector as the network input, and output the agent's heading angular velocity and forward acceleration in the current state; Step 3.3): The agent performs the action and obtains the state vector for the next time step; Step 3.4), repeat steps 3.1-3.4 until the lead aircraft reaches the preset target point, completing the distributed cluster and autonomous collision avoidance mission.

Images