Multi-fixed-wing unmanned aerial vehicle distributed cluster anti-collision flight control method

Abstract

The invention discloses an anti-collision flight control method for a multi-fixed-wing unmanned aerial vehicle distributed cluster. Firstly, entities in an observable range of each unmanned aerial vehicle are divided into different groups according to characteristics and attributes of the entities; a group invariant network architecture based on space-time attention is designed and is used for processing environment observation. A multi-agent depth deterministic strategy gradient algorithm is improved by using a parameter sharing and truncation double-Q learning technology so as to effectively optimize a control strategy. According to the method, a distributed cluster and anti-collision control problem of the extensible fixed-wing multiple unmanned aerial vehicles is modeled as a distributed partially observable Markov decision process model, and a space-time attention multi-agent actuator-evaluator is provided. In the algorithm, a group invariant network architecture based on space-time attention is designed, and the group invariant network architecture is composed of a local space attention module and a global time attention module. Through the architecture, the STAAC can adapt to the change of the number of the unmanned aerial vehicles and the intruding vehicles.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and unmanned aerial vehicle system technology, and in particular to an anti-collision flight control method for a distributed cluster of multiple fixed-wing unmanned aerial vehicles. Background Art

[0002] Compared with rotary-wing UAVs, fixed-wing UAVs have significant advantages in range, endurance and flight speed. Therefore, fixed-wing UAV swarms have broad application prospects, especially in industrial and military activities. However, due to the complex dynamics and non-complete constraints of fixed-wing UAVs, it is difficult for fixed-wing UAVs to achieve swarm flight. So far, swarm flight control of multiple fixed-wing UAVs still faces huge challenges.

[0003] In scenarios involving multiple drones, the collision avoidance problem becomes more challenging because both the collision between drones and obstacles and the collision between drones and intruders must be considered simultaneously. Currently, deep reinforcement learning is increasingly being used in this field. However, existing studies have only considered static obstacles in simple scenarios, and the effectiveness of existing methods in complex scenarios with dynamic obstacles needs further study. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a collision avoidance flight control method for a distributed cluster of multiple fixed-wing UAVs. The present invention considers the distributed clustering and collision avoidance flight control problem of multiple fixed-wing UAVs in a dynamic environment with different numbers of intruders. Compared with previous work, this problem is more challenging. First, this problem involves multiple optimization objectives, and it is difficult for UAVs to learn a clustering strategy that simultaneously satisfies UAV-UAV collision avoidance and UAV-intruder collision avoidance constraints. In addition, the size of the UAV formation and the number of intruders are dynamically changing, and the cluster controller must be scalable and able to adapt to changes in the number of intruders.

[0005] Technical solution: The present invention provides a method for anti-collision flight control of a distributed cluster of multiple fixed-wing UAVs, comprising the following steps:

[0006] Step 1), constructing a multi-agent deep reinforcement learning framework based on an actuator network and an evaluator network, and setting a group invariant network based on a spatiotemporal attention mechanism in the multi-agent deep reinforcement learning framework, wherein the group invariant network based on the spatiotemporal attention mechanism is used to aggregate drone state vectors of various dimensions input into the actuator network and the evaluator network;

[0007] Step 2), multiple drones continuously interact with the environment, obtain observation vectors from the environment, the actuator network outputs the action of each drone according to the observation vector, the agent executes the action and obtains the reward value; the above interaction data is saved in the experience pool, and the actuator network parameters and the evaluator network parameters are updated and saved;

[0008] Step 3), load the trained actuator network parameters, map the observation vector obtained by the UAV from the environment to the action space, and the UAV performs the action to avoid dynamic obstacles and complete the distributed clustering and autonomous collision avoidance tasks.

[0009] Furthermore, in step 1), the actuator network adopts a group-invariant network architecture based on a spatiotemporal attention mechanism, including:

[0010] Step 1.1.1), cluster the entities within the observable range of the agent into different groups according to their characteristics and attributes; specifically, from the perspective of each wingman i, the entities within the observation range are divided into four groups: the agent itself, the leader, the neighboring wingman, and the neighboring intruder;

[0011] Step 1.1.2), use the local spatial attention LSA module to calculate the spatial embedding of each group separately;

[0012] Step 1.1.3), use the global temporal attention GTA module to selectively aggregate temporal features of different groups;

[0013] In step 1.1.4), the global temporal attention embedding is mapped to the drone’s actions.

[0014] Furthermore, step 1.1.2) is specifically as follows: the spatial embedding of each group is calculated separately using the local spatial attention LSA module; the spatial embedding of the neighboring wingman i at time t can be defined as the aggregated state of all neighboring wingmen:

[0015]

[0016] Spatial attention weights Determined by the following formula;

[0017]

[0018] in, Indicates similarity, represents the state vector of wingman i at time t, represents the set of neighboring wingman state vectors of wingman i, (·) T is the transposition operation, FC(·) is the fully connected FC layer, and W fol is the parameter to be learned, d fol yes Similarly, calculate the aggregate spatial embedding of neighboring intruders at time t:

[0019]

[0020] Spatial attention weights Determined by the following formula:

[0021]

[0022] in, Indicates similarity, represents the state vector of the neighboring intruder k, W int is the parameter to be learned, d int yes Dimensions;

[0023] The spatial embeddings of the leader and wingman i at time step t are represented as;

[0024]

[0025] in, represents the state vector of the leader, represents the state vector of wingman i.

[0026] Furthermore, step 1.1.3) is specifically as follows: use the global temporal attention GTA module to selectively aggregate the temporal features of different groups; specifically, first use the long short-term memory network LSTM to extract temporal features separately from the historical observations of each group:

[0027]

[0028] Where LSTM(·) represents the LSTM layer,

[0029] The global temporal embedding of agent i at time τ Defined as:

[0030]

[0031] Where || represents a concatenated operation; the temporal attention mechanism is then used to distinguish the importance of different time slots, and the temporal attention weight The calculation is as follows:

[0032]

[0033] Global temporal attention embedding of agent i at time step t Defined as:

[0034]

[0035] Further, step 1.1.4) is specifically as follows:

[0036] The global temporal attention is embedded into v by the following formula i t Mapping to drone movements

[0037]

[0038] where MLP(·) is a two-layer multilayer perceptron MLP.

[0039] Furthermore, in step 1), the evaluator network adopts a group-invariant network architecture based on a spatiotemporal attention mechanism, including:

[0040] Step 1.2.1), use the spatiotemporal attention network to represent the environment state. The specific calculation method is the same as steps 1.1.1 to 1.1.3:

[0041]

[0042] where s t represents the environment state vector of agent i at time t, STAN(·) represents the spatiotemporal attention network, Represents the encoded environmental state feature embedding;

[0043] Step 1.2.2), use the FC layer as an action encoder to process the joint action The action encoder is defined as:

[0044]

[0045] Step 1.2.3), embed the global temporal attention of the environment state With action embedding Concatenate them and pass them to an MLP to approximate the Q-value of agent i:

[0046]

[0047] Further, step 2) specifically includes the following steps:

[0048] Step 2.1), observation and environment state vector acquisition: The observation of each agent is defined as follows:

[0049]

[0050] in is the state of the long machine at time step τ; is the state of wingman i at time step τ; is the state set of the neighboring wingmen of wingman i; is the state set of the neighboring intruders of wingman i; the system state includes the states of all entities:

[0051]

[0052] in is the state of the long machine at time step τ; is the state set of the wingman; is the state set of the intruder machine;

[0053] Step 2.2), define the action space: All wingmen have the same action space, and the action of each wingman i in the continuous space is defined as:

[0054]

[0055] The angular velocity of the heading Forward acceleration

[0056] Step 2.3), calculate the reward function: The reward function of each wingman i is defined as:

[0057]

[0058] in and They represent the leader following reward, UAV-UAV collision avoidance reward, and UAV-invader collision avoidance reward respectively; specifically, in order to encourage wingman i to gather with the leader, is defined as:

[0059]

[0060] in is the distance from the leader to the wingman i at time step t; P1>>0 is a large positive constant; w1>0 is an adjustment parameter; R a and R s They are the warning radius and minimum safety radius of the wingman respectively;

[0061] To penalize the collision between wingman i and its jth neighbor wingman, It is defined as;

[0062]

[0063] Where P2>>0 is a large positive constant;

[0064] w2>0 is an adjustment parameter; to prevent wingman i from colliding with the kth adjacent intruder, Defined as:

[0065]

[0066] Step 2.4), collect experience data: the agent continues to interact with the environment, and the historical trajectory generated in this process will be stored in the experience replay pool; use the shared experience pool to store the experience collected by all agents at the same time, and the experience is stored in the form of tuples (s t ,o t ,a t ,r t ,o t+1 ,s t+1 );in represents the observations of all agents at time step t; represents the actions of all agents at time step t, represents the rewards of all agents at time step t; the interaction process is as follows: the agent obtains the observation vector from the environment, the actuator network outputs the action of each wingman according to the observation vector, the agent executes the action to obtain the state vector at the next moment, and then calculates the reward value according to the reward function;

[0067] Step 2.5), network parameter update: randomly sample from the experience pool and extract small batches to update the evaluator and executor based on the spatiotemporal attention network architecture; the parameters θ of the policy network μ Update based on policy gradient:

[0068]

[0069] in represents the experience pool, μ represents the policy network, and Q1 represents the actuator network;

[0070] Evaluator parameters The optimization is done by minimizing the following loss:

[0071]

[0072] in represents the target Q value; to reduce the overestimation bias, the truncated double Q learning technique is introduced into the STAAC algorithm; specifically, the double Q learning technique with parameters is learned in the same way as Q1. The additional actuator Q2 forms the target Q value based on the minimum of the two Q values ​​estimated by Q1 and Q2;

[0073]

[0074] Smooth target motion Defined as;

[0075]

[0076] in and c represents Gaussian noise ∈ aThe variance and bounds of - and Represent the target networks of the strategy and the two executors, respectively, and the corresponding parameters and Update according to the following formula:

[0077]

[0078] Step 2.6), when the number of training rounds reaches a preset condition, the updating process is terminated and the actuator network parameters are saved.

[0079] Further, step 3) specifically includes the following steps:

[0080] Step 3.1), observation vector acquisition: the state vector includes four parts, including the leader state, the wingman state, the neighboring wingman state and the neighboring intruder state;

[0081] Step 3.2), construct an actuator network based on the spatiotemporal attention network architecture, load the trained network parameters, use the state vector of the agent at the current time step as the network input, and output the angular velocity and forward acceleration of the agent in the current state;

[0082] Step 3.3), the agent performs the action and obtains the state vector of the next time step;

[0083] Step 3.4), repeat steps 3.1 to 3.4 until the lead aircraft flies to the preset target point, completing the distributed clustering and autonomous collision avoidance tasks.

[0084] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0085] (1) This paper adopts a multi-agent deep reinforcement learning framework based on spatiotemporal attention mechanism to realize distributed collision avoidance control of drone clusters through actuator network and evaluator network. This framework can not only achieve collision avoidance between drones in dynamic and complex environments, but also effectively deal with the collision risk between drones and intruders, thus improving the security of the system.

[0086] (2) The local spatial attention module and the global temporal attention module are used to enable the system to automatically aggregate key spatial and temporal features and flexibly adjust strategies according to changes in the actual environment. By aggregating dynamic obstacles and environmental information through the spatiotemporal attention mechanism, the drone can better adapt to different numbers and distributions of intruders, thus having strong scalability and robustness.

[0087] (3) By introducing optimization methods such as experience pool and truncated double Q learning, the present invention can efficiently use historical interaction data to improve learning efficiency and reduce the overestimation bias problem. At the same time, based on the sharing mechanism of the experience pool, multiple agents can acquire, share and reuse data in parallel, thereby effectively reducing data requirements and training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 This is the block diagram of the spatiotemporal attention multi-agent actuator-evaluator algorithm;

[0089] Figure 2 It is a population-invariant network architecture based on the spatiotemporal attention mechanism;

[0090] Figure 3 This is the trajectory of the wingman using the learned STAAC strategy to avoid collisions with 10 intruder aircraft while clustering with the leader aircraft. The final position and direction of movement of the entities are marked on the trajectory with dots and arrows.

[0091] Figure 4 is the performance curve of the learned STAAC strategy in the test. DETAILED DESCRIPTION

[0092] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0093] A distributed swarm and collision avoidance flight control method for multiple fixed-wing UAVs based on spatiotemporal attention multi-agent reinforcement learning, which specifically includes:

[0094] 1. The clustering and collision avoidance problem of a scalable fixed-wing UAV formation in a dynamic environment is described as a distributed partially observable Markov decision process (Dec-POMDP). The observation, state, action, and reward of each wingman are defined as follows:

[0095] 1.1 Observation Space:

[0096] Given the limited perception capabilities of the wingman, the environment is partially observable. To alleviate this problem, the present invention stacks the latest four frames of historical information for decision making. The stacked observations at time step t can be defined as follows:

[0097]

[0098] 1.2 State representation: The system state includes the states of all entities. The system state can be expressed as

[0099]

[0100] 1.3 Action Space: All followers have the same action space. The action of each wingman i in the continuous space is defined as:

[0101]

[0102] 1.4 Reward function: In order to prevent the wingman from colliding with the neighboring wingman and the intruder when flying with the leader, the reward function of each wingman i is defined as:

[0103]

[0104] 2. Since the UAV formation is scalable and the number of intruders is variable, the observation dimension of each wingman is dynamic. In order to represent the dynamic dimensional observation as a fixed-size embedding, a novel spatiotemporal attention-based group-invariant network architecture is designed.

[0105] 2.1.1 Spatiotemporal Attention Network: The spatiotemporal attention network first clusters entities in the environment into different groups. Then, the spatial embedding of each group is calculated separately using the Local Spatial Attention (LSA) module. Finally, a Global Temporal Attention (GTA) module is designed to extract the temporal embeddings of different groups.

[0106] 2.1.2 Entity Clustering: Multi-UAV swarm flight scenarios in dynamic environments involve different types of entities. Considering the heterogeneity and differences of different entities, it is more effective for the agent to handle these entities individually. Therefore, firstly, the entities within the observable range of the agent are clustered into different groups according to the characteristics and attributes of the entities. Specifically, from the perspective of each wingman i, the entities within the observation range can be divided into four groups: the self, the leader, the neighboring wingman, and the neighboring intruder. Next, they are processed individually using the LSA module.

[0107] 2.1.3 Local Spatial Attention (LSA): Entities in the same group have different spatial features, and the spatial features can be better extracted by distinguishing the importance of different entities. Taking the neighboring wingman group as an example, the spatial embedding of the neighboring wingman at time t can be defined as the aggregated state of all neighboring wingmen:

[0108]

[0109] Spatial attention weights Determined by the following formula:

[0110]

[0111] in(·) T is the transpose operation, FC(·) is the fully connected (FC) layer, and W fol is the parameter to be learned, d fol yes Similarly, we can calculate the aggregate spatial embedding of neighboring intruders at time t:

[0112]

[0113] It is worth noting that the dimension of the spatial embedding of each group is independent of the number of entities in the group. When there is only one entity in a group, the spatial attention weight becomes a scalar 1, which shows that the local spatial attention operation LSA(·) mentioned above can be degenerated into the FC(·) function of a group with a single entity. Therefore, the spatial embeddings of the leader and wingman i themselves at time step t can be expressed as:

[0114]

[0115] 2.1.4 Global Temporal Attention (GTA): After the LSA module, the GTA module is used to selectively aggregate the temporal features of different groups. In order to improve some observability issues, the past four frames of observation data are stacked for each wingman in the swarm flight and collision avoidance control tasks. Since the roles and contributions of observation data at different times are different, clearly distinguishing the importance of data at different times is beneficial for the wingman to achieve its control goals. To this end, a recurrent neural network with a temporal attention mechanism is integrated into the GTA module. Specifically, a long short-term memory network (LSTM) is first used to extract temporal features separately from the historical observations of each group:

[0116]

[0117] Where LSTM(·) represents the LSTM layer,

[0118] The global temporal embedding of agent i at time τ can be defined as:

[0119]

[0120] Here, || represents a concatenation operation. Then, the temporal attention mechanism is used to distinguish the importance of different time slots. Temporal attention weight It can be calculated as follows:

[0121]

[0122] The global temporal attention embedding of agent i at time step t can be defined as:

[0123]

[0124] 2.2 Policy Network: Now define the structure of the policy network. First, use the spatiotemporal attention network to represent the observation of the agent:

[0125]

[0126] Where STAN(·) represents the spatiotemporal attention network defined in 2.1.1. Then, embed The global temporal attention is mapped to the action space:

[0127]

[0128] where MLP(·) is a two-layer multilayer perceptron (MLP).

[0129] 2.3 Value Network: The structure of the value network is similar to that of the policy network, except that the value network adds an FC layer as an action encoder to handle joint actions. Specifically, we first use a spatiotemporal attention network to represent the state of the environment:

[0130]

[0131] The action encoder is defined as:

[0132]

[0133] Then, the global temporal attention is embedded into With action embedding Concatenate them and pass them to an MLP to approximate the Q value of agent i:

[0134]

[0135] 3. STAAC algorithm update process: In order to improve learning efficiency and scalability, STAAC introduces a parameter sharing mechanism, that is, different agents share the same control strategy. Therefore, STAAC only needs to learn a shared strategy and a centralized evaluator for all agents. At the same time, a shared experience pool is used to store the experience collected by all agents at the same time. Specifically, the experience is stored in the form of tuples (s t ,o t ,a t ,r t ,o t+1 ,s t+1 ).in represents the observations of all agents at time step t. Similarly,

[0136] Update the policy and evaluator by randomly sampling mini-batches from the experience pool. Assume represents the experience pool, μ represents the policy network, and the parameter θ of the policy network μ Update based on policy gradient:

[0137]

[0138] Where Q1 represents the actuator network. The parameters of the evaluator are The optimization is done by minimizing the following loss:

[0139]

[0140] in represents the target Q value. In order to reduce the overestimation bias, the truncated double Q learning technique is introduced into the STAAC algorithm. Specifically, the truncation double Q learning technique is used to learn the target Q value in the same way as Q1. The additional actuator Q2 forms the target Q value based on the minimum of the two Q values ​​estimated by Q1 and Q2;

[0141]

[0142] Smooth target motion Defined as;

[0143]

[0144] in and c represents Gaussian noise ∈ a The variance and bounds of . It should be noted that μ - and Represent the target networks of the strategy and the two executors, respectively, and the corresponding parameters and Update according to the following formula:

[0145]

[0146] Specific implementation steps:

[0147] Step 1: Set the maximum capacity N of the experience pool and the maximum time step N of each learning stage. s , batch size N b , update frequency C;

[0148] Step 2 Randomly initialize the strategy and the network parameters θ of the two evaluators μ and

[0149] Step 3: Initialize the target network:

[0150] Step 4: Initialize the experience pool with maximum capacity N

[0151] Step 5 enters the following loop;

[0152] Step 5-1-1 randomly initialize the states of all entities;

[0153] Step 5-1-2 Receive the initial state s 1 and the initial observation value o 1 ;

[0154] Environmental state s at time t t The definition of is as follows;

[0155]

[0156] in is the state set of the wingman; is the state set of the intruder machine.

[0157] is defined as follows:

[0158]

[0159] in is the state of the long machine at time step τ; is the state of wingman i at time step τ; is the state set of the neighboring wingmen of wingman i; is the state set of the neighboring intruders of wingman i;

[0160] Step 5-2-1 From time step t=1 to N s Perform the following cycle;

[0161] Step 5-2-1-1 Select actions based on the current strategy and exploration noise:

[0162] action The definition of is as follows;

[0163]

[0164] in

[0165] Step 5-2-1-2 Execute the joint action and update the system status s t+1 ;

[0166] Step 5-2-1-3 Calculate the instant reward r i t according to:

[0167]

[0168] in and They represent the leader following reward, UAV-UAV collision avoidance reward, and UAV-invader collision avoidance reward respectively. Specifically, in order to encourage wingman i to gather with the leader, It is defined as;

[0169]

[0170] in is the distance from the leader to the wingman i at time step t; P1>>0 is a large positive constant; w1>0 is an adjustment parameter; R a and R s They are the wingman's early warning radius and minimum safety radius respectively.

[0171] To penalize the collision between wingman i and its jth neighbor wingman, It is defined as;

[0172]

[0173] Where P2>>0 is a large positive constant and w2>0 is a tuning parameter. Similarly, to prevent wingman i from colliding with the kth neighboring intruder, Defined as;

[0174]

[0175] Step 5-2-1-4 Receive new observation results o t+1

[0176] Step 5-2-1-6 will (s t ,o t ,a t ,r t ,o t+1 ,s t+1 )Deposit in represents the observations of all agents at time step t. Similarly,

[0177] Step 5-2-1-7 Update the state space and observation space s t ←s t+1 ;o t ←o t+1 ;

[0178] Step 5-2-1-8 Randomly sample size N from the experience pool b Samples of

[0179] Step 5-2-1-9 Update by minimizing the loss function

[0180] Specifically, the parameters of the evaluator Optimize by minimizing loss;

[0181]

[0182] in represents the target Q value. In order to reduce the overestimation bias, truncated double Q learning is introduced into the STAAC algorithm. Specifically, first learn with parameters in the same way as Q1 The additional evaluator Q2 is then used to form the target Q value, using the minimum of the two Q values ​​estimated by Q1 and Q2.

[0183]

[0184] Step 5-2-2 Perform the following steps every C steps:

[0185] Step 5-2-2-1 Update the policy parameter θ using the following gradient function μ ;

[0186]

[0187] Where Q1 represents the evaluator network.

[0188] Step 5-2-2-2 Update the target network parameters according to the following formula and

[0189]

[0190] Figure 2 Schematic diagram of the network structure used in the embodiment of the present invention. In the LSA module, the FC layers of the leader, ego wingman, neighboring wingman, and neighbor intruder have 64, 64, 128, and 128 neurons respectively. All FC layers are activated by the rectified linear unit (ReLU) function. Learnable parameters and In the GTA module, the LSTM layers of the leader, ego wingman, neighbor wingman, and neighbor intruder have 64, 64, 128, and 128 hidden units, respectively. For both the actuator and the evaluator, the MLP network contains two FC layers. Their hidden layers have 256 neurons, but the output layer has 2 and 1 neurons, activated by hyperbolic tangent (tanh) and linear functions, respectively.

[0191] Figure 3 The flight trajectory of the fixed-wing UAV cluster obtained in the embodiment of the present invention is shown. In the test scenario of this embodiment, 10 intruders are distributed in a rectangular mission area of ​​1200m×400m. The leader and the intruder fly along the predetermined trajectory, and the 10 wingmen follow the leader using the learned STAAC strategy. The test mission lasts for 60 seconds. As can be seen from the figure, whether in the cluttered space at the beginning of the test or the free space at the end of the test, the wingmen can successfully keep up with the leader, stay away from the intruder, and successfully avoid collisions with each other.

[0192] The cluster performance curve of the STAAC strategy in the test is as follows: Figure 4 As shown. It can be seen that the distance between the lead aircraft and the wingman is kept within the range of 50 to 100 meters most of the time. In addition, the minimum distance between the wingmen at each time step and the minimum distance between the intruder and the wingman are greater than the minimum safety radius R of the wingman. s =15m. This means that during the entire test process, no collision occurred between wingmen or between wingmen and intruders. The above results show that the method proposed in the present invention can enable the UAV cluster to fly safely in a dynamic obstacle environment.

[0193] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0194] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A collision avoidance flight control method for a distributed cluster of multiple fixed-wing UAVs, characterized in that: The steps include: Step 1), constructing a multi-agent deep reinforcement learning framework based on an actuator network and an evaluator network, and setting a group invariant network based on a spatiotemporal attention mechanism in the multi-agent deep reinforcement learning framework, wherein the group invariant network based on the spatiotemporal attention mechanism is used to aggregate drone state vectors of various dimensions input into the actuator network and the evaluator network; Step 2), multiple drones continuously interact with the environment, obtain observation vectors from the environment, the actuator network outputs the action of each drone according to the observation vector, the agent executes the action and obtains the reward value; the above interaction data is saved in the experience pool, and the actuator network parameters and the evaluator network parameters are updated and saved; Step 3), load the trained actuator network parameters, map the observation vector obtained by the UAV from the environment to the action space, and the UAV performs the action to avoid dynamic obstacles and complete the distributed clustering and autonomous collision avoidance tasks.

2. The anti-collision flight control method for a distributed cluster of multiple fixed-wing UAVs according to claim 1 is characterized in that: In step 1), the actuator network adopts a group-invariant network architecture based on a spatiotemporal attention mechanism, including: Step 1.1.1), cluster the entities within the agent's observable range into different groups based on their characteristics and attributes; from the perspective of each wingman i, the entities within the observation range are divided into four groups: the agent itself, the leader, the neighboring wingman, and the neighboring intruder; Step 1.1.2), use the local spatial attention LSA module to calculate the spatial embedding of each group separately; Step 1.1.3), use the global temporal attention GTA module to selectively aggregate temporal features of different groups; In step 1.1.4), the global temporal attention embedding is mapped to the drone’s actions.

3. The anti-collision flight control method for a distributed cluster of multiple fixed-wing UAVs according to claim 2 is characterized in that: Step 1.1.2) is as follows: Use the local spatial attention LSA module to calculate the spatial embedding of each group separately; the spatial embedding of the neighboring wingman i at time t can be defined as the aggregated state of all neighboring wingmen: Spatial attention weights Determined by the following formula: in, Indicates similarity, represents the state vector of wingman i at time t, represents the set of neighboring wingman state vectors of wingman i, (·) T is the transposition operation, FC(·) is the fully connected FC layer, and W fol is the parameter to be learned, d fol yes Similarly, calculate the aggregate spatial embedding of neighboring intruders at time t: Spatial attention weights Determined by the following formula: in, Indicates similarity, represents the state vector of the neighboring intruder k, W int is the parameter to be learned, d int yes Dimensions; The spatial embeddings of the leader and wingman i at time step t are expressed as: in, represents the state vector of the leader, represents the state vector of wingman i.

4. The anti-collision flight control method for a distributed cluster of multiple fixed-wing UAVs according to claim 2 is characterized in that: Step 1.1.3) is as follows: Use the global temporal attention GTA module to selectively aggregate the temporal features of different groups; specifically, first use the long short-term memory network LSTM to extract temporal features from each group's historical observations separately: Where LSTM(·) represents the LSTM layer, The global temporal embedding of agent i at time τ Defined as: Where || represents a concatenated operation; the temporal attention mechanism is then used to distinguish the importance of different time slots, and the temporal attention weight The calculation is as follows: Global temporal attention embedding of agent i at time step t Defined as:

5. The anti-collision flight control method for a distributed cluster of multiple fixed-wing UAVs according to claim 2 is characterized in that: Step 1.1.4) is as follows: The global temporal attention is embedded into v by the following formula i t Mapping to drone movements where MLP(·) is a two-layer multilayer perceptron MLP.

6. The anti-collision flight control method for a distributed cluster of multiple fixed-wing UAVs according to claim 1, characterized in that: In step 1), the evaluator network adopts a group-invariant network architecture based on a spatiotemporal attention mechanism, including: Step 1.2.1), use the spatiotemporal attention network to represent the environment state. The specific calculation method is the same as steps 1.1.1 to 1.1.3: where s t represents the environment state vector of agent i at time t, STAN(·) represents the spatiotemporal attention network, Represents the encoded environmental state feature embedding; Step 1.2.2), use the FC layer as an action encoder to process the joint action The action encoder is defined as: Step 1.2.3), embed the global temporal attention of the environment state With action embedding Concatenate them and pass them to an MLP to approximate the Q-value of agent i:

7. The anti-collision flight control method for a distributed cluster of multiple fixed-wing UAVs according to claim 1, characterized in that: Step 2) specifically includes the following steps: Step 2.1), observation and environment state vector acquisition: The observation of each agent is defined as follows: in is the state of the long machine at time step τ; is the state of wingman i at time step τ; is the state set of the neighboring wingmen of wingman i; is the state set of the neighboring intruders of wingman i; the system state includes the states of all entities: in is the state of the long machine at time step τ; is the state set of the wingman; is the state set of the intruder machine; Step 2.2), define the action space: All wingmen have the same action space, and the action of each wingman i in the continuous space is defined as: The angular velocity of the heading Forward acceleration Step 2.3), calculate the reward function: The reward function of each wingman i is defined as: in and They represent the leader following reward, UAV-UAV collision avoidance reward, and UAV-invader collision avoidance reward respectively; specifically, in order to encourage wingman i to gather with the leader, is defined as: in is the distance from the leader to the wingman i at time step t; P1>>0 is a large positive constant; w1>0 is an adjustment parameter; R a and R s They are the warning radius and minimum safety radius of the wingman respectively; To penalize the collision between wingman i and its jth neighbor wingman, is defined as: Where P2>>0 is a large positive constant; w2>0 is the adjustment parameter; To prevent wingman i from colliding with the kth adjacent intruder, Defined as Step 2.4), collect experience data: the agent continues to interact with the environment, and the historical trajectories generated in this process will be stored in the experience replay pool; A shared experience pool is used to store the experience collected by all agents at the same time. The experience is stored in the form of tuples (s t ,o t ,a t ,r t ,o t+1 ,s t+1 );in represents the observations of all agents at time step t; represents the actions of all agents at time step t, represents the rewards of all agents at time step t; the interaction process is as follows: the agent obtains the observation vector from the environment, the actuator network outputs the action of each wingman according to the observation vector, the agent executes the action to obtain the state vector at the next moment, and then calculates the reward value according to the reward function; Step 2.5), network parameter update: randomly sample from the experience pool and extract small batches to update the evaluator and executor based on the spatiotemporal attention network architecture; the parameters θ of the policy network μ Update based on policy gradient: in represents the experience pool, μ represents the policy network, and Q1 represents the actuator network; Evaluator parameters The optimization is done by minimizing the following loss: in represents the target Q value; to reduce the overestimation bias, the truncated double Q learning technique is introduced into the STAAC algorithm; specifically, the double Q learning technique with parameters is learned in the same way as Q1. The additional actuator Q2 forms the target Q value based on the minimum of the two Q values ​​estimated by Q1 and Q2; Smooth target motion Defined as; in and c represents Gaussian noise ∈ a Variance and bounds of μ - and Represent the target networks of the strategy and the two executors, respectively, and the corresponding parameters and Update according to the following formula: Step 2.6), when the number of training rounds reaches a preset condition, the updating process is ended and the actuator network parameters are saved.

8. The anti-collision flight control method for a distributed cluster of multiple fixed-wing UAVs according to claim 1, characterized in that: Step 3) specifically includes the following steps: Step 3.1), observation vector acquisition: the state vector includes four parts, including the leader state, the wingman state, the neighboring wingman state and the neighboring intruder state; Step 3.2), construct an actuator network based on the spatiotemporal attention network architecture, load the trained network parameters, use the state vector of the agent at the current time step as the network input, and output the angular velocity and forward acceleration of the agent in the current state; Step 3.3), the agent performs the action and obtains the state vector of the next time step; Step 3.4), repeat steps 3.1 to 3.4 until the lead aircraft flies to the preset target point, completing the distributed clustering and autonomous collision avoidance tasks.

Images