A UAV Control Method Based on Embedded Attention Strategy and Multi-Level Encoding Network

Abstract

This invention discloses a UAV control method based on an embedded attention strategy and a multi-level encoding network. It includes: Step 1: Constructing a super-network embedded attention strategy network; Step 2: Constructing a multi-level encoded Transformer value network based on the super-network embedded attention strategy network constructed in Step 1. This invention solves the problems of weak representation ability, poor adaptability, large parameters, and low learning efficiency in existing reinforcement learning algorithms for UAV control.

Description

Technical Field

[0001] This invention relates to the field of multi-agent system / unmanned aerial vehicle (UAV) control technology, and in particular to a UAV control method based on embedded attention strategy and multi-level coding network. Background Technology

[0002] In recent years, artificial intelligence has brought new developments to the field of drones, especially in intelligent adversarial decision-making and control. Commonly used reinforcement learning methods such as Qmix, MADDPG, and MAPPO can all make cooperative decisions for multi-drone swarms. However, drone swarm environments are usually dynamic, diverse, and large-scale. For example, in the same task scenario, the number of obstacles and the number of neighboring agents observed by agents in the swarm are different at different times. Especially in highly dynamic scenarios, the number of agents increases or decreases with task requirements and enemy-ally conflicts, further leading to different amounts of information observed by agents; after completing a task, the swarm should quickly adapt to the new environment and complete the next different task. However, there are corresponding drawbacks: during the same training, it is impossible to expand the number of observed neighbors during the interaction process. When the number of neighboring agents and the total number of agents change in different scenarios or even in the same environment, different numbers of agents must be retrained, and it is impossible to fine-tune parameters based on existing training results, i.e., poor generalization. As the number of agents increases, the dimensionality of the state space grows exponentially, and the number of network layers increases accordingly, causing the curse of dimensionality and long computation time. State space concatenation / convolution cannot effectively extract useful feature information. Currently, some methods exist for handling dynamic scenarios, such as ordinary attention methods, Transformer methods, UPDET methods, and DeepSet algorithms. These methods use shared neural networks, do not address the dimensionality of reinforcement learning commentator networks, and suffer from weak learning efficiency due to limited parameter representation capabilities. Existing methods with strong representation capabilities require additional neural network parameters, lack adaptive features, and have low sampling efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a drone control method based on an embedded attention strategy and a multi-level coding network. This invention addresses the problems of weak representation ability, poor adaptability, large parameters, and low learning efficiency in existing reinforcement learning algorithms for drone control.

[0004] Technical solution. A drone control method based on embedded attention strategy and multi-level coding network, comprising:

[0005] Step 1: Construct an embedded attention policy network for the super network;

[0006] Step 2: Construct a multi-layered encoded Transformer value network based on the embedded attention strategy network of the super network built in Step 1.

[0007] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, the construction process of the embedded attention strategy network of the super network in step 1 is as follows:

[0008] Step 11. Reduce the original high-dimensional observation space O to the latent variable space Z;

[0009] Step 12. In the latent variable space Z, for each agent i and its observation O i Assign corresponding weights;

[0010] Step 13. Construct a small network based on supernetwork coding;

[0011] Step 14. Using the output of the small network as the key value, calculate the degree of influence of the agent's observed neighbors / opponents on the agent.

[0012] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, step 11 is as follows:

[0013] The original high-dimensional observation space O is reduced in dimension and mapped to the latent variable space Z, that is:

[0014] f:O→Z

[0015] Where dim(O) < dim(Z); O represents the original observation space, Z represents the mapped latent variable space, and dim(·) represents the dimension of the space.

[0016] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, step 12 is as follows:

[0017] For each agent i and its observation O i Assign corresponding weights w i The weights are allocated as follows:

[0018] w i =g(O i ;θ)

[0019] Where g is a learnable function, θ is its parameter, and O i It is the observation of agent i, w i That is the corresponding weight.

[0020] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, step 13 is as follows:

[0021] Step 131. Encode the agent's neighbors and opponents using HyperNet, let θ hyper These are the parameters of HyperNet;

[0022] Step 132. HyperNet dynamically generates small network parameters φ based on the input x:

[0023] φ = HyperNet(x; θ) hyper )

[0024] The small network f(x; φ) changes with the input x:

[0025] y=f(x;φ)=f(x;HyperNet(x;θ hyper ))

[0026] Where x is the input data, θ hyper φ represents the parameters of the supernetwork, φ represents the parameters of the dynamically generated subnetwork, and y represents the output of the subnetwork.

[0027] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, step 14 is as follows:

[0028] Step 141. Use the output of the small network as the key value K = f(x; φ);

[0029] Step 142. Calculate the query value Q and the value V;

[0030] Step 143. Calculate attention weights:

[0031]

[0032] Where, d k It is the dimension of the key;

[0033] Step 144. Calculate the degree of influence of the agent's observed neighbors / adversaries on the agent:

[0034] output = αV

[0035] Where K, Q, and V are the key, query, and value matrices in the attention mechanism, respectively, and d k α is the dimension of the key, α is the attention weight, and output is the output of the attention mechanism.

[0036] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, step 2 is as follows:

[0037] Step 21. Agent Encoding: Encode the agent itself (s), its neighbor (n), and its opponent (o) to obtain the encoding E.s E n and E o The encoding process is represented as follows:

[0038] E s =f s (s), E n =f n (n), E o =f o (o)

[0039] Where f s f n f o These are their respective encoding functions. s, n, and o represent the raw information of the agent itself, its neighbors, and its opponent, respectively, and E... s E n E o This is the corresponding encoding result;

[0040] Step 22. Combining attention mechanisms; the multi-head attention mechanism is represented as:

[0041] MultiHead(Q,K,V)=Concat(head1,...,head h W O

[0042] Among them, head i =Attention(QW i Q KW i K VW i V )

[0043] Where Q, K, and V are the input query, key, and value matrices, respectively, and W... i Q, W i K W i V W is the parameter matrix of the i-th attention head. O is the output linear transformation matrix, and h is the number of attention heads;

[0044] Step 23. Transformer multi-head structure composition; combining three types of encoding (E s E n E o Finally, a new multi-head structure is formed from the Transformer structure:

[0045] MH(E s E n E o) = MultiHead(Concat(E s E n E o ))

[0046] Among them, E s E n E o These are the encodings of the agent itself, its neighbors, and its opponents, respectively. MH represents the multi-head attention structure.

[0047] Step 24. Encode neighbors and adversaries based on local attention calculation and global attention calculation;

[0048] Step 25. Optimize the Transformer structure based on neighbor and rival encoding.

[0049] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, the local attention calculation process in step 24 is as follows:

[0050] 7) Through shared network f k Calculate the key value for each neighbor / rival:

[0051] K i =f k (n i ) or K i =f k (o i )

[0052] 8) Calculate the average value of the query:

[0053]

[0054] Where N is the number of neighbors / opponents;

[0055] 9) Calculate attention weights:

[0056]

[0057] 10) Obtain the attention score using the softmax function:

[0058] α i =Softmax(wi)

[0059] 11) Use another shared network f v Calculated value:

[0060] V i =f v (n i ) or V i =f v (o i )

[0061] 12) Calculate the weighted sum:

[0062]

[0063] Among them, f k and f v It's a shared network, K i Q is the key value of the i-th neighbor / opponent. avg It is the average query value, w l It is the attention weight, α l It is the normalized attention score, V l It represents the value, N is the number of neighbors / opponents, and output is the final weighted output.

[0064] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, the global attention calculation process in step 24 is as follows:

[0065] 6) Use the weighted value of each agent calculated by the local attention mechanism as the global key value K. global .

[0066] 7) Calculate the global average - query value:

[0067]

[0068] Where M is the total number of agents;

[0069] 8) Through shared network f g Calculate the global value:

[0070] V global =f g (K global)

[0071] 9) Calculate the global attention weights:

[0072]

[0073] 10) Calculate the global weighted sum using softmax:

[0074] output global =softmax(w global V global

[0075] Here, K global It is a global key-value pair, Q global It is the global average query value, M is the total number of agents, and f is the global average query value. g It is a globally shared network, V global It is the overall value, w globalThese are global attention weights, output. global It is the output of global attention.

[0076] In the aforementioned UAV control method based on embedded attention strategy and multi-level coding network, step 25 is as follows:

[0077] 3) Concatenate the encoded self, neighbor, and adversary networks to form the multi-head part of the new Transformer:

[0078] MH = Concat(E s E n E o )

[0079] 4) Replace the traditional Add calculation method with a gating mechanism:

[0080] output=LayerNorm(Gate(mH,LayerNorm(mH)))

[0081] Among them, E s E n E o These are the encodings of the agent itself, its neighbors, and its adversaries, respectively. MH is the output of multi-head attention, GateLayer is the gate layer, LayerNorm is the layer normalization operation, and output is the final output.

[0082] Beneficial Effects: This invention first provides a method for an embedded attention policy network within a super-network. It first maps the original observation space to a smaller latent variable space using a super-attention network, then assigns different weights to each agent and its observations, and adapts to dynamic changes in the number of agents and the dimensionality of observations, enabling more powerful policy learning and improving the overall efficiency of policy learning. Secondly, it provides a multi-layer encoded Transformer value network: It improves upon the traditional Transformer by encoding each observation to form a new multi-head part, where each encoding employs a multi-layer attention mechanism. This significantly reduces the dimensionality of the critic network, promoting more stable policy learning. In summary, this invention dynamically adapts to dynamic scenarios through the embedded attention mechanism of a super-network, improving representational ability and learning efficiency; and utilizes a multi-layer encoded Transformer value network to reduce the dimensionality of the critic network, promoting policy learning. Specifically, compared with existing reinforcement learning algorithms, this invention achieves the following advantages:

[0083] (a) Compared with existing reinforcement learning models, this invention has the ability to adapt to dynamic scenes without adding extra neural networks, has higher representation ability and learning efficiency, and can avoid dimensionality explosion.

[0084] Existing reinforcement learning models use a shared neural network to encode the observed input, resulting in all agents and observations sharing the same set of neural network parameters. However, the limited number of fixed shared neural network parameters restricts the agent's representational ability, easily leading to local optima and reducing learning efficiency. Therefore, the algorithm used in this invention is more targeted, thus possessing higher representational ability and learning efficiency. In particular, this invention designs an embedded attention policy network based on a supernetwork, which can assign different network parameters to each agent and observation, eliminating the need for additional neural networks to achieve adaptive network parameters, and even adapting to situations where sensor observation information is missing, thus ignoring input dimensionality limitations. Furthermore, this invention reduces the parameter space, avoids the dimensionality explosion problem, reduces computational time costs, and improves learning efficiency. Therefore, this method is more suitable for the high-dynamic, high-adversarial scenarios required in actual air combat.

[0085] (b) Compared with existing Transformer models, this invention has a smaller parameter space, is suitable for multi-task scenarios, and promotes more stable policy learning.

[0086] This invention effectively captures the "long-term" information and dependencies of agents. Global attention keeps the input dimension of the commentator network fixed, preventing its dimension from growing exponentially with the number of agents, thus avoiding dimensionality explosion. Simultaneously, thanks to the characteristics of the attention mechanism, it can adapt to scenarios with different numbers of agents even with a fixed dimension. Local attention, on the other hand, allows it to adapt to scenarios with different numbers of observations even with a fixed dimension.

[0087] Depend on Figure 3 As can be seen, when this representation module is applied to different algorithms, it improves the efficiency of the original reinforcement learning and yields higher rewards. This indicates that the present invention has excellent portability and generalization characteristics, and can improve the representation ability and learning effect of existing algorithms.

[0088] Depend on Figure 4 As can be seen, when the mission scenario changes from eight to one hundred and sixty-four agents, and the observation size also changes accordingly, this invention can adapt well to different numbers of agents and observation dimensions. The curve trend shows that the entire reward curve is relatively stable, indicating that the method has more stable policy learning. Furthermore, the rise time / steps required for different numbers of agents in the graph show that the convergence time is greatly reduced, saving computational time costs. At the same time, this trend indicates that this invention has strong generalization ability, adapting to the control requirements of different numbers of agents. Attached Figure Description

[0089] Figure 1 This is a network diagram of an embedded attention strategy for a super network.

[0090] Figure 2 A value network diagram for a multi-layered encoded Transformer;

[0091] Figure 3 The trend of improved efficiency in reinforcement learning under different algorithms;

[0092] Figure 4 The reward value and convergence time vary with the number of agents. Detailed Implementation

[0093] Example 1. A UAV control method based on embedded attention strategy and multi-layered encoding network, to solve the problems of poor representation ability, low adaptability, low learning efficiency, and susceptibility to dimensionality explosion in existing reinforcement learning methods. See [link to example]. Figure 1-4 ,

[0094] Firstly, please see Figure 1 This invention provides an embedded attention strategy network graph for a super network, comprising: mapping the original observation space to a smaller latent variable space; and assigning different weights to each agent and its observations.

[0095] In encoding the agent's neighbors and adversaries, a super network HyerNet is used to output small networks for different agents and each agent's different observed neighbors. These small networks have different weight parameters and change with the input. This allows for the provision of different networks without increasing the network parameters, ignoring input limitations and providing an unrestricted network with better representation capabilities than shared networks with finite network parameters, while avoiding dimensionality explosion.

[0096] The results of the input small network are used as the key value. Then, the query and value are calculated using common attention methods. Finally, the degree of influence of the agent's neighbors / adversaries on the agent is obtained.

[0097] On the other hand, please see Figure 2 This invention provides a multi-layer encoded Transformer value network graph, including: encoding the agent itself, its neighbors, and its adversaries separately; the encoding adopts a combination of local and global multi-head attention mechanisms; the three encodings finally form a new multi-head structure of the transformer.

[0098] For agents whose neighbors and adversaries are encoded in the same way, a combination of local and global multi-head attention mechanisms is used, as follows:

[0099] In local attention computation, since each agent has different attention weights towards its neighbors / opponents, the key value of each neighbor / opponent is first calculated separately through a shared network, and then all are averaged to obtain the average-query value. The average-query value is matched with the key value of the corresponding neighbor / opponent of the agent; that is, attention weights are calculated through dot products. The attention score of each agent's neighbors / opponents is obtained through a softmax function. Finally, the key values ​​of different neighbors / opponents are used as input to another shared network to calculate the value of each agent towards each neighbor / opponent, and the attention score is used as a weight to calculate a weighted sum of the values. If the number of observed neighbors changes during the interaction, although the network dimensions are fixed (three-dimensional relative position and relative velocity), the local attention mechanism only needs to calculate the sum of each newly added neighbor separately and calculate the influence of all neighbors on the agent through a weighted sum. Therefore, the number of neighbors can be expanded in a fixed-dimensional network, exhibiting good observation scalability.

[0100] The global attention mechanism still employs the dot product attention mechanism. Here, the weighted value of each agent calculated by the local attention mechanism is used as the global key, and all values ​​are averaged to obtain the global average-query value. The global value of each agent is then calculated through another shared network. Finally, the global weighted sum is calculated using dot product and softmax operations. Similarly, when the total number of agents changes, only the sum of the newly added agents needs to be calculated individually under the previously updated network parameters, and the impact of all agents is recalculated, allowing for fine-tuning of the network parameters. Therefore, this method exhibits good generalization properties. Furthermore, its network dimension and number of layers do not increase with the number of agents, reducing the computational burden.

[0101] Finally, the encoded self, neighbor, and adversary networks are concatenated to form the multi-head part of the new Transformer, and the traditional Add computation method is replaced with a gate. This structure has better representation capabilities than the traditional Transformer.

[0102] Example 2. A UAV control method based on embedded attention strategy and multi-layered encoding network, to solve the problems of poor representation ability, low adaptability, low learning efficiency, and susceptibility to dimensionality explosion in existing reinforcement learning methods. See [link to example]. Figure 1-4 ,

[0103] This strategy primarily comprises two core components: an embedded attention policy network within a supernetwork and a multi-layered encoded Transformer value network. These two components work together to effectively adapt to dynamic changes in the number of agents and the dimensionality of observations, improving learning efficiency and avoiding the curse of dimensionality.

[0104] 1. Embedded Attention Strategy Network for Supernetworks

[0105] See attached document Figure 1 The implementation steps of this network are as follows:

[0106] 1.1 Observation Space Mapping

[0107] Mapping the original high-dimensional observation space O to a smaller latent variable space Z, i.e.:

[0108] f:O→Z

[0109] Where dim(Z) < dim(O). Here, O represents the original observation space, Z represents the mapped latent variable space, and dim(·) represents the dimension of the space. This step significantly reduces computational complexity through dimensionality reduction while preserving key information. This mapping makes subsequent processing more efficient, especially when dealing with large-scale UAV swarms, effectively alleviating computational resource pressure.

[0110] 1.2 Weight Allocation

[0111] For each agent i and its observation o i Assign different weights w i The mathematical expression for weight allocation is as follows:

[0112] w i =g(o i ;θ)

[0113] Where g is a learnable function, θ is its parameter, and o i It is the observation of agent i, w i These are the corresponding weights. This differentiated weight allocation mechanism enables the system to better identify and respond to the states of key agents, improving the accuracy and adaptability of decision-making.

[0114] 1.3 Super Network Coding

[0115] 1) Use HyperNet to encode the agent's neighbors and opponents. Let θ hyper These are the parameters for HyperNet.

[0116] 2) HyperNet dynamically generates small network parameters φ based on the input x:

[0117] φ-HyperNet(x; θ) hyper )

[0118] The small network f(x; φ) changes with the input x:

[0119] y=f(x;φ)=f(x;HyperNet(x;θ hyper ))

[0120] Here, x is the input data, θhyper φ represents the parameters of the supernetwork, φ represents the parameters of the dynamically generated smaller networks, and y represents the output of the smaller networks. This dynamic network generation method greatly enhances the expressiveness and flexibility of the model, enabling it to adapt to combinations of agents of different sizes and types without pre-defining a fixed network structure.

[0121] 1.4 Application of Attention Mechanism

[0122] 1) Use the output of the small network as the key value K = f(x; φ).

[0123] 2) Calculate the query value Q and the value V.

[0124] 3) Calculate attention weights:

[0125]

[0126] Where d k It is the dimension of the key.

[0127] 4) Calculate the degree of influence of the agent's observed neighbors / adversaries on the agent:

[0128] output = αV

[0129] Here, K, Q, and V are the key, query, and value matrices in the attention mechanism, respectively, and d k Here, α is the dimension of the key, α is the attention weight, and output is the output of the attention mechanism. This attention mechanism enables the model to dynamically focus on important information sources, improving the efficiency and accuracy of information processing, especially when dealing with large-scale, complex drone swarm environments.

[0130] 2. Multi-layered coding Transformer value network

[0131] See attached document Figure 2 The implementation steps of this network are as follows:

[0132] 2.1 Agent Coding

[0133] Encode the agent itself (s), its neighbor (n), and its opponent (o) separately to obtain the code E. s E n and E o The encoding process can be represented as:

[0134] E s =f s (s), E n =f n (n), E o =f o (o)

[0135] Where f s f n f o These are their respective encoding functions. s, n, and o represent the raw information of the agent itself, its neighbors, and its opponent, respectively, and E... s E n E o This is the corresponding encoding result. This classification encoding method can preserve the features of various entities, providing a more discriminative input for subsequent attention mechanisms.

[0136] 2.2 Attention Mechanism Integration

[0137] The encoding process employs a combination of local and global multi-head attention mechanisms. Let the number of heads be h. The multi-head attention mechanism can be represented as:

[0138] MultiHead(Q,K,V)=Concat(head1,...,head h W O

[0139] Among them, head i =Attention(QW i Q KW i K VW i V ).

[0140] Here, Q, K, and V are the input query, key, and value matrices, respectively, and W... i Q W i K W i V W is the parameter matrix of the i-th attention head. O is the linear transformation matrix of the output, and h is the number of attention heads. This multi-head attention mechanism, which combines local and global attention, enables the model to capture both local details and global context simultaneously, greatly enhancing the model's expressive power and its ability to understand complex scenes.

[0141] 2.3 Composition of Transformer Multi-Head Structure

[0142] Three encodings (E) s E n E o Finally, a new multi-head structure is formed using Transformer:

[0143] MH(E s E n E o) = MultiHead(Concat(E s E n E o ))

[0144] Here, E s E n E o These represent the encodings of the agent itself, its neighbors, and its opponents, respectively, with MH representing a multi-head attention structure. This structural design enables the model to effectively integrate information from itself, its neighbors, and its opponents to form a comprehensive state representation.

[0145] 2.4 Detailed steps for coding neighbors and rivals

[0146] 2.4.1 Local Attention Calculation

[0147] 1) Through shared network f k Calculate the key value for each neighbor / rival:

[0148] K i =f k (n i ) or K i =f k (o i )

[0149] 2) Calculate the average value of the query:

[0150]

[0151] Where N is the number of neighbors / rivals.

[0152] 3) Calculate attention weights:

[0153]

[0154] 4) Obtain the attention score using the softmax function:

[0155] α i =softmax(w i )

[0156] 5) Use another shared network f v Calculated value:

[0157] V i =f v (n i ) or V i =f v (o i )

[0158] 6) Calculate the weighted sum:

[0159]

[0160] Here, f k and f v It's a shared network, K i Q is the key value of the i-th neighbor / opponent. avg It is the average query value, w i It is the attention weight, α i It is the normalized attention score, V i Here, N is the value, N is the number of neighbors / opponents, and output is the final weighted output. This local attention mechanism can effectively capture the interaction between the agent and its neighboring entities, improving the model's ability to perceive the local environment.

[0161] 2.4.2 Global Attention Calculation

[0162] 1) Use the weighted value of each agent calculated by the local attention mechanism as the global key value K. global .

[0163] 2) Calculate the global average - query value:

[0164]

[0165] Where M represents the total number of agents.

[0166] 3) Through shared network f g Calculate the global value:

[0167] V global =f g (K global )

[0168] 4) Calculate the global attention weights:

[0169]

[0170] 5) Calculate the global weighted sum using softmax:

[0171] output global =softmax(w global V global

[0172] Here, K global It is a global key-value pair, Q global It is the global average query value, M is the total number of agents, and f is the global average query value. g It is a globally shared network, V global It is the overall value, w global These are global attention weights, output. globalIt is the output of global attention. The global attention mechanism enables the model to consider the state of the entire system, which helps to make more globally optimal decisions.

[0173] 2.5 Transformer Structure Optimization

[0174] 1) Concatenate the encoded self, neighbor, and adversary networks to form the multi-head part of the new Transformer:

[0175] MH = Concat(E s E n E o )

[0176] 2) Replace the traditional Add calculation method with a gating mechanism:

[0177] output=LayerNorm(Gate(MH,LayerNorm(MH)))

[0178] Here, E s E n E o These represent the encodings of the agent itself, its neighbors, and its opponents, respectively. MH is the output of multi-head attention, GateLayer is the gate layer, LayerNorm is the layer normalization operation, and output is the final output. This optimized structure enhances the model's non-linear expressive power and, through the gating mechanism, enables dynamic fusion of different information sources, thereby improving the model's expressive power and training stability.

[0179] This method not only effectively addresses the problems of poor representation ability, low adaptability, low learning efficiency, and dimensionality explosion in existing reinforcement learning methods, but also possesses good observation scalability and generalization characteristics. In practical applications, the network structure and parameters, such as the number of attention heads h, the complexity of HyperNet, and the number of layers in the encoding network, can be flexibly adjusted according to the specific UAV swarm task scenario to achieve the best balance between computational efficiency and model performance.

Claims

1. A UAV control method based on embedded attention strategy and multi-level coding network, characterized in that, include: Step 1: Construct an embedded attention policy network for the super network: Step 11. Convert the original high-dimensional observation space O Dimensionality reduction mapping to latent variable space Z ; Step 12. In the latent variable space Z For each intelligent agent i and its observation O i Assign corresponding weights; Step 13. Construct a small network based on supernetwork coding; Step 14. Using the output of the small network as the key value, calculate the degree of influence of the agent's observed neighbors / opponents on the agent; Step 2: Construct a multi-layered encoded Transformer value network based on the embedded attention strategy network of the supernetwork built in Step 1: Step 21. Agent Encoding: Encoding the agent itself s ,Neighbor n and opponents o Encode them separately to obtain the codes. E s , E n and E o The encoding process is represented as follows: E s = f s ( s )， E n = f n ( n )， E o = f o ( o ) in f s , f n , f o Each is its own encoding function; s , n , o These represent the original information of the agent itself, its neighbors, and its adversaries, respectively. E s , E n , E o This is the corresponding encoding result; Step 22. Combining attention mechanisms; the multi-head attention mechanism is represented as: MultiHead( Q , K , V )=Concat(Head1,...,Head h ) W o Among them, Head i =Attention( QW i Q , KW i K , VW i V ) in, Q , K , V It is the input query, key, and value matrix. W i Q , W i K , W i V It is the first i The parameter matrix of each attention head, W o It is the output linear transformation matrix. h It's about the number of heads; Step 23. Transformer multi-head structure composition; combining three types of encoding ( E s , E n , E o Finally, a new multi-head structure is formed from the Transformer structure: MH( E s , E n , E o )=MultiHead(Concat( E s , E n , E o )) in, E s , E n , E o These are the encodings of the agent itself, its neighbors, and its opponents, respectively. MH represents the multi-head attention structure. Step 24. Encode neighbors and adversaries based on local attention calculation and global attention calculation; Step 25. Optimize the Transformer structure based on neighbor and rival encoding.

2. The UAV control method based on embedded attention strategy and multi-level coding network according to claim 1, characterized in that, Step 11 is as follows: The original high-dimensional observation space O Dimensionality reduction mapping to latent variable space Z ,Right now: f : O → Z Among them, dim ( O ) < dim ( Z ); O Represents the original observation space. Z This represents the potential variable space after mapping, and dim(·) represents the dimension of the space.

3. The UAV control method based on embedded attention strategy and multi-level coding network according to claim 1, characterized in that, Step 12 is as follows: For each intelligent agent i and its observation O i Assign corresponding weights w i The weights are allocated as follows: w i = g ( O i ； θ ) in g It is a learnable function. θ These are its parameters. O i It is an intelligent agent i The observation, w i That is the corresponding weight.

4. The UAV control method based on embedded attention strategy and multi-level coding network according to claim 1, characterized in that, Step 13 is as follows: Step 131. Encode the agent's neighbors and opponents using the HyperNet super network, assuming... θ hyper These are the parameters of HyperNet; Step 132. HyperNet, based on the input... x Dynamically generate small network parameters Φ : Φ= HyperNet( x ; θ hyper ) Small network f ( x ; Φ ) with input x Change with change: y=f ( x ; Φ ) =f ( x ; HyperNet( x ; θ hyper )) in, x It is the input data. θ hyper These are parameters of the super network. Φ These are dynamically generated small network parameters. y It is the output of a small network.

5. The UAV control method based on embedded attention strategy and multi-level coding network according to claim 1, characterized in that, Step 14 is as follows: Step 141. Use the output of the small network as the key value. K=f ( x ; Φ ); Step 142. Calculate the query value Q and value V ; Step 143. Calculate attention weights: ； in, d k It is the dimension of the key; Step 144. Calculate the degree of influence of the agent's observed neighbors / adversaries on the agent: output= αV in, K , Q , V These are the key, query, and value matrices in the attention mechanism. d k It is the dimension of the key. α These are the attention weights, and output is the output of the attention mechanism.

6. The UAV control method based on embedded attention strategy and multi-level coding network according to claim 1, characterized in that, In step 24, the local attention calculation process is as follows: 1) Through a shared network f k Calculate the key value for each neighbor / rival: K i = f k ( n i )or K i = f k ( o i ) 2) Calculate the average value minus the query value: ， in N Number of neighbors / rivals; 3) Calculate attention weights: ， 4) Obtain the attention score using the softmax function: α i =softmax( w i ) 5) Use another shared network f v Calculated value: V i = f v ( n i )or V i = f v ( o i ) 6) Calculate the weighted sum: ， in, f k and f v It is a shared network. K i It is the first i The key value of each neighbor / rival. Q avg This is the average query value. w i It is attention weight. α i It is the normalized attention score. V i It is value. N It represents the number of neighbors / opponents, and output is the final weighted output.

7. The UAV control method based on embedded attention strategy and multi-level coding network according to claim 1, characterized in that, In step 24, the global attention calculation process is as follows: 1) Use the weighted value of each agent calculated by the local attention mechanism as the global key value. K global ; 2) Calculate the global average - query value: ， in M The total number of agents; 3) Through a shared network f g Calculate the global value: V global = f g ( K global )， 4) Calculate the global attention weights: ， 5) Calculate the global weighted sum using softmax: ， here, K global It is a global key-value pair. Q global This is the global average query value, where M is the total number of agents. f g It is a globally shared network. V global It is the overall value. w global These are global attention weights, output. global It is the output of global attention.

8. The UAV control method based on embedded attention strategy and multi-level coding network according to claim 1, characterized in that, Step 25 is as follows: 1) Concatenate the encoded self, neighbor, and adversary networks to form the multi-head part of the new Transformer: MH=Concat( E s , E n , E o ), 2) Replace the traditional Add calculation method with a gating mechanism: output=LayerNorm(Gate(MH , LayerNorm(MH))), in, E s , E n , E o These are the encodings of the agent itself, its neighbors, and its adversaries, respectively. MH is the output of multi-head attention, GateLayer is the gate layer, LayerNorm is the layer normalization operation, and output is the final output.

Images