Unmanned aerial vehicle control method based on embedded attention strategy and multilevel coding network

Abstract

The invention discloses an unmanned aerial vehicle control method based on an embedded attention strategy and a multilevel coding network. Comprising the following steps: step 1, constructing an embedded attention strategy network of a super network; and step 2, constructing a multi-level coded Transform value network based on the embedded attention strategy network of the super network constructed in the step 1. According to the method, the problems of weak representation capability, poor self-adaption, large parameters and low learning efficiency in the existing reinforcement learning algorithm during unmanned aerial vehicle control are solved.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent system / unmanned aerial vehicle control technology, and in particular to a unmanned aerial vehicle control method based on an embedded attention strategy and a multi-level coding network. Background Art

[0002] In recent years, artificial intelligence has brought new developments to the field of drones, especially in intelligent confrontation decision-making and control. Common reinforcement learning methods such as Qmix, MADDPG, MAPPO, etc. can make cooperative decisions for multi-drone clusters. However, the drone cluster environment is usually dynamic, diverse, and large-scale. For example, in the same mission scenario, the number of obstacles and the number of neighboring agents observed by the agents in the cluster at different times are different. Especially in highly dynamic scenarios, the number of agents increases or decreases with task requirements and enemy conflicts, which further leads to different amounts of information observed by the agents; when a task is completed, the cluster should quickly adapt to the new environment and complete the next different task. However, there are the following defects: in the same training, the number of observed neighbors cannot be expanded during the interaction process. In different scenarios or even in the same environment, when the number of neighboring agents and the total number of agents change, different numbers of agents must be retrained, and parameters cannot be fine-tuned on the existing training results, that is, the generalization is poor. As the number of agents increases, the dimension of the state space grows exponentially, and the number of network layers increases accordingly, resulting in dimensional disasters and long computation time. The state space cannot effectively extract useful feature information by splicing / convolution. There are also some methods to solve dynamic scenes, such as ordinary attention method, Transformer method, UPDET method and DeepSet algorithm. These methods use shared neural networks and do not solve the dimension of the reinforcement learning critic network. In addition, due to the limited parameter representation ability, the learning efficiency is low. The existing methods with strong representation ability require additional neural network parameters, are not adaptive, and have low sampling efficiency. Summary of the invention

[0003] The purpose of the present invention is to provide a drone control method based on an embedded attention strategy and a multi-level coding network. The present invention solves the problems of weak representation ability, poor adaptability, large parameters and low learning efficiency in existing reinforcement learning algorithms for drone control.

[0004] Technical solution: A drone control method based on an embedded attention strategy and a multi-level coding network, comprising:

[0005] Step 1: Build an embedded attention strategy network of the super network;

[0006] Step 2: Construct a multi-level encoded Transformer value network based on the embedded attention strategy network of the super network constructed in step 1.

[0007] In the aforementioned drone control method based on embedded attention strategy and multi-level encoding network, in step 1, the construction process of the embedded attention strategy network of the super network is as follows:

[0008] Step 11. Reduce the dimension of the original high-dimensional observation space O and map it to the latent variable space Z;

[0009] Step 12. For each agent i and its observation O in the latent variable space Z i Assign corresponding weights;

[0010] Step 13. Construct a small network based on super network coding;

[0011] Step 14. Use the output of the small network as the key value to calculate the degree of influence of the neighbors / opponents observed by the agent on the agent.

[0012] In the aforementioned drone control method based on embedded attention strategy and multi-level encoding network, step 11 is as follows:

[0013] The original high-dimensional observation space O is reduced in dimension and mapped to the latent variable space Z, that is:

[0014] f:O→Z

[0015] Where dim(O)<dim(Z); O represents the original observation space, Z represents the latent variable space after mapping, and dim(·) represents the dimension of the space.

[0016] In the aforementioned drone control method based on embedded attention strategy and multi-level encoding network, step 12 is as follows:

[0017] For each agent i and its observation O i Assign the corresponding weight w i , the weight distribution is as follows:

[0018] w i =g(O i ;θ)

[0019] Where g is a learnable function, θ is its parameter, and O i is the observation of agent i, w i is the corresponding weight.

[0020] In the aforementioned drone control method based on embedded attention strategy and multi-level encoding network, step 13 is as follows:

[0021] Step 131. Use the HyperNet to encode the agent's neighbors and opponents. Let θ hyper is the parameter of HyperNet;

[0022] Step 132. The super network HyperNet dynamically generates small network parameters φ according to the input x:

[0023] φ=HyperNet(x;θ hyper )

[0024] The small network f(x; φ) changes as the input x changes:

[0025] y=f(x;φ)=f(x;HyperNet(x;θ hyper ))

[0026] Where x is the input data, θ hyper are the parameters of the super network, φ are the dynamically generated small network parameters, and y is the output of the small network.

[0027] In the aforementioned drone control method based on embedded attention strategy and multi-level encoding network, step 14 is as follows:

[0028] Step 141. Use the output result of the small network as the key value K=f(x;φ);

[0029] Step 142. Calculate query value Q and value V;

[0030] Step 143. Calculate attention weights:

[0031]

[0032] Among them, d k is the dimension of the key;

[0033] Step 144. Calculate the influence of the agent's observed neighbors / opponents on the agent:

[0034] output=αV

[0035] Among them, K, Q, and V are the key, query, and value matrices in the attention mechanism, respectively. k is the dimension of the key, α is the attention weight, and output is the output of the attention mechanism.

[0036] In the aforementioned drone control method based on embedded attention strategy and multi-level encoding network, step 2 is as follows:

[0037] Step 21. Agent encoding: Encode the agent itself s, neighbor n and opponent o respectively to obtain the encoding Es 、E n and E o ; The encoding process is expressed as:

[0038] E s =f s (s), E n =f n (n), E o =f o (o)

[0039] where f s , f n , f o are their respective encoding functions. s, n, o represent the original information of the agent itself, its neighbors, and its opponents, respectively. s 、E n 、E o is the corresponding encoding result;

[0040] Step 22. Attention mechanism combination; the multi-head attention mechanism is expressed as:

[0041] MultiHead(Q,K,V)=Concat(head 1 , ..., head h )W O

[0042] Among them, head i =Attention(QW i Q , KW i K , VW i V )

[0043] Among them, Q, K, V are the input query, key and value matrices, W i Q.W i K , W i V is the parameter matrix of the i-th attention head, W O is the output linear transformation matrix, h is the number of attention heads;

[0044] Step 23. Transformer multi-head structure; three encodings (E s 、E n 、E o )Finally, the new multi-head structure of Transformer is formed:

[0045] MH(E s , E n , E o)=MultiHead(Concat(E s , E n , E o ))

[0046] Among them, E s 、E n 、E o They are the encodings of the agent itself, its neighbors, and its opponents, respectively. MH represents the multi-head attention structure.

[0047] Step 24. Encode neighbors and opponents based on local attention calculation and global attention calculation;

[0048] Step 25. Optimize the Transformer structure based on neighbor and opponent encoding.

[0049] In the aforementioned drone control method based on embedded attention strategy and multi-level coding network, in step 24, the local attention calculation process is as follows:

[0050] 7) Through a shared network k Calculate the key value of each neighbor / opponent:

[0051] K i =f k (n i ) or K i =f k (o i )

[0052] 8) Calculate the average-query value:

[0053]

[0054] Where N is the number of neighbors / opponents;

[0055] 9) Calculate attention weight:

[0056]

[0057] 10) Obtain the attention score through the softmax function:

[0058] α i =Softmax(wi)

[0059] 11) Use another shared network v Calculate value:

[0060] V i =f v (n i ) or V i =f v (o i )

[0061] 12) Calculate the weighted sum:

[0062]

[0063] Among them, f k and f v It is a shared network, K i is the key value of the i-th neighbor / opponent, Q avg is the average query value, w l is the attention weight, α l is the normalized attention score, V l is the value, N is the number of neighbors / opponents, and output is the final weighted output.

[0064] In the aforementioned drone control method based on embedded attention strategy and multi-level coding network, in step 24, the global attention calculation process is as follows:

[0065] 6) The weighted value of each agent calculated by the local attention mechanism is used as the global key value K global .

[0066] 7) Calculate the global average-query value:

[0067]

[0068] Where M is the total number of agents;

[0069] 8) Through a shared network g Calculate the global value:

[0070] V global =f g (K global)

[0071] 9) Calculate global attention weight:

[0072]

[0073] 10) Calculate the global weighted sum through softmax:

[0074] output global =softmax(w global )V global

[0075] Here, K global is a global key value, Q global is the global average query value, M is the total number of agents, and f g is a global shared network, V global is the global value, w globalis the global attention weight, output global is the output of global attention.

[0076] In the aforementioned drone control method based on embedded attention strategy and multi-level encoding network, step 25 is as follows:

[0077] 3) Concatenate the encoded self, neighbor, and opponent networks to form the multi-head part of the new Transformer:

[0078] MH=Concat(E s , E n , ,E o )

[0079] 4) Replace the traditional Add calculation method with a gating mechanism:

[0080] output=LayerNorm(Gate(mH,LayerNorm(mH)))

[0081] Among them, E s 、E n 、E o They are the encodings of the agent itself, its neighbors, and its opponents, MH is the output of multi-head attention, GateLayer is the gating layer, LayerNorm is the layer normalization operation, and output is the final output.

[0082] Beneficial effects: The present invention first provides a method for an embedded attention strategy network of a super network, which first uses the super attention network to map the original observation space to a smaller latent variable space, and then assigns different weights to each agent and its observation, and adapts to the dynamic changes in the number of agents and the observation dimension, so as to perform strategy learning with stronger representation capabilities and improve the efficiency of the entire strategy learning. Secondly, a value network of a multi-layer encoded Transformer is provided: the traditional Transformer is improved, and each observation is encoded to form a new multi-head part, in which each encoding adopts a multi-layer attention mechanism. In turn, the dimension of the critic network is greatly reduced, and the strategy is promoted to be learned more stably. In general, the present invention dynamically adapts to dynamic scenes through the embedded attention mechanism of the super network, improves the representation ability and learning efficiency; and uses the multi-layer encoded Transformer value network to reduce the dimension of the critic network and promote strategy learning. Specifically, compared with the existing reinforcement learning algorithm model, the present invention has the following advantages:

[0083] (a) Compared with the existing reinforcement learning model, the present invention can adapt to dynamic scenes without adding additional neural networks, has higher representation ability and learning efficiency, and can avoid dimensionality explosion.

[0084] The existing reinforcement learning model uses a shared neural network to encode the observation input, causing all agents and observations to share the same set of neural network parameters, while the fixed shared neural network parameters are limited, which limits the representation ability of the agent and easily causes local optimality, reducing learning efficiency. Therefore, the algorithm used in the present invention has stronger pertinence, so it has higher representation ability and learning efficiency. In particular, the present invention designs an embedded attention strategy network based on a super network, which can allocate different network parameters to each agent and observation, without the need to add additional neural networks to achieve network parameter adaptation, and even the situation where the adaptive sensor observation information is missing, and the input dimension limit can be ignored. In addition, the present invention can reduce the parameter space, avoid the dimensionality explosion problem, reduce the computing time cost, and improve learning efficiency. Therefore, this method can better meet the scene requirements of high dynamics and high confrontation in actual air combat.

[0085] (b) Compared with the existing Transformer model, the present invention has a smaller parameter space, is suitable for multi-task scenarios, and promotes more stable strategy learning.

[0086] The present invention can effectively capture the "long-term" information and dependencies of agents, where global attention keeps the input dimension of the critic network fixed, that is, it does not increase exponentially with the increase in the number of agents, avoiding the dimensionality explosion of the critic network. At the same time, thanks to the characteristics of the attention mechanism, it can also adapt to scenarios with different numbers of agents under fixed dimensions. For local attention, it can adapt to scenarios with different numbers of observations under fixed dimensions.

[0087] Depend on Figure 3 It can be seen that when the representation module is applied to different algorithms, the original reinforcement learning efficiency is improved, and the reward return is higher. This shows that the present invention has good portability and generalization characteristics, and can improve the representation ability and learning effect of existing algorithms.

[0088] Depend on Figure 4 It can be seen that when the mission scenario changes from eight to one hundred and sixty-four aircraft and the observation size also changes accordingly, the present invention can adapt well to different numbers of agents and observation dimensions. From the curve trend, it can be seen that the entire reward curve is relatively stable, indicating that this method has more stable strategy learning. In addition, from the rise time / number of steps required for different numbers of agents in the figure, it can be seen that the convergence time is greatly reduced, saving the computing time cost. At the same time, this trend shows that the present invention has a strong generalization ability and can adapt to the control requirements of different numbers of intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A network diagram of embedded attention strategy for a super network;

[0090] Figure 2 It is a value network diagram of a multi-layer encoded Transformer;

[0091] Figure 3 The trend of improving the efficiency of reinforcement learning under different algorithms;

[0092] Figure 4 is the variation of reward value and convergence time with the number of agents. DETAILED DESCRIPTION

[0093] Example 1. A drone control method based on an embedded attention strategy and a multi-level coding network is used to solve the problems of poor representation ability, low adaptability, low learning efficiency, and easy dimensionality explosion in existing reinforcement learning methods. Figure 1-4 ,

[0094] First, see Figure 1 , an embodiment of the present invention provides an embedded attention strategy network diagram of a super network, including: mapping the original observation space to a smaller latent variable space; assigning different weights to each agent and its observation;

[0095] In encoding the neighbors and opponents of the intelligent agent, a super network HyerNet is used to output small networks of different intelligent agents and different observed neighbors of each intelligent agent. The small networks at this time have different weight parameters, and the small networks change with the change of input, so as to provide different networks without increasing network parameters, ignore input restrictions, have unrestricted networks, and have better representation capabilities than shared networks with limited network parameters, while avoiding dimensionality explosion.

[0096] Based on the result of inputting the small network as the key value, the query and value values ​​are calculated using the common attention method, and finally the degree of influence of the neighbors / opponents observed by the agent on the agent is obtained.

[0097] On the other hand, see Figure 2 The embodiment of the present invention provides a value network diagram of a multi-layer encoded Transformer, including: encoding the intelligent agent itself, neighbors, and opponents separately; encoding using a combination of local and global multi-head attention mechanisms; the three types of encoding finally form a new multi-head structure of the transformer.

[0098] The encoding method for the agent's neighbors and opponents is the same, using a combination of local and global multi-head attention mechanisms, as follows:

[0099] In the local attention calculation, since each agent has different attention weights for its neighbors / opponents, the key value of each neighbor / opponent is first calculated separately through the shared network, and then all are averaged to obtain the average-query value. The average-query value matches the key value of the corresponding neighbor / opponent of the agent, that is, the attention weight is calculated by the dot product. The attention score of each agent's neighbor / opponent is obtained by the softmax function. Finally, we use the key values ​​of different neighbors / opponents as input to another shared network, calculate the value of each agent to each of its neighbors / opponents separately, and use the attention score as the weight to calculate the weighted sum of the values. If the number of neighbors observed during the interaction changes, although the network dimensions are fixed (three-dimensional relative position and relative speed), the local attention mechanism only needs to calculate the sum of each newly added neighbor separately, and calculate the influence of all neighbors on the agent through the weighted sum. Therefore, the number of neighbors can be expanded in a fixed-dimensional network, with good observation scalability.

[0100] The global attention mechanism still uses the dot product attention mechanism. At this time, the weighted value of each agent calculated by the local attention mechanism is used as the global key value, and all are averaged to obtain the global average-query value, and the global value of each agent is calculated through another shared network. Finally, the global weighted sum is calculated by dot product calculation and softmax operation. Similarly, when the total number of agents changes, it is only necessary to calculate the sum of the newly added agents separately under the previously updated network parameters, and recalculate the influence of all agents, so that it can fine-tune the network parameters. Therefore, this method has good generalization characteristics. In addition, its network dimension and number of layers do not increase with the increase of agents, reducing the computational burden.

[0101] Finally, the encoded self, neighbor and opponent networks are concatenated to form a new Transformer multi-head part, and the traditional Add calculation method is replaced by the gate. This structure has better representation ability than the traditional Transformer.

[0102] Example 2. A drone control method based on an embedded attention strategy and a multi-level coding network is used to solve the problems of poor representation ability, low adaptability, low learning efficiency, and easy dimensionality explosion in existing reinforcement learning methods. Figure 1-4 ,

[0103] The strategy mainly consists of two core components: the embedded attention strategy network of the super network and the Transformer value network of multi-level encoding. These two components work together to effectively adapt to the dynamic changes in the number of agents and observation dimensions, improve learning efficiency and avoid dimensionality disaster.

[0104] 1. Embedded Attention Strategy Network of Super Network

[0105] See attached Figure 1 , the implementation steps of the network are as follows:

[0106] 1.1 Observation Space Mapping

[0107] Map the original high-dimensional observation space O to a smaller latent variable space Z, that is:

[0108] f: O→Z

[0109] Where dim(Z)<dim(O). Here, O represents the original observation space, Z represents the latent variable space after mapping, and dim(·) represents the dimension of the space. This step significantly reduces the computational complexity by reducing the dimension while retaining key information. This mapping makes subsequent processing more efficient, especially when dealing with large-scale drone clusters, which can effectively alleviate the pressure on computing resources.

[0110] 1.2 Weight Distribution

[0111] For each agent i and its observation o i Assign different weights w i The mathematical expression of weight distribution is as follows:

[0112] w i =g(o i ;θ)

[0113] Where g is a learnable function, θ is its parameter, and o i is the observation of agent i, w i This differentiated weight allocation mechanism enables the system to better identify and respond to the status of key intelligent agents and improve the accuracy and adaptability of decision-making.

[0114] 1.3 Super Network Coding

[0115] 1) Use the HyperNet to encode the agent’s neighbors and opponents. Let θ hyper are the parameters of HyperNet.

[0116] 2) HyperNet dynamically generates small network parameters φ based on the input x:

[0117] φ-HyperNet(x;θ hyper )

[0118] The small network f(x; φ) changes as the input x changes:

[0119] y=f(x;φ)=f(x;HyperNet(x;θ hyper ))

[0120] Here, x is the input data, θhyper is the parameter of the super network, φ is the dynamically generated small network parameter, and y is the output of the small network. This dynamic network generation method greatly enhances the expressiveness and flexibility of the model, enabling it to adapt to combinations of agents of different sizes and types without pre-defining a fixed network structure.

[0121] 1.4 Application of Attention Mechanism

[0122] 1) Use the output of the small network as the key value K = f(x; φ).

[0123] 2) Calculate the query value Q and value V.

[0124] 3) Calculate attention weight:

[0125]

[0126] where d k Is the dimension of the key.

[0127] 4) Calculate the influence of the agent's observed neighbors / opponents on the agent:

[0128] output=αV

[0129] Here, K, Q, and V are the key, query, and value matrices in the attention mechanism, respectively. k is the dimension of the key, α is the attention weight, and output is the output of the attention mechanism. This attention mechanism enables the model to dynamically focus on important information sources, improving the efficiency and accuracy of information processing, especially when dealing with large-scale and complex drone cluster environments.

[0130] 2. Multi-layer Encoding Transformer Value Network

[0131] See attached Figure 2 , the implementation steps of the network are as follows:

[0132] 2.1 Agent Coding

[0133] Encode the agent itself s, neighbor n and opponent o respectively to get the encoding E s 、E n and E o The encoding process can be expressed as:

[0134] E s =f s (s), E n =f n (n), E o =f o (o)

[0135] where f s , f n , f o are their respective encoding functions. s, n, o represent the original information of the agent itself, its neighbors, and its opponents, respectively. s 、E n 、E o This classification encoding method can retain the characteristics of various entities and provide more discriminative input for the subsequent attention mechanism.

[0136] 2.2 Attention Mechanism Integration

[0137] The encoding process uses a combination of local and global multi-head attention mechanisms. Let the number of heads be h. The multi-head attention mechanism can be expressed as:

[0138] MultiHead(Q,K,V)=Concat(head 1 , ..., head h )W O

[0139] Among them, head i =Attention(QW i Q , KW i K , VW i V ).

[0140] Here, Q, K, V are the input query, key, and value matrices, and W i Q , W i K , W i V is the parameter matrix of the i-th attention head, W O is the output linear transformation matrix, and h is the number of attention heads. This multi-head attention mechanism that combines local and global attention enables the model to capture local details and global context at the same time, greatly enhancing the model's expressiveness and ability to understand complex scenes.

[0141] 2.3 Transformer multi-head structure composition

[0142] The three codes (E s 、E n 、E o )Finally, the new multi-head structure of Transformer is formed:

[0143] MH(E s , E n , Eo )=MultiHead(Concat(E s , E n , E o ))

[0144] Here, E s 、E n 、E o They are the encodings of the agent itself, its neighbors, and its opponents, respectively. MH stands for the multi-head attention structure. This structural design enables the model to effectively integrate the information of itself, its neighbors, and its opponents to form a comprehensive state representation.

[0145] 2.4 Neighbor and opponent encoding detailed steps

[0146] 2.4.1 Local Attention Calculation

[0147] 1) Through a shared network k Calculate the key value of each neighbor / opponent:

[0148] K i =f k (n i ) or K i =f k (o i )

[0149] 2) Calculate the average query value:

[0150]

[0151] Where N is the number of neighbors / opponents.

[0152] 3) Calculate attention weight:

[0153]

[0154] 4) Obtain the attention score through the softmax function:

[0155] α i =softmax(w i )

[0156] 5) Use another shared network v Calculate value:

[0157] V i =f v (n i ) or V i =f v (o i )

[0158] 6) Calculate the weighted sum:

[0159]

[0160] Here, f k and f v It is a shared network, K i is the key value of the i-th neighbor / opponent, Q avg is the average query value, w i is the attention weight, α i is the normalized attention score, V i is the value, N is the number of neighbors / opponents, and output is the final weighted output. This local attention mechanism can effectively capture the interaction between the agent and its neighboring entities and improve the model's perception of the local environment.

[0161] 2.4.2 Global Attention Calculation

[0162] 1) The weighted value of each agent calculated by the local attention mechanism is used as the global key value K global .

[0163] 2) Calculate the global average-query value:

[0164]

[0165] Where M is the total number of agents.

[0166] 3) Through a shared network g Calculate the global value:

[0167] V global =f g (K global )

[0168] 4) Calculate the global attention weight:

[0169]

[0170] 5) Calculate the global weighted sum through softmax:

[0171] output global =softmax(w global )V global

[0172] Here, K global is a global key value, Q global is the global average query value, M is the total number of agents, and f g is a global shared network, V global is the global value, w global is the global attention weight, output globalis the output of global attention. The global attention mechanism enables the model to consider the state of the entire system and helps to make more globally optimal decisions.

[0173] 2.5 Transformer structure optimization

[0174] 1) Concatenate the encoded self, neighbor, and opponent networks to form the multi-head part of the new Transformer:

[0175] MH=Concat(E s , E n , E o )

[0176] 2) Replace the traditional Add calculation method with a gating mechanism:

[0177] output=LayerNorm(Gate(MH,LayerNorm(MH)))

[0178] Here, E s 、E n 、E o They are the encodings of the agent itself, neighbors, and opponents, MH is the output of multi-head attention, GateLayer is the gating layer, LayerNorm is the layer normalization operation, and output is the final output. This optimized structure enhances the nonlinear expression ability of the model, and at the same time realizes the dynamic fusion of different information sources through the gating mechanism, improving the expression ability and training stability of the model.

[0179] This method can not only effectively solve the problems of poor representation ability, low adaptability, low learning efficiency, and easy dimensionality explosion in existing reinforcement learning methods, but also has good observation scalability and generalization characteristics. In practical applications, the network structure and parameters, such as the number of attention heads h, the complexity of HyperNet, and the number of layers of the encoding network, can be flexibly adjusted according to the specific UAV cluster mission scenario to achieve the best balance between computational efficiency and model performance.

Claims

1. A drone control method based on embedded attention strategy and multi-level coding network, characterized in that: include: Step 1: Build an embedded attention strategy network of the super network; Step 2: Construct a multi-level encoded Transformer value network based on the embedded attention strategy network of the super network constructed in step 1.

2. The drone control method based on embedded attention strategy and multi-level coding network according to claim 1 is characterized in that: In step 1, the construction process of the embedded attention strategy network of the super network is as follows: Step 11. Reduce the dimension of the original high-dimensional observation space O and map it to the latent variable space Z; Step 12. For each agent i and its observation O in the latent variable space Z i Assign corresponding weights; Step 13. Construct a small network based on super network coding; Step 14. Use the output of the small network as the key value to calculate the degree of influence of the neighbors / opponents observed by the agent on the agent.

3. The drone control method based on embedded attention strategy and multi-level coding network according to claim 2 is characterized in that: Step 11 is as follows: The original high-dimensional observation space O is reduced and mapped to the latent variable space Z, that is: f:O→Z Where dim(O)<dim(Z); O represents the original observation space, Z represents the latent variable space after mapping, and dim(·) represents the dimension of the space.

4. The drone control method based on embedded attention strategy and multi-level coding network according to claim 2 is characterized in that: Step 12 is as follows: For each agent i and its observation O i Assign the corresponding weight w i , the weight distribution is as follows: w i =g(O i (i) Where g is a learnable function, θ is its parameter, and O i is the observation of agent i, w i is the corresponding weight.

5. The drone control method based on embedded attention strategy and multi-level coding network according to claim 2 is characterized in that: Step 13 is as follows: Step 131. Use the HyperNet to encode the agent's neighbors and opponents. Let θ hyper is the parameter of HyperNet; Step 132. The super network HyperNet dynamically generates small network parameters φ according to the input x: φ=HyperNet(x;θ hyper ) The small network f(x; φ) changes as the input x changes: y=f(x;φ)=f(x;HyperNet(x;θ hyper )) Where x is the input data, θ hyper are the parameters of the super network, φ are the dynamically generated small network parameters, and y is the output of the small network.

6. The drone control method based on embedded attention strategy and multi-level coding network according to claim 2 is characterized in that: Step 14 is as follows: Step 141. Use the output result of the small network as the key value K=f(x;φ); Step 142. Calculate query value Q and value V; Step 143. Calculate attention weights: Among them, d k is the dimension of the key; Step 144. Calculate the influence of the agent's observed neighbors / opponents on the agent: output=αV Among them, K, Q, and V are the key, query, and value matrices in the attention mechanism, respectively. k is the dimension of the key, α is the attention weight, and output is the output of the attention mechanism.

7. The drone control method based on embedded attention strategy and multi-level coding network according to claim 1 is characterized in that: Step 2 is as follows: Step 21. Agent encoding: Encode the agent itself s, neighbor n and opponent o respectively to obtain the encoding E s 、E n and E o ; The encoding process is expressed as: E s =f s (s),E n =f n (n),E o =f o (o) where f s , f n , f o are their respective encoding functions; s, n, o represent the original information of the agent itself, its neighbors, and its opponents, respectively. s 、E n 、E o is the corresponding encoding result; Step 22. Attention mechanism combination; the multi-head attention mechanism is expressed as: MultiHead(Q,K,V)=Concat(head1,...,head h )W O in, Among them, Q, K, V are the input query, key and value matrices, is the parameter matrix of the i-th attention head, W O is the output linear transformation matrix, h is the number of attention heads; Step 23. Transformer multi-head structure; three encodings (E s 、E n 、E o )Finally, the new multi-head structure of Transformer is formed: MH (E s ,HAVE BEEN n ,HAVE BEEN o )=MultiHead(Conca(E s ,HAVE BEEN n ,HAVE BEEN o )) Among them, E s 、E n 、E o They are the encodings of the agent itself, its neighbors, and its opponents, respectively. MH represents the multi-head attention structure. Step 24. Encode neighbors and opponents based on local attention calculation and global attention calculation; Step 25. Optimize the Transformer structure based on neighbor and opponent encoding.

8. The drone control method based on embedded attention strategy and multi-level coding network according to claim 7 is characterized in that: In step 24, the local attention calculation process is as follows: 1) Through a shared network k Calculate the key value of each neighbor / opponent: K i =f k (n i ) or K i =f k (o i ) 2) Calculate the average query value: Where N is the number of neighbors / opponents; 3) Calculate attention weight: 4) Obtain the attention score through the softmax function: a i =softmax(w i ) 5) Use another shared network v Calculate value: V i =f v (n i ) or V i =f v (o i ) 6) Calculate the weighted sum: Among them, f k and f v It is a shared network, K i is the key value of the i-th neighbor / opponent, Q avg is the average query value, w i is the attention weight, α i is the normalized attention score, V i is the value, N is the number of neighbors / opponents, and output is the final weighted output.

9. The drone control method based on embedded attention strategy and multi-level coding network according to claim 7, characterized in that: In step 24, the global attention calculation process is as follows: 1) The weighted value of each agent calculated by the local attention mechanism is used as the global key value K global ; 2) Calculate the global average-query value: Where M is the total number of agents; 3) Through a shared network g Calculate the global value: V global =f g (K global ) 4) Calculate the global attention weight: 5) Calculate the global weighted sum through softmax: output global =softmax(w global )V global Here, K global is a global key value, Q global is the global average query value, M is the total number of agents, and f g is a global shared network, V global is the global value, w global is the global attention weight, output global is the output of the global attention.

10. The drone control method based on embedded attention strategy and multi-level coding network according to claim 7, characterized in that: Step 25 is as follows: 1) Concatenate the encoded self, neighbor, and opponent networks to form the multi-head part of the new Transformer: MH=Concat(E s ,E n ,E o ) 2) Replace the traditional Add calculation method with a gating mechanism: output=LayerNorm(Gate(MH,LayerNorm(MH))) Among them, E s 、E n 、E o They are the encodings of the agent itself, its neighbors, and its opponents, MH is the output of multi-head attention, GateLayer is the gating layer, LayerNorm is the layer normalization operation, and output is the final output.

Images