A Method for Allocating Interference Resources in UAV Swarms Based on Pre-trained Attention Encoders

Abstract

This application relates to a method for allocating interference resources for a drone swarm based on a pre-trained attention encoder. The method includes: setting an objective function for the interference resource allocation problem using an interference effect evaluation model and the interference power consumed by the drone swarm; modeling the interference resource allocation problem as a constrained combinatorial optimization problem by considering the limitations on drone communication, energy, and flight speed under denial conditions; modeling the interference decision-making process in the combinatorial optimization problem as a distributed locally observable Markov game model; mapping drones as agents; and solving the model using the MAPPO algorithm based on a pre-trained attention encoder to obtain the interference resource allocation scheme for the drone swarm. This method can solve the problems of state space dimensionality explosion and environmental non-stationarity in multi-agent collaborative decision-making processes.

Description

Technical Field

[0001] This application relates to the field of UAV interference resource allocation technology, and in particular to a method for UAV swarm interference resource allocation based on a pre-trained attention encoder. Background Technology

[0002] In recent years, unmanned aerial vehicles (UAVs) have been widely used in search and rescue, target reconnaissance, and other missions due to their low cost, flexibility, autonomous control, and reusability. However, when facing complex, changeable, and locally observable environments, a single UAV's detection, survivability, and combat capabilities are significantly reduced due to limitations in its perception and computing resources. UAV swarms, formed by networking multiple UAVs, can extend the capabilities of individual UAVs, enhancing their survivability, combat effectiveness, and overall combat capability. However, as the scale of UAV swarms increases, the cooperation and competition within the swarm become quite complex, and there are still many challenges to overcome in UAV swarm collaborative decision-making technology. In the case of collaborative jamming resource allocation scenarios, effective jamming by UAV swarms through intelligent jamming power control relies on information exchange between swarms. In denial environments with limited communication and navigation, the intelligent decision-making capabilities of the swarm will be greatly affected. Therefore, to achieve intelligent interference resource allocation for UAV swarms in complex electromagnetic environments, the algorithm needs to be able to operate under denial conditions. This is an unavoidable problem in communication interference countermeasures and is also a problem that previous work has not considered. Current mainstream swarm intelligent decision-making algorithms include multi-agent reinforcement learning (MARL), game theory-based methods, and swarm intelligence optimization algorithms. However, these methods require repeated runs to solve problems in continuous decision-making situations, have poor generalization, and are not suitable for intelligent decision-making under denial conditions. Summary of the Invention

[0003] Therefore, it is necessary to address the aforementioned technical problems by providing a method for allocating UAV swarm interference resources based on a pre-trained attention encoder, which can achieve UAV swarm interference resource allocation under denial conditions, improve resource allocation efficiency, and increase interference success rate.

[0004] A method for allocating interference resources in UAV swarms based on a pre-trained attention encoder, the method comprising:

[0005] The reconnaissance area of ​​the UAV swarm and the reconnaissance mission of the UAV swarm against multiple target radio stations are obtained; the UAV swarm includes multiple UAVs; based on the interference power transmitted by the UAVs during the reconnaissance, the antenna gain of the target radio stations, and the path loss between the UAVs and the target radio stations, an evaluation model for the interference effect of the UAV swarm's coordinated interference under denial conditions is established.

[0006] The objective function of the interference resource allocation problem is set by using the interference effect evaluation model and the interference power consumed by the UAV swarm. Combined with the limited communication, energy and flight speed of UAVs under the denial condition, the interference resource allocation problem is modeled as a constrained combinatorial optimization problem.

[0007] This paper models the interference decision-making process in the combinatorial optimization problem as a distributed locally observable Markov game, maps drones to intelligent agents, and uses the MAPPO algorithm based on a pre-trained attention encoder to solve the combinatorial optimization problem. Based on the pre-trained attention encoder, features of the environmental states of the action network and comment network in the distributed locally observable Markov game model are extracted and compressed to obtain the updated environmental states. Interference resources are allocated based on the updated environmental states, and the discounted reward and advantage function for each drone in the updated environmental states are calculated. The loss functions of the action network and comment network are calculated based on the discounted reward, advantage function, and the loss function defined by the algorithm, and the Adam optimizer is used to update the action network and comment network to obtain the updated action network and comment network. The solution to the combinatorial optimization problem is output based on the updated action network; that is, the combinatorial optimization problem of interference resource allocation schemes for drone swarms.

[0008] The aforementioned UAV swarm interference resource allocation method based on a pre-trained attention encoder establishes an interference effect evaluation model for cooperative UAV swarm interference under denial conditions based on the interference power transmitted during UAV reconnaissance, the antenna gain of the target radio station, and the path loss between the UAV and the target radio station. Using the interference effect evaluation model and the interference power consumed by the UAV swarm, the objective function of the interference resource allocation problem is set. Considering the limitations on UAV communication, energy, and flight speed under denial conditions, the interference resource allocation problem is modeled as a constrained combinatorial optimization problem. This achieves modeling of the target's motion and communication capabilities, and models the interference resource allocation problem under communication, navigation, and resource constraints as a constrained combinatorial optimization problem. To solve this combinatorial optimization problem using a multi-agent reinforcement learning algorithm... The perturbation decision-making process in combinatorial optimization problems is modeled as a locally observable Markov process, and UAVs are mapped as agents. A pre-trained autoencoder (APSE) based on an attention mechanism is proposed as a pre-unit of the multi-agent reinforcement learning algorithm to perform deep feature extraction and dimensionality reduction on the input environment state, so as to reduce the impact of environmental non-stationarity and excessively large state space on the performance of the multi-agent reinforcement learning algorithm. The pre-training mechanism and self-attention mechanism introduced in APSE can significantly improve its feature extraction ability and training stability. The optimization problem of cooperative perturbation resource allocation is successfully solved by adopting the CTDE paradigm and combining it with the multi-agent reinforcement learning algorithm MAPPO. Compared with the original MAPPO and other algorithms, the MAPPO-APSE algorithm has significantly improved the efficiency of perturbation resource allocation and average reward, and has better decision performance. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating a method for allocating interference resources in a drone swarm based on a pre-trained attention encoder, as shown in one embodiment.

[0010] Figure 2 This is a schematic diagram of the structure of a pre-trained attention encoder in one embodiment;

[0011] Figure 3 This is a schematic diagram of the pre-training process of a pre-trained attention encoder in one embodiment;

[0012] Figure 4 This is a schematic diagram of the framework of the MAPPO algorithm based on a pre-trained attention encoder in another embodiment. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] In one embodiment, such as Figure 1 As shown, a method for allocating interference resources in UAV swarms based on a pre-trained attention encoder is provided, including the following steps:

[0015] Step 102: Obtain the reconnaissance area of ​​the UAV cluster and the reconnaissance mission of the UAV cluster against multiple target radio stations; the UAV cluster includes multiple UAVs; establish an evaluation model of the interference effect of UAV cluster cooperative interference under denial conditions based on the interference power transmitted during UAV reconnaissance, the antenna gain of the target radio station, and the path loss between the UAV and the target radio station.

[0016] Firstly, this application proposes to use a swarm of quadcopter drones to perform anti-jamming tasks. The drones have limitations on flight speed, jamming power, and battery capacity. Furthermore, the drone swarm possesses communication capabilities, enabling it to communicate with drones within a certain radius centered on itself. The maximum communication radius can be calculated using the following formula:

[0017]

[0018] in, P is the maximum communication power of the drone. I,N T represents the sum of interference and noise experienced by the drone. h To establish a signal-to-noise ratio (SNR) threshold, normal communication is possible when the SNR of the signal received by UAV i from UAV k is greater than this threshold.

[0019] To simulate communication-constrained scenarios, this application approximates this by setting an upper limit on the number of other drones each drone can exchange information with. Furthermore, drones can only interact with other drones within a specified range for a short period; beyond this range, multiple hops are required, resulting in varying degrees of lag in information exchange.

[0020] To simulate navigation-constrained scenarios, in the event of navigation failure, the self-localization error of the UAV caused by inertial navigation can be modeled as a Gaussian noise vector with a mean of 0 and a variance of 1. In this case, the self-localization error ΔS of UAV i is... i It can be represented as:

[0021]

[0022] Where, ΔS i,0 Let ΔS be the self-localization error of the UAV at time 0. i,t′ Let be the cumulative self-localization error of UAV i at time t′. Meanwhile, in swarm mode, each UAV can obtain target location information from surrounding reachable UAVs. Through comprehensive estimation, the final estimated target location information is obtained, which can be expressed as:

[0023]

[0024] in, For the true location information of target j, d kj Let λ be the distance from drone k to target j. k Let K be the direction-finding error coefficient of UAV k. The number of drones directly connected to drone i.

[0025] In this application, targets are divided into two categories: ordinary communication radio targets and time-sensitive targets. Ordinary communication radio targets possess mobility, moving randomly within an area, and can periodically communicate with other N-1 communication radios without interference. That is, in the initial stage, a communication period of length T... radio The communication lasts for a period of time, then switches to a silent state and continues for T′. radio Time, then cycles between communication and silent states. Time-sensitive targets are short-lived communication targets, lacking mobility, and are in a silent state during the initial stage of system operation. T tct It appears after time T′, and then remains for T′. tct Disappears after a certain time (T′) tct <T′ radio Time-sensitive targets appear for short periods, are more random, and pose a greater threat. Therefore, within the time frame of a time-sensitive target's appearance, it needs to be given a higher interference priority, and successful interference will yield a greater reward. The UAV swarm cooperative jamming network radio system studied in this application considers executing a reconnaissance and jamming mission involving N UAVs against M target radio stations within a 5km × 5km area. The UAVs are fixed at an altitude of h = 1km to jam the target radio stations. The UAV set is n = {1, 2, 3, ..., N}, and the target radio station set is m = {1, 2, 3, ..., M}. Since the altitudes of the UAVs and target radio stations are relatively fixed, the position of UAV i in this scenario can be simplified to three-dimensional coordinates (x, y ... i ,y i The location of the target radio station j can be simplified to (x, h), where x is the target radio station j. j ,y j Initially, both the drone and the target radio spawn in a fixed area.

[0026] In the system studied in this application, the UAV uses a directional antenna for jamming, and the jamming effect is improved by increasing the gain of the transmitting antenna. In this case, the UAV's jamming capability is related to its own attitude and the relative position of the UAV and the jamming target. Assume that UAV i jams target radio station j, and the UAV's spatial position is s. i =(x i ,y i ,z i The power of the transmitted interference signal is The target radio station is located in the space of s. j =(x j ,y j ,z j The received interference signal power is The power units here are all in dBm. The Friesian transport equation is used to construct... and The relationship between them is as follows:

[0027]

[0028] in, For the antenna gain of UAV i against interference signals, For the target radio station's antenna gain against interfering signals, PL ij This represents the path loss. In free space, the signal path loss is proportional to the Euclidean distance between the UAV and the target. Considering the complex environment around the target and the multipath effect in signal propagation, the path attenuation of the signal can be generalized to be proportional to the distance raised to the power of τ∈[3,5]. The path loss in this case can be modeled as:

[0029] PL ij =10τlog 10 d ij +K pl (5)

[0030] Among them, K pl Let d be the path loss constant. ij =‖s i -s j Let || represent the distance between UAV i and target j, and ||·| denote the 2-norm of the vector.

[0031] In the system constructed in this application, each target communication radio station establishes a communication link with other communication radio stations. Each radio station is not only affected by interference from UAVs, but also by non-target signals from other units and environmental noise. Therefore, it is more reasonable to use the signal-to-interference-plus-noise ratio as an evaluation index of the interference effect.

[0032] Assuming that the multiple interference signals received by the target radio station conform to the linear superposition condition, the signal-to-interference-plus-noise ratio (SIR) of the signal received by the target radio station j is:

[0033]

[0034] Where N represents the number of drones. P represents the interference power of UAV i against target radio station j. jk The power of the communication signal received by the target radio station j. Let k be the transmit antenna gain of the target radio station. For the receiving antenna gain of target radio station j, PL ij The path loss N between target radio station j and UAV i noise The mean is 0 and the variance is σ. 2 Gaussian white noise.

[0035] To ensure the effectiveness of UAV jamming, it was estimated that when the signal-to-interference-plus-noise ratio (SINR) of the signal received by a certain communication link of target radio station j... j When the signal-to-interference-plus-noise ratio (SIR) is ≤H, the interference is considered effective, and the link will be disconnected. Here, H is the SIR threshold for effective interference.

[0036] Step 104: Using the interference effect evaluation model and the interference power consumed by the UAV swarm, set the objective function of the interference resource allocation problem. Combined with the limited communication, energy and flight speed of UAVs under the denial condition, the interference resource allocation problem is modeled as a constrained combinatorial optimization problem.

[0037] This application investigates the optimization problem of maximizing interference effectiveness through interference resource allocation under rejection conditions. Maximizing interference effectiveness means minimizing the interference power used while achieving effective interference.

[0038] Therefore, the interference resource allocation optimization model established in this application is as follows.

[0039]

[0040] Where λ1 and λ2 are weight constants, and λ1,λ2∈[0,1], they are used to represent the relative importance between the effectiveness of the interference and the amount of resource consumption; the larger the weight, the greater the importance. ω i This represents the threat coefficient of the communication link. In actual combat scenarios, the jamming task of the drone swarm is often the highest priority, so the weight λ1 of the jamming effectiveness is usually set to be larger.

[0041] In summary, the problem of allocating disturbed resources under rejection conditions can be modeled as a constrained combinatorial optimization problem, specifically expressed as:

[0042]

[0043] in, Let N represent the interference power emitted by UAV i against target radio j, M represent the number of UAVs, λ1 and λ2 are weighting constants, and λ1,λ2∈[0,1], used to represent the relative importance between interference effectiveness and resource consumption, ω i The threat factor for the communication link. This is a resource allocation function, related to the signal-to-interference-plus-noise ratio (SIR) of the target radio station j; the higher the SIR, the better the interference effect. P represents the interference power of UAV i against target radio station k. max C represents the maximum interference power, ratio represents the maximum communicable percentage, and C represents the maximum interference power. i The number of other drones that can communicate with the i-th drone is represented by H, the successful interference threshold is represented by H, C1 indicates that the interference power of the drone is limited, C2 indicates that each drone can only interfere with one target at a time, i.e., the number of interference targets is limited, and C3 indicates that the global information exchange of the drone swarm is limited.

[0044] Step 106: The interference decision-making process in the combinatorial optimization problem is modeled as a distributed locally observable Markov game model. The drones are mapped as agents. The MAPPO algorithm based on a pre-trained attention encoder is used to solve the combinatorial optimization problem. Features of the environmental states of the action network and comment network in the distributed locally observable Markov game model are extracted and compressed using the pre-trained attention encoder to obtain the updated environmental states. Interference resources are allocated based on the updated environmental states, and the discounted reward and advantage function for each drone in the updated environmental states are calculated. The loss functions of the action network and comment network are calculated based on the discounted reward, advantage function, and the loss function defined by the algorithm. The Adam optimizer is used to update the action network and comment network to obtain the updated action network and comment network. The solution to the combinatorial optimization problem is output based on the updated action network; that is, the combinatorial optimization problem of interference resource allocation schemes for a drone swarm.

[0045] A locally observable Markov game can be represented in the form of a seven-tuple. in:

[0046] 1) Denotes the state space set, and In the model of this application, the agent's observation state consists of three parts, namely, the enemy radio station's location information M estimated by the UAV at time t. i (t), the relative positions and communication status between drones G i (t) and the previous UAV jamming strategy E i (t). For unreachable targets in UAV t communication, the corresponding value is set to 0. With the above settings, the state space of the environment is as follows:

[0047]

[0048] 2) Let n be the set of intelligent agents, which is defined here as the set of drone swarms, n = {1, 2, 3, ..., N}.

[0049] 3) For the action space set, Defined as the action taken by agent i. Let i be the set of actions that agent i can take. It is a Cartesian product. In this application, the drone's actions are represented by the magnitude of interference power, and the drone's actions are quantified into 5 interference levels, with higher levels indicating higher interference power.

[0050] 4) P is the state transition probability. This indicates that the agent is in the current state S. t Take joint action Transition to the next state S t+1 The probability of.

[0051] 5) Let be the set of reward functions. Specifically, the reward of agent i at time t can be expressed as: In this application, the reward value of the UAV is related to its consumed interference power, the interference effect produced, and the type of interference target. Furthermore, the reward received by the UAV varies depending on the type of interference target. The reward function designed based on these principles is as follows:

[0052]

[0053] Among them, T j Given the type of enemy target j, a normal radio target takes the value 1, and a time-sensitive target takes the value 2. (sum) j The number of drones participating in jamming enemy target j. sgn(·) is the sign function, when H-SINR j A value >0 indicates that the drone swarm successfully interfered with enemy target j. In the event of successful interference, the lower the power consumption of the drones, the greater the reward.

[0054] 6) γ∈[0,1] is the reward discount factor, used to weigh the impact of past actions on the current reward value.

[0055] Considering the complexity of UAV swarm interference scenarios under denial conditions, the large dimensionality of environmental states, and the higher requirements for UAV information processing and feature understanding and discrimination capabilities, the main objectives of this application are: 1) to perform feature dimensionality reduction of environmental states using an autoencoder; 2) to utilize an attention mechanism to calculate the relationships between various sub-states in the environmental state, thereby improving the feature extraction capability of the autoencoder; and 3) to initialize the autoencoder using pre-training, designing a pre-trained attention encoder (APSE), the structure of which is as follows: Figure 2As shown, APSE's input is the environment state, which can be obtained through the interaction between the agent and the environment, saving significant time in data collection, labeling, and processing. It should be noted that when APSE is formally used to solve the problem of interfering resource allocation, it only uses the pre-trained encoder part. The APSE encoder part adopts a structure similar to the encoder part in Transformer, the difference being that it omits word embedding operations and directly uses the environment state as the encoder input. Figure 2 Multi-Head Attention is a multi-head attention mechanism that calculates the relationship between input and environment states to obtain the relationship between states and distinguish the more important features; Add&Norm are residual connections and layer normalization operations to alleviate the gradient vanishing and network degradation problems in deep networks; Feedforward is a multi-layer feedforward neural network, mainly used for scaling feature dimensions, specifically including:

[0056] For the environment state matrix S = [s1, s2, ..., s...] n By using an attention mechanism to calculate the relationships between sub-states, a better state representation can be extracted for the algorithm. Each sub-state s in S... i Multiplying ∈S by the three coefficient matrices respectively yields the initial representations of its Q, K, and V values:

[0057]

[0058] Here, Q (Query) can be understood as a query vector, used to match other substates; K (Key) is used to match Q, and can be understood as the keyword of the substate; V (Value) represents the important information or features of the substate.

[0059] Any substate s in the state matrix S i The degree of association between the state and other substates can be calculated using Q and K, as follows:

[0060]

[0061] Where `score` is the calculated self-attention score, `·` represents the dot product operation, and `d` is the matrix dimension of Q and K. In the self-attention mechanism, these two matrix dimensions are consistent. (Divided by...) The purpose is to prevent the dot product result from being too large. Then, the attention score is normalized to attention weights using the softmax function. Finally, the attention weights are multiplied by V to obtain the output of the self-attention mechanism:

[0062]

[0063] In the MARL algorithm employing the Actor-Critic structure, the Actor network receives the local state observed by the agent, while the Critic network, being a centralized network, receives the global state of the environment. This global state has a larger dimension and is relatively more complex. Therefore, this application designs different APSEs for different data dimensions processed by the networks. The APSE used for the Critic network has a larger number of attention heads and outputs a larger feature dimension.

[0064] Pre-training is an indispensable step in the APSE proposed in this application. On the one hand, directly adding a randomly initialized APSE to the MARL algorithm can lead to training instability due to incorrect feature extraction. On the other hand, since the APSE is at the forefront of the entire neural network, its gradient during backpropagation is small, and its parameter updates are slow, resulting in poor performance when training the APSE directly. In view of these problems, this application first uses pre-training to allow the APSE to learn some common characteristics of the environment state. Then, the encoder part of the pre-trained APSE is added to the MARL algorithm, and the encoder is fine-tuned through the rewards generated by the interaction between the agent and the environment, achieving the goal of feature learning and thus alleviating the training instability problem caused by randomly initialized APSEs.

[0065] Figure 3 The pre-training process for APSE is divided into a data collection phase and a training phase. During pre-training, the main goal of the autoencoder is to extract the features most conducive to reconstructing the original input; therefore, the optimization objective of APSE is:

[0066]

[0067] Where S represents the output state of the environment, and S′ represents the reconstructed environment state by APSE. The goal of the optimization function is to minimize the difference between the two. The pre-training pseudocode for APSE is shown in Table 1.

[0068] Table 1

[0069]

[0070] During the pre-training process, APSE uses environmental state data generated by random actions or interactions between the agent and the environment. This ensures input-output consistency when combined with the MARL algorithm and is also easy to implement. APSE is trained end-to-end using MARL and can be used with all MARL algorithms. Then, the APSE-based MAPPO algorithm is designed. The MAPPO algorithm adopts an Actor-Critic architecture and CTDE approach, with the ultimate goal of finding the optimal policy π for each agent through training. θ Sum function V φ(s). In this process, MAPPO reduces V by training a global Critic network. φ (s) variance, while relying on the constraints of this network to enable individual agents to cooperate with each other, mitigating environmental nonstationarity in multi-agent systems. This network is only used during the training phase. For each agent i, MAPPO is based on its local observations s i and sharing strategy π θ (a i |s i Choose an action a i To maximize discount rewards:

[0071]

[0072] MAPPO follows the PPO algorithm to train a separate policy function (Actor network) for each agent. The optimization objective of the Actor network is:

[0073]

[0074] in This is the result calculated using the generalized advantage function, where S is the policy entropy and σ is the entropy coefficient. The calculation method is as follows:

[0075]

[0076] in, For state Take action below Action value function, For state The value function. This is achieved by calculating the advantage function. Only actions with values ​​greater than the average value are encouraged, thus reducing variance during training. A weighted average of the n advantage functions is introduced using parameters λ∈[0,1] to balance the bias and variance introduced during value function estimation. The optimization objective of the Critic network is:

[0077]

[0078] in The reward is a discount. In the loss function above, B represents the batch size, and N is the number of agents. During training, this application follows common practices in PPO: state normalization, gradient clipping, layer normalization, orthogonally initialized activation functions, and large batch sizes, etc. The APSE-based MAPPO algorithm framework is as follows: Figure 4As shown, in this algorithm, APSE is used as the encoder. It first extracts and compresses features from the states generated by the environment before inputting them into the Critic and Actor networks. The entire algorithm is divided into two stages: data collection and algorithm update. In this stage, the agent interacts with the environment to generate rewards and new environmental states, which are then stored in a data cache pool. When the amount of data in the cache pool reaches a threshold, the algorithm update stage begins. In this stage, the previously collected data is used to calculate the losses of the Critic and Actor networks, and the algorithm is updated simultaneously. The training process is shown in Table 2.

[0079] Table 2

[0080]

[0081] By utilizing the Adam optimizer to update the action network and the optimization objective, the updated action network is then used to adjust the environment state based on the current input. Allocate jamming resources for each drone. Obtain the interference resource allocation scheme for the drone swarm.

[0082] The aforementioned UAV swarm interference resource allocation method based on a pre-trained attention encoder establishes an interference effect evaluation model for cooperative UAV swarm interference under denial conditions based on the interference power transmitted during UAV reconnaissance, the antenna gain of the target radio station, and the path loss between the UAV and the target radio station. Using the interference effect evaluation model and the interference power consumed by the UAV swarm, the objective function of the interference resource allocation problem is set. Considering the limitations on UAV communication, energy, and flight speed under denial conditions, the interference resource allocation problem is modeled as a constrained combinatorial optimization problem. This achieves modeling of the target's motion and communication capabilities, and models the interference resource allocation problem under communication, navigation, and resource constraints as a constrained combinatorial optimization problem. To solve the above combinatorial optimization problem using a multi-agent reinforcement learning algorithm, the combinatorial... The perturbation decision-making process in the optimization problem is modeled as a locally observable Markov process, mapping the UAV to an agent. A pre-trained autoencoder (APSE) based on an attention mechanism is proposed as a pre-unit of the multi-agent reinforcement learning algorithm to perform deep feature extraction and dimensionality reduction on the input environment state, so as to reduce the impact of environmental non-stationarity and excessively large state space on the performance of the multi-agent reinforcement learning algorithm. The pre-training mechanism and self-attention mechanism introduced in APSE can significantly improve its feature extraction ability and training stability. The optimization problem of cooperative perturbation resource allocation is successfully solved by adopting the CTDE paradigm and combining it with the multi-agent reinforcement learning algorithm MAPPO. Compared with the original MAPPO and other algorithms, the MAPPO-APSE algorithm has significantly improved the efficiency of perturbation resource allocation and average reward, and has better decision performance in combinatorial optimization problems.

[0083] In one embodiment, an evaluation model for the interference effect of coordinated UAV swarm interference under denial conditions is established based on the interference power transmitted during UAV reconnaissance, the antenna gain of the target radio station, and the path loss between the UAV and the target radio station, including:

[0084] Based on the jamming power transmitted during UAV reconnaissance, the antenna gain of the target radio station, and the path loss between the UAV and the target radio station, an evaluation model for the jamming effect of UAV swarm cooperative jamming under denial conditions is established.

[0085]

[0086] Where N represents the number of drones. P represents the interference power of UAV i against target radio station j. jk The power of the communication signal received by the target radio station j. Let k be the transmit antenna gain of the target radio station. For the receiving antenna gain of target radio station j, PL ij The path loss N between target radio station j and UAV i noise The mean is 0 and the variance is σ. 2 Gaussian white noise.

[0087] In one embodiment, the interference resource allocation problem is modeled as a constrained combinatorial optimization problem, including:

[0088] The problem of interfering resource allocation is modeled as a constrained combinatorial optimization problem.

[0089]

[0090] in, Let N represent the interference power emitted by UAV i against target radio j, M represent the number of UAVs, λ1 and λ2 are weighting constants, and λ1,λ2∈[0,1], used to represent the relative importance between interference effectiveness and resource consumption, ω i The threat factor for the communication link. This is a resource allocation function, related to the signal-to-interference-plus-noise ratio (SIR) of the target radio station j; the higher the SIR, the better the interference effect. P represents the interference power of UAV i against target radio station k. max C represents the maximum interference power, ratio represents the maximum communicable percentage, and C represents the maximum interference power. i The number of other drones that can communicate with the i-th drone is represented by H, the successful interference threshold is represented by H, C1 indicates that the interference power of the drone is limited, C2 indicates that each drone can only interfere with one target at a time, i.e., the number of interference targets is limited, and C3 indicates that the global information exchange of the drone swarm is limited.

[0091] In one embodiment, the training process of the pre-trained attention encoder includes: acquiring the environmental state from the reconnaissance area of ​​the UAV swarm and placing it into a data cache pool; inputting the environmental state in the data cache pool into the autoencoder and calculating the attention score between each sub-state using an attention mechanism; then performing feature compression and dimensionality reduction through a residual network and normalization operations to obtain a low-dimensional representation of the environmental state; decoding the low-dimensional representation of the environmental state to output the reconstructed environmental state and establishing the optimization objective of the autoencoder; updating and training the optimization objective according to the Adam optimizer to obtain the pre-trained attention encoder.

[0092] In one embodiment, the environment state includes multiple sub-states; the environment state in the data cache pool is input into the autoencoder, and an attention mechanism is used to calculate the attention score between each sub-state. Then, feature compression and dimensionality reduction are performed through a residual network and normalization operations to obtain a low-dimensional representation of the environment state, including:

[0093] The environment state from the data cache pool is input into the autoencoder, and the attention score of the substates in the environment state is calculated using an attention mechanism.

[0094]

[0095] Where Q is the query vector of the substate of the environment state, K is the keyword of the substate of the environment state, V represents the important information or features of the substate of the environment state, and d k Let Q and K be the matrix dimensions;

[0096] Then, by utilizing feedforward neural networks, residual networks, and normalization operations, further feature extraction and dimensionality reduction are performed on the calculated attention scores to obtain a low-dimensional representation H of the environment state:

[0097] H=LayerNorm[Attention(Q,K,V)+FeedForward(Attention(Q,K,V))].

[0098] In one embodiment, the low-dimensional representation of the environment state is decoded to obtain the reconstructed environment state, and an optimization objective of the autoencoder is established, including:

[0099] The low-dimensional representation of the environment state is decoded to obtain the reconstructed environment state, and an autoencoder is established with the optimization objective as follows:

[0100]

[0101] Where S′ represents the reconstructed environment state, and S represents the output state of the environment.

[0102] In one embodiment, the discounted reward and advantage function for each UAV's allocation of interference resources under the updated environmental conditions are calculated, including:

[0103] The expression for calculating the discounted reward for each drone's allocation of interference resources under the updated environmental conditions is as follows:

[0104]

[0105] Among them, a t This represents the action chosen at time t based on the observed environmental state, i.e., the disturbance action. t Let R represent the observed environmental state at time t, γ represent the reward discount factor, and t represent time.

[0106] In one embodiment, the advantage function for each UAV to allocate interference resources under the updated environmental conditions is calculated as follows:

[0107]

[0108] Where λ∈[0,1] represents the parameters for weighted averaging of these n dominance functions. Environmental status Take action below Action value function, In environmental conditions The average value of the actions that can be taken, where γ represents the reward discount factor.

[0109] In one embodiment, the loss functions for the action network and the comment network are calculated based on the discount reward and the advantage function, as well as the loss function defined by the algorithm, including:

[0110] The loss function of the action network is calculated based on the advantage function and the loss function defined by the algorithm.

[0111]

[0112] Where B represents the batch size, and N is the number of drones, i.e., the number of agents in the comment network. This indicates importance sampling, representing the probability ratio between the old and new strategies. Let S represent the dominance function, ∈ represent the clipping constant, clip represent the clipping function, S represent the policy entropy, and σ represent the entropy coefficient. Represents intelligent agents The optimal strategy found Indicates the current strategy. This indicates the old strategy.

[0113] In one embodiment, the loss function of the review network is calculated based on the discount reward and the loss function defined by the algorithm.

[0114]

[0115] in, For the discount reward, B represents the batch size, and N represents the number of drones, i.e., the number of agents in the comment network. Indicates the environmental state The value function of the actions that can be taken. Let represent the old value function, and ε represent the clipping constant.

[0116] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this application, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for allocating interference resources in UAV swarms based on a pre-trained attention encoder, characterized in that, The method includes: The reconnaissance area of ​​the UAV swarm and the reconnaissance mission of the UAV swarm against multiple target radio stations are obtained; the UAV swarm includes multiple UAVs; an interference effect evaluation model of the cooperative interference of the UAV swarm under denial conditions is established based on the interference power transmitted by the UAVs during the reconnaissance, the antenna gain of the target radio stations, and the path loss between the UAVs and the target radio stations. Using the aforementioned interference effect evaluation model and the interference power consumed by the UAV swarm, the objective function of the interference resource allocation problem is set. Considering the limitations of UAV communication, energy, and flight speed under denial conditions, the interference resource allocation problem is modeled as a constrained combinatorial optimization problem. The interference decision-making process in the combinatorial optimization problem is modeled as a distributed locally observable Markov game model. UAVs are mapped as agents, and the MAPPO algorithm based on a pre-trained attention encoder is used to solve the combinatorial optimization problem. Features of the environmental states of the action network and comment network in the distributed locally observable Markov game model are extracted and compressed using the pre-trained attention encoder to obtain updated environmental states. Interference resources are allocated based on the updated environmental states, and the discounted reward and advantage function for each UAV in the updated environmental states are calculated. The loss functions of the action network and comment network are calculated based on the discounted reward, advantage function, and the loss function defined by the algorithm, and the Adam optimizer is used to update the action network and comment network to obtain updated action networks and comment networks. The solution to the combinatorial optimization problem is output based on the updated action network, resulting in an interference resource allocation scheme for the UAV swarm.

2. The method according to claim 1, characterized in that, Based on the interference power transmitted during the UAV reconnaissance process, the antenna gain of the target radio station, and the path loss between the UAV and the target radio station, an evaluation model for the interference effect of the UAV swarm cooperative interference under the denial condition is established, including: Based on the interference power transmitted during the UAV reconnaissance process, the antenna gain of the target radio station, and the path loss between the UAV and the target radio station, an evaluation model for the interference effect of the UAV swarm cooperative interference under the denial condition is established. Where N represents the number of drones. P represents the interference power of UAV i against target radio station j. jk The power of the communication signal received by the target radio station j. Let k be the transmit antenna gain of the target radio station. For the receiving antenna gain of target radio station j, PL ij The path loss N between target radio station j and UAV i noise The mean is 0 and the variance is σ. 2 Gaussian white noise, where the power unit is W.

3. The method according to claim 1, characterized in that, The problem of allocating disruptive resources is modeled as a constrained combinatorial optimization problem, including: The problem of interfering resource allocation is modeled as a constrained combinatorial optimization problem. in, Let represent the interference power transmitted by UAV i to target radio station j, N represent the number of UAVs, M represent the number of target radio stations, λ1 and λ2 are weighting constants, and λ1,λ2∈[0,1], used to represent the relative importance between interference effectiveness and resource consumption, ω i The threat factor for the communication link. This is a resource allocation function, related to the signal-to-interference-plus-noise ratio (SIR) of the target radio station j; the higher the SIR, the better the interference effect. P represents the interference power of UAV i against target radio station k. max C represents the maximum interference power, ratio represents the maximum communicable percentage, and C represents the maximum interference power. i The number of other drones that can communicate with the i-th drone is represented by H, the successful interference threshold is represented by H, C1 indicates that the interference power of the drone is limited, C2 indicates that each drone can only interfere with one target at a time, i.e., the number of interference targets is limited, and C3 indicates that the global information exchange of the drone swarm is limited.

4. The method according to any one of claims 1 to 3, characterized in that, The training process of the pre-trained attention encoder includes: The environmental state is obtained from the reconnaissance area of ​​the drone swarm and placed into a data cache pool. The environmental state in the data cache pool is input into an autoencoder, and the attention score between each sub-state is calculated using an attention mechanism. Then, feature compression and dimensionality reduction are performed through a residual network and normalization operation to obtain a low-dimensional representation of the environmental state. The low-dimensional representation of the environmental state is decoded and output to obtain the reconstructed environmental state, and an optimization objective of the autoencoder is established. The optimization objective is updated and trained according to the Adam optimizer to obtain a pre-trained attention encoder.

5. The method according to claim 4, characterized in that, The environment state includes multiple sub-states; the environment states in the data cache pool are input into the autoencoder, and an attention mechanism is used to calculate the attention scores between each sub-state. Then, feature compression and dimensionality reduction are performed through a residual network and normalization operations to obtain a low-dimensional representation of the environment state, including: The environment state in the data cache pool is input into the autoencoder, and the attention score of the sub-states in the environment state is calculated using an attention mechanism. Where Q is the query vector of the substate of the environment state, K is the keyword of the substate of the environment state, V represents the important information or features of the substate of the environment state, and d k Let Q and K be the matrix dimensions; Then, by utilizing feedforward neural networks, residual networks, and normalization operations, further feature extraction and dimensionality reduction are performed on the calculated attention scores to obtain a low-dimensional representation H of the environment state: H=LayerNorm[Attention(Q,K,V)+FeedForward(Attention(Q,K,V))].

6. The method according to claim 4, characterized in that, The low-dimensional representation of the environmental state is decoded and output to obtain the reconstructed environmental state, and an optimization objective of the autoencoder is established, including: The low-dimensional representation of the environment state is decoded to obtain the reconstructed environment state, and an autoencoder is established with the following optimization objective: Where S′ represents the reconstructed environment state, and S represents the output state of the environment.

7. The method according to claim 1, characterized in that, Calculate the discounted reward and advantage function for each UAV's allocation of interference resources under the updated environmental conditions, including: The expression for calculating the discounted reward for each drone's allocation of interference resources under the updated environmental conditions is as follows: Among them, a t This represents the action chosen at time t based on the observed environmental state, i.e., the disturbance action. t Let R represent the observed environmental state at time t, γ represent the reward discount factor, and t represent time.

8. The method according to claim 7, characterized in that, The method further includes: The advantage function for each UAV to allocate interference resources under the updated environmental conditions is calculated as follows: Where λ∈[0,1] represents the parameters for weighted averaging of these n dominance functions. Environmental status Take action below Action value function, In environmental conditions The average value of the actions that can be taken, where γ represents the reward discount factor.

9. The method according to claim 1, characterized in that, The loss functions of the action network and the comment network are calculated based on the discount reward and advantage function and the loss function defined by the algorithm, including: The loss function of the action network is calculated based on the advantage function and the loss function defined by the algorithm. Where B represents the batch size, and N is the number of drones, i.e., the number of agents in the comment network. This indicates importance sampling, representing the probability ratio between the old and new strategies. Let S represent the dominance function, ∈ represent the clipping constant, clip represent the clipping function, S represent the policy entropy, and σ represent the entropy coefficient. Represents intelligent agents The optimal strategy found Indicates the current strategy. This indicates the old strategy.

10. The method according to claim 9, characterized in that, The method further includes: The loss function of the comment network is calculated based on the discount reward and the loss function defined by the algorithm. in, For the discount reward, B represents the batch size, and N represents the number of drones, i.e., the number of agents in the comment network. Indicates the environmental state The value function of the actions that can be taken. Let represent the old value function, and ε represent the clipping constant.

Images