Unmanned aerial vehicle cluster anti-interference communication method based on meta reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster anti-interference communication method based on meta reinforcement learning, and the method comprises the steps: constructing an unmanned aerial vehicle cluster anti-interference communication environment, initializing network parameters, and generating a plurality of meta reinforcement learning tasks; sampling tasks from a meta-training set, and initializing an environment, an experience pool and a context sampling pool; each unmanned aerial vehicle inputs observation data interacting with the environment to an encoder for context task reasoning and a long-short-term memory network module to obtain context task information and historical memory, the context task information and historical memory are spliced with current observation of an intelligent agent to serve as input of a double-depth recursive Q network, namely DDRQN, actions are predicted, and interaction with the environment is achieved; finally, judging whether the environment reaches the maximum training step length or not, if so, ending the current round, and entering a meta-test stage; otherwise, repeating the next round of training. The method has the advantages of high expandability, high generalization performance, high learning efficiency, high stability, low transmission energy loss, low frequency hopping overhead and long network service life.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle (UAV) intelligent communication, and in particular relates to an anti-interference communication method for a UAV cluster based on meta-reinforcement learning. Background Art

[0002] As a new technology, drones have attracted much attention due to their high reliability, flexibility and low cost of use. However, in a complex electromagnetic environment, drone cluster network communications face two major challenges: malicious interference signals and scarce spectrum resources. Malicious interference signals will destroy the propagation of radio waves, while limited spectrum resources make the communication links between different clusters susceptible to co-frequency interference. Traditional anti-interference communication schemes cannot adapt to the dynamically changing interference patterns in complex electromagnetic environments. In addition, in the face of limited battery capacity, power control needs to be effectively implemented to extend the life of the network. Therefore, how to build a long-life drone cluster network with strong anti-interference capabilities and high transmission reliability has become one of the key issues that need to be urgently solved in the current drone cluster field.

[0003] Based on the concept of cognitive radio, dynamic spectrum anti-jamming communication technology can be used to improve the performance of UAV swarm networks. For example, the wireless system applies Q learning [Wang X, Jin T, Hu L, et al. Energy-efficient power allocation and Q-learning-based relay selection for relay-aided D2D communication [J]. IEEE Transactions on Vehicular Technology, 2020, 69 (6): 6452-6462.] to select relay nodes to transmit data to adjacent radio devices according to network topology and channel conditions. The system also uses the Lagrangian dual decomposition method to allocate transmission power to improve energy efficiency and anti-jamming capabilities. In addition, another solution worthy of attention is a UAV swarm communication method based on multi-agent reinforcement learning [Lv Z, Niu G, Xiao L, et al. Reinforcement learning based UAV swarm communications against jamming [C]. ICC 2023-IEEE International Conference on Communications. IEEE, 2023: 5204-5209.], which optimizes UAV relay selection and power allocation to enhance communication performance and save energy consumption. Another study proposed a dual deep Q-learning method for UAV cluster anti-jamming in the joint spectrum domain and power domain [Wu Z, Lin Y, Zhang Y, et al. Multi-domain energy-saving and anti-jamming communication of UAV cluster based on multi-agent cooperation [J]. Science China: Information Science (in Chinese), 2023 (12): 2511-2526.], which can more effectively reduce long-term transmission energy loss and frequency hopping overhead compared to the baseline algorithm, and at the same time improve the data transmission success rate. However, the dynamics of the environment and tasks bring new challenges to decision-making. These traditional reinforcement learning methods require retraining the model for each new task and cannot fully cope with changes in the environment and tasks.

[0004] Meta-reinforcement learning, as an important branch of reinforcement learning research in recent years, has received increasing attention. Meta-reinforcement learning aims to train a model that can quickly learn and adapt to new tasks with only a small amount of interaction. Traditional reinforcement learning requires relearning from scratch for each new task. For example, the model-agnostic meta-learning (MAML) proposed by Finn et al. [Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks [C]. International Conference on Machine Learning. PMLR, 2017: 1126-1135.] is a meta-learning method based on optimizing meta-objectives. It trains a set of shared meta-parameters θ across tasks, so that the model initialized for a new task can converge quickly with only a small amount of data. Duan et al. proposed an RL 2 Algorithm[Duan Y,Schulman J,Chen X,et al.RL 2 :Fastreinforcement learning via slow reinforcement learning[J].arXiv preprintarXiv:1611.02779,2016.], the algorithm uses recurrent neural networks to build a policy model and trains across multiple tasks, passing hidden states between iterations of the same task and parameters between different tasks, with the goal of using memory to optimize cross-task capabilities. Rakelly et al. proposed the Probabilistic Embeddings for Actor-critic meta-RL (PEARL) algorithm [Rakelly K, Zhou A, Finn C, et al. Efficient off-policy meta-reinforcement learning via probabilistic context variables[C]. International conference on machine learning. PMLR, 2019: 5331-5340.], which takes task history information as input and requires the agent to infer the characteristics of the current task, thereby mapping partially observable Markov decision process tasks to Markov decision processes to construct strategies.

[0005] In summary, the challenges and complexities faced by UAV communication technology are obvious. Although some reinforcement learning methods have been proposed and applied in this field, there are still many challenges in dealing with large-scale, dynamically changing environments and complex interference patterns. Meta-reinforcement learning embodies the potential and advantages in dealing with these problems. It is a more advanced learning method that attempts to learn how to quickly adapt to new tasks in order to achieve the greatest learning effect with a small amount of data. This makes meta-reinforcement learning have a unique advantage in dealing with interference problems in UAV communication technology. Summary of the invention

[0006] The purpose of the present invention is to provide an anti-interference communication method for UAV clusters that can solve new tasks from a small amount of experience, has strong transferability, high sample efficiency, strong anti-interference performance, low transmission energy loss, and low frequency hopping overhead.

[0007] The technical solution to realize the present invention is: a UAV cluster anti-interference communication method based on meta-reinforcement learning, comprising the following steps:

[0008] Step 1: Build an anti-interference communication environment for drone clusters;

[0009] Step 2: Initialize network parameters and generate multiple meta-reinforcement learning tasks;

[0010] Step 3: Sample tasks from the meta-training set and initialize the environment, experience pool, and context sampling pool;

[0011] Step 4: Each drone inputs the observation data of the interaction with the environment into the encoder and long short-term memory network module for contextual task reasoning, samples the contextual task information from the distribution of the encoder output, and uses the hidden state of the long short-term memory network as the historical memory of the current task;

[0012] Step 5: Concatenate the contextual task information and historical memory with the agent’s current observation as the input of DDRQN to predict actions and interact with the environment.

[0013] Step 6: Determine whether the environment has reached the maximum training step length. If so, end the current round, reset the multi-agent unmanned cluster anti-interference environment, initialize the experience pool and sampling pool, and enter the meta-test phase; otherwise, repeat step 3 to start the next round of training.

[0014] A UAV cluster anti-interference communication device based on meta-reinforcement learning, the device is used to implement the UAV cluster anti-interference communication method based on meta-reinforcement learning, the device includes:

[0015] Environment construction module, building an anti-interference communication environment for drone clusters;

[0016] Initialization module, which initializes network parameters and generates multiple meta-reinforcement learning tasks;

[0017] The task sampling module samples tasks from the meta-training set and initializes the environment, experience pool, and context sampling pool;

[0018] The observation data processing module inputs the observation data of each drone’s interaction with the environment into the encoder and long short-term memory network module for contextual task reasoning, samples the contextual task information from the distribution of the encoder output, and uses the hidden state of the long short-term memory network as the historical memory of the current task;

[0019] The splicing module splices the contextual task information and historical memory with the agent's current observation as the input of DDRQN, predicts actions and interacts with the environment;

[0020] The judgment module determines whether the environment has reached the maximum training step length. If so, the current round ends, the multi-agent unmanned cluster anti-interference environment is reset, the experience pool and sampling pool are initialized, and the meta-test phase begins; otherwise, the task sampling module is repeated to start the next round of training.

[0021] A mobile terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the unmanned aerial vehicle cluster anti-interference communication method based on meta-reinforcement learning is implemented.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the unmanned aerial vehicle cluster anti-interference communication method based on meta-reinforcement learning.

[0023] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention is highly modular and has strong scalability. It can be integrated into the existing algorithm as an independent module without large-scale modification or reconstruction of the original system; (2) It adopts a meta-learning mechanism with stronger generalization performance. When faced with new interference strategies, the model can converge in a shorter time, significantly improving the learning efficiency, and can obtain better convergence results, thereby improving the overall performance and stability; (3) It reduces transmission energy loss, reduces frequency hopping overhead, reduces communication power, and extends the working life of the entire network. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a UAV cluster anti-interference communication method based on meta-reinforcement learning in the present invention.

[0025] Figure 2 It is a schematic diagram of the convergence effect of the accumulated round rewards in the meta-training stage in an embodiment of the present invention.

[0026] Figure 3 It is a schematic diagram of the convergence effect of the accumulated round rewards in the meta-test phase in an embodiment of the present invention.

[0027] Figure 4 It is a schematic diagram of the convergence effect of the accumulated energy loss in the meta-test phase in an embodiment of the present invention.

[0028] Figure 5 It is a schematic diagram of the convergence effect of the accumulated frequency hopping overhead in the meta-test phase in an embodiment of the present invention.

[0029] Figure 6 It is a schematic diagram of the convergence effect of the accumulated communication success reward in the meta-test phase in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0031] Combination Figure 1 The present invention provides a UAV cluster anti-interference communication method based on meta-reinforcement learning, comprising the following steps:

[0032] Step 1: Build an anti-interference communication environment for drone clusters;

[0033] Step 2: Initialize network parameters and generate multiple meta-reinforcement learning tasks;

[0034] Step 3: Sample tasks from the meta-training set and initialize the environment, experience pool, and context sampling pool;

[0035] Step 4: Each drone inputs the observation data of the interaction with the environment into the encoder and long short-term memory network module for contextual task reasoning, samples the contextual task information from the distribution of the encoder output, and uses the hidden state of the long short-term memory network as the historical memory of the current task;

[0036] Step 5: Concatenate the contextual task information and historical memory with the agent’s current observation as the input of DDRQN to predict actions and interact with the environment.

[0037] Step 6: Determine whether the environment has reached the maximum training step length. If so, end the current round, reset the multi-agent unmanned cluster anti-interference environment, initialize the experience pool and sampling pool, and enter the meta-test phase; otherwise, repeat step 3 to start the next round of training.

[0038] As a specific example, in step 1, an anti-interference communication environment for a drone cluster is constructed, including a network model, an interference model, and a communication model, as follows:

[0039] (1) Network model: It includes N jammers and M drone clusters, where each drone cluster includes a cluster head and I cluster members. The jammer and cluster head move according to the Markov random movement model; the cluster members move according to the reference point group movement model, that is, the reference point of the cluster member moves within a circular area with a radius of R1 centered on the cluster head drone, and the cluster members move within a circular area with a radius of R2 centered on their own reference points; the sum of the moving radius of each reference point and the moving radius of the cluster members does not exceed the maximum communication distance D max , that is, satisfying R1+R2≤D max ;

[0040] (2) Jamming model: The jammer uses a Markov jamming pattern that follows the Markov property during channel switching. Transfer to channel The probability of pr t Initialize to a uniform distribution of [0,1], where L is the total number of channels; set:

[0041] Operation A: With equal probability Randomly select a channel from set up From the channel Transfer to channel The probability plus pr add =Sum pr / div, where div is the scaling factor;

[0042] Operation B: Continue to generate random numbers between [0,1]. If the random number is greater than the set threshold pr continue , then execute operation A once until the generated random number is no greater than pr continue until;

[0043] Operation C: pr t Normalization;

[0044] Generate training tasks using the combination of operations A, B, and C to obtain diverse meta-reinforcement learning tasks;

[0045] (3) Communication model: Consider dividing time into T time slots, the duration of each time slot is dt, and the power gain is composed of slow fading and fast fading. represents the fast fading gain on channel l in time slot t, is a standard normal distribution, then the fast fading gain between the cluster head UAV m and the cluster member UAV i is It is expressed as:

[0046]

[0047] Slow fading gain between cluster head UAV m and cluster member UAV i It is expressed as:

[0048]

[0049] Where A is the path loss constant at the reference distance of 1m, ζ is the attenuation index of the path loss component, and dis(i,m) represents the distance between cluster member UAV i and cluster head UAV m;

[0050] At time slot t, the channel power gain of cluster head UAV m and cluster member UAV i communicating on the lth channel is It is expressed as:

[0051]

[0052] Define the link gain information between jammer i and drone j at time slot t as The transmission power is The jammer channel is selected as Then at time slot t, the interference of channel l between cluster head drone m and cluster member drone j is It is expressed as:

[0053]

[0054] The first term of the above formula represents the inter-cluster interference between cluster q and cluster m on the same channel l, where represents the link gain of cluster q and drone j on channel l at time slot t, Then it represents the power of cluster q on the i-th channel; similarly, the second term of the above formula represents the interference of jammer n on drone j, and the third term represents the interference of drones k and j in the cluster;

[0055] Setting σ 2 is the power of Gaussian additive white noise, B is the bandwidth of each channel, then the data transmission rate between cluster head drone m and cluster member drone i on channel l in time slot t is for:

[0056]

[0057] As a specific example, in step 2, the network parameters are initialized and multiple meta-reinforcement learning tasks are generated as follows:

[0058] The optimization problem is modeled as a partially observable Markov decision process, where the observed value of the cluster head UAV m at time slot t is Defined as:

[0059]

[0060] in represents the channel power gain of the cluster head UAV m and all members in its cluster on all communication channels in time slot t;

[0061] The action of cluster head drone m in time slot t It is expressed as:

[0062]

[0063] in, Indicates the selection of the transmission channel, Indicates the transmit power;

[0064] Cluster head drone m Select actions based on Strategy It is expressed as:

[0065]

[0066] in, Indicates that agent m observes Next select action probability;

[0067] Assume that the transmission energy loss of agent m in time slot t is for:

[0068]

[0069] in, represents the actual transmission time between cluster head UAV m and its cluster member i on channel l at time slot t;

[0070] Assume that the frequency hopping cost of agent m in time slot t is for:

[0071]

[0072] Where W is the unit frequency hopping cost, is the communication channel between cluster head UAV m and its cluster member i at time slot t, is the communication channel between cluster head UAV m and its cluster member i at time slot t-1;

[0073] Since the jammer is in the data transmission stage T Rx The interference channel can be switched within 1 second, so the transmission time T Rx Divided into two time periods T1 and T2, satisfying T1+T2=T Rx ;

[0074] Assuming the size of the data packet transmitted in each time slot is K, then in time slot t, the actual transmission time between the cluster head drone m and its cluster member i on channel l is:

[0075]

[0076] in, They represent the data transmission rates between cluster head UAV m and its cluster member i on channel l in time periods T1 and T2 in time slot t respectively;

[0077] Setting auxiliary variables Then the cluster head drone m performs the action Local rewards received for:

[0078]

[0079] Where δ is the reward for successful data transmission, and β is the transmission energy loss The proportion of 1-β is the frequency hopping overhead The proportion of

[0080] The operations ABC of the interference model are used to generate training tasks and obtain diverse meta-reinforcement learning tasks.

[0081] As a specific example, in step 3, tasks are sampled from the meta-training set to initialize the environment, experience pool, and context sampling pool as follows:

[0082] Transfer one of the tasks Γ Deposit into experience pool and context sampling pool The experience pool and context sampling pool of each cluster head drone m are independent of each other.

[0083] As a specific example, in step 4, each drone inputs the observation data of the interaction with the environment into the encoder and the long short-term memory network module for contextual task reasoning, samples the contextual task information from the distribution of the encoder output, and the hidden state of the long short-term memory network is used as the historical memory of the current task, as follows:

[0084] Step 4.1: Add a new contextual meta-reinforcement learning framework based on the DDRQN algorithm and sample multiple sets of data from the context sampler As the input of the contextual task reasoning encoder, the hidden state h t-1 and the current observation value of the environment o t As the input of the LSTM network, the vector z obtained from the context task inference encoder and the output h of the LSTM network are t , the current observation value of the environment is connected as the input of DDRQN;

[0085] Step 4.2: Use the inference network q φ (z|c Γ ) estimates the posterior distribution p(z|c Γ ), q is converted to Φ (z|c 1:N ) is modeled as a series of distributions Ψ φ (z|c n ) is the product of:

[0086]

[0087] in It is a neural network with parameter φ and Estimate the initial mean μ and variance σ, and then randomly sample from a normal distribution;

[0088] Step 4.3: Use the long short-term memory network to process the sequence information. In the forget gate, f t =σ(W f [h t-1 ,o t ]+b f ), where f t is the output at time t, h t-1 is the hidden state at time t-1, o t is the state at time t, W f is the parameter of the forget gate, b f For bias.

[0089] As a specific example, step 5 concatenates the contextual task information and historical memory with the agent’s current observation as the input of DDRQN to predict actions and interact with the environment as follows:

[0090] Let the cumulative return G t =r t +γr t+1 +.....+γ k r t+k , where γ is the discount factor; in the DDRQN evaluation network, the action value function Q π (o t ,a t ;w t )=E[G t |o t ,a t ;w t ] is used to evaluate the network parameters w t The current observation o t and action a t Q evalValue, the target network action value function of DDRQN Used to evaluate Q at the next moment target value;

[0091] Define the loss function in and is the maximum Q value of the action under the current observation, evaluating the network Q π The parameter w is optimized using the Adam algorithm. The parameters are every N steps iteration From Q π Parameter copying.

[0092] Step 6: Determine whether the environment has reached the maximum training step length. If so, end the current round, reset the multi-agent unmanned cluster anti-interference environment, initialize the experience pool and sampling pool, enter the meta-test phase, and use the pre-trained model parameters to test on the meta-test set. Fine-tune the model parameters during the test process to test its ability to quickly adapt to the new environment and the final convergence effect; otherwise, repeat step 3 to start the next round of training.

[0093] The present invention also provides a UAV cluster anti-interference communication device based on meta-reinforcement learning, which is used to implement the UAV cluster anti-interference communication method based on meta-reinforcement learning. The device includes:

[0094] Environment construction module, building an anti-interference communication environment for drone clusters;

[0095] Initialization module, which initializes network parameters and generates multiple meta-reinforcement learning tasks;

[0096] The task sampling module samples tasks from the meta-training set and initializes the environment, experience pool, and context sampling pool;

[0097] The observation data processing module inputs the observation data of each drone’s interaction with the environment into the encoder and long short-term memory network module for contextual task reasoning, samples the contextual task information from the distribution of the encoder output, and uses the hidden state of the long short-term memory network as the historical memory of the current task;

[0098] The splicing module splices the contextual task information and historical memory with the agent's current observation as the input of DDRQN, predicts actions and interacts with the environment;

[0099] The judgment module determines whether the environment has reached the maximum training step length. If so, the current round ends, the multi-agent unmanned cluster anti-interference environment is reset, the experience pool and sampling pool are initialized, and the meta-test phase begins; otherwise, the task sampling module is repeated to start the next round of training.

[0100] The present invention also provides a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the unmanned aerial vehicle cluster anti-interference communication method based on meta-reinforcement learning is implemented.

[0101] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps in the unmanned aerial vehicle cluster anti-interference communication method based on meta-reinforcement learning are implemented.

[0102] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0103] Example

[0104] This embodiment is described in detail as follows, using Python simulation. The parameter setting does not affect the generality. The methods compared with the method are: (1) context-aware DDRQN meta-learning anti-interference algorithm; (2) DDRQN anti-interference algorithm; (3) meta-learning anti-interference algorithm that performs random actions.

[0105] The training and testing tasks are described in the interference model. A total of 300 tasks are generated according to the abcd operations. The probability of selecting each operation is the same: (a) Execute operation C. (b) Execute operation A, then execute operation C, where div = 1. (c) Execute operation B, then execute operation C, where div = 2, pr continue =0.7. (d) Perform operation A, then perform operation B, and finally perform operation C, where div A =2,div B = 3. The first 200 tasks are used for training, and the last 100 tasks are used for testing.

[0106] In the meta-training phase, all drones are restricted to a length of 500m, a width of 250m, and a height limit of 60m to 120m. The movement trajectory of the members in the cluster follows the reference point group movement model. The maximum step size of the training is set to 200, and the network parameters are updated every 10 steps. In the meta-testing phase, the maximum step size is set to 100, a smaller batch size of batch=16 is used, and a longer parameter update interval of 40 is used. Other simulation parameters are listed in Table 1.

[0107] like Figure 2As shown in the figure, in the meta-training stage, the meta-reinforcement learning scheme can robustly enhance the model performance, and often achieve better results when the model reaches convergence. This is because the encoder and long short-term memory network based on contextual task reasoning have excellent task inference ability and memory retention ability. The encoder can accurately capture the commonalities and differences between different tasks through an in-depth understanding of the task context, thereby providing strong support for subsequent strategy selection. LSTM, with its powerful memory function, effectively remembers the experience and knowledge accumulated in the historical interaction process, so that the model can quickly respond correctly when faced with similar situations.

[0108] Table 1 Simulation parameters

[0109]

[0110] like Figure 3 As shown in the figure, in the meta-testing stage, tasks are randomly sampled from the test set, and the performance of the trained model is tested on new tasks. The convergence speed of the context-aware DDRQN meta-learning anti-interference algorithm is significantly better than other methods. This is because meta-reinforcement learning enables the model to learn to adapt to new tasks more quickly by learning on multiple tasks. This generalization ability enables the model to perform better when facing new tasks, and the model learns the common features and patterns between tasks instead of just memorizing the data of specific tasks. This learning method makes the model more generalizable and better able to adapt to various tasks.

[0111] like Figure 4-5 As shown in the figure, the model of the context-aware DDRQN meta-learning anti-interference algorithm has a higher communication success reward, uses less power for communication while ensuring the communication quality, and extends the overall life of the entire network. In addition, the algorithm using meta-reinforcement learning can significantly reduce the frequency hopping overhead and achieve a higher communication success rate at the same frequency hopping overhead.

[0112] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A UAV cluster anti-interference communication method based on meta-reinforcement learning, characterized in that: The steps include: Step 1: Build an anti-interference communication environment for drone clusters; Step 2: Initialize network parameters and generate multiple meta-reinforcement learning tasks; Step 3: Sample tasks from the meta-training set and initialize the environment, experience pool, and context sampling pool; Step 4: Each drone inputs the observation data of the interaction with the environment into the encoder and long short-term memory network module for contextual task reasoning, samples the contextual task information from the distribution of the encoder output, and uses the hidden state of the long short-term memory network as the historical memory of the current task; Step 5: Concatenate the contextual task information and historical memory with the agent’s current observation as the input of DDRQN to predict actions and interact with the environment. Step 6: Determine whether the environment has reached the maximum training step length. If so, end the current round, reset the multi-agent unmanned cluster anti-interference environment, initialize the experience pool and sampling pool, and enter the meta-test phase; otherwise, repeat step 3 to start the next round of training.

2. The UAV cluster anti-interference communication method based on meta-reinforcement learning according to claim 1 is characterized in that: The construction of the anti-interference communication environment of the drone cluster described in step 1 includes a network model, an interference model, and a communication model, which are as follows: (1) Network model: It includes N jammers and M drone clusters, where each drone cluster includes a cluster head and I cluster members. The jammer and cluster head move according to the Markov random movement model; the cluster members move according to the reference point group movement model, that is, the reference point of the cluster member moves within a circular area with a radius of R1 centered on the cluster head drone, and the cluster members move within a circular area with a radius of R2 centered on their own reference points; the sum of the moving radius of each reference point and the moving radius of the cluster members does not exceed the maximum communication distance D max , that is, satisfying R1+R2≤D max ; (2) Jamming model: The jammer uses a Markov jamming pattern that follows the Markov property during channel switching. Transfer to channel The probability of pr t Initialized to a uniform distribution of [0,1], where L is the total number of channels; set up: Operation A: With equal probability Randomly select a channel from set up From the channel Transfer to channel The probability plus pr add =Sum pr / div, where div is the scaling factor; Operation B: Continue to generate random numbers between [0,1]. If the random number is greater than the set threshold pr continue , then execute operation A once until the generated random number is no greater than pr continue until; Operation C: pr t Normalization; Generate training tasks using the combination of operations A, B, and C to obtain diverse meta-reinforcement learning tasks; (3) Communication model: Consider dividing time into T time slots, the duration of each time slot is dt, and the power gain is composed of slow fading and fast fading. represents the fast fading gain on channel l in time slot t, is a standard normal distribution, then the fast fading gain between the cluster head UAV m and the cluster member UAV i is It is expressed as: Slow fading gain between cluster head UAV m and cluster member UAV i It is expressed as: Where A is the path loss constant at the reference distance of 1m, ζ is the attenuation index of the path loss component, and dis(i,m) represents the distance between cluster member UAV i and cluster head UAV m; At time slot t, the channel power gain of cluster head UAV m and cluster member UAV i communicating on the lth channel is It is expressed as: Define the link gain information between jammer i and drone j at time slot t as The transmission power is The jammer channel is selected as Then at time slot t, the interference of channel l between cluster head drone m and cluster member drone j is It is expressed as: The first term of the above formula represents the inter-cluster interference between cluster q and cluster m on the same channel l, where represents the link gain of cluster q and drone j on channel l at time slot t, Then it represents the power of cluster q on the i-th channel; similarly, the second term of the above formula represents the interference of jammer n on drone j, and the third term represents the interference of drones k and j in the cluster; Setting σ 2 is the power of Gaussian additive white noise, B is the bandwidth of each channel, then the data transmission rate between cluster head drone m and cluster member drone i on channel l in time slot t is for:

3. The UAV cluster anti-interference communication method based on meta-reinforcement learning according to claim 2 is characterized in that: Initialize the network parameters described in step 2 to generate multiple meta-reinforcement learning tasks, as follows: The optimization problem is modeled as a partially observable Markov decision process, where the observed value of the cluster head UAV m at time slot t is Defined as: in represents the channel power gain of the cluster head UAV m and all members in its cluster on all communication channels in time slot t; The action of cluster head drone m in time slot t It is expressed as: in, Indicates the selection of the transmission channel, Indicates the transmit power; Cluster head drone m Select actions based on Strategy It is expressed as: in, Indicates that agent m observes Next select action The probability of Assume that the transmission energy loss of agent m in time slot t is for: in, represents the actual transmission time between cluster head UAV m and its cluster member i on channel l at time slot t; Assume that the frequency hopping cost of agent m in time slot t is for: Where W is the unit frequency hopping cost, is the communication channel between cluster head UAV m and its cluster member i at time slot t, is the communication channel between cluster head UAV m and its cluster member i at time slot t-1; Since the jammer is in the data transmission stage T Rx The interference channel can be switched within 1 second, so the transmission time T Rx Divided into two time periods T1 and T2, satisfying T1+T2=T Rx ; Assuming the size of the data packet transmitted in each time slot is K, then in time slot t, the actual transmission time between the cluster head drone m and its cluster member i on channel l is: in, They represent the data transmission rates between cluster head UAV m and its cluster member i on channel l in time periods T1 and T2 in time slot t respectively; Setting auxiliary variables Then the cluster head drone m performs the action Local rewards received for: Where δ is the reward for successful data transmission, and β is the transmission energy loss The proportion of 1-β is the frequency hopping overhead The proportion of The operations ABC of the interference model are used to generate training tasks and obtain diverse meta-reinforcement learning tasks.

4. The UAV cluster anti-interference communication method based on meta-reinforcement learning according to claim 3 is characterized in that: The sampling task from the meta-training set described in step 3 initializes the environment, experience pool, and context sampling pool as follows: Transfer one of the tasks Γ Deposit into experience pool and context sampling pool The experience pool and context sampling pool of each cluster head drone m are independent of each other.

5. The UAV cluster anti-interference communication method based on meta-reinforcement learning according to claim 1 is characterized in that: In step 4, each drone inputs the observation data of the interaction with the environment into the encoder and long short-term memory network module for contextual task reasoning, samples the contextual task information from the distribution of the encoder output, and the hidden state of the long short-term memory network is used as the historical memory of the current task, as follows: Step 4.1: Add a new contextual meta-reinforcement learning framework based on the DDRQN algorithm and sample multiple sets of data from the context sampler As the input of the contextual task reasoning encoder, the hidden state h t-1 and the current observation value of the environment o t As the input of the LSTM network, the vector z obtained from the context task inference encoder and the output h of the LSTM network are t , the current observation value of the environment is connected as the input of DDRQN; Step 4.2: Use the inference network q φ (z|c Γ ) estimates the posterior distribution p(z|c Γ ), q is converted to Φ (z|c 1:N ) is modeled as a series of distributions Ψ φ (z|c n ) is the product of: in It is a neural network with parameter φ and Estimate the initial mean μ and variance σ, and then randomly sample from a normal distribution; Step 4.3: Use the long short-term memory network to process the sequence information. In the forget gate, f t =σ(W f [h t-1 ,o t ]+b f ), where f t is the output at time t, h t-1 is the hidden state at time t-1, o t is the state at time t, W f is the parameter of the forget gate, b f For bias.

6. The UAV cluster anti-interference communication method based on meta-reinforcement learning according to claim 1 is characterized in that: Step 5 describes the concatenation of contextual task information and historical memory with the agent’s current observation as the input to DDRQN to predict actions and interact with the environment, as follows: Let the cumulative return G t =r t +γr t+1 +.....+γ k r t+k , where γ is the discount factor; in the DDRQN evaluation network, the action value function Q π (o t ,a t ;w t )=E[G t |o t ,a t ;w t ] is used to evaluate the network parameters w t The current observation o t and action a t Q eval Value, the target network action value function of DDRQN Used to evaluate Q at the next moment target value; Defining the loss function in and is the maximum Q value of the action under the current observation, evaluating the network Q π The parameter w is optimized using the Adam algorithm. The parameters are every N steps iteration From Q π Parameter copying.

7. The UAV cluster anti-interference communication method based on meta-reinforcement learning according to claim 1 is characterized in that: The meta-testing phase described in step 6 is as follows: Use the pre-trained model parameters to test on the meta-test set, fine-tune the model parameters during the test process, and test the ability to quickly adapt to the new environment and the final convergence effect.

8. A UAV cluster anti-interference communication device based on meta-reinforcement learning, characterized in that: The device is used to implement the UAV cluster anti-interference communication method based on meta-reinforcement learning as described in any one of claims 1 to 7, and the device includes: Environment construction module, building an anti-interference communication environment for drone clusters; Initialization module, which initializes network parameters and generates multiple meta-reinforcement learning tasks; The task sampling module samples tasks from the meta-training set and initializes the environment, experience pool, and context sampling pool; The observation data processing module inputs the observation data of each drone’s interaction with the environment into the encoder and long short-term memory network module for contextual task reasoning, samples the contextual task information from the distribution of the encoder output, and uses the hidden state of the long short-term memory network as the historical memory of the current task; The splicing module splices the contextual task information and historical memory with the agent's current observation as the input of DDRQN, predicts actions and interacts with the environment; The judgment module determines whether the environment has reached the maximum training step length. If so, the current round ends, the multi-agent unmanned cluster anti-interference environment is reset, the experience pool and sampling pool are initialized, and the meta-test phase begins; otherwise, the task sampling module is repeated to start the next round of training.

9. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the unmanned aerial vehicle cluster anti-interference communication method based on meta-reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the steps in the UAV cluster anti-interference communication method based on meta-reinforcement learning as described in any one of claims 1 to 7 are implemented.

Images