Multi-agent and multi-domain anti-interference method for UAV swarm based on partially observable information

Abstract

The present invention discloses a multi-agent multi-domain anti-interference method for drone clusters based on partially observable information. The method utilizes partial observed environmental information of each agent, retains historical experience data through a long short-term memory network, inputs the deep recurrent Q network of each agent to perform action value function fitting, adopts the ε-greedy algorithm to select the channel and power corresponding to the maximum output Q value, and then continuously and independently trains the deep recurrent Q network of each agent, updates the Q value distribution, and finally learns the optimal decision of drone channels and transmission power that can adapt to minimize communication transmission energy consumption in unknown interference scenarios. The present invention uses historical experience data of partially observable information to achieve effective multi-agent anti-interference communication from the spectrum domain and power domain in two scenarios where the drone cluster network is in sweeping interference and Markov interference respectively; compared with the comparison scheme based on multi-agent deep Q learning, the proposed scheme can more efficiently reduce the long-term communication transmission energy consumption of the drone cluster network when the environmental information is partially observable.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, and in particular to a multi-agent multi-domain anti-interference method for a drone cluster based on partially observable information. Background Art

[0002] In recent years, with the rapid development of radio technology, many advantages of drone communication systems have been continuously highlighted. Drones have been widely used in emergency networks to alleviate terminal requirements in communication systems. Anti-interference technology for drone cluster networks is an important technology to protect drone communications from interference threats. Among them, frequency hopping anti-interference is one of the most common anti-interference technologies. Since traditional frequency hopping anti-interference technology cannot cope with problems such as unknown, highly dynamic and complex interference environments, frequency hopping anti-interference technology based on reinforcement learning has become a research hotspot for frequency hopping anti-interference technology in drone communication networks in recent years.

[0003] Most previous studies have used the Q-Learning (QL) algorithm, but it is only applicable to low-dimensional discrete action spaces. When the action space is large, it will face the problem of dimensionality curse. To address the above problems, Shangxing Wang et al. proposed a channel selection algorithm based on deep Q-network (DQN) online learning, which effectively improved the anti-interference performance of UAV communication networks in complex environments. Fuqiang Yao and Luliang Jia used the Markov Game Framework to establish a multi-agent Markov decision process (MDP) model for the UAV cluster communication system, reducing the communication overhead when applied to actual communication environments. However, the above anti-interference communication technology does not take into account the problem of partial observability of the communication environment. Summary of the invention

[0004] The present invention aims to provide a multi-agent and multi-domain anti-interference method for UAV clusters based on partially observable information. By using the Deep Recurrent Q-Network (DRQN) algorithm, the cluster head UAV, on the basis of establishing a Dec-POMDP model, retains historical information data by using a long short-term memory network to train DRQN, thereby achieving a convergence to the real environment model.

[0005] The technical solution to achieve the purpose of the present invention is: a multi-agent multi-domain anti-interference method for drone clusters based on partially observable information, the specific steps are:

[0006] Step 1: Initialize the deep recurrent Q learning algorithm parameters;

[0007] Step 2: Each cluster head UAV obtains the observation value of the current time slot by interacting with the environment, including the channel and transmission power selected by the member UAVs in its cluster in the previous time slot;

[0008] Step 3: Each cluster head UAV uses the ε-greedy algorithm to select the communication channel and transmission power of the current time slot for its member UAVs in the cluster according to the output value of the value network and the ε value of the current time slot;

[0009] Step 4: Each cluster head UAV calculates the communication speed with the member UAVs in the cluster according to the interference situation, obtains the total communication time of the current time slot, and thus calculates the total energy cost required for the communication process with the member UAVs in its cluster, and obtains the corresponding environmental reward value;

[0010] Step 5: After the environment jumps to the next state, each cluster head UAV stores the observation value of the current time slot, the joint action of the member UAVs in the cluster, the environmental reward and the observation value of the next time slot into its own experience pool;

[0011] Step 6: When the sample data in the experience pool is sufficient, each cluster head UAV randomly samples from its own experience pool to obtain several batches of historical information data, which are organized into trajectory sequences in chronological order. Several consecutive steps are extracted from the trajectory sequences, which are input into the value network of each cluster head UAV, compared with the calculated true Q value, and the gradient descent method is used to update the value network parameters;

[0012] Step 7: Every certain number of time slots, copy the parameters of the value network to form a new target network;

[0013] Step 8: Repeat steps 2 to 7 until 100 data transmissions are completed;

[0014] Step 9: Repeat step 8 until the total reward value of the drone cluster network converges and local training is completed.

[0015] Compared with the prior art, the present invention has the following significant advantages: (1) It proposes a multi-agent multi-domain anti-interference framework that can be applied to partially observable environments. By minimizing the long-term communication transmission energy consumption of the UAV cluster network as the goal, the multi-domain anti-interference decision-making process is modeled as a multi-agent partially observable Markov process, and the observation value, action, reward of the cluster head UAV in the current time slot and the observation value of the next time slot are used as historical experience to assist each UAV cluster agent to complete its own channel selection and transmission power allocation; (2) It proposes a multi-domain anti-interference algorithm based on a multi-agent deep recurrent Q network. By using a long short-term memory network to retain historical information data, it is then input into the deep recurrent Q network of each agent for action value function fitting, and the parameters of each deep recurrent Q network are updated, and finally the optimal decision of the UAV channel and transmission power that can adapt to the unknown interference scenario and minimize the communication transmission energy consumption is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the multi-agent and multi-domain anti-interference method of UAV cluster based on partially observable information of the present invention.

[0017] Figure 2 Schematic diagram of the learning convergence effect of different algorithms in the swept frequency interference mode.

[0018] Figure 3 Schematic diagram of the learning convergence effect of different algorithms under the Markov interference model.

[0019] Figure 4 This is a graph showing the relationship between the convergence value of the environment reward of different algorithms and the number of channels in the swept frequency interference mode.

[0020] Figure 5 This is a graph showing the relationship between the convergence value of the environment reward of different algorithms and the number of channels under the Markov interference mode.

[0021] Figure 6 This is a graph showing the relationship between the convergence value of the environment reward of different algorithms and the number of jammers in the sweeping frequency jamming mode.

[0022] Figure 7 This is a graph showing the relationship between the convergence value of the environment reward of different algorithms and the number of jammers under the Markov jamming mode. DETAILED DESCRIPTION

[0023] The multi-agent multi-domain anti-interference method of the drone cluster based on partial observable information of the present invention comprises the following specific steps:

[0024] Step 1: Initialize the deep recurrent Q learning algorithm parameters;

[0025] Step 2: Each cluster head UAV obtains the observation value of the current time slot by interacting with the environment, including the channel and transmission power selected by the member UAVs in its cluster in the previous time slot;

[0026] Step 3: Each cluster head UAV uses the ε-greedy algorithm to select the communication channel and transmission power of the current time slot for its member UAVs in the cluster according to the output value of the value network and the ε value of the current time slot;

[0027] Step 4: Each cluster head UAV calculates the communication speed with the member UAVs in the cluster according to the interference situation, obtains the total communication time of the current time slot, and thus calculates the total energy cost required for the communication process with the member UAVs in its cluster, and obtains the corresponding environmental reward value;

[0028] Step 5: After the environment jumps to the next state, each cluster head UAV stores the observation value of the current time slot, the joint action of the member UAVs in the cluster, the environmental reward and the observation value of the next time slot into its own experience pool;

[0029] Step 6: When the sample data in the experience pool is sufficient, each cluster head UAV randomly samples from its own experience pool to obtain several batches of historical information data, which are organized into trajectory sequences in chronological order. Several consecutive steps are extracted from the trajectory sequences, which are input into the value network of each cluster head UAV, compared with the calculated true Q value, and the gradient descent method is used to update the value network parameters;

[0030] Step 7: Every certain number of time slots, copy the parameters of the value network to form a new target network;

[0031] Step 8: Repeat steps 2 to 7 until 100 data transmissions are completed;

[0032] Step 9: Repeat step 8 until the total reward value of the drone cluster network converges and local training is completed.

[0033] Furthermore, the parameters of the deep recurrent Q learning algorithm in step 1 include learning rate δ, greed factor ε, discount factor γ, experience pool size μ, decay factor θ, value network parameter w, and target network parameter w'.

[0034] Furthermore, in step 2, each cluster head UAV obtains the observation value of the current time slot by interacting with the environment, including the channel and transmission power selected by the member UAVs in its cluster in the previous time slot, as follows:

[0035] The UAV anti-interference decision-making problem is modeled as a decentralized partially observable Markov decision process (Decentralized Partially Observable Markov Decision Process, Dec-POMDP).

[0036] The system model is modeled as Dec-POMDP<D,S,A,O,R> , where D is a set of multiple agents, S is a set of joint states, A is a set of joint actions, O is a set of joint observations, and R is a reward function; define D = {1, ..., N} as a set of N agents; define the current observation of the cluster head drone n at time slot t + 1 as: Joint Observation Set in is the channel of cluster member i of cluster head drone n in time slot t+1, is the transmission power selected by cluster head UAV n for its cluster member i in time slot t+1; the action of cluster head UAV n in time slot t is defined as Joint Observation Set in It is the cluster member i of cluster head drone n that hops to channel is the transmission power selected by cluster head UAV n for its cluster member i in time slot t Define the joint state set S as the total environment state information, and the joint observation set O as the partial information that N agents can observe. Therefore, the joint observation set O can be regarded as a subset of the joint state set S. Define is the reward value of cluster head UAV n in time slot t.

[0037] Furthermore, in step 3, each cluster head UAV uses the ε-greedy algorithm to select the communication channel and transmission power of the current time slot for its member UAVs in the cluster according to the output value of the value network and the ε value of the current time slot, as follows:

[0038] Step 3-1: The observation value of each cluster head drone is used as the input of its value network, and the Q value corresponding to each action is used as the output of its value network. In time slot t, cluster head drone n observes Next action The Q value of is the expectation of the cumulative future reward value starting from time slot t+1, as follows:

[0039]

[0040] Among them, s t is the environmental status information at time slot t, Take action for cluster head drone n in time slot t Environmental status t Transfer to s t+1 probability.

[0041] Step 3-2: Select actions according to the ε-greedy algorithm, as follows:

[0042]

[0043] Among them, p is a random number between 0 and 1, ε (0 < ε < 1) is the exploration probability, is the hidden layer state of the neural network of the cluster head drone n at time slot t, and w is the value network parameter. In this network, the output is not only related to the input, but also to the hidden layer state of time slot t. related, Used to store the past network status of cluster head drone n, including historical information. Hidden layer status At the beginning of the round, it is 0, which means it does not contain any historical information. As the round progresses, It will be updated iteratively, and the time slot t generated by the network It will be used as the hidden layer state of time slot t+1, thereby affecting the output of the value network of time slot t+1 and iterating step by step.

[0044] This strategy randomly selects an action in the action space with a probability of ε to avoid falling into the local optimum. ε is the exploration probability, and 1-ε is the utilization (selection of the current optimal strategy) probability. The larger the value of ε, the smaller the utilization probability. In the initial stage of the algorithm execution, due to the large state action space, the exploration probability should take a larger value. As the number of iterations increases, it gradually approaches the optimal strategy, and the utilization probability should increase accordingly.

[0045] Furthermore, in step 4, each cluster head UAV calculates the communication speed with the member UAVs in the cluster according to the interference situation, and obtains the total communication time of the current time slot, thereby calculating the total energy cost required for the communication process with the member UAVs in its cluster, and obtaining the corresponding environmental reward value, as follows:

[0046] remember and are the transmission powers of cluster member i and jammer j of cluster head UAV n in time slot t, is the transmission power of cluster member k of cluster head UAV m in time slot t (when m=n, k≠i), G U and G J are the antenna gains of the UAV and the jammer, is the Euclidean distance between cluster head UAV n and its cluster member i or between cluster head UAV n and jammer j in time slot t, ρ is the UAV noise coefficient, σ 2 The mean square value of environmental noise, is the fast fading between cluster head UAV n and its cluster member i or between cluster head UAV n and jammer j in time slot t, B is the channel bandwidth, T is the time required for a single communication transmission, s is the data size of a single communication transmission, is the maximum average information rate of error-free transmission between cluster head UAV n and its cluster member i in time slot t in the additive Gaussian white noise channel; the gain of the Rician fading channel is modeled by the real part with a mean of 0 and a variance of ξ 2 , the imaginary part is modeled as having a mean of 0 and a variance of ξ 2 Independent and identically distributed Gaussian random process, so the fast fading channel is recorded as a is the real part, b is the imaginary part; the energy cost of cluster head drone n is set to

[0047]

[0048] Among them, when the cluster member i of cluster head UAV n is in the same channel with the jammer j, β = 1, otherwise β = 0; when the cluster member i of cluster head UAV n is in the same channel with the cluster member k of cluster head UAV m, α = 1, otherwise α = 0. The total reward value of the environment in time slot t is

[0049]

[0050] The actual physical meaning of energy consumption is the energy consumed by the cluster head UAV n for a data transmission with all cluster member UAVs.

[0051] Furthermore, after the environment jumps to the next state in step 5, each cluster head UAV stores the observation value of the current time slot, the joint action of the member UAVs in the cluster, the environmental reward and the observation value of the next time slot into its own experience pool, as follows:

[0052] When cluster head UAV n is in time slot t according to After selecting the frequency hopping channel and transmission power of the cluster member drones, the environmental state is given by s t Jump to s t+1 , calculated by the reward value calculation formula in s t Next select action Rewards and observation The current time slot t Historical experience data is saved in the experience pool.

[0053] Furthermore, when the sample data in the experience pool is sufficient, each cluster head UAV randomly samples from its own experience pool to obtain several batches of historical information data, which are composed into trajectory sequences in chronological order. Several consecutive steps are extracted from them, and the trajectory sequences are input into the value network of each cluster head UAV. The values ​​are compared with the calculated true Q values, and the gradient descent method is used to update the value network parameters, as follows:

[0054] The neural network input of cluster head drone n is the observation value of time slot t The output is the Q value corresponding to each action in time slot t. In order to enhance the algorithm stability, the present invention adopts a dual network structure, where w is the value network parameter and w' is the target network parameter. In step 7, the target network parameter w' is updated every certain round.

[0055] Step 6-1: When each agent is training the value network, first randomly select a batch of historical experience data from the experience pool to form several time series, each of which is a complete communication round, and then randomly select a time slot in each sequence, and select several consecutive steps as training samples. In the time slot t of the sample, the action Q value function of the cluster head drone n in time slot t is calculated through the value network As the estimated Q value, the target network calculates the action Q value function of cluster head drone n at time slot t+1 in, is the observation value, action and hidden layer state of the cluster head drone n in time slot t. The true value of the action Q value function is calculated using the following formula:

[0056]

[0057] Step 6-2: Substitute the actual Q value and the estimated Q value into the following formula to calculate and update the value network parameter w, gradually reducing Right now

[0058]

[0059] The gradient descent method makes the Q value calculated by the value network closer to the real Q value. Before each agent trains the neural network, the hidden layer state needs to be reset to zero. Generated by network iteration.

[0060] Furthermore, in step 7, the parameters of the value network are copied every certain number of time slots to form a new target network, i.e., w←w'.

[0061] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Example

[0063] In this embodiment, the cluster head UAV and the jammer complete 100 movements and channel and transmission power selections as one round, that is, complete one communication task. In the round, each movement, channel selection and transmission power made by the cluster head UAV and the jammer is called a time slot.

[0064] Combination Figure 1 In this embodiment, the multi-agent multi-domain anti-interference method of a drone cluster based on partial observable information has the following specific steps:

[0065] Step 1: Initialize the deep recurrent Q-learning algorithm parameters.

[0066] The algorithm parameters include learning rate δ, greed factor ε, discount factor γ, experience pool size μ, decay factor θ, value network parameter w, and target network parameter w'.

[0067] Step 2: Each cluster head drone obtains the observation value of the current time slot by interacting with the environment, including the channel and transmission power selected by the member drones in its cluster in the previous time slot. The specific steps are as follows:

[0068] The UAV anti-interference decision-making problem is modeled as a decentralized partially observable Markov decision process (Decentralized Partially Observable Markov Decision Process, Dec-POMDP).

[0069] The system model is modeled as Dec-POMDP<D,S,A,O,R> , where D is a set of multiple agents, S is a set of joint states, A is a set of joint actions, O is a set of joint observations, and R is a reward function; define D = {1, ..., N} as a set of N agents; define the current observation of the cluster head drone n at time slot t + 1 as: Joint Observation Set in is the channel of cluster member i of cluster head drone n in time slot t+1, is the transmission power selected by cluster head UAV n for its cluster member i in time slot t+1; the action of cluster head UAV n in time slot t is defined as Joint Observation Set in It is the cluster member i of cluster head drone n in time slot t that hops to channel is the transmission power selected by cluster head UAV n for its cluster member i in time slot t Define the joint state set S as the total environment state information, and the joint observation set O as the partial information that N agents can observe. Therefore, the joint observation set O can be regarded as a subset of the joint state set S. Define is the reward value of cluster head UAV n in time slot t.

[0070] Step 3: Each cluster head drone uses the ε-greedy algorithm to select the communication channel and transmission power of the current time slot for its member drones in the cluster according to the output value of the value network and the ε value of the current time slot. The specific steps are as follows:

[0071] Step 3-1: The observation value of each cluster head drone is used as the input of its value network, and the Q value corresponding to each action is used as the output of its value network. Next action The Q value of is the expectation of the cumulative future reward value starting from time slot t+1, as follows:

[0072]

[0073] Among them, s t is the environmental status information at time slot t, Take action for cluster head drone n in time slot t Environmental status t Transfer to s t+1 probability.

[0074] Step 3-2: Select actions according to the ε-greedy algorithm, as follows:

[0075]

[0076] Among them, p is a random number between 0 and 1, ε (0<ε<1) is the exploration probability, is the hidden layer state of the neural network of the cluster head drone n at time slot t, and w is the value network parameter. In this network, the output is not only related to the input, but also to the hidden layer state of time slot t. related, Used to store the past network status of cluster head drone n, including historical information. Hidden layer status At the beginning of the round, it is 0, which means it does not contain any historical information. As the round progresses, It will be updated iteratively, and the time slot t generated by the network It will be used as the hidden layer state of time slot t+1, thereby affecting the output of the value network of time slot t+1 and iterating step by step.

[0077] This strategy randomly selects an action in the action space with a probability of ε to avoid falling into the local optimum. ε is the exploration probability, and 1-ε is the utilization (selection of the current optimal strategy) probability. The larger the value of ε, the smaller the probability of utilization. In the initial stage of the algorithm execution, due to the large state action space, the exploration probability should take a larger value. As the number of iterations increases, it gradually approaches the optimal strategy, and the utilization probability should increase accordingly. The probability ε is updated as follows:

[0078] ε=max{0.01,θ x}

[0079] Where x is the current round number.

[0080] Step 4: Each cluster head drone calculates the communication speed with the member drones in the cluster according to the interference situation, obtains the total communication time of the current time slot, and thus calculates the total energy cost required for the communication process with the member drones in its cluster, and obtains the corresponding environmental reward value. The specific steps are as follows:

[0081] remember and are the transmission powers of cluster member i and jammer j of cluster head UAV n in time slot t, is the transmission power of cluster member k of cluster head UAV m in time slot t (when m=n, k≠i), G U and G J are the antenna gains of the UAV and the jammer, is the Euclidean distance between cluster head UAV n and its cluster member i or between cluster head UAV n and jammer j in time slot t, ρ is the UAV noise coefficient, σ 2 The mean square value of environmental noise, is the fast fading between cluster head UAV n and its cluster member i or between cluster head UAV n and jammer j in time slot t, B is the channel bandwidth, T is the time required for a single communication transmission, s is the data size of a single communication transmission, is the maximum average information rate of error-free transmission between cluster head UAV n and its cluster member i in time slot t in the additive Gaussian white noise channel; the gain of the Rician fading channel is modeled by the real part with a mean of 0 and a variance of ξ 2 , the imaginary part is modeled as having a mean of 0 and a variance of ξ 2 Independent and identically distributed Gaussian random process, so the fast fading channel is recorded as a is the real part, b is the imaginary part; the energy cost of cluster head drone n is set to

[0082]

[0083] Among them, when the cluster member i of cluster head UAV n is in the same channel with the jammer j, β = 1, otherwise β = 0; when the cluster member i of cluster head UAV n is in the same channel with the cluster member k of cluster head UAV m, α = 1, otherwise α = 0. The total reward value of the environment in time slot t is

[0084]

[0085] The actual physical meaning of energy consumption is the energy consumed by the cluster head UAV n for a data transmission with all cluster member UAVs.

[0086] Step 5: After the environment jumps to the next state, each cluster head drone stores the observation value of the current time slot, the joint action of the member drones in the cluster, the environmental reward and the observation value of the next time slot into its own experience pool. The specific steps are as follows:

[0087] When cluster head UAV n is in time slot t according to After selecting the frequency hopping channel and transmission power of the cluster member drones, the environmental state is determined by s t Jump to s t+1 , calculated by the reward value calculation formula in s t Next select action Rewards and observation The current time slot t Historical experience data is saved in the experience pool.

[0088] Step 6: When the sample data in the experience pool is sufficient, each cluster head drone randomly samples from its own experience pool to obtain several batches of historical information data, which are organized into trajectory sequences in chronological order. Several consecutive steps are extracted from them, and the trajectory sequences are input into the value network of each cluster head drone, compared with the calculated true Q value, and the gradient descent method is used to update the value network parameters. The specific steps are as follows:

[0089] The neural network of the cluster head drone n consists of three neural units, the first neural unit is the Long Short-Term Memory (LSTM) unit. The LSTM structure is a special recurrent neural network structure that can use historical information to predict and process sequence data. LSTM consists of a forget gate, an input gate, and an output gate. The control parameters in the forget gate determine the historical information that needs to be discarded, the input gate determines the new information that is added, and the output gate determines the data output from this LSTM unit to the next unit.

[0090] Forget Gate:

[0091]

[0092] Input Gate:

[0093]

[0094] Output Gate:

[0095]

[0096] Among them, W i,f,c,o and b i,f,c,o The gate's input weights and biases, is the input of the LSTM unit at time slot t.

[0097] The LSTM structure uses three gates to determine the degree of retention of the input data sequence, which can predict the future through historical information. In the anti-interference scenario of the present invention, each agent only exchanges reward information, so it is impossible to determine the action information of other agents. The LSTM structure uses the experience of historical information to help each agent predict the actions of other agents, and can obtain a better anti-interference strategy for drone cluster network communication.

[0098] The neural network input of cluster head drone n is the observation value of time slot t The output is the Q value corresponding to each action in time slot t. In order to enhance the algorithm stability, the present invention adopts a dual network structure, where w is the value network parameter and w' is the target network parameter. In step 7, the target network parameter w' is updated every certain round.

[0099] Step 6-1: When each agent is training the value network, first randomly select a batch of historical experience data from the experience pool to form several time series, each of which is a complete communication round, and then randomly select a time slot in each sequence, and select several consecutive steps as training samples. In the time slot t of the sample, the action Q value function of the cluster head drone n in time slot t is calculated through the value network As the estimated Q value, the target network calculates the action Q value function of cluster head drone n at time slot t+1 in, is the observation value, action and hidden layer state of the cluster head drone n in time slot t. The true value of the action Q value function is calculated using the following formula:

[0100]

[0101] Step 6-2: Substitute the actual Q value and the estimated Q value into the following formula to calculate and update the value network parameter w, gradually reducing Right now

[0102]

[0103] The gradient descent method makes the Q value calculated by the value network closer to the real Q value. Before each agent trains the neural network, the hidden layer state needs to be reset to zero. Generated by network iteration.

[0104] The gradient descent process uses the Adaptive Moment Estimation (ADAM) method. During the value network parameter update process, only a batch of historical experience data is sampled for training in each iteration. Different data sets will result in different loss functions. Using the ADAM method can reduce the probability of converging to the local optimum. ADAM dynamically adjusts the learning rate for each parameter based on the first-order moment estimate and second-order moment estimate of the gradient of each parameter of the loss function. ADAM is based on the gradient descent method, but the learning step size of each iteration parameter has a certain range. A larger gradient will not lead to a larger learning step size, and the parameter value is relatively stable. The specific steps for implementing the ADAM algorithm are as follows:

[0105] Assume that at time slot t, the first-order derivative of the objective function with respect to the parameter is g t , first calculate the exponential moving mean:

[0106] m t =λ1m t-1 +(1-λ1)g t-1

[0107] Two more bias correction terms are calculated:

[0108]

[0109] The final gradient update method is:

[0110]

[0111] Finally, the result parameters related to the error function are returned In the algorithm t is the exponential moving mean, ω t is the square gradient, and the parameters λ1 and λ2 are the exponential decay rates that control these moving means, η is the learning step size, and τ is a constant, generally 10 -8 .

[0112] Step 7: Every certain number of time slots, copy the parameters of the value network to form a new target network;

[0113] Step 8: Repeat steps 2 to 7 until 100 data transmissions are completed;

[0114] Step 9: Repeat step 8 until the total reward value of the drone cluster network converges and local training is completed.

[0115] The present invention uses Python to implement the method. Set the total number of channels C = 4, the number of drones N = 9, the number of jammers J = 1, and the transmission power selected by the cluster head drone n for its cluster member i and the transmission power selected by the jammer j in the time slot t are respectively (The transmission power of cluster member drone i can be 27, 30, 33, 36dBm, and the transmission power of jammer j is usually 33dBm). The design of drone cluster requires cluster head drone n and cluster member drone i to complete q communications to complete a communication task. The anti-interference system under two different interference modes is simulated. Take the learning rate δ = 0.002, the discount factor γ = 0.99, the attenuation factor θ = 0.998, the experience pool size μ = 200, and the mean square value of the environmental noise σ 2 =-114dBm; the Rician fading channel gain is modeled with the real part as the mean value ξ 2 , the imaginary part is modeled as having a mean of 0 and a variance of ξ 2 Independent and identically distributed Gaussian random process, so the fast fading channel is recorded as a is the real part and b is the imaginary part.

[0116] The learning convergence effects in the sweep frequency and Markov interference modes are as follows: Figure 2-3 As shown. The frequency sweep is set to interfere with 1 channel at the same time, with a sweep step of 1MHz. A total of 4 interference states are set in the Markov interference mode. Each interference mode is randomly generated at the beginning of the simulation. The transition of any time slot interference mode follows the following state transfer matrix:

[0117]

[0118] Figure 2 and Figure 3The following are the convergence of reward values ​​in the channel and power random selection scheme, the DQN-based channel and power selection scheme, and the DRQN-based channel and power selection scheme under two interference modes: sweeping interference and Markov interference. As can be seen from the figure, the DRQN-based scheme has a higher convergence reward value than the DQN scheme, and the convergence result is more stable. This is because there is a long short-term memory network in DRQN. Each agent can obtain hidden information such as the action change rules of other agents and the environment change rules based on historical experience. The network output is not only determined by its own observations; while the output of DQN is completely determined by its own observations. Once the environment or other agents' decision-making rules change, it will cause fluctuations in the entire network. Comparison Figure 2 and Figure 3 The convergence of reward values ​​of the three channel and power selection schemes is roughly the same. Under Markov interference conditions, the performance of the DRQN-based scheme is 34.6% higher than that of the DQN-based scheme, and 54.5% higher than that of the random scheme; under sweeping frequency interference conditions, the performance of the DRQN-based scheme is 38.4% higher than that of the DQN-based scheme, and 56% higher than that of the random scheme.

[0119] Figure 4 and Figure 5 The relationship between the average reward convergence value and the number of channels of the three channel and power selection schemes under two interference modes, namely, sweeping interference and Markov interference. When the number of channels increases, the average reward convergence values ​​of the three schemes are improved. This is because when the number of channels increases, the occurrence of co-channel interference decreases, which reduces the energy cost of UAV communication. The average reward convergence value of the DRQN-based scheme changes less than that of other schemes, indicating that it is not sensitive to this environmental condition.

[0120] Figure 6 and Figure 7 The relationship between the reward convergence value and the number of jammers for the three channel and power selection schemes under the two interference modes of sweep frequency interference and Markov interference. As can be seen from the figure, under the two interference modes, as the number of jammers increases and the environment continues to deteriorate, the average reward convergence values ​​under the three schemes have a downward trend, but the average reward convergence value based on the DRQN ​​scheme is more stable, and the decline does not exceed 10%. Therefore, the DRQN ​​algorithm has good robustness.

Claims

1. The multi-agent multi-domain anti-interference method of UAV swarm based on partially observable information is characterized by In the following specific steps: Step 1: Initialize the deep recurrent Q learning algorithm parameters; Step 2: Each cluster head UAV obtains the observation value of the current time slot by interacting with the environment, including the channel and transmission power selected by the member UAVs in its cluster in the previous time slot; Step 3: Each cluster head UAV uses the ε-greedy algorithm to select the communication channel and transmission power of the current time slot for its member UAVs in the cluster according to the output value of the value network and the ε value of the current time slot; Step 4: Each cluster head UAV calculates the communication speed with the member UAVs in the cluster according to the interference situation, obtains the total communication time of the current time slot, and thus calculates the total energy cost required for the communication process with the member UAVs in its cluster, and obtains the corresponding environmental reward value; Step 5: After the environment jumps to the next state, each cluster head UAV stores the observation value of the current time slot, the joint action of the member UAVs in the cluster, the environmental reward and the observation value of the next time slot into its own experience pool; Step 6: When the sample data in the experience pool is sufficient, each cluster head UAV randomly samples from its own experience pool to obtain several batches of historical information data, which are organized into trajectory sequences in chronological order. Several consecutive steps are extracted from the trajectory sequences, which are input into the value network of each cluster head UAV, compared with the calculated true Q value, and the gradient descent method is used to update the value network parameters; Step 7: Every certain number of time slots, copy the parameters of the value network to form a new target network; Step 8: Repeat steps 2 to 7 until 100 data transmissions are completed; Step 9: Repeat step 8 until the total reward value of the drone cluster network converges and local training is completed.

2. The multi-agent multi-domain anti-interference method of UAV swarm based on partially observable information according to claim 1 is characterized in that: The parameters of the deep recurrent Q learning algorithm in step 1 include the learning rate δ, the greedy factor ε, the discount factor γ, the experience pool size μ, the decay factor θ, the value network parameter w, and the target network parameter w'.

3. The multi-agent multi-domain anti-interference method of UAV swarm based on partially observable information according to claim 1 is characterized in that: In step 2, each cluster head UAV obtains the observation value of the current time slot by interacting with the environment, including the channel and transmission power selected by the member UAVs in its cluster in the previous time slot, as follows: The UAV anti-interference decision-making problem is modeled as a decentralized partially observable Markov decision process (Decentralized Partially Observable Markov Decision Process, Dec-POMDP); The system model is modeled as Dec-POMDP<D,S,A,O,R> , where D is a set of multiple agents, S is a set of joint states, A is a set of joint actions, O is a set of joint observations, and R is a reward function; define D = {1, ..., N} as a set of N agents; define the current observation of the cluster head drone n at time slot t + 1 as: Joint Observation Set in is the channel of cluster member i of cluster head drone n in time slot t+1, is the transmission power selected by cluster head UAV n for its cluster member i in time slot t+1; the action of cluster head UAV n in time slot t is defined as Joint Observation Set in It is the cluster member i of cluster head drone n in time slot t that hops to channel is the transmission power selected by cluster head UAV n for its cluster member i in time slot t Define the joint state set S as the total environment state information, and the joint observation set O as the partial information that N agents can observe. Therefore, the joint observation set O can be regarded as a subset of the joint state set S. Define is the reward value of cluster head UAV n in time slot t.

4. The multi-agent multi-domain anti-interference method of UAV swarm based on partially observable information according to claim 1 is characterized in that: In step 3, each cluster head drone uses the ε-greedy algorithm to select the communication channel and transmission power of the current time slot for its member drones in the cluster according to the output value of the value network and the ε value of the current time slot, as follows: Step 3-1: The observation value of each cluster head drone is used as the input of its value network, and the Q value corresponding to each action is used as the output of its value network. Next action The Q value of is the expected cumulative future reward value starting from time slot t+1, as follows: Among them, s t is the environmental status information at time slot t, Take action for cluster head drone n in time slot t Environmental status t Transfer to s t+1 The probability of Step 3-2: Select actions according to the ε-greedy algorithm, as follows: Among them, p is a random number between 0 and 1, ε (0 < ε < 1) is the exploration probability, is the hidden layer state of the neural network of the cluster head drone n in time slot t, and w is the value network parameter; in this network, the output is not only related to the input, but also to the hidden layer state of time slot t related, Used to store the past network status of cluster head drone n, including historical information; hidden layer status At the beginning of the round, it is 0, that is, it does not contain any historical information; as the round progresses, It will be updated iteratively, and the time slot t generated by the network It will be used as the hidden layer state of time slot t+1, thereby affecting the output of the value network of time slot t+1, and iterating step by step; This strategy randomly selects an action in the action space with a probability of ε to avoid falling into the local optimum; ε is the exploration probability, and 1-ε is the utilization (selection of the current optimal strategy) probability; the larger the value of ε, the smaller the utilization probability; in the initial stage of the algorithm execution, due to the large state-action space, the exploration probability should take a larger value, and as the number of iterations increases, it gradually approaches the optimal strategy, and the utilization probability should increase accordingly.

5. The multi-agent multi-domain anti-interference method of UAV swarm based on partially observable information according to claim 1 is characterized in that: In step 4, each cluster head UAV calculates the communication speed with the member UAVs in the cluster according to the interference situation, and obtains the total communication time of the current time slot. From this, the total energy cost required for the communication process with the member UAVs in the cluster is calculated, and the corresponding environmental reward value is obtained, as follows: remember and are the transmission powers of cluster member i and jammer j of cluster head UAV n in time slot t, is the transmission power of cluster member k of cluster head UAV m in time slot t (when m=n, k≠i), G U and G J are the antenna gains of the UAV and the jammer, is the Euclidean distance between cluster head UAV n and its cluster member i or between cluster head UAV n and jammer j in time slot t, ρ is the UAV noise coefficient, σ 2 The mean square value of environmental noise, is the fast fading between cluster head UAV n and its cluster member i or between cluster head UAV n and jammer j in time slot t, B is the channel bandwidth, T is the time required for a single communication transmission, s is the data size of a single communication transmission, is the maximum average information rate of error-free transmission between cluster head UAV n and its cluster member i in time slot t in the additive Gaussian white noise channel; the real part of the Rician fading channel gain is modeled as 0 with a variance of ξ 2 , the imaginary part is modeled as having a mean of 0 and a variance of ξ 2 Independent and identically distributed Gaussian random process, so the fast fading channel is recorded as a is the real part, b is the imaginary part; the energy cost of cluster head drone n is set to Among them, when the cluster member i of cluster head UAV n is in the same channel with the jammer j, β = 1, otherwise β = 0; when the cluster member i of cluster head UAV n is in the same channel with the cluster member k of cluster head UAV m, α = 1, otherwise α = 0; the total reward value of the environment in time slot t is The actual physical meaning of energy consumption is the energy consumed by the cluster head UAV n for a data transmission with all cluster member UAVs.

6. The multi-agent multi-domain anti-interference method of UAV swarm based on partially observable information according to claim 1 is characterized in that: After the environment jumps to the next state in step 5, the observation value of each cluster head drone in the current time slot, the joint action of the member drones in the cluster, the environmental reward and the observation value of the next time slot are stored in their respective experience pools, as follows: When cluster head UAV n is in time slot t according to After selecting the frequency hopping channel and transmission power of the cluster member drones, the environmental state is given by s t Jump to s t+1 , calculated by the reward value calculation formula in s t Next select action Rewards and observation The current time slot t Historical experience data is saved in the experience pool.

7. The multi-agent multi-domain anti-interference method of UAV swarm based on partially observable information according to claim 1 is characterized in that: In step 6, when the sample data in the experience pool is sufficient, each agent randomly samples from its own experience pool to obtain several batches of historical information data, which are organized into trajectory sequences in chronological order. Several consecutive steps are extracted from them, and the trajectory sequences are input into the value network of each agent. They are compared with the calculated true Q value, and the gradient descent method is used to update the value network parameters, as follows: The neural network input of cluster head drone n is the observation value of time slot t The output is the Q value corresponding to each action in time slot t; in order to enhance the stability of the algorithm, the present invention adopts a dual network structure, w is denoted as the value network parameter, w' is denoted as the target network parameter, and the target network parameter w' is updated once every certain round in step 7; Step 6-1: When each agent is training the value network, a batch of historical experience data is randomly selected from the experience pool to form several time series, each of which is a complete communication round. Then, a time slot is randomly selected in each sequence, and several consecutive steps are selected as training samples. In the time slot t of the sample, the action Q value function of the cluster head drone n in time slot t is calculated through the value network. As the estimated Q value, the target network calculates the action Q value function of cluster head drone n at time slot t+1 in, is the observation value, action and hidden layer state of the cluster head UAV n in time slot t; the true value of the action Q value function is calculated using the following formula: Step 6-2: Substitute the actual Q value and the estimated Q value into the following formula to calculate and update the value network parameter w, gradually reducing Right now The gradient descent method makes the Q value calculated by the value network closer to the real Q value; before each agent trains the neural network each time, the hidden layer state needs to be set to zero, and the hidden layer state of the subsequent steps Generated by network iteration.

8. The multi-agent multi-domain anti-interference method of UAV swarm based on partially observable information according to claim 1 is characterized in that: In step 7, the parameters of the value network are copied every certain number of time slots to form a new target network, i.e., w←w'.

Images