Unmanned aerial vehicle cluster anti-jamming communication decision method based on deep reinforcement learning

Abstract

The application discloses a kind of unmanned aerial vehicle cluster anti-jamming communication decision-making methods based on deep reinforcement learning, belong to unmanned aerial vehicle communication technical field.The present application is to solve the problem of existing unmanned aerial vehicle communication in decision model static, insufficient adaptability to complex and changeable electromagnetic interference environment.It includes: each cluster head obtains current communication environment information and constitutes observation state;The observation state is input into the deep reinforcement learning network, and the decision result of channel and power is output;Each cluster head configures communication parameters according to the decision result to carry out data transmission;The reward value is calculated according to the communication feedback and the network parameters are updated;Repeat the above steps until the preset training round number is reached;In the application, channel and power are jointly encoded as a single discrete action, single unmanned aerial vehicle scene uses Dueling DDQN network, multi-unmanned aerial vehicle cluster scene uses ER-QMIX centralized training-distributed execution architecture, and multi-agent credit allocation is realized through hybrid network.The application improves the autonomous decision-making ability of unmanned aerial vehicle cluster in dynamic interference environment.

Description

Technical Field

[0001] This invention relates to a deep reinforcement learning-based anti-jamming communication decision-making method for UAV swarms, belonging to the field of UAV communication technology. Background Technology

[0002] With the rapid development of the low-altitude economy, drone swarms are increasingly widely used in logistics delivery, emergency rescue, and environmental monitoring. Drone swarm communication systems typically consist of multiple drone clusters, each containing a cluster head and several cluster members. The cluster head is responsible for making communication decisions for all its members. Drone swarm communication faces the following technical challenges: First, the electromagnetic environment is complex and variable, making communication links susceptible to various dynamic interferences such as full-band noise, periodic frequency sweeping, and random frequency hopping; second, spectrum resources are limited, and multiple drones sharing frequency bands leads to severe co-channel interference; third, drone onboard battery capacity is limited, and while high-power transmission can improve anti-interference capabilities, it significantly shortens flight time.

[0003] Existing anti-jamming communication decision-making methods mainly include traditional frequency-hopping spread spectrum methods, cognitive radio-based methods, and deep reinforcement learning-based methods. Traditional methods rely on pre-defined frequency-hopping patterns or spread spectrum sequences, with fixed decision rules, making them unable to adapt to dynamically changing interference environments. Cognitive radio methods depend on spectrum sensing results, making it difficult to guarantee sensing accuracy and real-time performance, and incurring significant overhead in multi-UAV scenarios. Single-UAV methods based on deep reinforcement learning are trained under fixed interference parameters, resulting in limited policy generalization ability and a significant performance degradation after environmental parameter changes. Existing multi-agent reinforcement learning methods decompose the multi-UAV problem into multiple independent single-UAV decisions, with each agent independently learning its own policy, lacking an effective collaborative decision-making mechanism.

[0004] Furthermore, most existing methods treat channel selection and transmit power control as two independent problems for decision-making, or use a fixed transmit power level, failing to fully consider the coupling relationship between channel selection and power control, and thus unable to achieve joint optimization decisions for both. Overall, most existing technologies are based on fixed interference parameters and empirical rules, relying excessively on pre-set knowledge and static decision models. The systems lack flexibility and adaptability, making it difficult to effectively cope with complex and ever-changing electromagnetic interference environments. Summary of the Invention

[0005] To address the problems of static decision-making models and insufficient adaptability to complex and ever-changing electromagnetic interference environments in existing UAV communication systems, this invention provides a UAV swarm anti-interference communication decision-making method based on deep reinforcement learning.

[0006] The present invention provides a method for anti-jamming communication decision-making in unmanned aerial vehicle (UAV) swarms based on deep reinforcement learning, comprising:

[0007] The drone swarm includes multiple drone clusters; the time axis of the communication system is divided into multiple time slots of equal length, and each time slot includes, in sequence, a spectrum sensing phase, a decision-making phase, a data transmission phase, and a learning phase;

[0008] Step 1: The cluster head of each UAV cluster scans the current spectrum environment during the spectrum sensing phase of each time slot to obtain current channel status information, including the signal-to-interference-plus-noise ratio of the adjacent previous communication channel, the selection of the adjacent previous communication channel, the transmit power of the adjacent previous transmitter, and the current channel selected by the current communication jammer.

[0009] Step 2: In the decision-making phase, the cluster head of each UAV cluster uses a deep reinforcement learning network to obtain the current decision results of all cluster members of each UAV cluster based on the current channel state information, including the current communication channel selection and the current transmitter transmission power;

[0010] Step 3: During the data transmission phase, the cluster head of each UAV cluster transmits the corresponding decision result to other cluster members based on the current decision result; each cluster member calculates the signal-to-interference-plus-noise ratio of the current communication channel and determines whether the communication status is successful.

[0011] Step 4: The cluster head of each drone cluster calculates the current instant reward based on the communication status, the cost of switching the current communication channel or transmitter power with the previous neighbor, the energy consumption cost of the transmitter power, and the collision status.

[0012] Then return to step one to obtain the channel state information for the next communication, and continue the iterative calculation for the next communication decision; until the preset communication duration or communication interruption condition is reached;

[0013] Simultaneously, the learning phase begins. Each time the channel state information is updated, the current channel state information, the current decision result, the current immediate reward, and the channel state information for the next communication are combined to form an experience sample. After each iteration reaches the preset duration, all current experience samples are stored in the experience replay pool to update the parameters of the deep reinforcement learning network.

[0014] According to the deep reinforcement learning-based anti-jamming communication decision-making method for UAV swarms of the present invention, the communication channel, the transmitter transmission power, and the jammer selection channel constitute the environmental observation state; the environmental observation state is preprocessed using one-hot coding and then input into the deep reinforcement learning network.

[0015] According to the deep reinforcement learning-based anti-interference communication decision-making method for UAV swarms of the present invention, the deep reinforcement learning network adopts the Dueling-DDQN algorithm framework, which includes a current network and a target network. During the iterative calculation process, the current network is used to obtain the decision result, and the target network is used to calculate the target Q reference value required for training. The current network decomposes the action value function into the sum of the state value function and the advantage function. The action selection adopts an ε-greedy strategy, randomly selecting actions with probability ε for exploration, and selecting the action with the largest output Q value of the current network with probability 1-ε. Using empirical samples, the loss function is calculated based on the output Q value of the current network and the target Q reference value, and the network parameters of the current network are updated. After a set number of iterations, the network parameters of the current network are synchronized to the target network.

[0016] According to the deep reinforcement learning-based anti-jamming communication decision-making method for UAV swarms of the present invention, the action value function is expressed as follows: :

[0017] ,

[0018] In the formula For channel state information, For the decision outcome, For the state value function, For the dominant function, The decision is a selectable decision within the action space of the decision outcome. This provides the action space for the decision outcome.

[0019] According to the deep reinforcement learning-based anti-jamming communication decision method for UAV swarms of the present invention, if the jammer switches channels during the data transmission phase, the data transmission phase is divided into a first time period and a second time period. The signal-to-interference-plus-noise ratio and interference power of the communication channels in the two time periods are calculated respectively, and the data transmission rate is calculated using the Shannon formula. The success of the communication status is determined based on the data transmission rate and data packet size in the two time periods.

[0020] According to the deep reinforcement learning-based anti-jamming communication decision-making method for UAV swarms of the present invention, the method is applied to the first... Each cluster head, in the time slot Implementation of decision results Instant rewards received afterward for:

[0021]

[0022] In the formula This is a function indicating successful communication. This is a function to indicate communication failure. To switch the penalty coefficient, This is a function indicating the switching of communication channel or transmitter transmit power. As the reward coefficient, This represents the transmitter's transmit power after linearization. This is the collision penalty coefficient. This is the collision indication function.

[0023] The method for anti-jamming communication decision-making of UAV swarms based on deep reinforcement learning according to the present invention involves training the deep reinforcement learning network and sampling samples in the experience replay pool as follows:

[0024] A priority empirical replay algorithm is used for sample sampling, with the absolute value of the temporal difference error used to measure sample priority; empirical samples... The probability of being sampled is :

[0025] ,

[0026] In the formula For empirical samples priority, For the first in the experience replay pool The priority of each sample , This represents the number of experience samples in the experience replay pool.

[0027] Importance sampling weights are used to correct for sampling bias in empirical samples.

[0028] According to the anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning of the present invention, the cluster heads of all UAV clusters make collaborative decisions based on a multi-agent reinforcement learning algorithm. The multi-agent reinforcement learning algorithm adopts a centralized training-distributed execution framework. During the training phase, each cluster head combines its individual output Q-values ​​into a global Q-value through a hybrid network and uses global environmental observation state information for credit allocation. During the execution phase, each cluster head makes decentralized independent decisions based on its own environmental observation state using the output of the GRU network.

[0029] The hybrid network has a Dueling structure. It takes global environmental observation state information as input and generates non-negative hybrid weights and biases through a supernetwork.

[0030] According to the UAV swarm anti-interference communication decision-making method based on deep reinforcement learning of the present invention, the multi-agent reinforcement learning algorithm adopts the QMIX algorithm with an independent entropy regularization mechanism.

[0031] Each cluster head is introduced with a learnable logarithmic temperature parameter. Based on the output of the GRU network, the policy distribution of independent decisions is obtained through a softmax function scaled by temperature. The temperature scaling factor is calculated based on the logarithmic temperature parameter.

[0032] The loss function of the multi-agent reinforcement learning algorithm adds a policy entropy regularization term to the QMIX loss.

[0033] According to the deep reinforcement learning-based anti-jamming communication decision-making method for UAV swarms of the present invention, the method is applied to the first... Each cluster head represents the learnable logarithmic temperature parameter as , The entropy temperature parameter is a learnable parameter.

[0034] initialization ;

[0035] During the training process, the first The policy distribution of each cluster head is approximated by a temperature-scaled softmax function:

[0036] ,

[0037] In the formula For policy distribution, This is information on the environmental observation status of the cluster head. For the first The individual output Q-value of each cluster head;

[0038] Policy Entropy The calculation is as follows:

[0039] ;

[0040] ER-QMIX total loss function for:

[0041] ,

[0042] In the formula For TD error loss, The entropy coefficient, The number of cluster heads;

[0043] ,

[0044] In the formula The global Q value, for Total instant reward for each cluster head As a discount factor, The global Q value for the next adjacent communication;

[0045] Learnable entropy temperature parameter Automatic adjustment via gradient descent.

[0046] The beneficial effects of the present invention are as follows: The method of the present invention, through joint decision-making on channel selection and transmit power, and closed-loop iteration of spectrum perception, anti-interference decision-making, data transmission and policy learning, can effectively cope with various complex dynamic electromagnetic interferences such as frequency sweeping, frequency hopping, and suppression, as well as intra-cluster co-frequency interference, thereby improving the autonomous decision-making ability of UAV clusters in dynamic interference environments and realizing adaptive anti-interference communication.

[0047] This invention achieves real-time perception of complex electromagnetic environments and dynamic adaptive adjustment of communication strategies through closed-loop iteration of spectrum sensing and deep reinforcement learning, effectively addressing various interference modes such as frequency sweeping and frequency hopping. Employing a joint optimization framework of channel selection and power control, it significantly reduces handover overhead and transmit energy consumption while ensuring communication quality, thereby improving spectrum and energy utilization efficiency.

[0048] This invention introduces a centralized training-distributed execution framework for multi-UAV swarm scenarios. It achieves global credit allocation through a hybrid network of the ER-QMIX algorithm, effectively solving the policy convergence problem in multi-agent cooperation. At the same time, each agent is configured with independent entropy and temperature parameters to achieve differentiated exploration, enabling the swarm to have good robustness and scalability in dynamic intelligent interference environments. Attached Figure Description

[0049] Figure 1 This is a principle block diagram of the anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning described in this invention;

[0050] Figure 2 This is a schematic diagram of the time slot structure division provided in a specific embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the single-agent Dueling-DDQN algorithm structure provided in a specific embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the cluster head multi-agent decision-making logic provided in a specific embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the interference switching logic provided in a specific embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of the reward function logic provided in a specific embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the deep reinforcement learning network training process provided in a specific embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram of the ε-greedy strategy and action selection logic provided in a specific embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Specific Implementation Method 1: Combination Figure 1 As shown, this invention provides a method for anti-jamming communication decision-making in UAV swarms based on deep reinforcement learning, including:

[0059] The drone swarm includes multiple drone clusters; the time axis of the communication system is divided into multiple time slots of equal length, and each time slot includes, in sequence, a spectrum sensing phase, a decision-making phase, a data transmission phase, and a learning phase;

[0060] Step 1: The cluster head of each UAV cluster scans the current spectrum environment during the spectrum sensing phase of each time slot to obtain current channel status information, including the signal-to-interference-plus-noise ratio of the adjacent previous communication channel, the selection of the adjacent previous communication channel, the transmit power of the adjacent previous transmitter, and the current channel selected by the current communication jammer.

[0061] Step 2: In the decision-making phase, the cluster head of each UAV cluster uses a deep reinforcement learning network to obtain the current decision results of all cluster members of each UAV cluster based on the current channel state information, including the current communication channel selection and the current transmitter transmission power;

[0062] Step 3: During the data transmission phase, the cluster head of each UAV cluster synchronously transmits the corresponding decision results to other cluster members based on the current decision results; each cluster member calculates the current signal-to-interference-plus-noise ratio of the communication channel and determines whether the communication status is successful.

[0063] Step 4: The cluster head of each drone cluster calculates the current instant reward based on the communication status, the cost of switching the current communication channel or transmitter power with the previous neighbor, the energy consumption cost of the transmitter power, and the collision status.

[0064] Then return to step one to obtain the channel state information for the next communication, and continue the iterative calculation for the next communication decision; until the preset communication duration or communication interruption condition is reached;

[0065] Simultaneously, the learning phase begins. Each time the channel state information is updated, the current channel state information, the current decision result, the current instant reward, and the channel state information of the next communication are combined to form an experience sample. After each iteration reaches the preset duration, all current experience samples are stored in the experience replay pool. Small batches of samples are sampled from the experience replay pool to train the deep reinforcement learning network and update its parameters.

[0066] The above spectrum sensing, anti-interference decision-making, data transmission, and policy learning steps are executed repeatedly until the preset communication duration or communication interruption condition is reached.

[0067] In this implementation, each cluster head acquires current communication environment information and forms an observation state; the observation state is input into a deep reinforcement learning network, which outputs decision results for channel and power; each cluster head configures communication parameters and transmits data according to the decision results; the reward value is calculated based on communication feedback and the network parameters are updated; the above steps are repeated until a preset number of training rounds are reached. Channel and power are jointly encoded into a single discrete action. A Dueling DDQN network is used in single-UAV scenarios, while an ER-QMIX centralized training-distributed execution architecture is used in multi-UAV swarm scenarios. Multi-agent credit allocation is achieved through a hybrid network, and an independent entropy temperature mechanism is introduced to provide personalized exploration strategies for each agent.

[0068] In this embodiment, the communication channel, the transmitter transmission power, and the jammer selection channel constitute the environmental observation state; the environmental observation state is preprocessed using one-hot coding and then input into the deep reinforcement learning network.

[0069] During the spectrum sensing phase of each time slot, the UAV cluster head acquires channel state information of the channels currently occupied by each cluster member, including the interference power of each channel, the channel and power selected in the previous time slot, the current frequency sweep channel and communication distance, providing environmental observation data for anti-interference decision-making.

[0070] Furthermore, the deep reinforcement learning network adopts the Dueling-DDQN algorithm framework, which includes a current network and a target network. During the iterative calculation process, the current network is used to obtain the decision result, and the target network is used to calculate the target Q reference value required for training. The current network decomposes the action value function into the sum of the state value function and the advantage function. Action selection adopts an ε-greedy strategy, randomly selecting actions for exploration with probability ε, and selecting the action with the largest output Q value of the current network with probability 1-ε. Using empirical samples, the loss function is calculated based on the output Q value of the current network and the target Q reference value, and the network parameters of the current network are updated. After a set number of iterations, the network parameters of the current network are synchronized to the target network.

[0071] Each cluster leader, based on channel state information acquired through spectrum sensing, independently decides the communication channel and transmit power for the current time slot for all cluster members using deep reinforcement learning. Each cluster leader inputs local observations into a deep Q-network to calculate the value of each joint action and select the optimal channel-power combination based on the exploration strategy. In multi-UAV swarm scenarios, a centralized training-distributed execution framework is adopted. During the training phase, the ER-QMIX algorithm is introduced, combining individual Q-values ​​into a global Q-value through a hybrid network. Credit allocation is achieved using the global state, resolving the policy convergence problem in multi-agent cooperation. Simultaneously, each agent is configured with an independent entropy temperature parameter to achieve differentiated exploration capabilities.

[0072] The action value function is expressed as :

[0073] ,

[0074] In the formula For channel state information, For the decision outcome, Let be the state value function, representing the state. Its fundamental value is unrelated to specific actions; Let be the dominance function, representing the state . Select action Advantages over average movement; The decision is a selectable decision within the action space of the decision outcome. This represents the action space for the decision outcome. By subtracting the mean of the advantage, it ensures... Its unique identifiability.

[0075] During the data transmission phase, if the jammer switches channels, the data transmission phase is divided into a first time period and a second time period. The signal-to-interference-plus-noise ratio (SINR) and interference power of the communication channel are calculated for each time period. The data transmission rate is calculated using the Shannon formula. The success of the communication is determined based on the data transmission rate and data packet size of the two time periods.

[0076] The actual transmission time depends on the data packet size and the transmission rate in both time periods. If the transmission cannot be completed in either time period, the communication is considered to have failed.

[0077] Furthermore, regarding the first Each cluster head, in the time slot Implementation of decision results Instant rewards received afterward for:

[0078]

[0079] In the formula This is a function that indicates successful communication. This is a function to indicate communication failure status. To switch the penalty coefficient, This is a function indicating the switching of communication channel or transmitter transmit power. As the reward coefficient, This represents the transmitter's transmit power after linearization. This is the collision penalty coefficient. This is the collision indication function.

[0080] Each cluster head calculates the immediate reward based on environmental feedback. The immediate reward comprehensively considers the benefits of successful communication, handover costs, energy consumption costs, and collision penalties within the same cluster. Experience samples are stored in the experience replay pool, and small batches of samples are sampled to update network parameters by minimizing the time-series difference error.

[0081] The method for training a deep reinforcement learning network and sampling samples in the experience replay pool is as follows:

[0082] A priority empirical replay algorithm is used for sample sampling, with the absolute value of the temporal difference error used to measure sample priority; empirical samples... The probability of being sampled is :

[0083] ,

[0084] In the formula For empirical samples priority, For the first in the experience replay pool The priority of each sample , This represents the number of experience samples in the experience replay pool.

[0085] Importance sampling weights are used to correct for sampling bias in empirical samples.

[0086] In multi-drone swarm scenarios, the cluster heads of all drone clusters make collaborative decisions based on a multi-agent reinforcement learning algorithm. The multi-agent reinforcement learning algorithm adopts a centralized training-distributed execution framework. During the training phase, each cluster head combines its individual output Q-values ​​into a global Q-value through a hybrid network and uses global environmental observation state information for credit allocation. During the execution phase, each cluster head makes decentralized independent decisions based on its own environmental observation state using the output of the GRU network.

[0087] The hybrid network features a Dueling structure, taking global environmental observation state information as input and generating non-negative hybrid weights and biases through a supernetwork. Each agent (cluster head) uses a GRU network to process local observation history.

[0088] The multi-agent reinforcement learning algorithm adopts the QMIX algorithm with an independent entropy regularization mechanism, namely the ER-QMIX algorithm; an independent entropy regularization mechanism is introduced on the basis of QMIX.

[0089] Each cluster head is introduced with a learnable logarithmic temperature parameter. Based on the output of the GRU network, the policy distribution of independent decisions is obtained through a softmax function scaled by temperature. The temperature scaling factor is calculated based on the logarithmic temperature parameter.

[0090] The loss function of the multi-agent reinforcement learning algorithm adds a policy entropy regularization term to the QMIX loss.

[0091] In this embodiment, for the first Each cluster head represents the learnable logarithmic temperature parameter as , The entropy temperature parameter is a learnable parameter.

[0092] initialization ;

[0093] During the training process, the first The policy distribution of each cluster head is approximated by a temperature-scaled softmax function:

[0094] ,

[0095] In the formula For policy distribution, This is information on the environmental observation status of the cluster head. For the first The individual output Q-value of each cluster head;

[0096] Policy Entropy The calculation is as follows:

[0097] ;

[0098] The total loss function of ER-QMIX is based on the TD error loss of QMIX, with the addition of a policy entropy regularization term. for:

[0099] ,

[0100] In the formula For TD error loss, The entropy coefficient, The number of cluster heads;

[0101] ,

[0102] In the formula The global Q value, for Total instant reward for each cluster head As a discount factor, The global Q value for the next adjacent communication;

[0103] Learnable entropy temperature parameter Automatic adjustment via gradient descent.

[0104] The following simulation will provide a detailed description of the embodiments of the present invention:

[0105] Combination Figure 2 As shown, the time axis of the UAV swarm anti-jamming communication system is divided into a finite number of equal-length time slots, each time slot having a length of [missing information]. Each time slot is further divided into four phases: broadband spectrum sensing phase, decision-making phase, data transmission phase, and learning phase. The duration of the broadband spectrum sensing phase is denoted as... During this period, each cluster head rapidly scans the current spectrum environment to obtain signal power and interference power information on each available channel. The duration of the decision-making phase is denoted as [missing information]. Each cluster head independently decides on the communication channel and transmit power for the next time slot based on the observation information obtained during the sensing phase. The duration of the data transmission phase is denoted as... Each cluster head synchronously transmits data to its cluster members based on the decision results. The duration of the learning phase is denoted as... Each cluster head updates the strategy network parameters based on environmental feedback. The four stages satisfy... .

[0106] jamming channel switching cycle of the jammer Independent of the anti-jamming decision-making cycle of the UAV, the jammer may switch channels at any stage. When the jammer switches channels during the data transmission phase, the data transmission phase is divided into two time periods. and ,satisfy .exist During the time period, the jammer operates on the original jamming channel before the switch; During the time period, the jammer switches to a new jamming channel. The drone may experience two different jamming environments within the same time slot, directly impacting the data transmission rate.

[0107] As an example, select the time slot length. ,in , , , .

[0108] Consider a drone swarm anti-jamming communication system, consisting of multiple drone swarms and a ground jamming unit. Each drone swarm comprises a Cluster Head (CH) and several Cluster Members (CMs), with the CH responsible for making communication decisions for all its members. All drones share the same spectrum resource, which is divided into... There are n orthogonal channels, and the channel set is denoted as . .

[0109] Number of channels The configuration is set to 4 or 5 depending on the scenario. This applies to both single-drone scenarios and single-cluster multi-drone scenarios. In multi-drone scenarios The jammer is located at a fixed position on the ground and can emit various types of malicious jamming signals. The location of the jammer is denoted as... It provides cluster heads on each channel. and its cluster members Interference channel gain The same path loss model as that used for communication links is employed for calculation.

[0110] As an example, the system comprises three UAV clusters and one jammer. Each cluster consists of one cluster head and three cluster members. The cluster heads are randomly distributed within a circle with a radius of 200m around the jammer. Each UAV cluster contains a cluster head and several cluster members. The cluster head is the decision-making center within the cluster, and in each time slot, it makes unified decisions regarding the communication channel and transmit power for all its cluster members. The cluster members, acting as data transmitters, configure their own communication parameters according to the cluster head's decision and transmit data to the cluster head.

[0111] In a single-UAV scenario, the system simplifies to a point-to-point uplink communication link between the UAV and the base station. Each time slot, the UAV selects a joint action from preset multi-level channels and power levels, and moves randomly to simulate the actual flight trajectory. In a multi-UAV swarm scenario, the system comprises multiple cluster heads and multiple cluster members. Co-channel interference arises between UAVs due to shared spectrum resources, including interference between members within a cluster and interference between different clusters. Cluster heads can only obtain local observation information and cannot know the complete status of other UAVs, demonstrating partially observable characteristics.

[0112] Each cluster head is treated as an agent, and collaborative decision-making among these agents is achieved through a centralized training-distributed execution framework. Assume the system has a total of... If there are 100 drone clusters, then the number of agents is 100. Setting up a single-cluster scenario Multi-cluster scenarios .

[0113] Divide the system's available spectrum into There are three mutually orthogonal communication channels, each with a bandwidth of [missing information]. Each drone and jammer shares this... There are multiple channels, and different communication links can reuse the same channel, but reusing the same channel will cause co-channel interference. The channel index number is... Each cluster head, within each time slot, from One channel is selected from each cluster head to transmit data to its cluster members. The transmit power of each cluster head is divided into... The set of discrete power levels is denoted as [a_discrete_levels]. The corresponding dBm values ​​for the power ratings are 10, 15, 20, 25, and 30, and the corresponding wattage values ​​are 0.01, 0.0316, 0.1, 0.316, and 1.0W, respectively.

[0114] The combined action space for channel selection and power control for each agent is: During the broadband spectrum sensing phase, each cluster head acquires information about the current communication environment using spectrum sensing technology. Specifically, the cluster head... In the time slot The received signal power on each available channel is measured using an energy detection method, and the signal-to-interference-plus-noise ratio (SIR) of each channel is calculated accordingly.

[0115] make Indicates time slot Inner cluster head and its cluster members The Euclidean distance between them Indicates time slot Inner cluster head and its cluster members In the channel The channel power gain consists of two parts: large-scale path loss and small-scale Rayleigh fading.

[0116] ,

[0117] in The Rayleigh fading component follows a complex Gaussian distribution with zero mean and unit variance, i.e. ; For large-scale path loss, a logarithmic distance model is adopted:

[0118] ,

[0119] In the formula The path loss constant is used as a reference distance of 1m. This is the path loss index.

[0120] Table 1 shows the basic parameters for the simulation scenario:

[0121] Table 1

[0122]

[0123] As an example, let the distance between the drone and the receiver be... The path loss is then... The channel gain is:

[0124] .

[0125] Cluster head In the channel The received signal power for: ;in For cluster head To its cluster members The transmission power of communication (unit: W).

[0126] Assuming the transmit power is 20dBmW (i.e., 0.1W) and the channel gain is... Then the received signal power is .

[0127] Cluster head In the channel Total interference power received This includes the superposition of three types of interference sources:

[0128]

[0129] in Intra-cluster co-channel interference caused when other cluster members within the same cluster use the same channel. Inter-cluster co-channel interference occurs when communication links from different clusters use the same channel. This refers to malicious interference emitted by a jammer on the same channel.

[0130] Cluster head In the channel Signal-to-interference-to-noise ratio for:

[0131]

[0132] in This represents the power of Gaussian white noise. The conditions for successful communication are... , .

[0133] As an example, let Then SINR is:

[0134] If the value is below the 10dB threshold, it is considered a communication failure.

[0135] Each cluster head formalizes the communication environment information acquired during the perception phase into the observation states of the agent in reinforcement learning. The agent is then defined. In the time slot Observation status An 11-dimensional vector:

[0136] .

[0137] The observation status consists of the following four parts:

[0138] Interference mode one-hot unique code This indicates the interference mode type for the current time slot. This implementation defines five interference modes, encoded as follows: full-band background noise interference is encoded as follows: Periodic sweep frequency narrowband jamming interference is encoded as One-hot unique code for frequency sweep channel. This indicates the channel currently occupied by the jammer's frequency sweep. In frequency sweeping narrowband jamming mode, the frequency sweeping channel... satisfy In weak full-band mode, the single-hot unique code of the frequency sweep channel is all zero. The power normalization value selected in the previous time slot itself. The power level selected from the previous time slot Normalization yields: ,in The normalized range is .

[0139] The channel hot unique code selected by the previous time slot , indicating the cluster head of the previous time slot Selected communication channel.

[0140] Observation status It serves as input to a deep reinforcement learning network, used to decide on the channel selection and transmit power for the current time slot.

[0141] As an example, suppose the current time slot is in strong frequency sweep mode, the frequency sweep channel is channel 2, the previous power is 20dBm (normalized value 0.5), and the previous channel is channel 1, then the observation state vector is:

[0142]

[0143] One-hot encoding is used to preprocess discrete state information of various types. One-hot encoding converts discrete categorical variables into binary vectors, which facilitates neural network processing.

[0144] One-hot coding of the channel: Each channel is encoded with a length of A binary vector, where the selected channel corresponds to a 1 in one position and the rest to 0. The number of channels is set. Channel 1 is coded as Channel 2 is encoded as Channel 3 is coded as Channel 4 is encoded as .

[0145] One-hot encoding of interference modes: Encodes the interference mode type as a binary vector of length 2. Full-band background noise interference is encoded as... Periodic sweep frequency narrowband jamming interference is encoded as When the system employs more interference mode types, the one-hot encoding length is extended accordingly.

[0146] Power Normalization: Transmit power levels are normalized instead of one-hot encoded to preserve the numerical order between power levels. Power Normalization Value ,in For power level index, For example, power level index 0 (10dBm) is normalized to 0, power level index 2 (20dBm) is normalized to 0.5, and power level index 4 (30dBm) is normalized to 1.

[0147] The above preprocessing method ensures that all input features have similar numerical ranges, which is beneficial for the training convergence of the neural network.

[0148] To address the real-time decision-making problem of a single unmanned aerial vehicle (UAV) under unknown dynamic disturbances, the Dueling DDQN algorithm is employed. This algorithm decomposes the action value function into the sum of the state value function and the advantage function.

[0149] Action value function Select Size of the action space. Action value function. The decomposition structure allows the network to independently learn state values ​​without altering the relative ranking of action advantages, thus accelerating convergence. Even if some actions are never sampled, the network can still learn... Assess the condition of the situation.

[0150] As an example, the input to the Dueling DDQN network is a 31-dimensional state vector, containing the current channel one-hot unique code (5-dimensional), the power normalized value (1-dimensional), the distance value (1-dimensional), and a 4-step historical channel and SINR sliding window (24-dimensional). The network structure is as follows: the shared feature layer consists of two fully connected layers, each containing 256 neurons, with ReLU activation function. The value stream branch consists of 128 neurons followed by one linear output neuron. The dominant stream branch consists of 128 neurons followed by 20 linear output neurons.

[0151] A dual-network mechanism is employed to mitigate the Q-value overestimation problem. Current network... Used for action selection, target network Used for value assessment. The two networks have identical structures, and the target network parameters are periodically copied from the current network.

[0152] TD target value The calculation method is as follows: This involves using the current network to select the optimal action for the next state, and then using the target network to evaluate the Q-value of that action. This decoupling of action selection and value evaluation effectively reduces the risk of overestimation.

[0153] The target network updates every 50 rounds. The loss function is the mean squared error.

[0154] ,in This is an experience replay pool.

[0155] Combination Figure 3 As shown, the joint optimization algorithm for single UAV Dueling DDQN anti-interference is as follows:

[0156] Input: Initial state of the drone, maximum number of training rounds Maximum number of moves per round Experience pool capacity;

[0157] Output: The trained policy ;

[0158] 1. Initialize the current network (Dueling structure) and target network The parameters are respectively and ,make ;

[0159] 2. Initialize an empty experience pool ;

[0160] 3. For to do

[0161] 4. Reset the environment and get the initial state. ;

[0162] 5. While do

[0163] 6. Change the current state Enter the current network Output the Q value for all actions;

[0164] 7. Based on Greedy strategy selects action ;

[0165] 8. Perform the action Environmental return rewards and the next state And determine whether the round has ended;

[0166] 9. Transfer the sample Store in experience pool ;

[0167] 10. If the number of samples in the experience pool is sufficient, then

[0168] 11. From Randomly sample small batches of samples;

[0169] 12. For each sample do

[0170] 13. Use the current network Calculate the Q-values ​​of all actions in the next state, and select the action corresponding to the largest Q-value: ;

[0171] 14. Use the target network Evaluate the Q-value of the action and calculate the TD objective: ,like If it is a terminated state, then ;

[0172] 15. End For

[0173] 16. Calculate the Q-value of the current network's prediction of the sampling action. Establish the mean squared error loss function: ;

[0174] 17. Update the current network parameters using gradient descent. ;

[0175] 18. End If

[0176] 19. If the current step count reaches the target network update interval, then let... ;

[0177] 20. Update exploration rate (Linear or exponential decay);

[0178] twenty one. ;

[0179] 22. End While

[0180] 23. End For

[0181] An ε-greedy strategy is employed to balance exploration and exploitation. In each decision slot, the agent... Randomly select actions, with probability Choose the action that maximizes the Q value:

[0182]

[0183] In this implementation, the exploration rate linearly decreases from 1.0 to 0.01 in the first 1600 rounds, and then remains constant at 0.01. The linear decrease formula is:

[0184] ,

[0185] ;

[0186] As an example, in the early stages of training ( The agent explores the environment completely randomly, collecting diverse experiences; in the later stages of training ( The intelligent agent mainly utilizes learned strategies to ensure the quality of decision-making.

[0187] In multi-drone swarm scenarios, the system includes Each agent (cluster head) has one intelligent agent. Based on local observations Select Action Environment Return Team Rewards ,in For intelligent agents Individual rewards.

[0188] Since each agent can only know local observations and cannot know the state and actions of other agents, and the decisions of each agent affect each other, the following core problems need to be solved: credit allocation reasonably attributes team rewards to the actions of each agent; environmental non-stationarity causes changes in transition probabilities and policy convergence due to the simultaneous updating of policies by each agent, that is, multiple agents learning similar policies leads to resource competition.

[0189] Combination Figure 4 As shown, a centralized training-distributed execution framework is employed to address the multi-agent collaborative decision-making problem. During the training phase, each agent has access to global state information and utilizes centralized information for collaborative training. During the execution phase, each agent makes independent decisions based solely on its own local observations, without relying on inter-agent communication. For the total number of agents, each agent... The observable history is processed using a gated cyclic unit (GRU) network. Let Q be the individual action value of the i-th agent, output by the GRU network, representing the value estimate of each action of the agent under the current local observation. It is a QMIX Mixing Network. For the TD objective, the target Q value of reinforcement learning; The team's instant total reward is obtained by summing the individual rewards of all cluster heads; discount factor. This represents the decay weight of future returns; The next global output of the target network It comes from the target hybrid network and is used to mitigate TD bias. Let m be the policy entropy of the m-th agent. The agent m has a random policy and outputs the probability distribution of each action. Prioritize experience replay. The TD error represents the deviation between the network's predicted value and the target value. It is a minimal constant, to prevent Time priority This results in the sample never being collected. This contains all trainable parameters for the main network, including the GRU network parameters for each agent and the parameters of the hybrid network. The dedicated entropy temperature controls the weight of the agent's policy entropy. The larger the value, the greater the level of exploration by the agent; The target network parameters include the target GRU network and the target hybrid network. Every 200 training epochs, the main network parameters are... Complete copy and assignment to the target network Hard update, stable training.

[0190] Global state It consists of a splicing of local observations from all agents (99 dimensions), a one-hot unique code for interference modes (2 dimensions), and a one-hot unique code for the swept-frequency channel (4 dimensions), totaling 105 dimensions. The global state is only used during the training phase.

[0191] Each agent Partially observable history is processed using a gated recurrent unit (GRU) network. The input to the GRU network is local observations. (11-dimensional), the output is the individual Q-value. ,in This represents the observation-action history.

[0192] The forward propagation calculation for the GRU network is as follows:

[0193] ,

[0194] ,

[0195] ,

[0196] in This is a hidden state for GRU. and The weight matrix is ​​learnable. The GRU network can memorize historical information, improving its adaptability to dynamic interference environments.

[0197] The QMIX algorithm is used to implement credit allocation among multiple agents. QMIX consists of two core components: a Mixing Network and a Hypernetwork.

[0198] The input to the hybrid network is the individual Q-values ​​of all agents. and global state Output the global Q value .

[0199] Hypernetworks with global state As input, generate the weights and biases for the hybrid network:

[0200] ,

[0201] ,

[0202] in , , , , The number of hidden units is determined by taking the absolute value of the weights to ensure they are non-negative.

[0203] First-layer hypernetwork:

[0204] ,

[0205] ;

[0206] Second-layer hypernetwork:

[0207] ,

[0208] ;

[0209] in To perform element-wise absolute value operations, the weights of the hybrid network must be non-negative, thus satisfying the following conditions: Monotonicity constraint. Bias. Non-negativity is not required, and absolute value operations are not used.

[0210] The training of the supernetwork is performed jointly with the main network. In each training step, the supernetwork parameters are processed using a loss function. The gradient is obtained through backpropagation and updated synchronously with the parameters of the agent network and the hybrid network. The hidden layer dimension of the supernetwork is chosen to be 64, with a total of 2 layers.

[0211] The global Q-value is calculated as follows:

[0212] ,

[0213] ELU is the exponential linear unit activation function.

[0214] Weights of hybrid networks and Taking the absolute value ensures that the partial derivative of the global Q-value with respect to the individual Q-value is non-negative.

[0215] .

[0216] To address the issues of homogenization of exploration behaviors and convergence of strategies in the QMIX algorithm, the ER-QMIX (Entropy-Regularized QMIX) algorithm is proposed, which introduces an independent entropy regularization mechanism.

[0217] Each agent It has a learnable entropy temperature parameter. , represented as initial value .

[0218] During training, the policy distribution of each agent is approximated using a temperature-scaled softmax approximation, and the entropy coefficient... The value is 0.01.

[0219] Entropy temperature parameter Automatic adjustment is achieved through gradient descent. When the policy entropy is lower than the target entropy, the temperature increases, the policy distribution becomes more uniform, and exploration is enhanced; when the policy entropy is higher than the target entropy, the temperature decreases, the policy distribution becomes more deterministic, and utilization is strengthened.

[0220] The method for updating the independent entropy temperature parameter is as follows:

[0221] Input: Number of agents Current network parameters of each agent Current entropy temperature parameter Entropy coefficient ;

[0222] Output: Updated entropy temperature parameters .

[0223] 1. For each agent

[0224] 2. Calculate the temperature

[0225] 3. Calculation Strategy Distribution

[0226] 4. Calculate the policy entropy

[0227] 5. Calculate the entropy regularization loss.

[0228] 6. Add to total loss

[0229] 7. Backpropagation to calculate gradient

[0230] 8. For each agent

[0231] 9. Update entropy temperature parameters

[0232] 10. Update actual temperature

[0233] 11. End For

[0234] 12. Return .

[0235] The target network update interval is set to 200 rounds, and the learning rate is... The experience pool capacity is 5000, and the batch size is 32.

[0236] As an example, in a multi-cluster drone scenario, after 5000 training rounds, the entropy temperature of each agent... The convergence to different values, ranging from 0.3 to 1.5, proves that the exploration needs of each agent are indeed different, and the independent entropy temperature mechanism achieves a personalized exploration-utilization balance. After completing the decision-making phase, each cluster head configures its own communication parameters according to the decision results and enters the data transmission phase. The duration of the data transmission phase is denoted as... Each cluster head synchronously transmits data to its cluster members.

[0237] jamming channel switching cycle of the jammer Independent of the anti-jamming decision-making cycle of the UAV, the jammer may switch jamming channels at any time during the data transmission phase. and It is irrelevant; the switching times of the jammers are evenly distributed during the data transmission phase.

[0238] As an example, the duration of the data transmission phase milliseconds, the jammer can be The jammer switches channels at any time within the data transmission phase, with the switching times uniformly and randomly distributed between 0 and 14 ms. When the jammer switches channels during the data transmission phase, the data transmission phase is divided into two time periods. and ,satisfy:

[0239] ,

[0240] exist During the time period, the jammer operates on the original jamming channel before the switch; During the time period, the jammer switches to a new jamming channel. Due to the different jamming environments in the two time periods, the UAV may experience two different jamming states within the same time slot.

[0241] If the jammer does not switch channels during the data transmission phase, then the entire data transmission phase... The interference environment remains unchanged, at this time , .

[0242] The probability of a jammer switching channels during the data transmission phase is determined by the jamming mode. In the periodic frequency sweeping narrowband blocking jamming mode, the jammer switches channels once per time slot, with the switching time being fixed; in the Markov random frequency hopping jamming mode, the jammer switches randomly within each time slot with a preset probability.

[0243] Channel power gain includes large-scale path loss and small-scale Rayleigh fading. Let... Indicates time slot Inner cluster head and its cluster members The Euclidean distance between them determines the channel power gain. for:

[0244] ,

[0245] in The Rayleigh fading component follows a complex Gaussian distribution with zero mean and unit variance; For large-scale path loss, a logarithmic distance model is adopted:

[0246] , .

[0247] As an example, let the communication distance be... Then path loss The channel gain is .

[0248] Due to limited spectrum resources, multiple communication links and jammers reuse the same channels. Cluster head and its cluster members In the channel During communication, it is affected by the superposition of three types of interference sources:

[0249] ;

[0250] Intra-cluster co-frequency interference Interference caused when other cluster members within the same cluster use the same channel:

[0251] ,

[0252] in The number of members per cluster, This is an indicator function; its value is 1 if the condition is true, and 0 otherwise. For cluster head For its first The channel selected by each cluster member This corresponds to the transmission power.

[0253] Inter-cluster co-frequency interference Interference that occurs when communication links from different clusters use the same channel:

[0254] ,

[0255] in The total number of clusters, For cluster head For its first The channel selected by each cluster member.

[0256] External interference Malicious interference emitted by a jammer on the same channel:

[0257] ,

[0258] in For the number of jammers, jammer Interference channels, This refers to the jammer's transmission power. For jammer to cluster head Cluster members In the channel Interference channel gain.

[0259] Because the jammer may switch channels during the data transmission phase. and The interference power differs between the two time periods. Let and Let represent the total interference power in the two time periods, respectively. Then, the signal-to-interference-plus-noise ratios (SIRs) in the two time periods are as follows:

[0260] ,

[0261] ,

[0262] in This represents the power of Gaussian white noise.

[0263] The conditions for successful communication are In this invention .

[0264] like Then in Internal data transfer successful; if Then in Internal data transmission was successful. If both time periods meet the success conditions, the communication is completely successful; if only one time period meets the conditions, it is partially successful; if neither condition is met, the communication fails.

[0265] Data transmission rate is calculated using the Shannon formula. The data transmission rates for the two time periods are as follows:

[0266] ,

[0267] ,

[0268] in For each channel bandwidth, in this invention .

[0269] Actual transmission time Depends on the size of the data packets to be sent (Unit: bit) and the transmission rate over two time periods.

[0270] If the jammer does not switch channels during the data transmission phase, the rate remains constant throughout the entire transmission phase. The actual transmission time is: ;

[0271] like If the transmission is successful, the transmission will succeed; otherwise, the transmission will fail.

[0272] If the jammer switches channels during the data transmission phase, the data packet will be... Transmit as much data as possible within the system. If... Then the transmission is in Completed internally: ;

[0273] Otherwise, the remaining data is in Continue transmitting:

[0274] ,

[0275] like If not, the communication will fail.

[0276] Combination Figure 5 As shown, five dynamic interference modes are established, and each interference mode switches cyclically with a preset number of steps. The interference channel switching cycle is independent of the UAV decision cycle.

[0277] Full-band background noise interference, jammer with power Simultaneous transmission of jamming signals on all channels. In this mode, any channel... All are subjected to the same interference power. Assume the jammer is in the time slot. The interference channel indication vector is ,but .

[0278] The total interference power of the drone receiver is:

[0279] ,

[0280] Weak interference power .

[0281] Periodic frequency sweeping narrowband jamming, the jammer uses power One time slot is reserved on each channel in sequence, and the frequency is swept cyclically in a fixed order. The frequency sweep sequence is periodic. Circular arrangement:

[0282] ,

[0283] In this mode, only one channel in each time slot is subjected to strong interference. The corresponding bit in the hot unique code of the interfering channel is set to 1, while the other bits are set to 0. If the UAV selects a channel... If it is strong, it will be subject to strong interference; otherwise, it will not be affected by the jammer.

[0284] Dual-channel alternating frequency sweep jamming: the jammer simultaneously jams a pair of adjacent channels in each time slot. The adjacent channel pairs rotate in a fixed order. Let... If the starting channel of the current time slot is indicated, then the set of interfered channels is:

[0285] ,

[0286] jammer with power Simultaneously transmit jamming on both channels.

[0287] Markov random frequency hopping interference occurs when the interference channel changes randomly in the form of a Markov chain. Let the current time slot interference channel be... Interference channel in the next time slot With probability Keeping constant, with probability Uniformly and randomly jump to other One of the channels. The transition probability is:

[0288] .

[0289] Select .

[0290] Fixed-frequency continuous blocking interference involves the jammer occupying a specific channel for a fixed number of consecutive time slots, then randomly switching channels at different times. Let the length of each fixed segment be... In the Within a fixed segment, interference channel from Randomly selected from, and kept Each time slot remains unchanged:

[0291] ,

[0292] choose step.

[0293] In a single UAV experiment, the combination and cycling of the above five interference modes were carried out continuously. Steps, randomly arranged to form a complete cycle Step, and repeat the loop.

[0294] Let pattern sequence for The random arrangement of time slots The corresponding pattern index is:

[0295] .

[0296] Dynamic interference and channel parameter definitions are shown in Table 2:

[0297] Table 2

[0298]

[0299] After completing the data transmission phase, each cluster head enters the learning phase based on environmental feedback. The duration of the learning phase is denoted as [duration missing]. Each cluster head updates the strategy network parameters using the interactive data of the current time slot.

[0300] The learning phase is executed after the data transmission phase of each time slot ends. Each cluster head calculates the instant reward based on whether the communication is successful and the transmission power. The observation state acquired in the perception phase, the joint action selected in the decision-making phase, the instant reward, and the observation state of the next time slot are combined into a quadruple and stored in the experience pool. Small batches of data are randomly sampled from the experience pool to update the network parameters.

[0301] As an example, a policy update is performed once at the end of each time slot, but network parameters are not updated at every step. When the number of samples in the experience pool reaches the preset batch size, a small batch of data is randomly sampled to perform the update.

[0302] Combination Figure 6 As shown, an instant reward function is designed that balances the benefits of successful communication, the cost of switching, and the energy consumption cost. (Agent) In the time slot Execute action Receive instant rewards afterwards .

[0303] Success reward: If communication is successful (SINR≥10dB), a reward of +1 is given; if communication fails (SINR<10dB), a reward of -1 is given. This reward incentivizes the agent to prioritize ensuring communication success.

[0304] .

[0305] Handover penalty: If the channel or power selected in the current time slot is different from that in the previous time slot, a penalty of -0.2 is applied.

[0306] .

[0307] Switching penalty coefficient This penalty encourages agents to maintain stable communication parameters and reduce unnecessary switching overhead.

[0308] Power Bonus: If communication is successful, an additional bonus inversely proportional to the transmission power will be awarded. These are the linearized power values, with 10dBm corresponding to 1, 15dBm to 1.5, 20dBm to 2, 25dBm to 2.5, and 30dBm to 3. Power bonus factor. The lower the power, the higher the reward, guiding the agent to choose a lower power level while ensuring success.

[0309] Collision penalty: If multiple cluster members within the same cluster select the same channel, each agent that selects that channel is penalized with a penalty of -1.0. Collision penalty coefficient. This penalty guides agents to stagger their channel selection, reducing intra-cluster co-channel interference.

[0310] As an example, suppose an agent successfully communicates, a channel switch occurs, the transmission power is 20 dBm (linear value 2), and no collision occurs. The immediate reward would be: .

[0311] If communication fails, no handover occurs, power is 10dBm, and no collision occurs, the immediate reward is... .

[0312] Using discount factors Weigh the importance of immediate rewards versus future rewards. The expected return of long-term cumulative discounts is defined as: ;

[0313] in As a team reward, Number of agents. Discount factor. This allows the agent to balance short-term gains and long-term cumulative gains when optimizing strategies.

[0314] The closer the discount factor is to 1, the more the agent focuses on long-term rewards, resulting in a longer-term strategic planning perspective; the smaller the discount factor, the more the agent focuses on immediate rewards, leading to more short-sighted decision-making. A balance was achieved between the two.

[0315] Prioritized Experience Replay (PER) algorithm is used to improve sample utilization efficiency. Unlike uniform random sampling, PER calculates priority based on the temporal difference error (TD error) of the samples, with samples having a higher probability of being sampled due to larger errors.

[0316] Each sample priority Calculated based on the absolute value of its TD error:

[0317] ,

[0318] in For TD error, To prevent small constants with zero priority.

[0319] Priority can also be expressed in power-law form:

[0320] ,

[0321] in It serves as a priority index, controlling the extent to which priority is used. It degenerates into uniform sampling at times. Sampling is performed entirely according to priority. In this invention... .

[0322] sample Probability of being sampled for: ;

[0323] Since prioritizing experience replay alters the sample distribution, it introduces bias. Importance Sampling (IS) weights are used to compensate for this bias: ,in For the experience pool capacity, It is an importance sampling index used to control the degree of bias compensation. Starting from 0.4, the value linearly increases to 1.0 during training. After each network parameter update, the TD error of the sampled samples is recalculated, and its priority is updated. The update method is as follows: .

[0324] For newly added samples to the experience pool, their priority is set to the highest priority in the current experience pool to ensure a high probability of sampling. The Adam optimizer is used to update network parameters. The Adam optimizer combines the advantages of momentum and adaptive learning rates, enabling it to adaptively adjust the learning rate in different parameter directions.

[0325] Target network parameters Periodically from current network parameters The copy, hard update method is: ;

[0326] Network parameter updates use mini-batch gradient descent: ;

[0327] in For the learning rate, single drone scenario Multi-drone swarm scenarios , For batch size.

[0328] Combination Figure 7 As shown, the training process of the anti-jamming communication decision-making method for UAV swarms based on deep reinforcement learning is as follows:

[0329] Step 1: Initialize the current network parameters for all agents. and target network parameters ,make Clear the experience pool .

[0330] Step 2: Reset the environment, initialize the position, channel, and power of each UAV, generate an interference pattern sequence, and obtain the initial observation state of each agent. .

[0331] Combination Figure 8 As shown, in step three, in the current time slot, each agent reports its observed state. Input the current network, and the network outputs the Q-values ​​for all actions, based on... Strategy selection action .

[0332] Step four: Each cluster head performs a joint action, configuring the communication channel and transmit power, and transmitting data to cluster members. The receiving end calculates the SINR and determines whether the communication was successful.

[0333] Step 5: Calculate the instant reward based on communication success and transmission power. The environment transitions to the next state, and the next observation state is obtained. .

[0334] Step 6: Combine the quadruplets Stored in the experience pool. In multi-agent scenarios, the quadruple of each agent is stored independently.

[0335] Step 7: When the number of samples in the experience pool reaches the preset batch size At that time, randomly sample (or sample according to priority) a small batch of samples and calculate the TD target value. The gradient descent method is used to update the current network parameters. .

[0336] Step 8: Every preset number of rounds, copy the current network parameters to the target network: .

[0337] Repeat steps three through eight until the preset number of training rounds is reached or the communication success rate converges.

[0338] Table 3 is the definition table of reinforcement learning training parameters:

[0339] Table 3

[0340]

[0341] The testing process is as follows:

[0342] Step 1: Load the pre-trained network parameters It is incorporated into the policy network of each agent.

[0343] Step 2: Reset the environment, initialize the test scenario, and set the interference mode and communication parameters.

[0344] Step 3: During the testing phase, each agent disables exploration and adopts a greedy strategy to select actions.

[0345] ;

[0346] Step four: Each cluster head performs joint actions, configures communication parameters for data transmission, and records performance indicators such as communication success rate and average transmit power.

[0347] Step 5: Repeat steps 2 to 4 until the preset number of test rounds is completed.

[0348] Testing phase The agent makes decisions entirely using the learned strategies. The number of test rounds is typically 50 to 100 to obtain stable performance statistics.

[0349] This invention encodes channel selection and transmit power control into a single discrete action and achieves joint optimization of the two through a deep reinforcement learning network. This avoids the problem of existing methods making separate decisions on channel and power and ignoring their coupling relationship, and achieves joint optimization of communication success rate and energy consumption.

[0350] This invention employs a Dueling DDQN network structure to learn state value and action advantage separately, enabling the agent to learn state value independently without changing the relative ranking of action advantages. This solves the problem of low learning efficiency caused by action space coupling in single UAV decision-making and significantly improves the decision-making efficiency of single UAVs in dynamic interference environments.

[0351] This invention achieves multi-agent credit allocation through a QMIX hybrid network and a super network structure, reasonably attributing team rewards to each agent. This solves the problems of policy convergence and environmental non-stationarity caused by the inability to distinguish individual contributions in team rewards during independent learning, and significantly enhances the collaborative decision-making capabilities of multi-UAV swarms.

[0352] This invention introduces an independent entropy-temperature mechanism based on QMIX, assigning a learnable entropy-temperature parameter to each agent. The temperature is adaptively adjusted during training to achieve a personalized exploration-exploitation balance. This solves the problems of policy convergence and low exploration efficiency caused by all agents sharing the same exploration strategy in traditional methods, effectively improving exploration efficiency and policy diversity.

[0353] This invention establishes a dynamic interference model covering five typical interference modes: full-band background noise, periodic frequency sweep, dual-channel alternating frequency sweep, Markov random frequency hopping, and fixed-frequency continuous blocking. It solves the problem of insufficient generalization ability caused by training under fixed interference parameters in existing methods and verifies the strong robustness of the method of this invention to dynamic interference environments.

[0354] This invention adopts a centralized training-distributed execution architecture. During the training phase, global information is used for collaborative optimization. During the execution phase, each agent makes independent decisions based only on local observations and does not rely on inter-agent communication. This solves the problem of lack of collaboration caused by existing multi-agent methods that break down multi-machine problems into independent single-machine decisions.

[0355] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A method for anti-interference communication decision-making in UAV swarms based on deep reinforcement learning, characterized in that... include, The drone swarm includes multiple drone clusters; the time axis of the communication system is divided into multiple time slots of equal length, and each time slot includes, in sequence, a spectrum sensing phase, a decision-making phase, a data transmission phase, and a learning phase; Step 1: The cluster head of each UAV cluster scans the current spectrum environment during the spectrum sensing phase of each time slot to obtain current channel status information, including the signal-to-interference-plus-noise ratio of the adjacent previous communication channel, the selection of the adjacent previous communication channel, the transmit power of the adjacent previous transmitter, and the current channel selected by the current communication jammer. Step 2: In the decision-making phase, the cluster head of each UAV cluster uses a deep reinforcement learning network to obtain the current decision results of all cluster members of each UAV cluster based on the current channel state information, including the current communication channel selection and the current transmitter transmission power; Step 3: During the data transmission phase, the cluster head of each UAV cluster transmits the corresponding decision result to other cluster members based on the current decision result; each cluster member calculates the signal-to-interference-plus-noise ratio of the current communication channel and determines whether the communication status is successful. Step 4: The cluster head of each drone cluster calculates the current instant reward based on the communication status, the cost of switching the current communication channel or transmitter power with the previous neighbor, the energy consumption cost of the transmitter power, and the collision status. Then return to step one to obtain the channel state information for the next communication, and continue the iterative calculation for the next communication decision; Until the preset communication duration or communication interruption condition is reached; Simultaneously, the learning phase begins. Each time the channel state information is updated, the current channel state information, the current decision result, the current immediate reward, and the channel state information for the next communication are combined to form an experience sample. After each iteration reaches the preset duration, all current experience samples are stored in the experience replay pool to update the parameters of the deep reinforcement learning network.

2. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, The communication channel, transmitter transmission power, and jammer channel selection constitute the environmental observation state; the environmental observation state is preprocessed using one-hot coding and then input into the deep reinforcement learning network.

3. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning network adopts the Dueling-DDQN algorithm framework, which includes a current network and a target network. During the iterative calculation process, the current network is used to obtain the decision result, and the target network is used to calculate the target Q reference value required for training. The current network decomposes the action value function into the sum of the state value function and the advantage function. Action selection employs an ε-greedy strategy, randomly selecting actions with probability ε for exploration, and selecting the action with the largest Q-value of the current network output with probability 1−ε. Empirical samples are used to calculate the loss function based on the current network output Q-value and the target Q-reference value, and the network parameters of the current network are updated. After a set number of iterations, the network parameters of the current network are synchronized to the target network.

4. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 3, characterized in that, The action value function is expressed as : ， In the formula For channel state information, For the decision outcome, For the state value function, For the dominant function, The decision is a selectable decision within the action space of the decision outcome. This provides the action space for the decision outcome.

5. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, During the data transmission phase, if the jammer switches channels, the data transmission phase is divided into a first time period and a second time period. The signal-to-interference-plus-noise ratio (SINR) and interference power of the communication channel are calculated for each time period. The data transmission rate is calculated using the Shannon formula. The success of the communication is determined based on the data transmission rate and data packet size of the two time periods.

6. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, For the Each cluster head, in the time slot Implementation of decision results Instant rewards received afterward for: In the formula This is a function indicating successful communication. This is a function to indicate communication failure. To switch the penalty coefficient, This is a function indicating the switching of communication channel or transmitter transmit power. As the reward coefficient, This represents the transmitter's transmit power after linearization. This is the collision penalty coefficient. This is the collision indication function.

7. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, The method for training a deep reinforcement learning network and sampling samples in the experience replay pool is as follows: A priority empirical replay algorithm is used for sample sampling, with the absolute value of the temporal difference error used to measure sample priority; empirical samples... The probability of being sampled is : ， In the formula For empirical samples priority, For the first in the experience replay pool The priority of each sample , This represents the number of experience samples in the experience replay pool. Importance sampling weights are used to correct for sampling bias in empirical samples.

8. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 4, characterized in that, All cluster heads of drones make collaborative decisions based on a multi-agent reinforcement learning algorithm. The multi-agent reinforcement learning algorithm adopts a centralized training-distributed execution framework. During the training phase, each cluster head combines its individual output Q-values ​​into a global Q-value through a hybrid network and uses global environmental observation state information for credit allocation. During the execution phase, each cluster head makes decentralized independent decisions based on its own environmental observation state using the output of the GRU network. The hybrid network has a Dueling structure. It takes global environmental observation state information as input and generates non-negative hybrid weights and biases through a supernetwork.

9. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 8, characterized in that, The multi-agent reinforcement learning algorithm adopts the QMIX algorithm, which incorporates an independent entropy regularization mechanism. Each cluster head is introduced with a learnable logarithmic temperature parameter. Based on the output of the GRU network, the policy distribution of independent decisions is obtained through a softmax function scaled by temperature. The temperature scaling factor is calculated based on the logarithmic temperature parameter. The loss function of the multi-agent reinforcement learning algorithm adds a policy entropy regularization term to the QMIX loss.

10. The anti-interference communication decision-making method for UAV swarms based on deep reinforcement learning according to claim 9, characterized in that, For the Each cluster head represents the learnable logarithmic temperature parameter as , The entropy temperature parameter is a learnable parameter. initialization ; During the training process, the first The policy distribution of each cluster head is approximated by a temperature-scaled softmax function: ， In the formula For policy distribution, This is information on the environmental observation status of the cluster head. For the first The individual output Q-value of each cluster head; Policy Entropy The calculation is as follows: ； ER-QMIX total loss function for: ， In the formula For TD error loss, The entropy coefficient, The number of cluster heads; ， In the formula The global Q value, for Total instant reward for each cluster head As a discount factor, The global Q value for the next adjacent communication; Learnable entropy temperature parameter Automatic adjustment via gradient descent.

Images