A method for anti-jamming communication of a drone cluster based on deep reinforcement learning

Abstract

The application relates to the field of unmanned aerial vehicle communication and discloses an unmanned aerial vehicle cluster anti-interference communication method based on deep reinforcement learning, wherein the weight of each risk index is adaptively determined by an entropy weight method, the index importance can be dynamically adjusted according to the discrete degree of the current channel state, the comprehensive risk value can accurately reflect the real-time interference level of the channel, different numbers of time slots are allocated to different priority services, and long-term failed low-priority service instances are temporarily compensated by combining an anti-starvation mechanism, so that the communication reliability of high-priority services is ensured, the problem that low-priority services are starved due to continuous failure is effectively alleviated, long-term fairness is realized, real samples and counterfactual samples are combined for training, so that the intelligent agent can learn from virtual experience, the conflict between the anti-starvation compensation rule and the deep Q network learning goal is effectively avoided, and the policy convergence is accelerated.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) communication, and in particular to a UAV swarm anti-jamming communication method based on deep reinforcement learning. Background Technology

[0002] With the widespread application of drone swarms in disaster relief, tactical reconnaissance, border patrol, and low-altitude logistics, the demand for multi-drone collaborative communication is rapidly increasing. Existing spectrum allocation technologies are mainly divided into two categories:

[0003] One type is based on deterministic rules or random allocation strategies, such as fixed channel allocation methods and random frequency hopping methods. These methods allocate channels by pre-designing or randomly selecting channels, and cannot use historical interference and service status information for adaptive optimization. They have limited responsiveness to spectrum changes in complex dynamic interference environments, and it is also difficult to dynamically adjust channel resources according to differences in service priorities.

[0004] Another type is learning-based methods based on heuristic rules or traditional deep reinforcement learning, such as greedy methods based on energy detection and ordinary deep Q-network methods. Although these methods can optimize channel selection using historical data, they cannot effectively guarantee the communication reliability of high-priority services in scenarios where multiple service priorities coexist. At the same time, low-priority services are prone to not getting communication opportunities for a long time, resulting in poor overall fairness in service priority. Summary of the Invention

[0005] The purpose of this invention is to address the technical problems of existing UAV spectrum allocation, which on the one hand cannot utilize historical information for adaptive optimization and is difficult to dynamically adjust channel resources according to service priorities; on the other hand, in scenarios where multiple service priorities coexist, it cannot effectively guarantee the communication reliability of high-priority services and easily leads to low-priority services not getting communication opportunities for a long time. The invention provides a UAV swarm anti-interference communication method based on deep reinforcement learning.

[0006] To achieve the above-mentioned objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A method for anti-jamming communication of UAV swarms based on deep reinforcement learning includes the following sub-steps:

[0008] The cluster-head UAV obtains the risk index vector of each candidate channel, uses the entropy weight method to obtain the weight of each risk index, and obtains the comprehensive risk of each candidate channel through the weight of each risk index and the risk index.

[0009] The comprehensive risk of all ranked candidate channels is divided using quantiles to obtain a dynamic threshold. The candidate channels are then divided into spectrum pools using the dynamic threshold. The spectrum pools include a high-priority spectrum pool, a normal spectrum pool, and a high-risk spectrum pool.

[0010] Obtain the number of time slots allocated to each type of service, which includes high-priority services, ordinary-priority services and low-priority services. Based on the anti-starvation mechanism, process low-priority service instances within low-priority services that are at risk of starvation, increase the number of time slots allocated to low-priority service instances and the access probability of different spectrum pools, and obtain the candidate channel set of low-priority service instances.

[0011] The cluster-headed UAV is used as an intelligent agent. Based on the state space, action space and instantaneous reward function, a deep Q-network is used to learn the predicted Q value of the intelligent agent.

[0012] The deep Q-network is iteratively trained so that the agent learns the optimal channel selection strategy based on counterfactual samples and real samples.

[0013] Compared with existing technologies, the beneficial effects of this application are as follows: By adaptively determining the weights of each risk indicator using the entropy weight method, the importance of the indicators can be dynamically adjusted according to the dispersion of the current channel state, ensuring that the comprehensive risk value accurately reflects the real-time interference level of the channel; by dynamically dividing the spectrum pool using the quantile method, the boundaries between low-risk and high-risk channels can be automatically determined according to the current channel risk distribution, allowing channels of different qualities to be adaptively classified into their corresponding pools; by allocating different numbers of time slots to different priority services and combining an anti-starvation mechanism to temporarily compensate for long-term failed low-priority service instances, the communication reliability of high-priority services is ensured, while effectively alleviating the problem of low-priority services being starved due to continuous failures, achieving long-term fairness; and by combining real samples and counterfactual samples for training, the agent can learn from virtual experience, effectively avoiding the conflict between the anti-starvation compensation rules and the learning objectives of the deep Q-network, thus accelerating policy convergence.

[0014] Furthermore, a method for anti-jamming communication in UAV swarms based on deep reinforcement learning, wherein the cluster leader UAV acquires a risk index vector for each candidate channel, obtains the weight of each risk index using the entropy weight method, and obtains the comprehensive risk of each candidate channel through the weight and risk index of each risk index, includes the following sub-steps:

[0015] The drone swarm uses the cluster leader drone to perform spectrum sensing on the candidate channel set at super-frame intervals to obtain the smoothed channel energy of each candidate channel;

[0016] The communication failure rate, smoothed sensing energy, collision level, and interference duration are normalized respectively. The normalized data are then integrated with the sudden interference prediction risk to obtain the risk index vector of each candidate channel at the current moment.

[0017] The entropy weight method is used to process the risk indicators within the risk indicator vector of each candidate channel to obtain the weight of each risk indicator at the current time.

[0018] Iterate through each risk indicator and its weight to obtain the comprehensive risk of each candidate channel at the current moment.

[0019] In the above scheme, spectrum sensing and energy smoothing are performed with superframes as the period, which can stably obtain channel state, suppress instantaneous noise fluctuations, and improve the reliability of input data. A five-dimensional index vector containing communication failure rate, smoothed energy, collision level, interference duration, and sudden interference prediction risk is constructed, which can comprehensively characterize channel quality. In particular, the sudden interference prediction mechanism can detect abnormal energy increases in advance and make up for the lag of historical statistics. The entropy weight method is used to adaptively calculate the weight of each risk index, and the importance is automatically assigned according to the dispersion of the index among channels, which completely avoids the subjective bias of manually setting weights and makes the comprehensive risk assessment more objective and accurate. The comprehensive risk value of each channel is obtained by weighted summation, and the channel quality is quantified in real time. This naturally reduces the selection probability of high-risk channels in subsequent decisions and guides the agent to prioritize the use of low-risk channels, thereby significantly improving the communication success rate.

[0020] Furthermore, a method for anti-jamming communication of UAV swarms based on deep reinforcement learning, which involves normalizing the communication failure rate, smoothed sensing energy, collision level, and interference duration, and then integrating the normalized data with the sudden interference prediction risk to obtain the risk index vector for each candidate channel at the current moment, includes the following sub-steps:

[0021] The communication failure rate is obtained by using the historical number of communication selections and the number of communication selection failures of the i-th candidate channel.

[0022] The collision level is obtained by using the historical communication selection count and collision count of the i-th candidate channel;

[0023] The duration of interference is obtained by using the historical continuous interference duration and the maximum truncation value of the interference duration of the i-th candidate channel;

[0024] By utilizing the long-term baseline energy and smoothed sensing energy of the i-th candidate channel, the risk of sudden interference prediction is obtained;

[0025] Where i is the candidate channel index.

[0026] In the above scheme, the communication failure rate is calculated by using the number of historical communication selections and failures, which can accurately reflect the comprehensive reliability of each channel in recent communications and provide long-term statistical basis for risk assessment. The collision level is obtained by using the number of historical communication selections and collisions, which specifically quantifies the degree of communication failure caused by interference or node occupation, complementing the communication failure rate and enhancing the ability to identify interference sources. The interference duration is calculated by using the duration of historical continuous interference and combining it with the maximum truncation value, which can capture the persistence characteristics of channel interference and avoid the impact of excessively large values ​​on the stability of the entropy weight method. The risk of sudden interference is predicted by using the ratio of long-term baseline energy to current smoothed energy, which can detect sudden abnormal increases in channel energy in advance, make up for the lag of relying solely on historical statistics, and enable risk assessment to have the ability to warn of transient interference.

[0027] Furthermore, a method for anti-jamming communication of UAV swarms based on deep reinforcement learning, wherein the process of using the entropy weight method to process the risk indexes within the risk index vector of each candidate channel to obtain the weight of each risk index at the current time includes the following sub-steps:

[0028] Traverse each risk index of the candidate channels to obtain the feature weight of the i-th candidate channel under the q-th risk index;

[0029] By iterating through the feature weights of each candidate channel under the q-th risk indicator, the information entropy of the q-th risk indicator is obtained;

[0030] Iterate through the information entropy of each risk indicator to obtain the weight of the q-th risk indicator;

[0031] Where q is the risk indicator index inside the risk indicator vector.

[0032] In the above scheme, by traversing each risk indicator and calculating the feature weight of each channel, the indicator values ​​of different dimensions can be normalized, laying the foundation for entropy weight calculation. Then, by traversing the feature weight of each channel to obtain information entropy, the dispersion of each indicator in different channels can be quantified. The more obvious the difference between the indicators, the smaller the information entropy, and vice versa. Finally, based on the information entropy, the entropy weight of each indicator is obtained, so that risk indicators with significant differences occupy a higher weight in the comprehensive risk, and the weight of indicators with insignificant differences is reduced accordingly.

[0033] Furthermore, a method for anti-jamming communication of UAV swarms based on deep reinforcement learning, wherein the comprehensive risk segmentation of all ranked candidate channels is obtained by using quantiles, and the candidate channels are divided into spectrum pools using the dynamic thresholds, wherein the spectrum pools include a high-priority spectrum pool, a normal spectrum pool, and a high-risk spectrum pool, includes the following sub-steps:

[0034] Arrange the overall risks of all candidate channels at the current moment in ascending order;

[0035] The comprehensive risk set is divided using quantiles to obtain the low-risk dynamic threshold and high-risk dynamic threshold at the current moment;

[0036] Based on the comprehensive risk of the i-th candidate channel and the comparison between the low-risk dynamic threshold and the high-risk dynamic threshold, the i-th candidate channel is assigned to the spectrum pool.

[0037] In the above scheme, all candidate channels are sorted by their overall risk from lowest to highest, avoiding interference from the original order in threshold selection. Then, quantiles are used to obtain low-risk and high-risk dynamic thresholds, ensuring that the spectrum pool boundary is entirely determined by the risk distribution of all channels at the current moment, without relying on empirically set fixed thresholds. This effectively avoids the problem of mismatch in subjective thresholds when the interference environment changes drastically. Finally, based on the comparison between the overall risk of each channel and the two dynamic thresholds, channels are adaptively assigned to high-priority, normal, or high-risk spectrum pools. Thus, the spectrum pool can be adjusted in real time according to changes in the interference environment and channel quality distribution, ensuring that high-priority services have sufficient low-risk channel resources while achieving dynamic quality-based grading of spectrum resources, improving adaptability and spectrum utilization efficiency in complex electromagnetic environments.

[0038] Furthermore, a method for anti-jamming communication of UAV swarms based on deep reinforcement learning, wherein obtaining the number of time slots allocated to each type of service, wherein the services include high-priority services, ordinary-priority services, and low-priority services, and processing low-priority service instances within low-priority services that are at risk of starvation according to an anti-starvation mechanism, thereby increasing the number of time slots allocated to low-priority service instances and the access probability of different spectrum pools, and obtaining the candidate channel set of low-priority service instances includes the following sub-steps:

[0039] Based on the preset priority weights of each type of service, obtain the number of time slots allocated to each type of service within a single superframe;

[0040] Record the number of consecutive communication failures for each low-priority service instance;

[0041] Based on the failure count threshold and the number of consecutive communication failures of low-priority service instances, determine whether low-priority service instances are at risk of starvation.

[0042] Based on the preset priority weights of each type of service, obtain the number of time slots allocated to each type of service within a single superframe;

[0043] Record the number of consecutive communication failures for each low-priority service instance;

[0044] Based on the failure count threshold and the number of consecutive communication failures of low-priority service instances, determine whether low-priority service instances are at risk of starvation.

[0045] like If the p-th low-priority business instance is at risk of starvation, the anti-starvation compensation mechanism will be triggered.

[0046] like If the p-th low-priority service instance does not have a starvation risk, the anti-starvation compensation mechanism will not be triggered.

[0047] Temporarily compensate the service weight of the p-th low-priority service instance to obtain a dynamic service weight;

[0048] Increase the probability that the p-th low-priority service instance accesses the high-priority spectrum pool;

[0049] Renormalize the probability that the p-th low-priority service instance accesses the high-priority spectrum pool, the normal spectrum pool, and the high-risk spectrum pool;

[0050] Based on the normalized probability and random number of the p-th low-priority service instance accessing different spectrum pools, obtain the candidate channel set of the p-th low-priority service instance at the current time;

[0051] in, The threshold for the number of failures. Let p be the number of consecutive communication failures of the p-th low-priority service instance at time t, where p is the index of the low-priority service instance.

[0052] In the above scheme, the number of time slots within a superframe is allocated to high-priority, normal-priority, and low-priority services based on the preset priority weights of each service type. This allows high-priority services to obtain more communication opportunities, ensuring the communication reliability of critical services in the time domain. By recording the number of consecutive communication failures of each low-priority service instance and comparing it with a failure threshold, service instances at risk of starvation can be accurately identified, avoiding indiscriminate compensation for all low-priority services. For low-priority service instances at risk of starvation, their service weight is temporarily increased, thereby increasing the number of time slots allocated to them within the superframe and increasing their probability of accessing the high-priority spectrum pool. After normalization, a candidate channel set is generated using random numbers. This anti-starvation compensation mechanism allows low-priority services that have experienced long-term failures to obtain more transmission opportunities and have the chance to use higher-quality channels. This avoids the service being blocked for a long time due to consecutive failures and does not force the allocation of specific channels, thus compromising the autonomous decision-making capability of the deep Q network. Therefore, without compromising the performance of high-priority services, it effectively improves long-term fairness and overall communication efficiency.

[0053] Furthermore, a method for anti-interference communication of UAV swarms based on deep reinforcement learning is provided, wherein the candidate channel set of the p-th low-priority service instance at the current time is obtained based on the normalized probability and random number of the p-th low-priority service instance accessing different spectrum pools, and the formula is as follows;

[0054] ;

[0055] in, Let be the candidate channel set for the p-th low-priority service instance at time t. Let be the set of candidate channels in the high-priority spectrum pool at time t. Let r be the probability that the p-th low-priority service instance accesses the high-priority spectrum pool after normalization at time t, where r is a random number. , Let be the set of candidate channels in the ordinary spectrum pool at time t. Let be the probability that the p-th low-priority service instance accesses the ordinary spectrum pool after normalization at time t. Let be the set of candidate channels in the high-risk spectrum pool at time t. If, For the i-th candidate channel, The spectrum pool is defined for the i-th candidate channel at time t. For high-priority spectrum pool, For ordinary spectrum pools, This is a high-risk spectrum pool.

[0056] In the above scheme, by introducing random numbers and comparing them with the normalized accumulated access probabilities, low-priority service instances probabilistically select one from the high-priority, ordinary, or high-risk spectrum pools as a candidate channel set for the current time slot, rather than directly specifying a specific channel. This transforms anti-starvation compensation into an enhancement of access opportunities to high-quality spectrum pools, rather than a forced replacement of the deep Q-network's decision-making. This effectively avoids fixed preferences or channel congestion caused by deterministic rules. The use of random numbers ensures the objectivity and repeatability of probabilistic selection, allowing the candidate channel set to differ across time slots. This enhances the exploration diversity during the deep Q-network training process, enabling it to learn the optimal selection strategy from channel sets of varying quality, further improving the agent's adaptability to dynamic interference environments. While temporarily increasing the chances of low-priority services obtaining good channels, the anti-starvation compensation mechanism retains the flexibility of the deep Q-network's autonomous decision-making, achieving effective synergy between the anti-starvation compensation mechanism and the reinforcement learning objective.

[0057] Furthermore, a method for anti-interference communication of UAV swarms based on deep reinforcement learning, wherein the UAV swarm leader is used as an agent, and the predicted Q value of the agent is obtained by learning with a deep Q-network based on the state space, action space and immediate reward function, includes the following sub-steps;

[0058] Based on the smoothed sensing energy, comprehensive risk, communication failure rate of each candidate channel, and the service status of the current service instance, construct the state space at the current moment.

[0059] Based on the candidate channel set and selected action for the current service instance, obtain the action space;

[0060] Construct an instant reward function for the current moment based on the communication success flag, collision flag, channel comprehensive risk, and fairness compensation term;

[0061] A deep Q-network is constructed based on the main network and the target network. The state space is then input into the deep Q-network after standardization to obtain the predicted Q-value of the agent at the current moment.

[0062] In the above scheme, a state space is constructed based on the smooth sensing energy, comprehensive risk, communication failure rate of each candidate channel, and the service status of the current service instance. This enables the deep Q network to simultaneously perceive the channel quality distribution and service demand characteristics, providing rich and multi-dimensional input information for intelligent decision-making. By constraining the action space through the candidate channel set of the current service instance, high-priority services select channels only from the high-priority spectrum pool, ordinary services select channels only from the ordinary pool, and low-priority services at risk of starvation access different pools according to probability. This achieves effective matching between service priority and channel quality, avoiding conflicts such as low-priority services blindly occupying high-quality channels or high-priority services having no channels available. An instant reward function is constructed based on communication success flags, collision flags, comprehensive channel risk, and fairness compensation terms. The fairness compensation term only takes effect when low-priority services fail continuously for more than a threshold. This enables the deep Q network to perceive the fairness requirements of long-term failed services during the learning process, guiding the strategy to optimize in a direction that balances the reliability of high-priority services with the fairness of low-priority services. A dual structure of master network and target network is used to construct the deep Q network. The standardized state space is input into the network to obtain the predicted Q value, effectively alleviating the value function oscillation problem during training and improving learning stability and convergence speed.

[0063] Furthermore, a method for anti-jamming communication in UAV swarms based on deep reinforcement learning, wherein iterative training of a deep Q-network to enable the agent to learn the optimal channel selection strategy based on counterfactual samples and real samples includes the following sub-steps:

[0064] Initialize the main network parameters, target network parameters, experience replay pool, random exploration probability, learning rate, discount factor, and maximum number of training epochs;

[0065] intelligent agents with - Greedy strategy selects actions: with 1- The probability is used to select the action with the largest predicted Q value from the candidate channel set of the current time-deep Q-network and the b-th service instance, so as to... The probability is that an action is randomly selected from the candidate channel set of the b-th service instance for exploration.

[0066] Based on the action, the communication is performed on the channel selected from the candidate channel set, and the state space of the next time step, the communication success flag, the collision flag, and the immediate reward function are obtained.

[0067] Based on whether the b-th business instance has implemented the anti-starvation compensation mechanism, determine whether to generate counterfactual actions and counterfactual rewards;

[0068] If the b-th business instance implements an anti-starvation compensation mechanism, then the real sample experience at the current moment will be used. and counterfactual sample experience Store the data in the experience replay pool D. If the b-th business instance does not have an anti-starvation compensation mechanism, then the real sample experience at the current moment will be stored in the pool. The data is stored in the experience replay pool D to complete one experience sample collection; the experience samples include real samples and counterfactual samples.

[0069] The experience sample collection is performed in a loop, and the experience playback pool is updated as needed. When the number of samples in the pool reaches the sample capacity, B empirical samples are randomly sampled from the empirical replay pool D, and the B empirical samples are used to form an offline training batch.

[0070] For each empirical sample in the offline training batch, the target Q value is calculated through the target network;

[0071] The mean squared error loss is calculated by using the predicted Q value and the target Q value;

[0072] The deep Q-network is trained iteratively based on the mean squared error loss until the training converges or the maximum number of training rounds is reached.

[0073] in, is the probability of random exploration, and b is the index of the business instance.

[0074] In the above scheme, a stable and controllable basic environment for deep Q-network training is established by initializing hyperparameters such as main network and target network parameters, experience replay pool, and exploration probability. The agent adopts a greedy strategy to select actions, balancing the use of existing knowledge with the exploration of unknown channels, and avoiding getting trapped in local optima. The next state, communication success flag, collision flag, and immediate reward are obtained based on the actual communication results, providing realistic environmental feedback for experience samples. When a business instance triggers the anti-starvation compensation mechanism, additional counterfactual actions and their counterfactual rewards are generated, and real samples and counterfactual samples are stored together in the experience replay pool, effectively alleviating the conflict between the anti-starvation scheduling rules and the learning objective of the deep Q-network, while expanding the diversity of training data. By cyclically collecting experience samples and randomly sampling offline training batches after the sample size reaches capacity, the temporal correlation between continuous samples is broken, improving training stability. The target Q-value is calculated using the target network, and the main network parameters are iteratively updated using the mean squared error loss between the predicted Q-value and the target Q-value. The main network parameters are periodically copied to the target network, which significantly suppresses value function oscillations during training and accelerates convergence.

[0075] Furthermore, a method for anti-interference communication of UAV swarms based on deep reinforcement learning, wherein determining whether to generate counterfactual actions and counterfactual rewards based on whether the b-th service instance has implemented an anti-starvation compensation mechanism includes the following sub-steps:

[0076] If the b-th service instance implements an anti-starvation compensation mechanism, then a counterfactual action is randomly selected from the candidate channel set for removing the selected channel of the b-th service instance. Based on a comparison between the overall risk of a counterfactual action and a low-risk dynamic threshold, the success of the counterfactual action in communication is estimated, and the judgment criteria are as follows:

[0077] like Then it is considered a counterfactual action. Communication successful;

[0078] like Then it is considered a counterfactual action. Communication failed;

[0079] Construct the counterfactual reward for the current moment based on the communication success flag, collision flag, and comprehensive risk and fairness compensation items for the counterfactual action;

[0080] in, For the counterfactual action at time t, The combined risk of counterfactual events at time t. The low-risk dynamic threshold is defined at time t.

[0081] In the above scheme, when a service instance triggers the anti-starvation compensation mechanism, an action is randomly selected from the candidate channel set after removing the selected channel for the current service instance as a counterfactual action. This avoids the bias that may be caused by manual selection and ensures the objectivity and diversity of counterfactual samples. The success or failure of communication is predicted based on the comparison between the comprehensive risk of the counterfactual action and the low-risk dynamic threshold. Virtual communication results can be obtained without actually executing the communication, significantly reducing the cost of acquiring counterfactual samples. A counterfactual reward is constructed based on the predicted communication success flag, collision flag, comprehensive risk, and fairness compensation item of the counterfactual action, ensuring that fairness considerations for long-term failed services are also incorporated into the counterfactual samples. Introducing counterfactual samples provides additional training data for the deep Q network without changing the real-world interaction environment, alleviating the conflict between the anti-starvation compensation rules and the learning objective of the deep Q network. It also enhances the perception of fairness requirements for low-priority services, improves the utilization efficiency of training samples, and enhances the generalization performance of the strategy. Attached Figure Description

[0082] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a flowchart of a deep reinforcement learning-based anti-jamming communication method for drone swarms.

[0084] Figure 2 This is a comparison curve showing the success rate of business communication between this method and the comparison method.

[0085] Figure 3 This is a comparison curve showing the overall communication success rate of our method and the comparative method.

[0086] Figure 4 This is a graph comparing the average reward values ​​of our method and the comparative methods.

[0087] Figure 5 This is a comparison curve between our method and the communication collision rate. Detailed Implementation

[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0089] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between elements or an indirect connection via other elements.

[0090] This invention is achieved through the following technical solutions, such as... Figure 1 As shown, a method for anti-jamming communication in UAV swarms based on deep reinforcement learning includes the following steps:

[0091] S1: The cluster-head UAV obtains the risk index vector of each candidate channel, uses the entropy weight method to obtain the weight of each risk index, and obtains the comprehensive risk of each candidate channel through the weight of each risk index and the risk index.

[0092] S11: The drone swarm uses the cluster leader drone to perform spectrum sensing on the candidate channel set in super-frames to obtain the smoothed channel energy of each candidate channel.

[0093] The drone swarm consists of a single leader drone and several follower drones;

[0094] The cluster-head UAV is used for spectrum environment perception, interference risk assessment, spectrum resource classification, time slot scheduling, and communication decision-making.

[0095] The follower drone is used to perform communication tasks based on the allocation results of the cluster leader drone.

[0096] The candidate channel set is as follows: ;

[0097] in, For candidate channel set, For the i-th candidate channel, is the total number of candidate channels, and i is the index of the candidate channel;

[0098] The superframe consists of K consecutive time slots, as shown in the formula:

[0099] ;

[0100] in, For superframe, For the z-th time slot within the superframe, =1,2,3...K, where K is the total number of time slots within a superframe, and z is the time slot index. t is the current time slot number. Modulo operation;

[0101] S111: The cluster-head UAV performs spectrum sensing on the i-th candidate channel and obtains the sensing energy at the current moment, as shown in the formula:

[0102] ;

[0103] in, Let be the sensing energy of the i-th candidate channel at time t. Let be the energy of the legitimate communication information of the i-th candidate channel at time t. Let be the interference signal energy of the i-th candidate channel at time t. Let be the environmental noise energy of the i-th candidate channel at time t, where t is the current time slot number.

[0104] It should be noted that legitimate communication information energy refers to the communication information energy sent by this UAV cluster or other cooperative communication nodes that should be received normally.

[0105] S112: The sensing energy of the i-th candidate channel is smoothed using a sliding window to obtain the smoothed sensing energy at the current time, as shown in the formula:

[0106] ;

[0107] Among them, Let be the smoothed sensing energy of the i-th candidate channel at time t. The length of the sliding window. The time slot number within the sliding window. Let be the sensing energy of the i-th candidate channel at time k.

[0108] It is important to note that smoothing through a sliding window can reduce the impact of environmental noise energy on perceived energy.

[0109] S12: Normalize the communication failure rate, smoothed sensing energy, collision level, and interference duration. Integrate the normalized values ​​with the burst interference prediction risk to obtain the risk index vector for each candidate channel at the current moment. The formula is:

[0110] ;

[0111] in, Let be the risk index vector of the i-th candidate channel at time t. Let be the normalized, smoothed sensing energy of the i-th candidate channel at time t. Let be the normalized communication failure rate of the i-th candidate channel at time t. Let be the collision level of the i-th candidate channel at time t after normalization. Let this be the normalized interference duration of the i-th candidate channel at time t. The burst interference prediction risk for the i-th candidate channel at time t.

[0112] It should be noted that using Min-Max for normalization is an existing technique, and will not be elaborated on here.

[0113] S121: Calculate the communication failure rate using the historical number of communication selections and failures of the i-th candidate channel, using the following formula:

[0114] ;

[0115] in, Let be the communication failure rate of the i-th candidate channel at time t. The slot number within the historical window. The length of the history window (the number of time slots in the previous superframe). This is an indicator function; it takes a value of 1 when the i-th candidate channel is selected at time j and communication fails, and a value of 0 otherwise. This is an indicator function; it takes a value of 1 when the i-th candidate channel is selected at time j, and a value of 0 otherwise. It is a very small positive number.

[0116] In the embodiments, .

[0117] S122: The collision level is obtained by using the historical communication selection count and collision count of the i-th candidate channel, using the following formula:

[0118] ;

[0119] in, Let be the collision level of the i-th candidate channel at time t. This is an indicator function. It takes the value 1 when the i-th candidate channel is selected and a collision occurs at time j, and 0 otherwise.

[0120] S123: Utilizing the historical duration of continuous interference and the maximum truncation value of the interference duration for the i-th candidate channel, the interference duration is obtained using the following formula:

[0121] ;

[0122] in, Let be the duration of interference from the i-th candidate channel at time t. To obtain the minimum value, This represents the number of time slots in which the i-th candidate channel continuously satisfies the interference condition, tracing back from time t. This is the maximum truncation value for the interference duration. Let be the interference exponential function of the i-th candidate channel at time t, which takes a value of 1 when interference occurs and 0 otherwise. For indicator functions, when The value is 1 if the condition is met, and 0 otherwise. The interference threshold is set to twice the ambient noise energy.

[0123] It is important to note that the maximum truncation value for interference duration is used to limit the upper limit of the interference duration, preventing an excessively large value from affecting the stability of subsequent entropy weight method calculations. .

[0124] S124: Using the long-term baseline energy and smoothed sensing energy of the i-th candidate channel, the sudden interference prediction risk is obtained, as shown in the formula:

[0125] ;

[0126] in, This represents the long-term baseline energy up to time t-1. To find the maximum value, The smoothing factor (e.g., 0.01) , .

[0127] S13: Use the entropy weight method to process the risk indicators within the risk indicator vector of each candidate channel to obtain the weight of each risk indicator at the current time.

[0128] S131: Traverse each risk index of the candidate channels to obtain the feature weight of the i-th candidate channel under the q-th risk index, as shown in the formula:

[0129] ;

[0130] in, Let be the feature weight of the i-th candidate channel under the q-th risk indicator at time t. Here are the risk indicator indices within the risk indicator vector, q=1,2..5. Let q be the risk index inside the risk index vector of the i-th candidate channel at time t.

[0131] S132: Iterate through the feature weights of each candidate channel under the q-th risk indicator to obtain the information entropy of the q-th risk indicator, as shown in the formula:

[0132] ;

[0133] in, Let be the information entropy of the q-th risk indicator at time t.

[0134] S133: Iterate through the information entropy of each risk indicator to obtain the weight of the q-th risk indicator, using the following formula:

[0135] ;

[0136] in, Let q be the weight of the q-th risk indicator at time t.

[0137] S14: Iterate through each risk indicator and its weight to obtain the comprehensive risk of each candidate channel at the current time. The formula is:

[0138] ;

[0139] in, Let be the comprehensive risk of the i-th candidate channel at time t.

[0140] It should be noted that for channels that are in a state of high energy, high collision, low success rate or continuous interference for a long time, the comprehensive risk value will increase with the observation results. In the subsequent deep Q network channel selection, the selection probability will be reduced in the action space, thereby improving the communication reliability of UAV swarms in dynamic interference environment.

[0141] S2: Use quantiles to divide the overall risk of all ranked candidate channels to obtain a dynamic threshold. Use the dynamic threshold to divide the candidate channels into spectrum pools, which include a high-priority spectrum pool, a normal spectrum pool, and a high-risk spectrum pool.

[0142] S21: Arrange the comprehensive risks of all candidate channels at the current moment in ascending order, using the following formula:

[0143] ;

[0144] in, The comprehensive risk set sorted at time t. The overall risk of ranking the first candidate channel at time t. The overall risk of sorting the last candidate channels at time t;

[0145] S22: Utilize quantiles to partition the comprehensive risk set, and obtain the low-risk dynamic threshold and high-risk dynamic threshold at the current moment. The formula is:

[0146] ;

[0147] in, The low-risk dynamic threshold at time t. The high-risk dynamic threshold at time t. It is a quantile function. Return the comprehensive risk set sorted at time t. No. Quantiles Return the sorted comprehensive risk set at time t. No. Quantiles For low-risk quantile parameters, For high-risk quantile parameters, .

[0148] In the embodiments, , .

[0149] S23: Based on the comprehensive risk of the i-th candidate channel and the comparison between the low-risk dynamic threshold and the high-risk dynamic threshold, the i-th candidate channel is assigned to the spectrum pool, as follows:

[0150] ;

[0151] in, The spectrum pool is defined for the i-th candidate channel at time t. For high-priority spectrum pool, For ordinary spectrum pools, This is a high-risk spectrum pool.

[0152] The candidate channels in the high-priority spectrum pool have lower overall risk and are given priority for use by high-priority services (including delay-sensitive services).

[0153] The overall risk of the candidate channels in the ordinary spectrum pool is at a moderate level, and they are used for regular communication.

[0154] Candidate channels in the high-risk spectrum pool have a high overall risk and are usually not given priority. They are only used as backup channels when there is insufficient candidate channel resources in the high-priority spectrum pool or the ordinary spectrum pool, service congestion, or special communication needs.

[0155] It is important to note that after each superframe ends, or when an increase in channel communication failure rate, collision level, sensed energy, or abnormal increase in interference duration is detected, the comprehensive risk value of each candidate channel will be recalculated, and the low-risk dynamic threshold and high-risk dynamic threshold will be updated. This dynamically updates the spectrum pool for candidate channel allocation. The spectrum pool can adaptively adjust with changes in the external electromagnetic interference environment and network communication status, thereby improving the system's anti-interference capability and spectrum utilization efficiency.

[0156] S3: Obtain the number of time slots allocated to each type of service, which includes high-priority services, ordinary-priority services and low-priority services. Based on the anti-starvation mechanism, process the low-priority service instances within the low-priority services that are at risk of starvation, increase the number of time slots allocated to the low-priority service instances and the access probability of different spectrum pools, and obtain the candidate channel set of the low-priority service instances.

[0157] S31: Based on the preset priority weights for each service type, obtain the number of time slots allocated to each service type within a single superframe, using the following formula:

[0158] ;

[0159] in, For the first The number of time slots allocated to a service type within a single superframe. For the first Priority weights for different business types For business indexing, ,when At that time, Category 1 services are high-priority services. At that time, the second type of service is a regular priority service. At that time, the third type of business is a low-priority business;

[0160] It is important to note that As a preset constant, .

[0161] S32: Records the number of consecutive communication failures for each low-priority service instance, using the following formula:

[0162] ;

[0163] in, Let p be the number of consecutive communication failures of the p-th low-priority service instance at time t+1, where p is the index of the low-priority service instance within the low-priority service. This is the communication failure indicator for the p-th low-priority service instance at time t. It is 1 if it fails and 0 otherwise.

[0164] S33: Based on the failure count threshold and the number of consecutive communication failures of low-priority service instances, determine whether the low-priority service instances are at risk of starvation. The determination criteria are as follows:

[0165] like If the p-th low-priority business instance is at risk of starvation, the anti-starvation compensation mechanism will be triggered.

[0166] like If the p-th low-priority service instance does not have a starvation risk, the anti-starvation compensation mechanism will not be triggered.

[0167] in, This is the threshold for the number of failures.

[0168] It is important to note that, for scenarios involving prolonged communication failures, the anti-starvation mechanism provides temporary resource compensation for low-priority service instances during prolonged communication failures by adjusting the number of consecutive communication failures, dynamic service weight compensation, spectrum pool access probability, and counterfactual reward correction. The anti-starvation mechanism only adjusts the service weight and candidate action set within each service type and does not directly replace the channel selection decision of the deep Q network.

[0169] S34: Temporarily compensate the service weight of the p-th low-priority service instance to obtain a dynamic service weight, using the following formula:

[0170] ;

[0171] in, Let be the dynamic service weight of the p-th low-priority service instance at time t, and 1 be the initial service weight of the p-th low-priority service instance. Let t be the anti-starvation compensation weight.

[0172] It should be noted that the dynamic service weight only adjusts the service weight of the p-th low-priority service instance that is at risk of starvation within the low-priority service. When the p-th low-priority service instance successfully communicates, the dynamic service weight is restored to the initial service weight of the p-th low-priority service instance. The initial service weight of each low-priority service instance is 1, while the priority weight of the low-priority service remains unchanged.

[0173] S35: Increase the probability that the p-th low-priority service instance accesses the high-priority spectrum pool, using the following formula:

[0174] ;

[0175] in, To increase the probability that the p-th low-priority service instance accesses the high-priority spectrum pool after time t. The probability that the p-th low-priority service instance accesses the high-priority spectrum pool. Adjust the access probability coefficient (e.g., 0.1);

[0176] S36: Renormalize the probability of the p-th low-priority service instance accessing the high-priority spectrum pool, the normal spectrum pool, and the high-risk spectrum pool.

[0177] It is important to note that the renormalization is intended to ensure that the sum of the probabilities of the p-th low-priority service instance accessing the high-priority spectrum pool, the normal spectrum pool, and the high-risk spectrum pool is 1. The anti-starvation compensation mechanism only changes the probability of the low-priority service instance itself accessing different spectrum pools, without directly specifying a specific channel.

[0178] S37: Based on the normalized probability of the p-th low-priority service instance accessing different spectrum pools and the random number, obtain the candidate channel set for the p-th low-priority service instance at the current time. The formula is:

[0179] ;

[0180] in, Let be the candidate channel set for the p-th low-priority service instance at time t. Let be the set of candidate channels in the high-priority spectrum pool at time t. Let r be the probability that the p-th low-priority service instance accesses the high-priority spectrum pool after normalization at time t, where r is a random number. , Let be the set of candidate channels in the ordinary spectrum pool at time t. Let be the probability that the p-th low-priority service instance accesses the ordinary spectrum pool after normalization at time t. Let t be the set of candidate channels in the high-risk spectrum pool at time t.

[0181] It is important to note that When a deep Q network processes the p-th low-priority service instance, it only does so within the candidate channel set. The candidate channel selection and anti-starvation compensation mechanism do not force the deep Q network to select a fixed channel; random numbers enable probabilistic access to different spectrum pools.

[0182] S4: Treat the cluster-headed drone as an intelligent agent, and use a deep Q-network to learn the predicted Q value of the agent based on the state space, action space and immediate reward function.

[0183] S41: Based on the smoothed sensing energy, comprehensive risk, communication failure rate, and service status of each candidate channel, construct the state space for the current moment, using the following formula:

[0184] ;

[0185] in, Let be the state space at time t. Let b be the business status of the b-th business instance at time t, where b is the index of the business instance.

[0186] The service status includes at least one of the following: service type, queue status, latency requirements, and number of consecutive communication failures.

[0187] S42: Based on the candidate channel set and selected action for the current service instance, obtain the action space, as shown in the formula:

[0188] ;

[0189] in, Let be the set of candidate channels for the b-th service instance at time t. Select an action at time t (i.e., select the i-th candidate channel at time t). Let t be the action space at time t.

[0190] Specifically, when the business type of a business instance is a high-priority business (i.e., a high-priority business instance). When the business type of the business instance is a normal priority business, When the business instance is a low-priority business instance that is at risk of starvation, When the business instance is a low-priority business instance that does not face the risk of starvation, .

[0191] S43: Construct the instantaneous reward function for the current moment based on the communication success flag, collision flag, channel integrated risk, and fairness compensation term. The formula is as follows:

[0192] ;

[0193] in, Let be the instantaneous reward function at time t. The reward weight is assigned to indicate successful communication. Let t be the communication success flag. It is 1 if the selected candidate channel successfully communicates, and 0 otherwise. The collision marker reward weight, This is the collision flag at time t. It is 1 when a collision occurs in the selected candidate channel, and 0 otherwise. For channel-wide risk weights, The overall risk of choosing an action at time t. To ensure fairness, the reward weighting is adjusted. Let b be the fairness compensation reward for the b-th business instance at time t. This represents the number of consecutive communication failures for the b-th service instance.

[0194] It should be noted that the fairness compensation term serves only as an auxiliary reward signal for the deep Q network and does not directly replace the channel selection result of the deep Q network.

[0195] S44: Construct a deep Q-network based on the main network and the target network. Input the state space into the deep Q-network after standardization to obtain the predicted Q value of the agent at the current time.

[0196] The main network and the target network have the same structure, both including an input layer, a hidden layer, and an output layer; the hidden layer includes a first hidden layer and a second hidden layer.

[0197] The processing steps of the main network are as follows:

[0198] The standardized state space is input into the input layer. Features extracted from the input layer are then output to the first hidden layer. The first hidden layer uses the ReLU function to activate features, alleviating the gradient vanishing problem during deep Q-network training and enhancing the non-linear fitting ability of the deep Q-network, enabling it to learn the complex mapping relationship between states and actions. The activated features from the first hidden layer are then input to the second hidden layer, which uses the ReLU function to activate features. By reducing the number of neurons, this compresses and refines features, removes redundant features, improves the generalization ability of the deep Q-network, and avoids overfitting. The reactivated features from the second hidden layer are then output to the output layer, which outputs the agent's predicted Q-value. ;

[0199] in, To predict the Q-value (state space at time t) Action at time t in the next execution action space (Expected cumulative rewards).

[0200] S5: Iteratively train the deep Q-network to enable the agent to learn the optimal channel selection strategy based on counterfactual samples and real samples.

[0201] S51: Initialize the main network parameters, target network parameters, experience replay pool, random exploration probability, learning rate, discount factor, and maximum number of training epochs;

[0202] S52: Intelligent agents with - Greedy strategy selects actions: with 1- The probability is used to select the action with the largest predicted Q value from the candidate channel set of the current time-deep Q-network and the b-th service instance, so as to... The probability of randomly selecting an action from the candidate channel set for the b-th service instance is given by the formula:

[0203] ;

[0204] in, To obtain Maximum value action , Main network parameters, These are dummy variables, representing iteration. Each action in the middle, For random exploration probability, Uniformly random numbers, , From One action is randomly selected evenly from the middle.

[0205] It should be noted that this can be understood as 1- The probability is determined to prioritize the channel allocation action that currently yields the highest estimated benefit. The probability is to avoid the greedy strategy getting trapped in a local optimum.

[0206] S53: Execute communication based on the channel selected from the candidate channel set according to the action, and obtain the state space of the next time step, the communication success flag of the current time step, the collision flag, and the immediate reward function;

[0207] S54: Based on whether the anti-starvation compensation mechanism has been implemented for the b-th business instance, determine whether to generate counterfactual actions and counterfactual rewards.

[0208] S541: If the b-th service instance implements an anti-starvation compensation mechanism, then a counterfactual action is randomly selected from the candidate channel set for removing the selected channel of the b-th service instance. Based on a comparison between the overall risk of a counterfactual action and a low-risk dynamic threshold, the success of the counterfactual action in communication is estimated, and the judgment criteria are as follows:

[0209] like Then it is considered a counterfactual action. Communication successful;

[0210] like Then it is considered a counterfactual action. Communication failed;

[0211] in, For the counterfactual action at time t, The combined risk of counterfactual events at time t.

[0212] S542: Construct the counterfactual reward for the current moment based on the communication success flag, collision flag, and comprehensive risk and fairness compensation item of the counterfactual action, using the following formula:

[0213] ;

[0214] in, For counterfactual rewards at time t, Let t be the flag indicating successful counterfactual action communication. The flag is 1 if counterfactual action communication is successful, and 0 otherwise. This is the collision flag for counterfactual actions at time t. It is 1 when a counterfactual action collides, and 0 otherwise (in counterfactual cases, it is assumed that a collision occurs if the action does not succeed).

[0215] S55: If the b-th business instance has implemented an anti-starvation compensation mechanism, then the real sample experience at the current moment will be used. and counterfactual sample experience Store the data in the experience replay pool D. If the b-th business instance does not have an anti-starvation compensation mechanism, then the real sample experience at the current moment will be stored in the pool. The data is stored in the experience replay pool D to complete one experience sample collection; the experience samples include real samples and counterfactual samples.

[0216] in, , .

[0217] S56: Loop through experience sample collection, when the experience playback pool... When the number of samples in the pool reaches the sample capacity, B empirical samples are randomly sampled from the empirical replay pool D, and these B empirical samples constitute the offline training batch.

[0218] S57: For each empirical sample in the offline training batch, calculate the target Q-value using the target network, with the following formula:

[0219] ;

[0220] in, Let Q be the target Q value at time t. As a discount factor, The predicted Q-value of the target network. These are dummy variables, representing iteration. Each action in the middle, These are the target network parameters.

[0221] It is important to note that, in order to avoid conflict between the anti-starvation compensation mechanism and the learning objectives of the deep Q network, a counterfactual reward correction is introduced. This allows the long-term failure state of low-priority services to participate in the deep Q network policy learning, while not changing the basic decision-making method of the deep Q network in channel selection within a single time slot.

[0222] S58: Calculate the mean squared error loss by using the predicted Q-value and the target Q-value, using the following formula:

[0223] ;

[0224] Where B is the total number of empirical samples in the offline training batch. For mean square error loss, Let be the target Q value of the b-th empirical sample at time t. Let Q be the predicted Q value for the b-th empirical sample. Let be the state space of the b-th empirical sample at time t. Let b be the action of the b-th empirical sample at time t.

[0225] S59: Iteratively train the deep Q-network based on the mean squared error loss until training converges or the maximum number of training rounds is reached.

[0226] It is important to note that the main network parameters are periodically copied to the target network parameters to stabilize the training.

[0227] like Figure 2 As shown, the proposed method achieves higher communication success rates for high-priority services than both the energy-detection-based greedy method (Greedy-ED) and the ordinary deep Q-network method (Full-DQN), and is also more stable for low-priority services compared to the energy-detection-based greedy method.

[0228] like Figure 3-5 As shown, the present invention outperforms the fixed channel allocation method (FCA), random frequency hopping method (RFH), polling frequency hopping (RH), energy detection-based greedy method, and ordinary deep Q network method in terms of overall communication success rate, average reward value (average of cumulative reward), and communication collision rate.

[0229] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for anti-interference communication of unmanned aerial vehicle (UAV) swarms based on deep reinforcement learning, characterized in that, Includes the following sub-steps: The cluster-head UAV obtains the risk index vector of each candidate channel, uses the entropy weight method to obtain the weight of each risk index, and obtains the comprehensive risk of each candidate channel through the weight of each risk index and the risk index. The comprehensive risk of all ranked candidate channels is divided using quantiles to obtain a dynamic threshold. The candidate channels are then divided into spectrum pools using the dynamic threshold. The spectrum pools include a high-priority spectrum pool, a normal spectrum pool, and a high-risk spectrum pool. Obtain the number of time slots allocated to each type of service, which includes high-priority services, ordinary-priority services and low-priority services. Based on the anti-starvation mechanism, process low-priority service instances within low-priority services that are at risk of starvation, increase the number of time slots allocated to low-priority service instances and the access probability of different spectrum pools, and obtain the candidate channel set of low-priority service instances. The cluster-headed UAV is used as an intelligent agent. Based on the state space, action space and instantaneous reward function, a deep Q-network is used to learn the predicted Q value of the intelligent agent. The deep Q-network is iteratively trained so that the agent learns the optimal channel selection strategy based on counterfactual samples and real samples. The iterative training of the deep Q-network, enabling the agent to learn the optimal channel selection strategy based on counterfactual and real samples, includes the following sub-steps: Initialize the main network parameters, target network parameters, experience replay pool, random exploration probability, learning rate, discount factor, and maximum number of training epochs; intelligent agents with - Greedy strategy selects actions: with 1- The probability is used to select the action with the largest predicted Q value from the candidate channel set of the current time-deep Q-network and the b-th service instance, so as to... The probability is that an action is randomly selected from the candidate channel set of the b-th service instance for exploration. Based on the action, the communication is performed on the channel selected from the candidate channel set, and the state space of the next time step, the communication success flag, the collision flag, and the immediate reward function are obtained. Based on whether the b-th business instance has implemented the anti-starvation compensation mechanism, determine whether to generate counterfactual actions and counterfactual rewards; If the b-th business instance implements an anti-starvation compensation mechanism, then the real sample experience at the current moment will be used. and counterfactual sample experience Store the data in the experience replay pool D. If the b-th business instance does not have an anti-starvation compensation mechanism, then the real sample experience at the current moment will be stored in the pool. The data is stored in the experience replay pool D to complete one experience sample collection; the experience samples include real samples and counterfactual samples. The experience sample collection is performed in a loop, and the experience playback pool is updated as needed. When the number of samples in the pool reaches the sample capacity, B empirical samples are randomly sampled from the empirical replay pool D, and the B empirical samples are used to form an offline training batch. For each empirical sample in the offline training batch, the target Q value is calculated through the target network; The mean squared error loss is calculated by using the predicted Q value and the target Q value; The deep Q-network is trained iteratively based on the mean squared error loss until the training converges or the maximum number of training rounds is reached. in, is the probability of random exploration, and b is the index of the business instance; The step of determining whether to generate a counterfactual action and counterfactual reward based on whether the b-th business instance has implemented an anti-starvation compensation mechanism includes the following sub-steps: If the b-th service instance implements an anti-starvation compensation mechanism, then a counterfactual action is randomly selected from the candidate channel set for removing the selected channel of the b-th service instance. Based on a comparison between the overall risk of a counterfactual action and a low-risk dynamic threshold, the success of the counterfactual action in communication is estimated, and the judgment criteria are as follows: like Then it is considered a counterfactual action. Communication successful; like Then it is considered a counterfactual action. Communication failed; Construct the counterfactual reward for the current moment based on the communication success flag, collision flag, and comprehensive risk and fairness compensation items for the counterfactual action; in, For the counterfactual action at time t, The combined risk of counterfactual events at time t. The low-risk dynamic threshold is defined at time t.

2. The anti-interference communication method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, The cluster-head UAV acquires the risk index vector for each candidate channel, uses the entropy weight method to obtain the weight of each risk index, and obtains the comprehensive risk of each candidate channel through the weight and risk index of each risk index, including the following sub-steps: The drone swarm uses the cluster leader drone to perform spectrum sensing on the candidate channel set at super-frame intervals to obtain the smoothed channel energy of each candidate channel; The communication failure rate, smoothed sensing energy, collision level, and interference duration are normalized respectively. The normalized data are then integrated with the sudden interference prediction risk to obtain the risk index vector of each candidate channel at the current moment. The entropy weight method is used to process the risk indicators within the risk indicator vector of each candidate channel to obtain the weight of each risk indicator at the current time. Iterate through each risk indicator and its weight to obtain the comprehensive risk of each candidate channel at the current moment.

3. The anti-interference communication method for UAV swarms based on deep reinforcement learning according to claim 2, characterized in that, The process of normalizing the communication failure rate, smoothed sensing energy, collision level, and interference duration, and then integrating the normalized data with the burst interference prediction risk to obtain the risk index vector for each candidate channel at the current moment includes the following sub-steps: The communication failure rate is obtained by using the historical number of communication selections and the number of communication selection failures of the i-th candidate channel. The collision level is obtained by using the historical communication selection count and collision count of the i-th candidate channel; The duration of interference is obtained by using the historical continuous interference duration and the maximum truncation value of the interference duration of the i-th candidate channel; By utilizing the long-term baseline energy and smoothed sensing energy of the i-th candidate channel, the risk of sudden interference prediction is obtained; Where i is the candidate channel index.

4. The anti-interference communication method for UAV swarms based on deep reinforcement learning according to claim 2, characterized in that, The process of using the entropy weight method to process the risk indexes within the risk index vector of each candidate channel to obtain the weight of each risk index at the current time includes the following sub-steps: Traverse each risk index of the candidate channels to obtain the feature weight of the i-th candidate channel under the q-th risk index; By iterating through the feature weights of each candidate channel under the q-th risk indicator, the information entropy of the q-th risk indicator is obtained; Iterate through the information entropy of each risk indicator to obtain the weight of the q-th risk indicator; Where q is the risk indicator index inside the risk indicator vector.

5. The anti-interference communication method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, The step of using quantiles to segment the overall risk of all ranked candidate channels to obtain a dynamic threshold, and then using the dynamic threshold to allocate candidate channels to spectrum pools, which include a high-priority spectrum pool, a normal spectrum pool, and a high-risk spectrum pool, includes the following sub-steps: Arrange the overall risks of all candidate channels at the current moment in ascending order; The comprehensive risk set is divided using quantiles to obtain the low-risk dynamic threshold and high-risk dynamic threshold at the current moment; Based on the comprehensive risk of the i-th candidate channel and the comparison between the low-risk dynamic threshold and the high-risk dynamic threshold, the i-th candidate channel is assigned to the spectrum pool.

6. The anti-interference communication method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, The step of obtaining the number of time slots allocated to each type of service, including high-priority services, normal-priority services, and low-priority services, and processing low-priority service instances within low-priority services that are at risk of starvation according to the anti-starvation mechanism, thereby increasing the number of time slots allocated to low-priority service instances and the access probability of different spectrum pools, and obtaining the candidate channel set of low-priority service instances includes the following sub-steps: Based on the preset priority weights of each type of service, obtain the number of time slots allocated to each type of service within a single superframe; Record the number of consecutive communication failures for each low-priority service instance; Based on the failure count threshold and the number of consecutive communication failures of low-priority service instances, determine whether low-priority service instances are at risk of starvation. like If the p-th low-priority business instance is at risk of starvation, the anti-starvation compensation mechanism will be triggered. like If the p-th low-priority service instance does not have a starvation risk, the anti-starvation compensation mechanism will not be triggered. Temporarily compensate the service weight of the p-th low-priority service instance to obtain a dynamic service weight; Increase the probability that the p-th low-priority service instance accesses the high-priority spectrum pool; Renormalize the probability that the p-th low-priority service instance accesses the high-priority spectrum pool, the normal spectrum pool, and the high-risk spectrum pool; Based on the normalized probability and random number of the p-th low-priority service instance accessing different spectrum pools, obtain the candidate channel set of the p-th low-priority service instance at the current time; in, The threshold for the number of failures. Let p be the number of consecutive communication failures of the p-th low-priority service instance at time t, where p is the index of the low-priority service instance.

7. The anti-interference communication method for UAV swarms based on deep reinforcement learning according to claim 6, characterized in that, The candidate channel set for the p-th low-priority service instance at the current time is obtained based on the probability and random number of the p-th low-priority service instance accessing different spectrum pools after normalization, as shown in the formula below. ； in, Let be the candidate channel set for the p-th low-priority service instance at time t. Let be the set of candidate channels in the high-priority spectrum pool at time t. Let r be the probability that the p-th low-priority service instance accesses the high-priority spectrum pool after normalization at time t, where r is a random number. , Let be the set of candidate channels in the ordinary spectrum pool at time t. Let be the probability that the p-th low-priority service instance accesses the ordinary spectrum pool after normalization at time t. Let be the set of candidate channels in the high-risk spectrum pool at time t. If, For the i-th candidate channel, The spectrum pool is defined for the i-th candidate channel at time t. For high-priority spectrum pool, For ordinary spectrum pools, This is a high-risk spectrum pool.

8. The anti-interference communication method for UAV swarms based on deep reinforcement learning according to claim 1, characterized in that, The step of using the cluster-head UAV as an intelligent agent and learning the predicted Q value of the intelligent agent based on the state space, action space and instant reward function using a deep Q network includes the following sub-steps; Based on the smoothed sensing energy, comprehensive risk, communication failure rate of each candidate channel, and the service status of the current service instance, construct the state space at the current moment. Based on the candidate channel set and selected action for the current service instance, obtain the action space; Construct an instant reward function for the current moment based on the communication success flag, collision flag, channel comprehensive risk, and fairness compensation term; A deep Q-network is constructed based on the main network and the target network. The state space is then input into the deep Q-network after standardization to obtain the predicted Q-value of the agent at the current moment.

Images