Distributed intelligent decision-making method, system and device for unmanned aerial vehicle group, and medium

Abstract

The invention relates to the technical field of unmanned aerial vehicle communication, and discloses a distributed intelligent decision-making method, system and device for an unmanned aerial vehicle group, and a medium. The method comprises the following steps: constructing a distributed intelligent decision model of an unmanned aerial vehicle group as a global model based on federated learning, and distributing the global model to each unmanned aerial vehicle in the unmanned aerial vehicle group; each unmanned aerial vehicle divides the DQN into two layers based on hierarchical reinforcement learning, selects an optimal communication channel through the first layer of DQN, and determines a transmitting power and a coding strategy when the unmanned aerial vehicle adopts the optimal communication channel for communication through the second layer of DQN, so as to carry out local training on the distributed global model and generate a local model; aggregating the local models obtained by training all the unmanned aerial vehicles so as to update the global model; and the global model after multiple rounds of federated learning iteration updating is obtained, distributed intelligent decision making of the unmanned aerial vehicle group is performed through the global model after iteration updating, and the anti-interference effect and the data privacy safety performance are relatively high.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) communication, and particularly to a distributed intelligent decision-making method, system, device and medium for a UAV swarm. Background Art

[0002] Compared with single-UAV communication, the cooperative communication of a UAV swarm demonstrates significant advantages in terms of task execution ability, efficiency, and robustness in complex environments. In a UAV swarm system, through cooperation, UAVs can share information, allocate tasks, and coordinate actions, enabling the entire system to handle more complex tasks that may be difficult or inefficient for a single UAV to complete. Although UAV swarm systems have shown remarkable capabilities and advantages in performing cooperative tasks, the environment they face during communication is more complex and more vulnerable to external malicious communication interference, which may disrupt the information exchange between UAVs, steal important information from us, and affect the efficiency and security of task execution.

[0003] In the communication system of a UAV swarm, a cluster-based architecture is usually adopted to organize UAVs. Each UAV node within a cluster can communicate with each other and share information to cooperate in completing specific tasks. However, the cluster-based communication architecture also introduces some challenges and limitations. First, since the information exchange between UAVs needs to be carried out through wireless channels, communication is vulnerable to external interference and eavesdropping, and it is difficult to ensure the security and privacy of information. Second, in a UAV swarm communication system, the receiver needs to scan each channel to determine which channel the transmitter is sending information on. This process not only consumes time but also consumes the energy of the receiver. Finally, the specific environmental conditions of each UAV are different, resulting in different levels of interference they face. Even for the same interference source, the interference intensity and arrival time felt by different UAVs may be different, thus affecting the timing and strategy of UAVs taking effective anti-interference measures.

[0004] In the prior art, an anti-interference communication algorithm based on cooperative multi-agent layered Q learning (MALQL) is used to solve the communication anti-interference problem. Among them, a layered QL architecture is adopted to divide the UAV network into different levels (such as intra-cluster and inter-cluster), and each level has different task objectives, such as intra-cluster interference management and inter-cluster resource coordination, thus simplifying the handling of complex problems; a cooperative multi-agent mechanism is adopted, and each UAV acts as an independent agent, sharing and optimizing strategies through cooperative reinforcement learning (Q-learning) to cope with the dynamic interference environment; by adaptively allocating spectrum resources and optimizing transmission power, the probability of communication conflicts and interference is significantly reduced, and the resource utilization rate is improved.

[0005] However, the above methods mainly optimize interference perception and resource allocation, and have limited effects on certain types of complex interference (such as advanced synchronization interference). For example, in the case of limited spectrum resources and complex and variable interference environments, a single anti-interference strategy cannot flexibly cope with all types of interference, especially for the compound interference situation with frequency interference, power interference, and coding interference existing simultaneously. At the same time, although the anti-interference algorithm and intelligent spectrum selection enhance the security of the communication link to a certain extent, privacy and security issues may be involved when multiple agents share information. This method has no special design in terms of privacy protection and security, and faces potential risks of malicious attacks. Summary of the Invention

[0006] The purpose of the present invention is to provide a distributed intelligent decision-making method, system, device, and medium for a drone swarm, which can ensure the data security of drones when collaborating to complete tasks, improve communication efficiency, realize the sharing of intelligent decisions among drones within a cluster, and enhance the anti-interference ability of the entire network.

[0007] To solve the above technical problems, an embodiment of the present invention provides a distributed intelligent decision-making method for a drone swarm, including the following steps: Based on federated learning, construct a distributed intelligent decision-making model for the drone swarm as a global model, and distribute the global model to each drone in the drone swarm; Each drone, based on hierarchical reinforcement learning, divides the deep Q-network (DQN) into two layers. Through the first-layer DQN, select a channel with the least interference from the available channels of the drone as the best communication channel, and through the second-layer DQN, minimize the transmission power when the drone communicates using the best communication channel, and select the coding strategy that maximizes the communication efficiency when the drone communicates using the best communication channel, so as to perform local training on the distributed global model and generate a local model; Aggregate the local models trained by all drones to update the global model; Obtain the global model after multiple rounds of federated learning iterative updates, so as to perform distributed intelligent decision-making for the drone swarm through the iteratively updated global model.

[0008] Optionally, the step of selecting a channel with the least interference from the available channels of the drone as the best communication channel through the first-layer DQN includes: Use the first-layer DQN to perform spectrum sensing on the current communication environment of the drone by means of energy detection to obtain the received signal energy of each channel, and compare the received signal energy with a preset threshold. If the received signal energy is greater than the preset threshold, the channel has interference, otherwise there is no interference, so as to determine the best communication channel; Minimizing the transmission power of the UAV when communicating through the best communication channel by the second-layer DQN, and selecting a coding strategy that maximizes the communication efficiency when the UAV communicates through the best communication channel, includes: Discretizing the preset transmission power by the second-layer DQN to divide the preset transmission power into multiple transmission power levels, and determining the transmission power of the UAV in each time slot when communicating through the best communication channel by the following formula: ; In the formula, P represents the transmission power of the UAV in time slot t, P min represents the minimum transmission power required for the UAV to transmit, P max represents the maximum transmission power required for the UAV to transmit, y(t) represents the transmission power level selected by the UAV in time slot t, and Y represents the number of transmission power levels; In each time slot when the UAV communicates through the best communication channel, select the modulation and demodulation method that maximizes the communication efficiency from multiple modulation and demodulation methods as the best modulation and demodulation method by the second-layer DQN, and maximize the modulation order of the best modulation and demodulation method on the premise of ensuring data accuracy.

[0009] Optionally, the modulation and demodulation methods include the following: binary phase shift keying, quadrature phase shift keying, 16-QAM, and 64-QAM.

[0010] Optionally, aggregating the local models trained by all UAVs to update the global model includes: Using the federated averaging algorithm to obtain the average value of the local models trained by all UAVs to update the global model.

[0011] Optionally, aggregating the local models trained by all UAVs to update the global model includes: Fusing the first-layer DQNs in the local models trained by all UAVs to update the best communication channel selected by the UAV in the global model.

[0012] Optionally, the UAV swarm adopts a cluster architecture, and each cluster contains a cluster head and multiple UAV nodes; Building a distributed intelligent decision-making model of the UAV swarm as the global model and distributing the global model to each UAV in the UAV swarm includes: Building a distributed intelligent decision-making model of the UAV swarm as the global model by the cluster head and distributing the global model to each UAV node in the UAV swarm; Optionally, aggregating the local models trained by all UAVs to update the global model includes: Each drone node uploads the locally trained model to the cluster head, enabling the cluster head to aggregate all the locally trained models of the drones to update the global model.

[0013] Optionally, the cluster head is provided with a model buffer for storing the number of locally trained models expected to be uploaded by the drones in each round of federated learning. The aggregating of all the locally trained models of the drones includes: If the number of locally trained models received by the cluster head reaches the number of locally trained models expected to be uploaded in the model buffer and no new locally trained models are received from the drones within the first preset time period, then the received locally trained models are aggregated. If the number of locally trained models received by the cluster head within the second preset time period does not reach the number of locally trained models expected to be uploaded in the model buffer, then the received locally trained models are directly aggregated.

[0014] An embodiment of the present invention further provides a distributed intelligent decision-making system for a drone swarm, including: A model establishment module for constructing a distributed intelligent decision-making model of the drone swarm as a global model based on federated learning and distributing the global model to each drone in the drone swarm. A model training module for each drone to divide the deep Q-network (DQN) into two layers based on hierarchical reinforcement learning, select a channel with the least interference from the available channels of the drone as the optimal communication channel through the first layer of DQN, minimize the transmission power when the drone communicates using the optimal communication channel through the second layer of DQN, and select an encoding strategy for the highest communication efficiency when the drone communicates using the optimal communication channel, so as to perform local training on the distributed global model to generate a local model. A model update module for aggregating all the locally trained models of the drones to update the global model. An intelligent decision-making module for obtaining the global model after multiple rounds of federated learning iteration updates to perform distributed intelligent decision-making on the drone swarm through the iteratively updated global model.

[0015] An embodiment of the present invention further provides a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, instructions executable by the at least one processor are stored in the memory, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the above-mentioned distributed intelligent decision-making method for a drone swarm.

[0016] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned distributed intelligent decision-making method for an unmanned aerial vehicle (UAV) swarm.

[0017] The distributed intelligent decision-making method for an unmanned aerial vehicle (UAV) swarm provided by the present invention has at least the following beneficial effects: The present invention combines federated learning and hierarchical reinforcement learning. First, a distributed intelligent decision-making model for the UAV swarm is constructed as a global model and distributed to each UAV in the UAV swarm for local training respectively. After the training is completed, the local models obtained from the local training of all UAVs are aggregated to update the original global model. Each participating UAV only uses its own private data for local training of the model without directly exchanging any original data with other devices, ensuring the privacy and security of the communication data of the UAVs.

[0018] Moreover, when each UAV conducts local training, it simultaneously considers selecting a channel with the least interference from the available channels of the UAV as the optimal communication channel (frequency domain), minimizing the transmission power when the UAV communicates using the optimal communication channel (power domain), and selecting a coding strategy that maximizes the communication efficiency when the UAV communicates using the optimal communication channel (code domain). From these three perspectives, the anti-interference ability during communication is enhanced, and even in the face of complex interference scenarios, it has a good anti-interference effect.

[0019] Therefore, the present invention can ensure the data security of UAVs when collaborating to complete tasks, improve the communication efficiency, realize the intelligent decision-making sharing among UAVs within the UAV swarm, and enhance the anti-interference ability of the entire network. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] One or more embodiments are illustrated by way of example in the accompanying drawings, and these exemplary illustrations do not limit the embodiments.

[0021] Figure 1 is a flowchart of a distributed intelligent decision-making method for an unmanned aerial vehicle (UAV) swarm according to an embodiment of the present invention; Figure 2 is a schematic diagram of the architecture of an unmanned aerial vehicle (UAV) swarm according to an embodiment of the present invention; Figure 3 is a schematic diagram of the framework of hierarchical reinforcement learning according to an embodiment of the present invention; Figure 4 is a flowchart of anti-interference of hierarchical reinforcement learning according to an embodiment of the present invention; Figure 5 is a schematic diagram of the framework of federated learning according to an embodiment of the present invention; Figure 6Schematic diagram of an adaptive model synchronization mechanism provided according to an embodiment of the present invention; Figure 7 Schematic diagram of a model buffer provided according to an embodiment of the present invention; Figure 8 Flowchart of anti-interference in federated learning provided according to an embodiment of the present invention. Detailed implementation manners

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that in the embodiments of the present invention, many technical details are presented for the readers to better understand the present invention. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed by the present invention can still be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation to the specific implementation manners of the present invention. The various embodiments can be combined and cross-referenced with each other on the premise of not being contradictory.

[0023] An embodiment of the present invention relates to a distributed intelligent decision-making method for an unmanned aerial vehicle (UAV) swarm. The specific process of the distributed intelligent decision-making method for the UAV swarm in this embodiment can be as Figure 1 shown and includes: Step 101: Based on federated learning, construct a distributed intelligent decision-making model for the UAV swarm as a global model, and distribute the global model to each UAV in the UAV swarm.

[0024] Step 102: Each UAV, based on hierarchical reinforcement learning, divides the deep Q-network (DQN) into two layers. The first layer of DQN is used to select a channel with the least interference from the available channels of the UAV as the best communication channel, and the second layer of DQN is used to minimize the transmission power when the UAV communicates using the best communication channel, and select an encoding strategy that maximizes the communication efficiency when the UAV communicates using the best communication channel, so as to locally train the distributed global model to generate a local model.

[0025] Step 103: Aggregate the local models trained by all UAVs to update the global model.

[0026] Step 104: Obtain the global model after multiple rounds of federated learning iterative update, so as to perform distributed intelligent decision-making for the UAV swarm through the iteratively updated global model.

[0027] In this embodiment, federated learning and hierarchical reinforcement learning are combined. First, a distributed intelligent decision-making model for the UAV swarm is constructed as the global model and distributed to each UAV in the UAV swarm for local training respectively. After the training is completed, the local models obtained from the local training of all UAVs are aggregated to update the original global model. Each participating UAV only uses its own private data for local training of the model without directly exchanging any original data with other devices, ensuring the privacy and security of UAV communication data. Moreover, when each UAV conducts local training, it simultaneously considers selecting a channel with the least interference from the available channels of the UAV as the best communication channel (frequency domain), minimizing the transmission power when the UAV communicates using the best communication channel (power domain), and selecting a coding strategy that maximizes the communication efficiency when the UAV communicates using the best communication channel (code domain). From these three perspectives, the anti-interference ability during communication is enhanced, and even in the face of complex interference scenarios, it has a good anti-interference effect. Therefore, this embodiment can ensure the data security of UAVs when collaborating to complete tasks, improve communication efficiency, realize the sharing of intelligent decisions among UAVs within the UAV swarm, and enhance the anti-interference ability of the entire network.

[0028] The implementation details of the distributed intelligent decision-making method for the UAV swarm in this embodiment are specifically described below. The following content is only the implementation details provided for convenience of understanding and is not necessary for implementing this solution.

[0029] The UAV swarm in this embodiment adopts a clustered architecture. Each cluster contains a cluster head and multiple UAV nodes, as Figure 2 shown. In the UAV network, an environment interfered by a jammer is constructed, and the number of jammers is set to 1. The network consists of N S nodes, and these nodes are assigned to N different clusters according to the number of nodes N C within the cluster. The number of each cluster is obtained by dividing the total number of nodes by the number of nodes within the cluster, that is, N = N S / N C . In each cluster, there is a UAV with a high level and strong data processing ability as the cluster head, and the cluster head acts as a local control center within the cluster. As the gathering point of information, the cluster head processes and analyzes the data collected from each UAV to obtain a comprehensive understanding of the current environment and task requirements within the cluster.

[0030] In this embodiment, a distributed intelligent decision-making model for a drone swarm is constructed based on federated learning. The distributed intelligent decision-making model is used as the global model in federated learning, and the global model is distributed to each drone in the drone swarm, so that each drone can perform local training on the distributed federated learning respectively to form a local model. The local models obtained by training all drones are aggregated to update the original global model. In this way, by obtaining the global model updated through multiple rounds of federated learning iteration, the distributed intelligent decision-making of the drone swarm can be carried out.

[0031] First, the process of each drone performing local training on the distributed federated learning respectively to form a local model will be specifically described: In this embodiment, based on hierarchical reinforcement learning, the DQN process is divided into two layers, where the first layer focuses on the channel selection problem. The key task of this layer (frequency domain layer) is to identify and select an optimal communication path from the available channels, that is, the channel with the least noise and interference. When entering the second layer, the channel selection decision made in the frequency domain layer will be used as the key state input, providing the necessary context information for the agent to adjust the transmission power (power domain) and select the modulation and demodulation method (code domain) on this basis. The process framework of the anti-interference method based on hierarchical reinforcement learning is as Figure 3 shown.

[0032] In specific implementation, in the frequency domain, intelligent frequency decision-making is performed to select a channel with less interference, which can ensure communication quality. Therefore, in this embodiment, the first-layer DQN uses the energy detection method to perform spectrum sensing on the current communication environment of the drone to obtain the received signal energy of each channel, and compares the received signal energy with a preset threshold. If the received signal energy is greater than the preset threshold, the channel has interference; otherwise, there is no interference, so as to determine the optimal communication channel.

[0033] Among them, the energy detection method is a commonly used technique in spectrum sensing for detecting whether there is a signal in a certain frequency band channel. The mathematical expression of the energy detection method is shown as follows: represents the detection statistic, that is, the average energy of the signal; represents the total number of samples; represents the received signal sample. Let η be the threshold set based on the required detection performance. If Λ>η, it is considered that there is noise or interference in this channel; otherwise, it is considered that there is no noise or interference.

[0034] In the power domain, multiple transmission power levels are set. The higher the level, the greater the power used, and correspondingly, the stronger the anti-interference ability, while the energy consumption is also higher. On the premise of ensuring the accuracy and stability of data transmission, reasonably controlling the transmission power can reduce the energy consumption of the UAV. Among them, the accuracy and stability of data transmission can be reflected by performance indicators such as the bit error rate and retransmission rate during data transmission.

[0035] Therefore, in this embodiment, the second-layer DQN discretizes the preset transmission power to divide the preset transmission power into multiple transmission power levels, and determines the transmission power of the UAV in each time slot when communicating through the best communication channel through the following formula: ; In the formula, P represents the transmission power of the UAV in time slot t, P min represents the minimum transmission power required for the UAV to transmit, P max represents the maximum transmission power required for the UAV to transmit, y(t) represents the transmission power level selected by the UAV in time slot t, and Y represents the number of transmission power levels.

[0036] In the code domain, the data transmission rate is improved through an efficient coding strategy. The modulation order is maximized on the premise of ensuring the correctness of the transmitted data, and the communication efficiency of the UAV is improved. Therefore, in this embodiment, the second-layer DQN selects the modulation and demodulation method that can achieve the highest communication efficiency from multiple modulation and demodulation methods as the best modulation and demodulation method in each time slot when the UAV communicates through the best communication channel, and maximizes the modulation order of the best modulation and demodulation method on the premise of ensuring data accuracy. The modulation and demodulation methods include the following: Binary Phase Shift Keying (BPSK), Quadrature Phase Shift Keying (QPSK), 16-QAM (Quadrature Amplitude Modulation), and 64QAM. For example, the UAV corresponds these four different modulation methods to four levels respectively. Level 1 corresponds to BPSK, Level 2 corresponds to QPSK, Level 3 corresponds to 16QAM, and Level 4 corresponds to 64QAM. The higher the level, the higher the transmission rate. At the same time, the anti-interference ability is lower.

[0037] Among them, the Markov Decision Process (MDP) model of the first-layer DQN is: 1) State space: The current state is the spectrum state. The channel state is sensed by the energy detection method. The current states of all channels are initialized to 0. If it is 1, it means that there is noise or interference in the current channel, otherwise it is 0. There are c channels in total, so the size of the state set is c.

[0038] 2) Action space: The action selection is the available channels. C = {1, 2, 3, ..., c} represents the available channels, where c represents the total number of available channels. The size of the action space is defined as c.

[0039] 3) State transition probability: The state transition probability is denoted as P: S × A × S → [0, 1], which represents that when the current state s k belongs to the state space S, an action a k belongs to the action space A, and then the probability of transferring to the next state s k+1 is considered, where k represents the k-th group of transmitted data.

[0040] 4) Discount factor γ: 0 < γ ≤ 1.

[0041] 5) Reward function R: If the transmission is successful, then R = 1; otherwise, R = 0.

[0042] The Markov decision process MDP model of the second-layer DQN is as follows: 1) State space: The state is the communication channel selected in the first layer. All channels are initialized to 0. When a communication channel is set to 1, and there are c channels in total, the size of the state set is c.

[0043] 2) Action space: The action selection is power and modulation / demodulation mode. P = {1, 2, ..., p} represents the power levels, where p represents the maximum power level, and M = {1, 2, ..., m} represents the modulation levels, where m represents the maximum modulation level. The size of the action space is defined as p * m.

[0044] 3) State transition probability: The state transition probability is denoted as P: S × A × S → [0, 1].

[0045] 4) Discount factor γ: 0 < γ ≤ 1.

[0046] 5) Reward function R: If the transmission is successful, then R = α × power + β × modulation, where power is the transmission power level, modulation is the modulation order level, and α and β are the weights of the transmission power level and the modulation order level respectively, and α > β. The purpose is to ensure the data transmission rate as much as possible. First, adjust the transmission power, and then adjust the modulation level. Otherwise, R = 0.

[0047] The basic process of UAV decision-making is as follows. Specifically, at the beginning of each time slot, the first-layer DQN is responsible for selecting the optimal transmission channel and performing spectrum sensing on the channel using the energy detection method. The second-layer DQN then determines the transmission power and modulation and demodulation method based on the selected channel, and then transmits data on the selected channel with the selected power and modulation method. Next, the base station, as the receiver, is responsible for receiving data from the UAV and returning the communication result according to the received quality of the data. According to the success or failure of the communication, the reward values of the two-layer DQN will be updated accordingly to feedback the learning results. This process will continue in multiple interactions with the communication environment until the set maximum number of updates is reached. By continuously learning the environmental feedback, DQN optimizes the decision-making strategy, enabling the UAV to autonomously make the best communication decisions in a changing environment.

[0048] Therefore, the anti-interference algorithm process of a single UAV based on hierarchical reinforcement learning is as Figure 4 shown, including: Input: Training data, experience pool.

[0049] Output: Optimal policy estimate π.

[0050] Step 1: Establish a value neural network and a target neural network, an experience pool E, initialize the communication environment, initialize the DQN state space and action space, and initialize the agent parameters according to the experience pool size, discount factor, exploration rate, and learning rate.

[0051] Step 2: At the beginning of each time slot, the UAV first uses the energy detection method to obtain the current spectrum state and determines the channel for the current time slot through the first-layer DQN using the sensing result.

[0052] Step 3: The second-layer DQN takes the channel selected by the first-layer DQN as the state of the second layer and selects the transmit power and modulation and demodulation method for the current transmission channel.

[0053] Step 4: Transmit on the selected channel with the selected power and modulation method. If the transmission is successful, the first layer obtains a reward value R = 1, and the second layer obtains a reward value R = α×power + β×modulation.

[0054] Step 5: If there is interference conflict, directly interrupt the transmission. The first layer obtains a reward value R = 0, and the second layer obtains a reward value R = 0.

[0055] Step 6: Return the current state and reward to the hierarchical DQN. The DQN selects the channel for the next moment, the transmission time on this channel, the transmission power, and the modulation method according to the current state.

[0056] Next, a detailed description is given of the process of constructing a global model based on federated learning, distributing the global model to each drone in the drone swarm, and aggregating the local models obtained from the training of all drones to update the original global model: In this embodiment, a drone swarm intelligent anti-jamming decision-making communication model is constructed in combination with federated learning. Federated learning is added to the first layer of DQN, that is, the channel selection layer. Each drone within a cluster holds its independent dataset, that is, it is trained in its respective independent environment, and without revealing the privacy of personal data, it shares the model parameter updates with the cluster head. The server then is responsible for summarizing and integrating these model parameters from different devices to form a global model. In the whole process, each participating drone only uses its own private data for local training of the model, and there is no need to directly exchange any raw data with other devices. Even if the other party knows other communication parameters, such as transmission power and modulation and demodulation methods, it is impossible to infer our channel selection network and it is difficult to effectively perform interference or eavesdropping. And after the network parameters of the drones are synchronized, if the inputs are the same, then the same outputs will be obtained, which ensures that each drone will select the same channel when transmitting and receiving, saving time and energy consumption. The federated learning framework is as Figure 5 shown.

[0057] This framework includes a cluster head and multiple drones. Among them, the cluster head plays the role of a coordination server. Each drone represents a data node. They independently run the deep reinforcement learning algorithm to optimize their decision-making processes. These drones do not directly exchange or share their flight data or observation results, but locally train and update the model parameters. In this process, the cluster head plays a key role. It is not responsible for processing the private data of any individual data node, but serves as a centralized aggregation point, responsible for receiving model updates from each drone. Then, the cluster head performs a secure aggregation operation to improve the global model. When fusing the drone models, only the first layer of the hierarchical DQN is fused, aiming to synchronize the channel selection. Once the global model is updated, the cluster head then sends this updated model back to all drones, so that each drone can use the latest knowledge to improve the efficiency and effectiveness of its operations.

[0058] In a specific implementation, when aggregating the local models trained by all drones to update the global model, the Federated Averaging Algorithm (FedAvg) is used to update the global model. The FedAvg algorithm allows multiple clients to collaboratively train a shared model while maintaining the privacy of their respective data, thereby achieving the sharing of network knowledge and the centralized optimization of decision-making capabilities. The core idea of this algorithm is to allow multiple clients to independently train models on their local data and then send the updates of these local models to a central server. After the server receives these model updates from each client, it calculates their average value to update the global model. Subsequently, the updated global model is distributed to each client again and used as the starting point for the next round of training. This process is iterated until a predetermined accuracy is reached or a specified number of iterations are performed.

[0059] In the drone swarm network, the process of FedAvg is as follows: 1) Initialization phase: The cluster head acts as the central server, initializes the global model, and distributes this initial model to all drones in the network. These drones, as clients, are responsible for locally training the model based on the data they collect.

[0060] 2) Local training phase: Each drone independently trains the model on its own dataset.

[0061] 3) Upload model update: After completing a certain number of training iterations, each drone uploads the update (parameter difference) of its model to the cluster head. This upload process usually occurs after a fixed period when each drone finishes local training to reduce the network burden caused by frequent communication.

[0062] 4) Aggregate model update: The cluster head collects the model updates from all drones participating in this round of training and then calculates the average value of these updates. This averaging operation is based on the assumption that the contributions of each drone are equally important, thereby generating a new global model. The aggregated model update is shown as follows:

[0063] ； In the formula, K represents the total number of drones participating in the current round of training. The k-th drone receives the model of the t-th round sent by the cluster head locally, uses its own data to train its own model and then uploads it to the cluster head. The cluster head aggregates the models of each drone collected and obtains the model for the next round .

[0064] 5) Distribute the global model: The cluster head distributes the updated global model to all drones, and the drones use this new model as the starting point for the next round of local training.

[0065] 6) Iterative optimization: The above process is repeated, and each iteration aims to improve the performance and accuracy of the global model. Through multiple rounds of local training, model uploading, aggregation update, and model distribution, the global model gradually converges to an optimized state.

[0066] Meanwhile, this embodiment assists model aggregation based on an adaptive model synchronization mechanism, and the adaptive model synchronization mechanism is as Figure 6 shown. Each data node, that is, the drone, independently performs local training and uploads the updated model parameters to the cluster head after completion. At the same time, the cluster head does not need to wait for the model parameters of each drone to start aggregation, but immediately performs it after collecting a sufficient number of model parameters, so as to improve the aggregation efficiency.

[0067] Among them, the cluster head is equipped with a model buffer Buffer for temporarily storing the model parameters locally updated by each drone collected in each round of training. This buffer is set with a sliding window of dynamically variable length, representing the expected number of model updates to be received. The cluster head sets a storage area for the number of models received in each round and adds the number of models participating in the training in each round to the storage area. If the number of models received in each round is equal to the expected number of models to be received, then the expected number of models to be received remains unchanged; if not, then take the most recently stored x numbers from the storage area for averaging as the new expected number of models to be received. The schematic diagram of the model buffer is as Figure 7 shown.

[0068] The cluster head sets two key time parameters: one is the maximum waiting time wait max for each round of training, and the other is the additional waiting time wait rec after receiving the expected number of models. At the beginning of each round of federated learning, the cluster head distributes the global model to the online drones for local training. After the training is completed, the drones send the updated model parameters back to the cluster head. The cluster head collects these model parameters and monitors the status of the model collection area. Once the number of models collected reaches the preset expected number, the cluster head will start an additional waiting timer with a duration of wait rec . If no new models are added during this additional waiting time, the cluster head will immediately perform the aggregation update of the models, thereby generating a new round of global models.

[0069] On the other hand, if during wait maxWithin a certain period of time, if the number of received models never reaches the expected number, the cluster head will also perform model aggregation to ensure the continuity and progress of the learning process. Such a design aims to address the issue of update delays caused by uneven computing capabilities or unstable communication environments among drones, and to prevent the learning efficiency of the entire network from being affected by the delays of individual drones. If the buffer remains empty after exceeding the set maximum waiting time, the cluster head will default the global model of the previous round as the latest to avoid the stagnation of the training process. Such a mechanism ensures the continuity and smoothness of training, while reducing the waiting time caused by network delays or differences in drone processing capabilities, and significantly improving the overall training efficiency.

[0070] Therefore, the process of the anti-interference algorithm for a swarm of drones based on federated learning in this embodiment can be as Figure 8 shown, including: Input: Training data, experience pool.

[0071] Output: The optimal policy estimate π.

[0072] Step 1: All drones within the cluster establish a value neural network and a target neural network, an experience pool E, initialize the communication environment, initialize the DQN state space and action space, and initialize the agent parameters according to the experience pool size, discount factor, exploration rate, and learning rate. The cluster head initializes a model buffer Buffer and sets two key time parameters: one is the maximum waiting time wait max , and the other is the additional waiting time wait rec after receiving the expected number of models.

[0073] Step 2: The cluster head distributes the initial network to the drone nodes within the cluster.

[0074] Step 3: The drones within the cluster update the model according to the single-drone anti-interference algorithm, as Figure 3 shown. The drones that reach the maximum number of local updates upload their local model parameters to the cluster head.

[0075] Step 4: If the cluster head buffer is not full or the cluster head update time has not been reached, the cluster head adds the model to the buffer Buffer.

[0076] Step 5: If the number of collected models reaches the expected number preset in the Buffer, the cluster head will start an additional waiting timer with a duration of wait rec . If no new models are added during this additional waiting time, the cluster head stops receiving model parameters. If wait maxIf the number of received models never reaches the expected number, the cluster head will also stop receiving model parameters to ensure the continuity and progress of the learning process. If the buffer is still empty after exceeding the set maximum waiting time, the cluster head will default the global parameters of the previous round as the latest global parameters to avoid the stagnation of the training process.

[0077] Step 6: The cluster head retrieves the model parameters from the Buffer for aggregation and update, thereby generating a new round of global models. The buffer Buffer is set with a dynamically variable-length sliding window, representing the expected number of model updates to be received. The cluster head sets a storage area for the number of models received per round and adds the number of models participating in the training per round to the storage area. If the number of models received per round is equal to the expected number of models received, then the expected number of models to be received remains unchanged; if not, then the most recently stored x numbers are averaged from the storage area as the new expected number of models to be received.

[0078] Step 7: The cluster head distributes the latest global model parameters to the UAVs within the cluster.

[0079] Step 8: After receiving the global model parameters, the UAVs synchronously update the network model and perform a new round of model training.

[0080] In summary, the distributed intelligent decision-making method for the UAV swarm in this embodiment can be implemented through the following process: Step 1: Initialize relevant parameters.

[0081] This step requires initializing the communication environment, initializing the DQN state space and action space, and initializing the agent parameters according to the experience pool size, discount factor, exploration rate, and learning rate.

[0082] Step 2: After initialization, the cluster head distributes the initial network model to the UAVs within the cluster. The distributed network model is the model responsible for channel selection.

[0083] The network parameters regularly synchronized by the cluster head in this step ensure the consistency of the UAVs within the cluster when selecting channels, reduce communication conflicts and interference caused by improper channel selection by the receiver, and improve the stability and efficiency of communication.

[0084] Step 3: Initialize the model buffer Buffer in the cluster head UAV. The cluster head is equipped with a model cache Buffer for temporarily storing the model parameters updated locally by each UAV in each round of training.

[0085] Among them, a sliding window with a dynamically variable length is set in this buffer, representing the number of model updates expected to be received. Therefore, the cluster head does not need to wait for the model parameters of each drone to start aggregation, but can start immediately after collecting a sufficient number of model parameters, so as to improve the aggregation efficiency and better meet the dynamic requirements of actual combat scenarios.

[0086] Step 4: During each round of federated learning, the drones within the cluster use the hierarchical DQN algorithm to interact with their respective communication environments and update their local models. To reduce the burden of network communication, the updated model parameters are only uploaded to the cluster head after the device has completed a certain number of local training rounds.

[0087] Among them, since this study is based on the frequency domain, power domain, and code domain, the action space is large and the algorithm convergence speed is slow. To improve the algorithm convergence speed and anti-interference ability, an anti-interference model based on a hierarchical reinforcement learning framework is used. In the design of this model, the DQN process is divided into two layers, where the first layer focuses on the channel selection problem. The key task of this layer is to identify and select an optimal communication path from the available channels, that is, the channel with the least noise and interference. When entering the second layer, the channel selection decision made in the frequency domain layer will be used as the key state input, providing the necessary context information for the agent to adjust the transmission power and select the appropriate modulation and demodulation method on this basis. Such a hierarchical design ensures that the formulation of the entire communication strategy is carried out under the optimal or most suitable communication environment conditions, thus greatly improving the communication efficiency and quality.

[0088] Step 5: If the buffer is not full or the cluster head update time has not been reached, the drones will send the trained model parameters to the cluster head, and the cluster head will add the model parameters to the storage area.

[0089] Among them, the cluster head sets two key time parameters: one is the maximum waiting time wait for each round of training max , and the other is the additional waiting time wait after receiving the expected number of models rec . The cluster head collects these model parameters and monitors the status of the model collection area.

[0090] Step 6: If the buffer is full or the cluster head update time has been reached, the cluster head stops receiving the model parameters sent by the drones within the cluster in this round.

[0091] In this step, if the number of models collected reaches the expected number preset in the Buffer, the cluster head will start an additional waiting timer with a duration of wait rec . If no new models are added during this additional waiting time, the cluster head will immediately perform the aggregation update of the models, thereby generating a new round of global models. If it reaches wait max, if the number of received models never reaches the expected number, the cluster head will also perform model aggregation to ensure the continuity and progress of the learning process. Such a design aims to address the update delay problem caused by uneven computing capabilities or unstable communication environments of drones, and avoid the impact of individual drone delays on the learning efficiency of the entire network. If the buffer is still empty after exceeding the set maximum waiting time, the cluster head will default the global model of the previous round as the latest to prevent the training process from stagnating. Such a mechanism ensures the continuity and smoothness of training, while reducing the waiting time caused by network delays or differences in drone processing capabilities, and significantly improving the overall training efficiency.

[0092] Step 7: The cluster head retrieves the received model parameters from the Buffer for aggregation and update to obtain new global model parameters.

[0093] In this step, if the number of received models per round is equal to the expected number of received models, then the expected number of received models remains unchanged; if not, then the most recently stored quantity is retrieved from the storage area for averaging as the new expected number of received models.

[0094] Step 8: The cluster head distributes the new global model to the drones within the cluster.

[0095] Among them, even if some drones within the cluster participate in uploading model parameters, the scope faced by the cluster head when distributing global model parameters is all drones within the cluster to ensure the synchronization of all drone information.

[0096] Step 9: After receiving the global model, the drones synchronously update the network model and perform a new round of model training.

[0097] Among them, the drones within the cluster use the parameters of the distributed global model to update the parameters of the local model to ensure model synchronization, and perform private model training within the node using the locally collected data information.

[0098] The distributed intelligent decision-making method for a swarm of drones of the present invention has the following beneficial effects: (1) The present invention adopts a multi-domain joint approach, starting from the frequency domain, power domain, and code domain, to increase the flexibility and anti-interference performance of anti-interference and improve the information transmission efficiency.

[0099] (2) The present invention is based on hierarchical reinforcement learning. Since this research is based on the frequency domain, power domain, and code domain, although it can reduce energy consumption and improve the transmission rate, the action space is large, resulting in a slow convergence speed of the algorithm. To improve the convergence speed of the algorithm and enhance the anti-interference ability, an anti-interference model based on the hierarchical reinforcement learning framework provides an effective interference management strategy for the communication of the UAV swarm system. Hierarchical reinforcement learning decomposes high-dimensional decision-making tasks into multiple low-dimensional subtasks, and each subtask is controlled by a sub-policy, so that the large action space can be decomposed into several smaller and more manageable sub-spaces.

[0100] (3) The present invention considers the design of the model. In the design of the local model of the UAVs within a cluster, the DQN process is divided into two layers, where the first layer focuses on the channel selection problem. The key task of this layer is to identify and select an optimal communication path from the available channels, that is, the channel with the least noise and interference. When entering the second layer, the channel selection decision made in the frequency domain layer before serves as the key state input, providing the necessary context information for the UAV to adjust the transmission power and select the appropriate modulation and demodulation method on this basis. The hierarchical design ensures that the formulation of the communication strategy is carried out under the optimal or most suitable communication environment conditions, thus greatly improving the communication efficiency and quality.

[0101] (4) The present invention is based on federated learning, which combines federated learning with hierarchical reinforcement learning. The transmission channel is the core secret of communication. If the enemy does not know the transmission channel selected by us, even if they master other communication parameters, such as transmission power and modulation and demodulation methods, it is difficult to effectively perform interference or eavesdropping. Therefore, federated learning is added to the first layer DQN of hierarchical reinforcement learning, that is, the channel selection layer. The network parameters regularly synchronized by the cluster head ensure the consistency of the UAVs within the cluster when selecting channels, reducing communication conflicts and interference caused by improper channel selection by the receiver, and improving the stability and efficiency of communication.

[0102] (5) The present invention is based on an adaptive model synchronization mechanism. Considering the complexity of the communication environment and the different performances of UAVs, the training time of each UAV is different. Therefore, the adaptive model synchronization mechanism allows the model parameters from different UAVs to be collected after local training by establishing a central model collection buffer, and at the same time gives the system flexibility to dynamically adjust the required number of models according to the actual situation for the update of the global model.

[0103] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split into multiple steps. As long as the same logical relationship is included, it is within the protection scope of the present invention; adding insignificant modifications to the algorithm or process or introducing insignificant designs, but without changing the core design of the algorithm and process, is within the protection scope of the invention.

[0104] Another embodiment of the present invention relates to a distributed intelligent decision-making system for a drone swarm. The implementation details of the distributed intelligent decision-making system for the drone swarm in this embodiment will be specifically described below. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing this solution. The distributed intelligent decision-making system for the drone swarm in this embodiment includes: A model establishment module, configured to build a distributed intelligent decision-making model for the drone swarm as a global model based on federated learning, and distribute the global model to each drone in the drone swarm; A model training module, for each drone, based on hierarchical reinforcement learning, divides the deep Q-network (DQN) into two layers. The first layer of DQN is used to select a channel with the least interference from the available channels of the drone as the best communication channel, and the second layer of DQN is used to minimize the transmission power when the drone communicates using the best communication channel, and select an encoding strategy that maximizes the communication efficiency when the drone communicates using the best communication channel, so as to locally train the distributed global model to generate a local model; A model update module, configured to aggregate the local models trained by all drones to update the global model; An intelligent decision-making module, configured to obtain the global model after multiple rounds of federated learning iteration updates, so as to perform distributed intelligent decision-making for the drone swarm through the iteratively updated global model.

[0105] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in cooperation with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiment are still valid in this embodiment. To avoid repetition, they will not be elaborated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0106] It is worth mentioning that each module involved in this embodiment is a logical module. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovative part of the present invention, units that are not closely related to solving the technical problems proposed by the present invention are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0107] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the distributed intelligent decision-making method for the drone swarm in the above embodiments.

[0108] Wherein, the memory and the processor are connected by a bus. The bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and the memory together. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, etc., which are well known in the art, and thus will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be an element or multiple elements, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on the transmission medium. The data processed by the processor is transmitted over the wireless medium through the antenna. Further, the antenna also receives data and transmits the data to the processor.

[0109] The processor is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory may be used to store data used by the processor when executing operations.

[0110] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method embodiments described above are implemented.

[0111] That is, those skilled in the art can understand that all or part of the steps in implementing the above method embodiments can be completed by instructing relevant hardware through a program. The program is stored in a storage medium, including several instructions for enabling a device (which may be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0112] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes may be made in form and details without departing from the spirit and scope of the present invention.

Claims

1. A distributed intelligent decision-making method for drone swarms, characterized in that: The method comprises: Based on federated learning, a distributed intelligent decision-making model of the drone swarm is constructed as a global model, and the global model is distributed to each drone in the drone swarm; Each drone divides the deep Q network DQN into two layers based on hierarchical reinforcement learning. The first layer DQN selects a channel with the least interference from the available channels of the drone as the best communication channel, and the second layer DQN minimizes the transmission power when the drone communicates using the best communication channel, and selects a coding strategy for the drone to achieve the highest communication efficiency when using the best communication channel, so as to locally train the distributed global model and generate a local model. Aggregate the local models trained by all drones to update the global model; Obtain a global model after multiple rounds of federated learning iterative updates, and use the iteratively updated global model to make distributed intelligent decisions for the drone swarm.

2. The distributed intelligent decision-making method for drone swarms according to claim 1, characterized in that: The method of selecting a channel with the least interference from the available channels of the drone as the best communication channel through the first layer of DQN includes: The first layer of DQN uses the energy detection method to perform spectrum perception on the current communication environment of the drone to obtain the received signal energy of each channel and compare the received signal energy with the preset threshold. If the received signal energy is greater than the preset threshold, there is interference in the channel, otherwise there is no interference, so as to determine the best communication channel; The method of minimizing the transmission power of the UAV when communicating through the optimal communication channel by the second layer DQN, and selecting a coding strategy for achieving the highest communication efficiency when the UAV communicates through the optimal communication channel, includes: The preset transmit power is discretized through the second layer of DQN to divide the preset transmit power into multiple transmit power levels, and the transmit power in each time slot when the drone uses the optimal communication channel for communication is determined by the following formula: ； Where P represents the transmission power of the UAV at time slot t, P min represents the minimum transmission power required for UAV transmission, P max represents the maximum transmit power required for the UAV transmission, y(t) represents the transmit power level selected by the UAV in time slot t, and Y represents the number of transmit power levels; Through the second layer of DQN, in each time slot when the drone adopts the optimal communication channel to communicate, the adaptive demodulation method that makes the communication efficiency highest is selected from a variety of adaptive demodulation methods as the optimal adaptive demodulation method, and the modulation order of the optimal adaptive demodulation method is maximized under the premise of ensuring data accuracy.

3. The distributed intelligent decision-making method for drone swarms according to claim 2, characterized in that: The adaptive demodulation modes include the following: binary phase shift keying, quadrature phase shift keying, 16-order quadrature amplitude modulation and 64-order quadrature amplitude modulation.

4. The distributed intelligent decision-making method for drone swarms according to claim 1, characterized in that: The local models trained by all drones are aggregated to update the global model, including: The federated average algorithm is used to obtain the average value of the local models trained by all drones to update the global model.

5. The distributed intelligent decision-making method for drone swarms according to claim 4, characterized in that: The local models trained by all drones are aggregated to update the global model, including: The first layer DQN in the local models trained by all drones is fused to update the optimal communication channel selected by the drones in the global model.

6. The distributed intelligent decision-making method for drone swarms according to claim 4 or 5, characterized in that: The drone swarm adopts a cluster architecture, and each cluster contains a cluster head and multiple drone nodes; The distributed intelligent decision-making model of the drone swarm is constructed as a global model, and the global model is distributed to each drone in the drone swarm, including: The distributed intelligent decision-making model of the drone swarm is constructed through the cluster head as a global model, and the global model is distributed to each drone node in the drone swarm; The local models trained by all drones are aggregated to update the global model, including: Each drone node uploads the trained local model to the cluster head, so that the cluster head aggregates the local models trained by all drones to update the global model.

7. The distributed intelligent decision-making method for drone swarms according to claim 6, characterized in that: The cluster head is provided with a model cache area for storing the number of local models expected to be uploaded by the drone in each round of federated learning; The local models trained by all drones are aggregated, including: If the number of local models received by the cluster head reaches the number of local models expected to be uploaded in the model cache, and no new local models uploaded by the drone are received within the first preset time, the received local models are aggregated; If the number of local models received by the cluster head within the second preset time period does not reach the number of local models expected to be uploaded by the model cache, the received local models are directly aggregated.

8. A distributed intelligent decision-making system for drone swarms, characterized in that: The system comprises: The model building module is used to build a distributed intelligent decision-making model of the drone swarm as a global model based on federated learning, and distribute the global model to each drone in the drone swarm; The model training module is used for each drone to divide the deep Q network DQN into two layers based on hierarchical reinforcement learning, select a channel with the least interference from the available channels of the drone as the optimal communication channel through the first layer DQN, and minimize the transmission power of the drone when communicating using the optimal communication channel through the second layer DQN, and select a coding strategy for the drone to communicate with the highest communication efficiency when using the optimal communication channel, so as to locally train the distributed global model and generate a local model; The model update module is used to aggregate the local models trained by all drones to update the global model; The intelligent decision-making module is used to obtain the global model after multiple rounds of federated learning iterative updates, so as to make distributed intelligent decisions of the drone swarm through the iteratively updated global model.

9. A computer device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed intelligent decision-making method for the drone swarm as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the distributed intelligent decision-making method for a drone swarm as described in any one of claims 1 to 7 is implemented.

Images