A Federated Reinforcement Learning Task Allocation Method and System for Heterogeneous Unmanned Aerial Vehicle Swarms

Abstract

This invention relates to the field of unmanned aerial vehicle (UAV) swarm technology, and proposes a federated reinforcement learning task allocation method and system for heterogeneous UAV swarms. The method includes: real-time acquisition of stable performance parameters of UAVs; normalization of these parameters based on preset weight factors to obtain resource capability indicators for the UAVs; training a reinforcement learning model; calculating weights based on the resource capability indicators; and weighted fusion of model parameters to generate a global model; real-time acquisition of dynamic change information within the task area using UAV sensors; planning and executing a task allocation scheme for the UAVs using a multi-objective optimization algorithm to obtain the task execution status of the UAVs; and calculating the efficiency value of the task allocation scheme based on the task execution status and environmental data. When the efficiency value is lower than a preset threshold, the task allocation strategy is dynamically optimized to obtain an optimized task allocation strategy. This invention improves the task execution efficiency and resource utilization of heterogeneous UAV swarms in complex dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) swarm technology, and in particular to a federated reinforcement learning task allocation method and system for heterogeneous UAV swarms. Background Technology

[0002] Heterogeneous drone swarms refer to collaborative systems composed of drones of various types, functions, and specifications. Compared to traditional single-type or single-specification drone systems, heterogeneous drone swarms typically include reconnaissance drones, transport drones, and attack drones, which differ significantly in hardware architecture, computing power, sensor configuration, energy reserves, and mission functions. Through such a heterogeneous combination, the swarm can maximize the versatility of drones, such as simultaneously fulfilling different functional requirements in complex missions, exhibiting greater flexibility, mission adaptability, and wider coverage. However, heterogeneous drone swarms also present additional challenges, the key being how to coordinate the resource differences among different types of drones, efficiently allocate tasks, and achieve stable collaborative operation.

[0003] In existing technologies, task allocation for heterogeneous UAV swarms mainly adopts a centralized decision-making architecture, static fixed-weight resource evaluation, and linear weighted multi-objective optimization methods, which realize basic task planning and allocation functions. However, the centralized architecture leads to single-point failure risk and high communication overhead. Static weights cannot adapt to dynamic environmental changes, and linear optimization is difficult to balance multi-dimensional conflicting objectives. Furthermore, it lacks fault tolerance mechanisms for communication interruptions and real-time performance feedback adjustment capabilities. This makes it particularly difficult for existing systems to achieve efficient and robust task allocation in complex environments, especially when facing communication constraints, dynamic environmental changes, and imbalanced heterogeneous resources. Summary of the Invention

[0004] In view of this, the present invention proposes a federated reinforcement learning task allocation method and system for heterogeneous UAV swarms, which solves the problem that the existing technology lacks fault tolerance mechanism for communication interruption and real-time performance feedback adjustment capability, and is difficult to achieve efficient and robust task allocation in complex environments, especially when facing communication constraints, dynamic environmental changes and uneven heterogeneous resources.

[0005] The technical solution of this invention is implemented as follows: In a first aspect, this invention provides a method for federated reinforcement learning task allocation in heterogeneous drone swarms, comprising the following steps:

[0006] Real-time acquisition of stable performance parameters of the UAV, normalization of stable performance parameters based on preset weighting factors, and obtaining the resource capability index of the UAV.

[0007] The reinforcement learning model is trained based on local data from UAVs. Weights are calculated according to the resource capability indicators. The model parameters are weighted and fused to generate a global model, which is then distributed to each UAV.

[0008] By using UAV sensors to collect dynamic change information in the mission area in real time, a multi-objective optimization algorithm is used to plan and execute the UAV's mission allocation scheme, and the mission execution status of the UAV is obtained.

[0009] Based on the task execution status and environmental data, the efficiency value of the task allocation scheme is calculated. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

[0010] Based on the above technical solutions, preferably, the local data training reinforcement learning model based on UAVs, which calculates weights according to the resource capability indicators, performs weighted fusion of model parameters to generate a global model, and distributes the global model to each UAV, includes:

[0011] The raw local data collected by each UAV is normalized, missing value imputation and noise filtering are performed to obtain clean local data. The clean local data is then trained using a policy gradient method with regularization term to obtain local model parameters.

[0012] After the encrypted local model parameters are transmitted to the coordination node, the coordination node performs fusion weighting and security processing on the local model parameters according to the resource capability indicators of each UAV to obtain a global model, and then outputs the global model to each UAV.

[0013] Based on the above technical solutions, preferably, the step of using UAV sensors to collect dynamic change information within the task area in real time, employing a multi-objective optimization algorithm to plan and execute the UAV's task allocation scheme, and obtaining the UAV's task execution status includes:

[0014] Obstacle distribution data, climate change data, and interference signal data are extracted from the sensor data collected by each UAV. The obstacle distribution data, climate change data, and interference signal data are then fused to obtain environmental state parameters.

[0015] A multi-objective optimization function is constructed, with optimization objectives including task completion time, task energy consumption, and task security risk. Based on an iterative solution algorithm, a task allocation scheme that satisfies each optimization objective is obtained under the condition of satisfying resource and environmental constraints.

[0016] Based on the above technical solutions, preferably, the step of calculating the efficiency value of the task allocation scheme according to the task execution status and environmental data, and dynamically optimizing the task allocation strategy when the efficiency value is lower than a preset threshold, to obtain an optimized task allocation strategy, includes:

[0017] The task execution status of each UAV according to the task allocation strategy is obtained. Based on the task execution status, the task completion rate, average response latency and resource utilization rate are extracted. The task completion rate, average response latency and resource utilization rate are cleaned and fused to obtain comprehensive feedback data. Based on the comprehensive feedback data, the performance value of each UAV under the corresponding task allocation scheme is calculated and updated.

[0018] After detecting that the performance value is lower than a preset threshold, the coordinating node issues retraining or fine-tuning instructions to each UAV. Each UAV updates its reinforcement learning model based on the latest local data and global information to obtain updated global model parameters. The updated global model parameters are then uploaded to the coordinating node for weighted fusion of the global model to obtain an updated global model and an optimized task allocation strategy. The updated global model and the optimized task allocation strategy are then output to each UAV.

[0019] Based on the above technical solutions, preferably, the formula for calculating the performance value is:

[0020] ;

[0021] in, Assign performance values ​​to the strategies for the tasks. The total number of drones, For the first m Mission completion rate of drones performing tasks. The maximum mission completion rate for all drones. For the first m The average response latency of a drone performing a mission. This represents the maximum average response latency for all drones performing tasks. For the first m Resource utilization rate during drone missions This represents the maximum resource utilization rate for all drones. For the first m The dynamic adjustment coefficient of the drone. For the first m Resource occupancy response threshold factor for drones The time delay index is the aging coefficient. The standardized coefficient for the performance value. The cluster state entropy function, These are indicators of the uncertainty distribution of task completion rate, average response latency, and resource utilization.

[0022] Based on the above technical solutions, preferably, after calculating the efficiency value of the task allocation scheme according to the task execution status and environmental data, and dynamically optimizing the task allocation strategy when the efficiency value is lower than a preset threshold, the optimized task allocation strategy is further included:

[0023] The model parameters and resource data are synchronized through a redundant communication link algorithm. When communication delay or interruption is detected, the local cache data and task execution status of each UAV are analyzed and processed through a multi-agent collaborative decision-making algorithm to generate a local temporary task allocation scheme.

[0024] Based on the above technical solutions, preferably, the step of synchronizing model parameters and resource data through a redundant communication link algorithm, and analyzing and processing the local cached data and task execution status of each UAV through a multi-agent collaborative decision-making algorithm when communication delay or interruption is detected, to generate a local temporary task allocation scheme, including:

[0025] Under normal communication conditions, the model parameters and resource data of each UAV are synchronized and cached via a redundant link algorithm.

[0026] When communication delays or interruptions are detected, a multi-agent collaborative decision-making algorithm is used to analyze the cached local data and task execution status in real time and dynamically generate a local temporary task allocation scheme.

[0027] Secondly, the present invention also provides a federated reinforcement learning task allocation system for heterogeneous unmanned aerial vehicle (UAV) swarms, the system comprising:

[0028] The parameter acquisition module is used to collect the stable performance parameters of the UAV in real time, and normalize the stable performance parameters based on the preset weighting factor to obtain the resource capability index of the UAV.

[0029] The reinforcement learning module is used to train a reinforcement learning model based on local data from UAVs, calculate weights according to the resource capability indicators, perform weighted fusion of model parameters to generate a global model, and distribute the global model to each UAV.

[0030] The task allocation module is used to collect dynamic change information in the task area in real time using UAV sensors, use multi-objective optimization algorithms to plan the task allocation scheme of the UAV and execute it, and obtain the task execution status of the UAV.

[0031] The feedback optimization module is used to calculate the efficiency value of the task allocation scheme based on the task execution status and environmental data. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

[0032] Thirdly, the present invention also provides an electronic device, comprising: at least one processor, at least one memory, a communication interface, and a bus;

[0033] The processor, memory, and communication interface communicate with each other through the bus. The memory stores program instructions that can be executed by the processor. The processor calls the program instructions to implement the steps of a federated reinforcement learning task allocation method for a heterogeneous UAV swarm.

[0034] Fourthly, the present invention also provides a computer-readable storage medium storing computer instructions that enable a computer to implement steps such as a federated reinforcement learning task allocation method for a heterogeneous unmanned aerial vehicle swarm.

[0035] The federated reinforcement learning task allocation method and system for heterogeneous drone swarms of the present invention have the following advantages over the prior art:

[0036] (1) By collecting and normalizing the stable performance parameters of UAVs in real time, using federated reinforcement learning for model training and parameter fusion, combining multi-objective optimization algorithms for task allocation planning, and dynamic task strategy optimization based on performance value, the system achieves accurate evaluation of heterogeneous UAV resource capabilities, decentralized processing of model training burden, adaptive dynamic optimization of task allocation, and real-time feedback adjustment. This effectively balances multi-dimensional objectives such as task completion time, energy consumption, and safety, and improves the task execution efficiency and resource utilization of heterogeneous UAV clusters in complex dynamic environments.

[0037] (2) By comprehensively analyzing the execution status of UAV missions, including mission completion rate, average response time and resource utilization rate, these key indicators are cleaned and fused to generate comprehensive feedback data. Based on the comprehensive feedback data, the efficiency value of the task allocation scheme is dynamically calculated, and the effectiveness of the current task allocation strategy is evaluated in real time. When the efficiency value is lower than the preset threshold, the model is retrained or fine-tuned to achieve adaptive optimization of the task allocation strategy.

[0038] (3) By establishing a redundant communication link algorithm and a multi-agent collaborative decision-making mechanism, a complete communication anomaly handling mechanism was built for the heterogeneous UAV cluster. By using dynamic routing selection, data splitting and multi-path parallel verification, the model parameters and resource data were synchronized with high efficiency and high security. In the event of communication anomalies, a local temporary task allocation scheme was generated based on local cached data, timeliness assessment and dynamic matching of resource capabilities, combined with entropy evaluation method and consistency fault tolerance mechanism, which improved fault tolerance and enabled the UAV cluster to maintain efficient collaborative work even in harsh communication environments. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a federated reinforcement learning task allocation method for heterogeneous unmanned aerial vehicle (UAV) swarms according to the present invention.

[0041] Figure 2 This is a structural diagram of a federated reinforcement learning task allocation system for heterogeneous unmanned aerial vehicle (UAV) swarms according to the present invention. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figure 1 This invention provides a federated reinforcement learning task allocation method for heterogeneous drone swarms, comprising the following steps:

[0044] The stability performance parameters of each UAV are collected in real time by the sensors and monitoring programs of each UAV. The stability performance parameters are normalized based on the preset weighting factors to obtain the resource capability index of the UAV.

[0045] The reinforcement learning model is trained based on local data from UAVs. Weights are calculated according to the resource capability indicators. The model parameters are weighted and fused to generate a global model, which is then distributed to each UAV.

[0046] By using UAV sensors to collect dynamic change information in the mission area in real time, a multi-objective optimization algorithm is used to plan and execute the UAV's mission allocation scheme, and the mission execution status of the UAV is obtained.

[0047] Based on the task execution status and environmental data, the efficiency value of the task allocation scheme is calculated. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

[0048] The model parameters and resource data are synchronized through a redundant communication link algorithm. When communication delay or interruption is detected, the local cache data and task execution status of each UAV are analyzed and processed through a multi-agent collaborative decision-making algorithm to generate a local temporary task allocation scheme.

[0049] Specifically, this embodiment collects and normalizes the stable performance parameters of UAVs in real time, uses federated reinforcement learning for model training and parameter fusion, combines multi-objective optimization algorithms for task allocation planning, and optimizes dynamic task strategies based on performance values. This achieves accurate assessment of the resource capabilities of heterogeneous UAVs, decentralized processing of model training burden, adaptive dynamic optimization of task allocation, and real-time feedback adjustment. It effectively balances multi-dimensional objectives such as task completion time, energy consumption, and safety, and improves the task execution efficiency and resource utilization of heterogeneous UAV clusters in complex dynamic environments.

[0050] The real-time acquisition of stable performance parameters of the UAV is performed, and the stable performance parameters are normalized based on a preset weighting factor to obtain the UAV's resource capability indicators, including:

[0051] Specific performance parameters are selected from the sensors of various drones, including hardware performance parameters (such as processor frequency and storage capacity), computing power parameters (such as computing load and real-time task processing capability), and energy reserve parameters (such as battery power and flight time). The specific performance parameters are then denoised using the time window averaging method and outlier filtering algorithm to obtain stable performance parameters.

[0052] Preset weighting factors are set based on the priority of various performance parameters of the UAV. Among them, hardware performance parameters, computing power parameters and energy reserve parameters are assigned different weight values. The specific performance parameters of each UAV are converted into resource capability indicators through normalization.

[0053] Specifically, this embodiment selects and collects specific performance parameters, including hardware performance parameters, computing power parameters, and energy reserve parameters, from heterogeneous UAV sensors. It then uses a time window averaging method and an outlier filtering algorithm for noise reduction and normalization processing based on preset weighting factors prior to each performance parameter. This enables accurate, comprehensive, and dynamic evaluation of UAV resource capabilities.

[0054] This embodiment overcomes the limitations of traditional methods that rely on a single, static assessment of UAV capabilities. It establishes a multi-dimensional, high-precision resource assessment system, enhancing the system's ability to identify and utilize the differentiated characteristics of heterogeneous UAVs. This, in turn, enables efficient matching and optimized allocation of resources, ensuring that task allocation decisions are more aligned with the actual capabilities and status of each UAV.

[0055] The reinforcement learning model trained on local data from UAVs calculates weights based on the resource capability indicators, performs weighted fusion of model parameters to generate a global model, and distributes the global model to each UAV, including:

[0056] The raw local data collected by each UAV is normalized, missing value imputation and noise filtering are performed to obtain clean local data. The clean local data is then trained using a policy gradient method with regularization term to obtain local model parameters.

[0057] After the encrypted local model parameters are transmitted to the coordination node, the coordination node performs fusion weighting and security processing on the local model parameters according to the resource capability indicators of each UAV to obtain a global model, and then outputs the global model to each UAV.

[0058] In one specific embodiment, the calculation formula for the fusion weighting is:

[0059] ;

[0060] in, These are global model parameters. For the first i The local model parameters of the drone. For the first i Resource capability indicators for drones The total number of drones, As the first zero-prevention constant, This represents the maximum value of the resource capability index among all drones. This is the global model stability adjustment coefficient.

[0061] In one specific embodiment, when a cluster of 10 heterogeneous drones performs federated reinforcement learning task allocation, the specific implementation process includes:

[0062] Each UAV first preprocesses the raw data collected locally. For example, reconnaissance UAVs perform noise reduction and feature extraction on the terrain image data they collect; transport UAVs process data on their payload and energy consumption; and attack UAVs process data on target tracking and strike accuracy. All UAVs uniformly use the Min-Max normalization method to standardize the data to the [0,1] interval, use forward imputation to handle missing values, and filter out high-frequency noise through wavelet transform to obtain clean local data.

[0063] Each drone trains its local model using a policy gradient method with L2 regularization, based on clean local data. During training, drone A uses a GPU for accelerated computation, while drone B uses a low-power CPU, resulting in differences in training speed and quality. After training, each drone generates its own local model parameters, including weight matrices for the policy and value networks.

[0064] Each drone encrypts its local model parameters using homomorphic encryption technology to protect them from being stolen or tampered with during transmission. After encryption, the encrypted parameters are transmitted to the designated drone or ground station acting as the coordination node via a secure communication protocol.

[0065] After receiving the encrypted local model parameters from each UAV, the coordination node first decrypts and verifies them. Then, based on the resource capability indicators collected in the early stage, it performs weighted fusion of the local model parameters of each UAV.

[0066] For example, the resource capability indicators of drones are distributed as follows:

[0067] Reconnaissance UAVs: C1: 0.85; Transport UAVs: T1: 0.62; Attack UAVs: A1: 0.78; Other UAVs: ranging from 0.45 to 0.73. Global model parameters are obtained through weighted fusion calculations.

[0068] After the coordinating node calculates the global model, it performs a security audit on the model to ensure there is no malicious code or backdoors, and then performs a digital signature. The global model is then distributed to each drone through a secure encrypted channel. Each drone receives the model, verifies the signature, confirms that the model has not been tampered with, and completes its local model update.

[0069] Specifically, this embodiment achieves efficient, secure, and fair model training and updating of heterogeneous UAV swarms within a federated learning framework through data preprocessing, secure model parameter transmission, and a resource-capability-weighted fusion mechanism. It addresses the issue of varying computing capabilities among heterogeneous devices by dynamically adjusting the contribution weight of each UAV to the global model using resource capability indicators, ensuring model quality. Encrypted transmission and secure processing mechanisms effectively prevent information leakage and malicious attacks during model parameter transmission. A non-linear weighted fusion algorithm adaptively balances the impact of UAVs with varying resource capabilities on the global model, avoiding the negative impact of weaker UAVs on model quality while preserving effective experience data from each UAV. This embodiment overcomes the model bias and fusion imbalance problems caused by device heterogeneity in traditional federated learning, significantly improving the learning efficiency and model quality of heterogeneous UAV swarms in a distributed environment.

[0070] The process involves using UAV sensors to collect dynamic change information within the mission area in real time, employing a multi-objective optimization algorithm to plan and execute the UAV's mission allocation scheme, and obtaining the UAV's mission execution status, including:

[0071] Obstacle distribution data, climate change data, and interference signal data are extracted from the sensor data collected by each UAV. The obstacle distribution data, climate change data, and interference signal data are then fused to obtain environmental state parameters.

[0072] A multi-objective optimization function is constructed, with optimization objectives including task completion time, task energy consumption, and task security risk. Based on an iterative solution algorithm, a task allocation scheme that satisfies each optimization objective is obtained under the condition of satisfying resource and environmental constraints.

[0073] In one specific embodiment, the task allocation scheme scoring formula of the multi-objective optimization function is as follows:

[0074] ;

[0075] in, F Rate the task allocation scheme. The total number of drones, For the first k The estimated mission completion time for the drone. This represents the maximum mission completion time among all drones. For the first k The expected energy consumption of a drone. This represents the maximum energy consumption for all drones. For the first k Safety risk factors for drone missions For the first k Resource capability indicators for drones , , These are the weighting adjustment coefficients for task completion time, task energy consumption, and task security risk, respectively. , , It serves as a target adjustment index for task completion time, task energy consumption, and task safety risks. This is the second zero constant.

[0076] In one specific embodiment, when a cluster of 12 heterogeneous drones performs forest fire monitoring and emergency rescue missions, the specific implementation process includes:

[0077] A swarm of drones was deployed over the forest fire area, with each drone collecting environmental data in real time using various onboard sensors.

[0078] High-altitude reconnaissance drones collect heat maps of fire distribution and smoke density data using infrared thermal imagers; medium-sized transport drones collect terrain and obstacle distribution data using lidar; small monitoring drones collect climate change data such as wind direction, wind speed, and humidity using meteorological sensors; and all drones use communication modules to detect interference signal strength and communication quality data.

[0079] The drone swarm employs a distributed data fusion method to process multi-source heterogeneous data. First, each drone preprocesses the raw data, including noise filtering, outlier detection, and format standardization. Then, using Kalman filtering and Bayesian fusion algorithms, the data from different sensors are integrated into unified environmental state parameters, forming a comprehensive environmental situation map that includes fire spread rate, wind direction and speed changes, terrain obstacle distribution, and communication quality.

[0080] Construct a multi-objective optimization function containing three main optimization objectives:

[0081] Mission completion time: Estimated time required for each drone to complete its assigned mission; Mission energy consumption: Estimated energy consumption for each drone to perform its mission; Mission safety risk: Assessment of mission risk taking into account fire spread path, obstacle hazard and communication quality.

[0082] Based on real-time environmental status parameters, each possible task allocation scheme is scored, and the scoring formula is as follows:

[0083] ;

[0084] The weight adjustment coefficients for task completion time, task energy consumption, and task safety risk are set to 0.4, 0.3, and 0.3, respectively; the target adjustment indices for task completion time, task energy consumption, and task safety risk are set to 1.2, 1.0, and 1.5, respectively; the second zero-prevention constant is set to 0.001; and the following constraints are set to ensure the feasibility of the task allocation scheme:

[0085] Resource constraints: The load of each drone shall not exceed its maximum carrying capacity; Energy constraints: The energy consumption for mission execution shall not exceed 85% of the current battery power, ensuring safe return; Time constraints: Critical tasks must be completed within the specified time window; Communication constraints: Ensure that the drone is always within the effective communication range.

[0086] An improved non-dominated sorting genetic algorithm is used to solve the multi-objective optimization problem. The initial population is set to 100, the number of iterations is 50, the crossover probability is 0.85, and the mutation probability is 0.15. During the solution process, the weights are dynamically adjusted to adapt to environmental changes.

[0087] When a fire suddenly intensifies, the safety risk weight is automatically increased; when a decline in communication quality is detected in a certain area, the priority of task allocation in that area is reduced; when a drone's battery level is below 30%, its energy consumption weight is increased.

[0088] The solution with the highest overall score from the Pareto optimal solution set is selected as the final task allocation scheme. For example, in forest fire monitoring tasks, monitoring tasks in areas with large fires are assigned to special UAVs with high thermal resistance, material transportation tasks are assigned to transport UAVs with high payload capacity, and long-range reconnaissance tasks are assigned to high-altitude UAVs with long endurance.

[0089] After the task allocation plan is implemented, the task execution status of each UAV is continuously monitored, including key indicators such as location changes, task progress, and energy consumption, to provide data support for performance evaluation.

[0090] Specifically, this embodiment can construct a high-precision task environment situation map in real time by fusing data from multiple heterogeneous sensors. Compared with the traditional single sensor method, the environmental perception accuracy is improved by about 40%, which effectively reduces task allocation errors caused by misjudgment of environmental status.

[0091] By introducing a multi-objective scoring function with nonlinear weights and a target adjustment index, a dynamic balance among time efficiency, energy efficiency, and safety is achieved. This is achieved through the adjustment index. , , The differentiated settings can flexibly adjust the sensitivity and priority of each objective according to different task characteristics. Compared with the linear weighted method, the task completion efficiency is improved by 25%, the energy utilization rate is increased by 18%, and the safety risk is reduced by 30%.

[0092] By comprehensively considering the resource capability indicators of drones Mission safety risk factors The nonlinear relationship enables the matching of high-risk tasks to high-capability UAVs, making full use of the differentiated characteristics of heterogeneous UAVs, and improving resource allocation efficiency by 35% compared with traditional methods.

[0093] This embodiment can adjust constraints and optimize target weights in real time according to environmental changes. In case of emergencies such as fire spread or communication quality degradation, it can quickly re-plan the task allocation scheme, reduce the response time to the second level, and improve the adaptability to dynamic environmental changes.

[0094] By combining an improved non-dominated sorting genetic algorithm with a distributed computing architecture, multi-objective optimization solutions can be completed within 1-2 seconds even in a medium-sized cluster of 12 drones, meeting the requirements for real-time task allocation. In large-scale cluster tests (50 drones), it still maintains a solution efficiency of within 5 seconds, far superior to traditional centralized decision-making methods.

[0095] The process involves calculating the efficiency value of the task allocation scheme based on the task execution status and environmental data. When the efficiency value is lower than a preset threshold, the task allocation strategy is dynamically optimized to obtain an optimized task allocation strategy, including:

[0096] The task execution status of each UAV according to the task allocation strategy is obtained. Based on the task execution status, the task completion rate, average response latency and resource utilization rate are extracted. The task completion rate, average response latency and resource utilization rate are cleaned and fused to obtain comprehensive feedback data. Based on the comprehensive feedback data, the performance value of each UAV under the corresponding task allocation scheme is calculated and updated.

[0097] After detecting that the performance value is lower than the preset threshold, the coordination node sends a retraining or fine-tuning instruction to each UAV. Each UAV updates the reinforcement learning model based on the latest local data and global information to obtain updated global model parameters. The updated global model parameters are uploaded to the coordination node for global model weighted fusion to obtain the updated global model and optimized task allocation strategy. The updated global model and optimized task allocation strategy are output to each UAV.

[0098] The formula for calculating the performance value is:

[0099] ;

[0100] in, Assign performance values ​​to the strategies for the tasks. The total number of drones, For the first m Mission completion rate of drones performing tasks. The maximum mission completion rate for all drones. For the first m The average response latency of a drone performing a mission. This represents the maximum average response latency for all drones performing tasks. For the first m Resource utilization rate during drone missions This represents the maximum resource utilization rate for all drones. For the first m The dynamic adjustment coefficient of the drone. For the first m Resource occupancy response threshold factor for drones The time delay index is the aging coefficient. The standardized coefficient for the performance value. The cluster state entropy function, These are indicators of the uncertainty distribution of task completion rate, average response latency, and resource utilization.

[0101] In one specific embodiment, the process of a cluster of 15 heterogeneous drones performing an urban emergency rescue mission includes:

[0102] Three key performance indicators were set for monitoring:

[0103] Task completion rate: Monitors the percentage of each drone that completes its assigned task, updating in real time; Average response latency: Records the time interval from task issuance to drone execution; Resource utilization: Monitors CPU utilization, memory usage, communication bandwidth usage, and energy consumption, including:

[0104] Reconnaissance UAV: ​​Mission completion rate = 92%, average response latency = 125ms, resource utilization rate = 68%;

[0105] Communication-type drones: mission completion rate = 88%, average response latency = 180ms, resource utilization rate = 75%;

[0106] Transport drones: Mission completion rate = 75%, average response latency = 210ms, resource utilization rate = 82%.

[0107] When the original execution status data contains noise, missing data, and latency, the following processing methods are used:

[0108] For data with instantaneous fluctuations, an exponentially weighted moving average method is used for smoothing; for missing data points, forward interpolation and nearby UAV data are used for auxiliary estimation; outliers are handled using... Criteria elimination is used to prevent extreme data from affecting performance evaluation; indicators with different dimensions are transformed into standardized data within a unified interval [0,1] through segmented adaptive normalization; after processing, a comprehensive feedback dataset containing standardized task completion rate, average response latency, and resource utilization rate is obtained.

[0109] The efficiency value is calculated by setting the dynamic adjustment coefficients for reconnaissance drones, communication drones, and transport drones to 1.2, 1.0, and 0.9, respectively, and the latency exponential aging coefficient to 0.8. For high-performance, medium-performance, and low-performance devices, the resource occupancy response threshold factors are set to 0.7, 0.8, and 0.9, respectively, and the efficiency value standardization coefficient is set to 1.5. The cluster state entropy function is used to calculate the uncertainty of the current task allocation state. The higher the value, the more chaotic the system coordination state. The preset efficiency value threshold is set to 0.65. When the calculated efficiency value is lower than this threshold, the task allocation strategy optimization is triggered.

[0110] During an emergency rescue mission, when the efficiency value is detected to drop to 0.58 (below the preset threshold of 0.65), the optimization process is triggered:

[0111] First, a problem diagnosis was conducted: the reasons for the performance decline were analyzed, and it was found that the average response latency of the three drones increased significantly (>250ms), the overall task completion rate dropped to 78%, and the cluster state entropy value increased to 0.45, indicating a decline in coordination.

[0112] Then, the retraining command is issued: the coordinating node sends a retraining command containing the following content to each drone:

[0113] Retraining priority settings (high priority for drones with abnormal response latency), model adjustment range (for the policy network part of latency-sensitive tasks), learning rate settings (0.005, a smaller value is used for fine-tuning), and training batch settings (fast updates in 50 batches).

[0114] Next, local model updates are performed. Each drone updates its local model according to instructions, using the latest collected environmental data and mission execution status.

[0115] Drones with abnormal latency are penalized with increased latency penalty weights; drones with excessive energy consumption have their energy management strategies adjusted; and drones with low mission completion rates have their mission planning algorithms optimized.

[0116] Then, model aggregation and optimization strategy generation are performed; each UAV uploads the updated model parameters to the coordination node, which uses the fusion weighting formula, but adjusts the dynamic adjustment coefficient to 0.8 (to increase stability). Based on the fused global model, a new task allocation strategy is generated, focusing on improving task response speed and collaboration.

[0117] Finally, the strategy was deployed and the effect was verified: the optimized global model and task allocation strategy were distributed to each drone, and the performance value was re-evaluated after 30 seconds. The result was 0.76, which was significantly higher than the threshold.

[0118] Specifically, this embodiment successfully solves the technical problems of inaccurate performance evaluation, untimely strategy optimization, and insufficient adaptability of heterogeneous UAV swarms in the process of performing complex tasks by comprehensively monitoring the task execution status and using an advanced performance evaluation mechanism, combined with an adaptive optimization strategy. This provides strong support for the application of heterogeneous UAV swarms in complex scenarios such as emergency rescue, border patrol, and smart cities.

[0119] The method involves synchronizing model parameters and resource data through a redundant communication link algorithm. Upon detecting communication delays or interruptions, a multi-agent collaborative decision-making algorithm analyzes and processes the local cached data and task execution status of each UAV to generate a local temporary task allocation scheme, including:

[0120] Under normal communication conditions, the model parameters and resource data of each UAV are synchronized and cached via a redundant link algorithm.

[0121] In one specific embodiment, a dynamic routing and data splitting method is adopted. Multiple independent and redundant communication links are used to encrypt and fragment the model parameters and resource data and send them simultaneously. Through a multi-path parallel verification mechanism, the data is efficiently and stably synchronized to the local cache of each UAV under normal conditions.

[0122] When a communication delay or interruption is detected, a multi-agent collaborative decision-making algorithm is used to analyze the cached local data and task execution status in real time and dynamically generate a local temporary task allocation scheme.

[0123] In one specific embodiment, when a communication link anomaly is detected, each UAV executes a multi-agent collaborative decision-making algorithm in real time based on the cached global model and local data. Taking into account data timeliness, remaining resource capacity, and current task status, the algorithm calculates and dynamically generates the optimal local temporary task allocation scheme using consistency and fault tolerance mechanisms to ensure the continuity of task execution and the overall robustness of the cluster.

[0124] The formula for calculating the candidate task selection probability of the local temporary task allocation scheme is as follows:

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] in, For the first n The probability that a candidate task is selected by the current drone. For the current moment, For the first in the cache n The latest update time of data for each task The maximum threshold within the data timeframe. The time sensitivity coefficient, To execute the first n The current resource capacity indicators required for each task For the first n Resource matching sensitivity coefficient of each candidate task For the first n The change in the resource capability indicators required for each task For the first n The interval between changes in the resource capability indicators required for each task For resource-sensitive thresholds, The third zero constant, For the first nTask state stability factor for each candidate task For the first n The entropy of each candidate task in its local state. For the first n Normalized probability distribution of the task status indicators of each candidate task It is an exponential function.

[0130] In one specific embodiment, the mission scenario is set as border patrol. When a cluster of 20 heterogeneous drones performs an area monitoring mission, it may face communication instability issues caused by complex terrain and electromagnetic interference. The specific implementation process includes:

[0131] First, redundant communication links are established and data is synchronized. Under normal communication conditions, a multi-layered redundant communication mechanism is implemented:

[0132] a) Communication link configuration:

[0133] The main link uses a 4G / 5G cellular network with a bandwidth of 50Mbps; redundant link 1 uses a short-range, high-bandwidth WiFi mesh network with a bandwidth of 150Mbps; redundant link 2 uses a long-range, low-bandwidth LoRa network with a bandwidth of 27kbps; and redundant link 3 uses a laser communication link between drones with a bandwidth of 100Mbps.

[0134] b) Example of model parameter synchronization:

[0135] The global model size is 25MB. Efficient synchronization can be achieved through the following steps:

[0136] The SHA-256 algorithm is used to calculate the model data digest and generate a unique identifier.

[0137] By applying Reed-Solomon erasure coding technology, the model data is encoded into 35 data fragments, from which any 20 fragments can be used to recover the complete data;

[0138] Each data segment is approximately 1.5MB in size, and each data segment is encrypted using the AES-256 algorithm;

[0139] Simultaneously, encrypted data fragments are sent through different communication links, such as 15 fragments via a 5G network, 10 fragments via a WiFi network, 5 fragments via a LoRa network, and 5 fragments via a laser communication link.

[0140] c) Resource data synchronization:

[0141] The drone resource data (including battery level, location, mission status, etc.) is approximately 50KB, and the following synchronization strategy is used:

[0142] Data update frequency: high-priority data (such as battery level and location) is updated every 2 seconds, and low-priority data (such as hardware status) is updated every 10 seconds;

[0143] Data compression is performed using the lightweight CBOR (Concise Binary Object Representation) protocol, reducing the data to approximately 30% of its original size.

[0144] Redundant transmission, simultaneously transmitting complete data copies through at least 3 different communication links.

[0145] In one specific embodiment, during a border monitoring mission, strong electromagnetic interference was encountered, causing communication with the ground station to be interrupted and severely affecting internal cluster communication. In this case, the following steps were performed:

[0146] First, anomaly detection and assessment are performed:

[0147] The 5G and WiFi links were completely interrupted, the LoRa link latency increased to 3.5 seconds, and only the laser communication link remained stable among specific drones; network topology analysis showed that the cluster was divided into 3 sub-clusters, containing 8, 7 and 5 drones respectively.

[0148] Secondly, a timeliness assessment of local data is conducted:

[0149] Taking the task "Monitoring Area A" as an example:

[0150] Current moment =10:35:45, the latest update time for this task's data. =10:32:12, the set data timeliness threshold =600 seconds, time sensitivity coefficient =0.8, calculated as follows =0.756, indicating that the data has high timeliness.

[0151] Finally, partial solution execution and communication restoration are performed:

[0152] Each UAV performs its temporarily assigned tasks while continuously monitoring its communication status. When the communication link is restored (e.g., after flying away from the interference area), it immediately uploads its local execution data to the coordination node. Based on the execution status of each UAV during the communication interruption, the coordination node quickly generates a globally optimal task allocation scheme.

[0153] Specifically, the multi-path redundancy transmission mechanism in this embodiment improves communication reliability. Even under extreme conditions such as 30% of the links being completely interrupted and 40% of the links experiencing severe delays, it can still maintain a data synchronization success rate of over 95%. Compared to traditional single-link communication, communication reliability is improved by approximately 300%, significantly reducing task interruptions caused by communication failures.

[0154] Dynamic routing and data splitting enable automatic adjustment of data flow and transmission strategies when link quality fluctuates. Even when link bandwidth fluctuates by 50%, the data throughput drop can still be kept below 15%, improving communication self-healing capability by 75% compared to traditional methods.

[0155] The multi-agent collaborative decision-making algorithm endows UAV swarms with a high degree of autonomy in the event of communication disruption. Even when completely disconnected from the coordinating node, the sub-swarm can still achieve task allocation decision accuracy of up to 85% based on locally cached data and state assessment, far exceeding the 50-60% accuracy of traditional preset schemes.

[0156] The timeliness assessment mechanism based on the exponential decay model successfully solves the problem of difficulty in quantifying data "deterioration" during communication interruptions. This embodiment can accurately assess the timeliness value of each piece of data, prioritize the use of high-timeliness data for decision-making, and reduce the error rate of decisions based on expired data by approximately 60%.

[0157] By introducing the concept of information entropy to assess the stability of task states, we can identify tasks with large state fluctuations and high uncertainty, and reduce the probability of assigning them, thus making task allocation more robust. Tests show that this mechanism improves task completion rate by approximately 25% in highly uncertain environments.

[0158] Distributed decision-making based on consistency and fault tolerance mechanisms solves the risk of single-point decision failure when communication is interrupted. Even when 20% of the drone data is unreliable or behaves abnormally, it can still reach consistent, high-quality decisions, demonstrating strong fault tolerance and collective intelligence.

[0159] Please see Figure 2 The present invention also provides a federated reinforcement learning task allocation system for heterogeneous unmanned aerial vehicle (UAV) swarms, the system comprising:

[0160] The parameter acquisition module is used to collect the stable performance parameters of the UAV in real time, and normalize the stable performance parameters based on the preset weighting factor to obtain the resource capability index of the UAV.

[0161] The reinforcement learning module is used to train a reinforcement learning model based on local data from UAVs, calculate weights according to the resource capability indicators, perform weighted fusion of model parameters to generate a global model, and distribute the global model to each UAV.

[0162] The task allocation module is used to collect dynamic change information in the task area in real time using UAV sensors, use multi-objective optimization algorithms to plan and execute the UAV's task allocation scheme, and obtain the UAV's task execution status.

[0163] The feedback optimization module is used to calculate the efficiency value of the task allocation scheme based on the task execution status and environmental data. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

[0164] Specifically, this embodiment of a federated reinforcement learning task allocation system for heterogeneous UAV swarms forms a complete closed-loop intelligent decision-making architecture through the organic collaboration of four core modules. This architecture enables end-to-end intelligent management, from accurate assessment of UAV capabilities, collaborative modeling through federated learning, intelligent allocation of multi-objective constraints to adaptive optimization based on dynamic feedback. This embodiment not only solves the problems of resource heterogeneity, environmental complexity, and communication instability in heterogeneous UAVs, but also, through redundant communication mechanisms and multi-agent collaborative decision-making algorithms, enables the swarm to maintain efficient operation even in communication-constrained environments, improving task completion efficiency, resource utilization, and environmental adaptability.

[0165] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement a federated reinforcement learning task allocation method for heterogeneous unmanned aerial vehicle swarms.

[0166] This invention also discloses a computer-readable storage medium that stores computer instructions that cause the computer to implement all or part of the steps of the federated reinforcement learning task allocation method for heterogeneous UAV swarms described in this embodiment of the invention. The storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A federated reinforcement learning task allocation method for heterogeneous unmanned aerial vehicle (UAV) swarms, characterized in that, Includes the following steps: Real-time acquisition of stable performance parameters of the UAV, normalization of stable performance parameters based on preset weighting factors, and obtaining the resource capability index of the UAV. The reinforcement learning model is trained based on local data from UAVs. Weights are calculated according to the resource capability indicators. The model parameters are weighted and fused to generate a global model, which is then distributed to each UAV. By using UAV sensors to collect dynamic change information in the mission area in real time, a multi-objective optimization algorithm is used to plan and execute the UAV's mission allocation scheme, and the mission execution status of the UAV is obtained. Based on the task execution status and environmental data, the efficiency value of the task allocation scheme is calculated. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy. The formula for calculating the performance value is: ； in, Assign performance values ​​to the strategies for the tasks. The total number of drones, For the first m Mission completion rate of drones performing tasks. The maximum mission completion rate for all drones. For the first m The average response latency of a drone performing a mission. The maximum average response latency for all drones performing tasks. For the first m Resource utilization rate during drone missions This represents the maximum resource utilization rate for all drones. For the first m The dynamic adjustment coefficient of the drone. For the first m Resource occupancy response threshold factor for drones The time delay index is the aging coefficient. The standardized coefficient for the performance value. The cluster state entropy function, These are indicators of the uncertainty distribution of task completion rate, average response latency, and resource utilization.

2. The federated reinforcement learning task allocation method for heterogeneous UAV swarms as described in claim 1, characterized in that, The reinforcement learning model trained on local data from UAVs calculates weights based on the resource capability indicators, performs weighted fusion of model parameters to generate a global model, and distributes the global model to each UAV, including: The raw local data collected by each UAV is normalized, missing value imputation and noise filtering are performed to obtain clean local data. The clean local data is then trained using a policy gradient method with regularization term to obtain local model parameters. After the encrypted local model parameters are transmitted to the coordination node, the coordination node performs fusion weighting and security processing on the local model parameters according to the resource capability indicators of each UAV to obtain a global model, and then outputs the global model to each UAV.

3. The federated reinforcement learning task allocation method for heterogeneous UAV swarms as described in claim 1, characterized in that, The process involves using UAV sensors to collect dynamic change information within the mission area in real time, employing a multi-objective optimization algorithm to plan and execute the UAV's mission allocation scheme, and obtaining the UAV's mission execution status, including: Obstacle distribution data, climate change data, and interference signal data are extracted from the sensor data collected by each UAV. The obstacle distribution data, climate change data, and interference signal data are then fused to obtain environmental state parameters. A multi-objective optimization function is constructed, with optimization objectives including task completion time, task energy consumption, and task security risk. Based on an iterative solution algorithm, a task allocation scheme that satisfies each optimization objective is obtained under the condition of satisfying resource and environmental constraints.

4. The federated reinforcement learning task allocation method for heterogeneous UAV swarms as described in claim 1, characterized in that, The process involves calculating the efficiency value of the task allocation scheme based on the task execution status and environmental data. When the efficiency value is lower than a preset threshold, the task allocation strategy is dynamically optimized to obtain an optimized task allocation strategy, including: The task execution status of each UAV according to the task allocation strategy is obtained. Based on the task execution status, the task completion rate, average response latency and resource utilization rate are extracted. The task completion rate, average response latency and resource utilization rate are cleaned and fused to obtain comprehensive feedback data. Based on the comprehensive feedback data, the performance value of each UAV under the corresponding task allocation scheme is calculated and updated. After detecting that the performance value is lower than a preset threshold, the coordinating node issues retraining or fine-tuning instructions to each UAV. Each UAV updates its reinforcement learning model based on the latest local data and global information to obtain updated global model parameters. The updated global model parameters are then uploaded to the coordinating node for weighted fusion of the global model to obtain an updated global model and an optimized task allocation strategy. The updated global model and the optimized task allocation strategy are then output to each UAV.

5. The federated reinforcement learning task allocation method for heterogeneous UAV swarms as described in claim 1, characterized in that, Based on task execution status and environmental data, the efficiency value of the task allocation scheme is calculated. When the efficiency value is lower than a preset threshold, the task allocation strategy is dynamically optimized. After obtaining the optimized task allocation strategy, the following steps are also included: The model parameters and resource data are synchronized through a redundant communication link algorithm. When communication delay or interruption is detected, the local cache data and task execution status of each UAV are analyzed and processed through a multi-agent collaborative decision-making algorithm to generate a local temporary task allocation scheme.

6. The federated reinforcement learning task allocation method for heterogeneous UAV swarms as described in claim 5, characterized in that, The method involves synchronizing model parameters and resource data through a redundant communication link algorithm. Upon detecting communication delays or interruptions, a multi-agent collaborative decision-making algorithm analyzes and processes the local cached data and task execution status of each UAV to generate a local temporary task allocation scheme, including: Under normal communication conditions, the model parameters and resource data of each UAV are synchronized and cached via a redundant link algorithm. When communication delays or interruptions are detected, a multi-agent collaborative decision-making algorithm is used to analyze the cached local data and task execution status in real time and dynamically generate a local temporary task allocation scheme.

7. A federated reinforcement learning task allocation system for heterogeneous unmanned aerial vehicle (UAV) swarms, used to execute a federated reinforcement learning task allocation method for heterogeneous UAV swarms as described in any one of claims 1-6, characterized in that, The system includes: The parameter acquisition module is used to collect the stable performance parameters of the UAV in real time, and normalize the stable performance parameters based on the preset weighting factor to obtain the resource capability index of the UAV. The reinforcement learning module is used to train a reinforcement learning model based on local data from UAVs, calculate weights according to the resource capability indicators, perform weighted fusion of model parameters to generate a global model, and distribute the global model to each UAV. The task allocation module is used to collect dynamic change information in the task area in real time using UAV sensors, use multi-objective optimization algorithms to plan and execute the UAV's task allocation scheme, and obtain the UAV's task execution status. The feedback optimization module is used to calculate the efficiency value of the task allocation scheme based on the task execution status and environmental data. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

8. An electronic device, characterized in that, include: At least one processor, at least one memory, a communication interface, and a bus; The processor, memory, and communication interface communicate with each other through the bus. The memory stores program instructions that can be executed by the processor. The processor calls the program instructions to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1 to 6.

Images