Federal reinforcement learning task allocation method and system for heterogeneous unmanned aerial vehicle cluster

Abstract

The invention relates to the technical field of unmanned aerial vehicle clusters, and provides a federal reinforcement learning task allocation method and system for a heterogeneous unmanned aerial vehicle cluster, and the method comprises the steps: collecting the stability performance parameters of unmanned aerial vehicles in real time, carrying out the normalization processing of the stability performance parameters based on a preset weight factor, and obtaining the resource capability indexes of the unmanned aerial vehicles; training a reinforcement learning model, calculating a weight according to the resource capability index, and performing weighted fusion on model parameters to generate a global model; dynamic change information in a task area is collected in real time through an unmanned aerial vehicle sensor, a task allocation scheme of the unmanned aerial vehicle is planned and executed through a multi-target optimization algorithm, and the task execution state of the unmanned aerial vehicle is obtained; and calculating an efficiency value of the task allocation scheme according to the task execution state and the environment data, and when the efficiency value is lower than a preset threshold value, dynamically optimizing the task allocation strategy to obtain an optimized task allocation strategy. The task execution efficiency and the resource utilization rate of the heterogeneous unmanned aerial vehicle cluster in a complex dynamic environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of drone cluster technology, and in particular to a method and system for allocating federated reinforcement learning tasks for heterogeneous drone clusters. Background Art

[0002] A heterogeneous drone swarm refers to a collaborative system composed of drones of varying types, functions, and specifications. Compared to traditional drone systems consisting of a single type or specification, heterogeneous drone swarms typically include reconnaissance drones, transport drones, attack drones, and other drones, each with significant differences in hardware architecture, computing power, sensor configuration, energy reserves, and mission capabilities. Leveraging this heterogeneous combination, the swarm can maximize the diverse advantages of drones, enabling them to simultaneously fulfill diverse functional requirements within complex missions, resulting in greater flexibility, mission adaptability, and coverage. However, heterogeneous drone swarms also present additional challenges, the key being how to coordinate the resource differences among different drone types, efficiently allocate tasks, and achieve stable collaborative operations.

[0003] In existing technologies, task allocation for heterogeneous drone swarms mainly adopts centralized decision-making architecture, static fixed-weight resource assessment, and linear weighted multi-objective optimization methods to achieve basic task planning and allocation functions. However, the centralized architecture leads to single-point failure risks and high communication overhead, static weights cannot adapt to dynamic environmental changes, linear optimization has difficulty balancing multi-dimensional conflicting objectives, and lacks fault-tolerant mechanisms for communication interruptions and real-time performance feedback adjustment capabilities. This makes it difficult for existing systems to achieve efficient and robust task allocation in complex environments, especially when faced with limited communication, dynamic environmental changes, and imbalanced heterogeneous resources. Summary of the Invention

[0004] In view of this, the present invention proposes a federated reinforcement learning task allocation method and system for heterogeneous drone clusters, which solves the problem that the existing technology lacks a fault-tolerant mechanism for communication interruptions and real-time performance feedback adjustment capabilities, and is difficult to achieve efficient and robust task allocation in complex environments, especially when faced with limited communication, dynamic environmental changes, and imbalanced heterogeneous resources.

[0005] The technical solution of the present invention is implemented as follows: In a first aspect, the present invention provides a method for allocating federated reinforcement learning tasks for a heterogeneous UAV cluster, comprising the following steps: Collect the stability performance parameters of the UAV in real time, normalize the stability performance parameters based on the preset weight factors, and obtain the resource capability index of the UAV; A reinforcement learning model is trained based on the local data of the drone, weights are calculated according to the resource capability indicators, model parameters are weightedly integrated to generate a global model, and the global model is distributed to each drone; The drone sensors are used to collect dynamic change information in the mission area in real time, and a multi-objective optimization algorithm is used to plan and execute the drone's task allocation plan to obtain the drone's task execution status; According to the task execution status and environmental data, the efficiency value of the task allocation plan is calculated. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

[0006] Based on the above technical solution, preferably, the reinforcement learning model is trained based on the local data of the UAV, weights are calculated according to the resource capability indicators, model parameters are weightedly integrated to generate a global model, and the global model is sent to each UAV, including: The raw local data collected by each drone is normalized, missing value filled, and noise filtered to obtain clean local data. The clean local data is trained using a policy gradient method with a regularization term to obtain local model parameters. After the encrypted local model parameters are transmitted to the coordination node, the coordination node performs fusion weighting and security processing on the local model parameters according to the resource capability indicators of each drone to obtain a global model, and outputs the global model to each drone.

[0007] Based on the above technical solution, preferably, the method utilizes the drone sensor to collect dynamic change information in the mission area in real time, adopts a multi-objective optimization algorithm to plan and execute the drone's task allocation plan, and obtains the drone's task execution status, including: Obstacle distribution data, climate change data, and interference signal data are extracted from the sensor data collected by each drone, and the obstacle distribution data, climate change data, and interference signal data are fused and processed to obtain environmental state parameters; A multi-objective optimization function is constructed. The optimization objectives include task completion time, task energy consumption and task safety risk. Based on the iterative solution algorithm, a task allocation plan that meets each optimization objective is solved while satisfying resource and environmental constraints.

[0008] Based on the above technical solution, preferably, the performance value of the task allocation scheme is calculated according to the task execution status and environmental data, and when the performance value is lower than a preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy, including: Obtaining the task execution status of each UAV for the task allocation strategy, extracting the task completion rate, average response delay, and resource utilization rate based on the task execution status, cleaning and fusing the task completion rate, average response delay, and resource utilization rate to obtain comprehensive feedback data, and calculating and updating the performance value of each UAV under the corresponding task allocation scheme based on the comprehensive feedback data; After detecting that the performance value is lower than the preset threshold, a retraining or fine-tuning instruction is sent to each drone through the coordination node. Each drone updates the reinforcement learning model based on the latest local data and global information to obtain updated global model parameters. The updated global model parameters are uploaded to the coordination node, and global model weighted fusion is performed to obtain an updated global model and optimized task allocation strategy. The updated global model and optimized task allocation strategy are output to each drone.

[0009] On the basis of the above technical solution, preferably, the calculation formula of the efficacy value is: ; in, The effectiveness value of the task allocation strategy, is the total number of drones, For the m The mission completion rate of UAVs performing missions, is the maximum mission completion rate of all UAVs, For the m The average response delay of the UAV in executing the task, is the maximum value of the average response delay of all UAVs performing tasks, For the m The resource utilization rate of the drone in performing the mission, is the maximum resource occupancy rate of all drones, For the m The dynamic adjustment coefficient of the UAV, For the m The resource occupation response threshold factor of the UAV, is the exponential aging coefficient of the time delay, is the standardization coefficient of the efficacy value, is the cluster state entropy function, It is the uncertainty distribution indicator of task completion rate, average response delay and resource occupancy rate.

[0010] Based on the above technical solution, preferably, after calculating the performance value of the task allocation scheme according to the task execution status and environmental data, and dynamically optimizing the task allocation strategy when the performance value is lower than a preset threshold, and obtaining the optimized task allocation strategy, the method further includes: Model parameters and resource data are synchronized through a redundant communication link algorithm. When communication delay or interruption is detected, the local cache data and task execution status of each drone are analyzed and processed through a multi-agent collaborative decision-making algorithm to generate a local temporary task allocation plan.

[0011] Based on the above technical solution, preferably, the model parameters and resource data are synchronized through the redundant communication link algorithm. When a communication delay or interruption is detected, the local cache data and task execution status of each drone are analyzed and processed through the multi-agent collaborative decision-making algorithm to generate a local temporary task allocation plan, including: Under normal communication conditions, multi-path synchronization and caching of model parameters and resource data of each UAV are performed through redundant link algorithms; When communication delay or interruption is detected, the cached local data and task execution status are analyzed in real time through a multi-agent collaborative decision-making algorithm to dynamically generate a local temporary task allocation plan.

[0012] In a second aspect, the present invention further provides a federated reinforcement learning task allocation system for heterogeneous drone clusters, the system comprising: The parameter acquisition module is used to collect the stability performance parameters of the UAV in real time, normalize the stability performance parameters based on the preset weight factors, and obtain the resource capability index of the UAV; A reinforcement learning module is used to train a reinforcement learning model based on the local data of the UAV, calculate weights according to the resource capability indicators, perform weighted fusion of model parameters to generate a global model, and distribute the global model to each UAV; The task allocation module is used to use the drone's sensors to collect dynamic change information in the task area in real time, use a multi-objective optimization algorithm to plan the drone's task allocation plan and execute it, and obtain the drone's task execution status; The feedback optimization module is used to calculate the performance value of the task allocation plan based on the task execution status and environmental data. When the performance value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

[0013] In a third aspect, the present invention further provides an electronic device comprising: at least one processor, at least one memory, a communication interface, and a bus; Among them, the processor, memory, and communication interface communicate with each other through the bus, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement steps such as a federated reinforcement learning task allocation method for a heterogeneous drone cluster.

[0014] In a fourth aspect, the present invention also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to implement the steps of a method for allocating federated reinforcement learning tasks for a heterogeneous drone cluster.

[0015] The method and system for allocating federated reinforcement learning tasks for heterogeneous drone clusters of the present invention have the following beneficial effects compared to the prior art: (1) By collecting the stable performance parameters of drones in real time and normalizing them, using federated reinforcement learning for model training and parameter fusion, combining multi-objective optimization algorithms for task allocation planning, and optimizing dynamic task strategies based on performance values, we can achieve accurate evaluation of heterogeneous drone resource capabilities, decentralized processing of model training burdens, adaptive dynamic optimization of task allocation, and real-time feedback adjustment. This effectively balances multi-dimensional goals such as task completion time, energy consumption, and safety, and improves the task execution efficiency and resource utilization of heterogeneous drone clusters in complex dynamic environments. (2) By integrating the UAV task execution status, including task completion rate, average response delay, and resource occupancy rate, these key indicators are cleaned and integrated to generate comprehensive feedback data. Based on the comprehensive feedback data, the efficiency value of the task allocation scheme is dynamically calculated, and the effectiveness of the current task allocation strategy is evaluated in real time. When the efficiency value is lower than the preset threshold, the model is retrained or fine-tuned to achieve adaptive optimization of the task allocation strategy. (3) By establishing a redundant communication link algorithm and a multi-agent collaborative decision-making mechanism, a complete communication anomaly handling mechanism is constructed for heterogeneous UAV clusters. By utilizing dynamic routing selection, data splitting, and multi-path parallel verification, high-efficiency and high-security synchronization of model parameters and resource data is achieved. In the event of communication anomalies, a local temporary task allocation plan is generated based on local cached data, timeliness evaluation, and dynamic matching of resource capabilities, combined with entropy evaluation method and consistency fault tolerance mechanism, which improves fault tolerance and enables UAV clusters to maintain efficient collaborative work even in harsh communication environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a flow chart of a method for allocating federated reinforcement learning tasks for heterogeneous drone clusters according to the present invention; Figure 2 This is a structural diagram of a federated reinforcement learning task allocation system for a heterogeneous drone cluster of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] See also Figure 1 , the present invention provides a federated reinforcement learning task allocation method for a heterogeneous UAV cluster, comprising the following steps: The stability performance parameters of each drone are collected in real time through the sensors and monitoring program of each drone, and the stability performance parameters are normalized based on the preset weight factors to obtain the resource capacity index of the drone; A reinforcement learning model is trained based on the local data of the drone, weights are calculated according to the resource capability indicators, model parameters are weightedly integrated to generate a global model, and the global model is distributed to each drone; The drone sensors are used to collect dynamic change information in the mission area in real time, and a multi-objective optimization algorithm is used to plan and execute the drone's task allocation plan to obtain the drone's task execution status; According to the task execution status and environmental data, the efficiency value of the task allocation plan is calculated. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy; Model parameters and resource data are synchronized through a redundant communication link algorithm. When communication delay or interruption is detected, the local cache data and task execution status of each drone are analyzed and processed through a multi-agent collaborative decision-making algorithm to generate a local temporary task allocation plan.

[0020] Specifically, this embodiment collects the stable performance parameters of drones in real time and normalizes them, uses federated reinforcement learning for model training and parameter fusion, combines multi-objective optimization algorithms for task allocation planning, and optimizes dynamic task strategies based on performance values. It achieves accurate assessment of heterogeneous drone resource capabilities, decentralized processing of model training burdens, adaptive dynamic optimization of task allocation, and real-time feedback adjustment, effectively balancing multi-dimensional goals such as task completion time, energy consumption, and safety, and improving the task execution efficiency and resource utilization of heterogeneous drone clusters in complex dynamic environments.

[0021] The real-time collection of the stability performance parameters of the UAV and the normalization of the stability performance parameters based on a preset weight factor to obtain the resource capability index of the UAV include: Specific performance parameters are selected from the sensor data of each drone, including hardware performance parameters (such as processor frequency and storage capacity), computing power parameters (such as computing load and real-time task processing capabilities), and energy reserve parameters (such as battery power and flight time). These specific performance parameters are then de-noised using a time window averaging method and an outlier filtering algorithm to obtain stable performance parameters. A preset weight factor is set based on the priority of each performance parameter of the drone, among which hardware performance parameters, computing power parameters and energy reserve parameters are assigned different weight values ​​respectively, and the specific performance parameters of each drone are converted into resource capacity indicators through normalization.

[0022] Specifically, this embodiment selects and collects specific performance parameters including hardware performance parameters, computing power parameters and energy reserve parameters from heterogeneous drone sensors, and uses time window averaging method and outlier filtering algorithm for denoising, and combines the preset weight factors based on the priority of each performance parameter for normalization, thereby achieving an accurate, comprehensive and dynamic evaluation of drone resource capabilities.

[0023] This embodiment overcomes the limitations of traditional methods of single and static drone capability assessment, establishes a multi-dimensional, high-precision resource assessment system, improves the system's ability to identify and utilize the differentiated characteristics of heterogeneous drones, and thus achieves efficient matching and optimized allocation of resources, ensuring that task allocation decisions are more in line with the actual capabilities and status of each drone.

[0024] The reinforcement learning model is trained based on the local data of the UAV, weights are calculated according to the resource capability indicators, model parameters are weightedly integrated to generate a global model, and the global model is distributed to each UAV, including: The raw local data collected by each drone is normalized, missing value filled, and noise filtered to obtain clean local data. The clean local data is trained using a policy gradient method with a regularization term to obtain local model parameters. After the encrypted local model parameters are transmitted to the coordination node, the coordination node performs fusion weighting and security processing on the local model parameters according to the resource capability indicators of each drone to obtain a global model, and outputs the global model to each drone.

[0025] In a specific embodiment, the calculation formula of the fusion weight is: ; in, is the global model parameter, For the i The local model parameters of the UAV, For the i Resource capability indicators for drones, is the total number of drones, is the first anti-zero constant, is the maximum value of resource capability index among all drones, is the global model stability adjustment coefficient.

[0026] In a specific embodiment, when a cluster of 10 heterogeneous drones performs federated reinforcement learning task allocation, the specific implementation process includes: Each drone first preprocesses the raw data collected locally. For example, reconnaissance drones perform noise reduction and feature extraction on their terrain imagery; transport drones process data on the relationship between payload and energy consumption; and attack drones process data on target tracking and strike accuracy. All drones use the Min-Max normalization method to normalize the data to the [0, 1] range, forward fill in missing values, and filter out high-frequency noise using wavelet transforms to produce clean local data.

[0027] Each drone trains its local model using a policy gradient method with L2 regularization based on clean local data. During training, drone A's computing device uses a GPU for accelerated computing, while drone B uses a low-power CPU. This results in differences in training speed and quality. After training, each drone generates its own local model parameters, including the weight matrices for the policy network and value network.

[0028] Each drone uses homomorphic encryption to encrypt its local model parameters, protecting them from theft or tampering during transmission. After encryption, the encrypted parameters are transmitted to the drone or ground station designated as the coordination node via a secure communication protocol.

[0029] After receiving the encrypted local model parameters of each drone, the coordination node first performs decryption verification. Then, based on the resource capability indicators collected in the early stage, the local model parameters of each drone are weighted and integrated.

[0030] For example, the resource capability indicators of drones are distributed as follows: Reconnaissance UAVs have a C1 of 0.85, transport UAVs have a T1 of 0.62, attack UAVs have an A1 of 0.78, and other UAVs range from 0.45 to 0.73. Global model parameters are calculated through weighted fusion.

[0031] After the coordination node calculates the global model, it conducts a security audit to ensure it contains no malicious code or backdoors, and then digitally signs it. The global model is then distributed to each drone via a secure encrypted channel. Each drone verifies the signature upon receiving it, confirming that the model has not been tampered with, and completes the local model update.

[0032] Specifically, this embodiment achieves efficient, secure, and fair model training and updating for heterogeneous drone clusters within a federated learning framework through data preprocessing, secure model parameter transmission, and a resource capability weighted fusion mechanism. It addresses the issue of differences in computing power among heterogeneous devices, dynamically adjusting the contribution weight of each drone to the global model through resource capability indicators to ensure model quality. Through encrypted transmission and secure processing mechanisms, it effectively prevents information leakage and malicious attack risks during model parameter transmission. Using a nonlinear weighted fusion algorithm, it adaptively balances the impact of drones with varying resource capabilities on the global model, avoiding the negative impact of weaker drones on model quality while retaining valid empirical data from each drone. This embodiment overcomes the model bias and fusion imbalance problems caused by device heterogeneity in traditional federated learning, significantly improving the learning efficiency and model quality of heterogeneous drone clusters in a distributed environment.

[0033] The method utilizes the drone sensor to collect dynamic change information in the mission area in real time, adopts a multi-objective optimization algorithm to plan and execute the drone's mission allocation plan, and obtains the drone's mission execution status, including: Obstacle distribution data, climate change data, and interference signal data are extracted from the sensor data collected by each drone, and the obstacle distribution data, climate change data, and interference signal data are fused and processed to obtain environmental state parameters; A multi-objective optimization function is constructed. The optimization objectives include task completion time, task energy consumption and task safety risk. Based on the iterative solution algorithm, a task allocation plan that meets each optimization objective is solved while satisfying resource and environmental constraints.

[0034] In a specific embodiment, the task allocation scheme scoring calculation formula of the multi-objective optimization function is: ; in, Score the task assignment proposals, is the total number of drones, For the k The estimated mission completion time of the UAV, is the maximum mission completion time of all drones, For the k The expected energy consumption of a drone, is the maximum energy consumption of all drones, For the k Mission safety risk factors for drones, For the k Resource capability indicators for drones, 、 、 are the weight adjustment coefficients of task completion time, task energy consumption, and task safety risk, respectively. 、 、 It is the target adjustment index of task completion time, task energy consumption, and task safety risk. is the second anti-zero constant.

[0035] In a specific embodiment, when a swarm of 12 heterogeneous drones performs forest fire monitoring and emergency rescue tasks, the specific implementation process includes: A swarm of drones is deployed over the forest fire area, and each drone collects environmental data in real time through a variety of sensors on board: High-altitude reconnaissance drones use infrared thermal imagers to collect fire distribution heat maps and smoke density data; medium-sized transport drones use lidar to collect terrain and obstacle distribution data; small monitoring drones use meteorological sensors to collect climate change data such as wind direction, wind speed, and humidity; all drones use communication modules to detect interference signal strength and communication quality data.

[0036] The drone swarm uses a distributed data fusion approach to process multi-source, heterogeneous data. First, each drone preprocesses the raw data, including noise filtering, outlier detection, and format standardization. Then, using Kalman filtering and Bayesian fusion algorithms, the disparate sensor data is integrated into unified environmental state parameters, forming a comprehensive environmental situation map that includes information on fire spread, wind direction and speed, terrain obstacle distribution, and communication quality.

[0037] Construct a multi-objective optimization function with three main optimization objectives: Mission completion time: Estimate the time required for each drone to complete the assigned mission; Mission energy consumption: Estimate the energy consumption of each drone to perform the mission; Mission safety risk: Comprehensively consider the fire spread path, obstacle danger and communication quality to assess the mission risk.

[0038] Based on the real-time environment status parameters, each possible task allocation scheme is scored. The scoring formula is: ; The weight adjustment coefficients of task completion time, task energy consumption, and task safety risk are set to 0.4, 0.3, and 0.3, respectively; the target adjustment indexes of task completion time, task energy consumption, and task safety risk are set to 1.2, 1.0, and 1.5, respectively; the second zero-proof constant is set to 0.001; and the following constraints are set to ensure the feasibility of the task allocation scheme: Resource constraints: The load of each drone must not exceed its maximum carrying capacity; energy constraints: The energy consumption of mission execution must not exceed 85% of the current power to ensure safe return; time constraints: Critical tasks must be completed within the specified time window; communication constraints: Ensure that the drone is always within the effective communication range.

[0039] An improved non-dominated sorting genetic algorithm is used for multi-objective optimization. The initial population of the algorithm is set to 100, the number of iterations is 50, the crossover probability is 0.85, and the mutation probability is 0.15. During the solution process, the weights are dynamically adjusted to adapt to environmental changes: When the fire suddenly intensifies, the safety risk weight will be automatically increased; when the communication quality in a certain area is detected to be degraded, the priority of the task allocation in that area will be lowered; when the battery level of a drone is lower than 30%, its energy consumption weight will be increased.

[0040] The solution with the highest overall score from the Pareto optimal solution set is selected as the final task allocation solution. For example, in a forest fire monitoring task, monitoring large fire areas can be assigned to specialized drones with high thermal resistance, material delivery tasks can be assigned to transport drones with high load capacity, and long-range reconnaissance tasks can be assigned to high-altitude drones with long endurance.

[0041] After the task allocation plan is executed, the task execution status of each drone is continuously monitored, including key indicators such as position changes, task progress, and energy consumption, to provide data support for performance evaluation.

[0042] Specifically, this embodiment can build a high-precision mission environment situation map in real time through the fusion of multi-source heterogeneous sensor data. Compared with the traditional single-sensor method, the environmental perception accuracy is improved by about 40%, effectively reducing task allocation errors caused by misjudgment of environmental conditions.

[0043] By introducing a multi-objective scoring function with nonlinear weights and target adjustment indexes, a dynamic balance among time efficiency, energy efficiency, and security is achieved. 、 、 The differentiated settings can flexibly adjust the sensitivity and priority of each target according to the characteristics of different tasks. Compared with the linear weighted method, the task completion efficiency is improved by 25%, the energy utilization rate is increased by 18%, and the safety risk is reduced by 30%.

[0044] By comprehensively considering the resource capability indicators of drones Mission safety risk factors By leveraging the nonlinear relationship between the two, it is possible to assign high-risk tasks to high-capability drones, fully utilizing the differentiated characteristics of heterogeneous drones and improving resource allocation efficiency by 35% compared with traditional methods.

[0045] This embodiment can adjust constraints and optimize target weights in real time according to environmental changes. In emergencies such as fire spread and communication quality degradation, it can quickly re-plan task allocation plans, reducing response time to seconds and improving adaptability to dynamic environmental changes.

[0046] By combining an improved non-dominated sorting genetic algorithm with a distributed computing architecture, even in a moderately sized swarm of 12 drones, the system can complete multi-objective optimization within 1-2 seconds, meeting the demands of real-time task allocation. In large-scale swarm testing (50 drones), the system can still achieve a solution within 5 seconds, significantly outperforming traditional centralized decision-making methods.

[0047] The method of calculating the performance value of the task allocation scheme according to the task execution status and environmental data, and dynamically optimizing the task allocation strategy when the performance value is lower than a preset threshold, to obtain an optimized task allocation strategy, includes: Obtaining the task execution status of each UAV for the task allocation strategy, extracting the task completion rate, average response delay, and resource utilization rate based on the task execution status, cleaning and fusing the task completion rate, average response delay, and resource utilization rate to obtain comprehensive feedback data, and calculating and updating the performance value of each UAV under the corresponding task allocation scheme based on the comprehensive feedback data; After detecting that the efficiency value is lower than a preset threshold, the coordination node sends a retraining or fine-tuning instruction to each drone, and each drone updates the reinforcement learning model based on the latest local data and global information to obtain updated global model parameters. The updated global model parameters are uploaded to the coordination node, and global model weighted fusion is performed to obtain an updated global model and an optimized task allocation strategy. The updated global model and the optimized task allocation strategy are output to each drone; The calculation formula of the efficacy value is: ; in, The effectiveness value of the task allocation strategy, is the total number of drones, For the m The mission completion rate of UAVs performing missions, is the maximum mission completion rate of all UAVs, For the m The average response delay of the UAV in executing the task, is the maximum value of the average response delay of all UAVs performing tasks, For the m The resource utilization rate of the drone in performing the mission, is the maximum resource occupancy rate of all drones, For the mThe dynamic adjustment coefficient of the UAV, For the m The resource occupation response threshold factor of the UAV, is the exponential aging coefficient of the time delay, is the standardization coefficient of the efficacy value, is the cluster state entropy function, It is the uncertainty distribution indicator of task completion rate, average response delay and resource occupancy rate.

[0048] In a specific embodiment, when a swarm of 15 heterogeneous drones performs an urban emergency rescue mission, the specific implementation process includes: Set up three key performance indicators for monitoring: Mission Completion Rate: monitors the percentage of each drone completing assigned missions, updated in real time. Average Response Latency: records the time interval from when a mission is issued to when the drone executes the operation. Resource Utilization: monitors CPU usage, memory usage, communication bandwidth usage, and energy consumption, including: Reconnaissance UAV: ​​Mission completion rate = 92%, average response latency = 125ms, resource utilization = 68%; Communication UAV: ​​mission completion rate = 88%, average response delay = 180ms, resource utilization = 75%; Transport UAV: ​​Mission completion rate = 75%, average response latency = 210ms, resource utilization = 82%. When the original execution status data is noisy, missing, or delayed, the following processing methods are used: For instantaneous fluctuating data, the exponential weighted moving average method is used for smoothing; for missing data points, forward interpolation and adjacent drone data are used for auxiliary estimation; for outliers, Criteria are eliminated to prevent extreme data from affecting performance evaluation; indicators of different dimensions are converted into standardized data within a unified interval [0,1] through piecewise adaptive normalization; after processing, a comprehensive feedback data set including standardized task completion rate, average response delay, and resource occupancy rate is obtained.

[0049] The efficiency value is calculated, and the dynamic adjustment coefficients of reconnaissance drones, communication drones, and transport drones are set to 1.2, 1.0, and 0.9, respectively. The delay exponential aging coefficient is set to 0.8. For high-performance devices, medium-performance devices, and low-performance devices, the resource occupation response threshold factors are set to 0.7, 0.8, and 0.9, respectively. The efficiency value normalization coefficient is set to 1.5. The cluster state entropy function calculates the uncertainty of the current task allocation state. The higher the value, the more chaotic the system coordination state. The preset efficiency value threshold is set to 0.65. When the calculated efficiency value is lower than this threshold, the task allocation strategy optimization is triggered.

[0050] During an emergency rescue mission, when the effectiveness value was detected to be 0.58 (lower than the preset threshold of 0.65), the optimization process was triggered: First, we conducted problem diagnosis: analyzing the reasons for the performance degradation, we found that the average response delay of the three drones increased significantly (>250ms), the overall task completion rate dropped to 78%, and the cluster state entropy value increased to 0.45, indicating a decrease in coordination.

[0051] Then the retraining instruction is issued: the coordination node sends a retraining instruction containing the following content to each drone: Retraining priority setting (high priority is given to drones with abnormal response delays), model adjustment range (policy network part for delay-sensitive tasks), learning rate setting (0.005, smaller values ​​are used for fine-tuning), training batch setting (50 batches for fast updates).

[0052] Next, the local model is updated. Each drone uses the latest collected environmental data and mission execution status to update the local model according to the instructions: Drones with abnormal delays will have their delay penalty weight increased, drones with excessively high energy consumption will have their energy management strategies adjusted, and drones with low mission completion rates will have their mission planning algorithms optimized.

[0053] Then, model aggregation and optimization strategy generation are performed; each UAV uploads the updated model parameters to the coordination node. The coordination node uses the fusion weighted formula, but adjusts the dynamic adjustment coefficient value to 0.8 (to increase stability). Based on the fused global model, a new task allocation strategy is generated, focusing on improving task response speed and coordination.

[0054] Finally, the strategy was deployed and its effectiveness was verified: the optimized global model and task allocation strategy were distributed to each drone. After 30 seconds, the performance value was re-evaluated, and the result reached 0.76, which was significantly higher than the threshold.

[0055] Specifically, this embodiment successfully solves the technical problems of inaccurate performance evaluation, untimely strategy optimization, and insufficient adaptability of heterogeneous drone clusters in the process of executing complex tasks through comprehensive task execution status monitoring and advanced performance evaluation mechanism, combined with adaptive optimization strategy. It provides strong support for the application of heterogeneous drone clusters in complex scenarios such as emergency rescue, border patrol, and smart cities.

[0056] The redundant communication link algorithm is used to synchronize model parameters and resource data. When a communication delay or interruption is detected, the multi-agent collaborative decision-making algorithm is used to analyze and process the local cache data and task execution status of each drone to generate a local temporary task allocation plan, including: Under normal communication conditions, multi-path synchronization and caching of model parameters and resource data of each UAV are performed through a redundant link algorithm.

[0057] In a specific embodiment, a dynamic routing selection and data splitting method is adopted, and multiple independent and redundant communication links are used to encrypt and fragment the model parameters and resource data and send them simultaneously. Through a multi-path parallel verification mechanism, the data is efficiently and stably synchronized to the local cache of each drone under normal conditions.

[0058] When a communication delay or interruption is detected, a multi-agent collaborative decision-making algorithm is used to analyze the cached local data and task execution status in real time, and dynamically generate a local temporary task allocation plan; In a specific embodiment, when a communication link anomaly is detected, each drone executes a multi-agent collaborative decision-making algorithm in real time based on the cached global model and local data, comprehensively considering data timeliness, resource surplus capacity, and current task status, and using consistency and fault-tolerant mechanisms to calculate and dynamically generate the optimal local temporary task allocation plan to ensure the continuity of task execution and the overall robustness of the cluster.

[0059] The calculation formula for the candidate task selection probability of the local temporary task allocation scheme is: ; ; ; ; in, For the n The probability that a candidate task is selected by the current drone, For the current moment, For the cache n The latest update time of the task data, is the maximum threshold within the data timeliness, is the time sensitivity coefficient, To implement the n The current resource capacity indicators required for each task, For the n The resource matching sensitivity coefficient of candidate tasks, For the n The change in resource capability indicators required for each task, For the n The change interval of the resource capability indicator required for each task, is the resource sensitivity threshold, is the third anti-zero constant, For the n The task state stability factor of each candidate task, For the n The entropy of the local state of the candidate tasks, For the n The normalized probability distribution of the task status indicators of candidate tasks, is an exponential function.

[0060] In a specific embodiment, the mission scenario is set as border patrol. When a swarm of 20 heterogeneous drones performs regional monitoring tasks, they may face communication instability caused by complex terrain and electromagnetic interference. The specific implementation process includes: First, redundant communication links are established and data is synchronized. Under normal communication conditions, a multi-level redundant communication mechanism is implemented: a) Communication link configuration: The main link uses a 4G / 5G cellular network with a bandwidth of 50Mbps; redundant link 1 uses a short-distance, high-bandwidth WiFi mesh network with a bandwidth of 150Mbps; redundant link 2 uses a long-distance, low-bandwidth LoRa network with a bandwidth of 27kbps; redundant link 3 uses a laser communication link between drones with a bandwidth of 100Mbps.

[0061] b) Model parameter synchronization example: The global model size is 25MB and is efficiently synchronized using the following steps: Use the SHA-256 algorithm to calculate the model data digest and generate a unique identifier; Applying Reed-Solomon erasure coding technology, the model data is encoded into 35 data fragments, of which any 20 fragments can restore the complete data; Each data fragment is approximately 1.5MB in size and is encrypted using the AES-256 algorithm. The encrypted data fragments are sent simultaneously through different communication links, for example: 15 fragments are sent through the 5G network, 10 fragments are sent through the WiFi network, 5 fragments are sent through the LoRa network, and 5 fragments are sent through the laser communication link.

[0062] c) Resource data synchronization: The drone resource data (including battery level, location, mission status, etc.) is approximately 50KB and uses the following synchronization strategy: Data update frequency: high-priority data (such as battery level and location) is updated every 2 seconds, and low-priority data (such as hardware status) is updated every 10 seconds; Data compression, using the lightweight CBOR (Concise Binary Object Representation) protocol to compress data to about 30% of its original size; Redundant transmission, transmitting complete copies of data simultaneously through at least 3 different communication links.

[0063] In a specific embodiment, during a border monitoring mission, strong electromagnetic interference was encountered, resulting in a communication interruption with the ground station and a serious impact on internal cluster communication. In this case, the following steps were performed: First, perform anomaly detection and evaluation: It was detected that the 5G and WiFi links were completely interrupted, the LoRa link delay increased to 3.5 seconds, and only the laser communication link remained stable between specific drones; network topology analysis showed that the cluster was divided into 3 sub-clusters, containing 8, 7 and 5 drones respectively.

[0064] Secondly, evaluate the timeliness of local data: Take the task "Monitoring Area A" as an example: Current moment =10:35:45, the latest update time of the task data =10:32:12, the set data age threshold =600 seconds, time sensitivity coefficient =0.8, calculated =0.756, indicating that the data has high timeliness.

[0065] Finally, perform local solution execution and communication recovery processing: Each UAV performs a temporarily assigned task while continuously monitoring the communication status. When it detects that the communication link has been restored (for example, after flying out of the interference area), it immediately uploads the local execution data to the coordination node. The coordination node quickly generates a globally optimal task allocation plan based on the execution status of each UAV during the communication interruption.

[0066] Specifically, the multipath redundant transmission mechanism in this embodiment improves communication reliability, maintaining a data synchronization success rate of over 95% even under extreme conditions, such as 30% complete link interruption and 40% severe link delay. Compared to traditional single-link communication, this improves communication reliability by approximately 300%, significantly reducing task interruptions caused by communication failures.

[0067] Dynamic routing selection and data splitting methods enable automatic adjustment of data flow and transmission strategies when link quality fluctuates. Even when link bandwidth fluctuates by 50%, the data throughput can still be kept at no more than 15% drop, improving communication self-healing capabilities by 75% compared to traditional methods.

[0068] The multi-agent collaborative decision-making algorithm gives drone swarms a high degree of autonomy even in the event of communication interruptions. Even when completely disconnected from the coordination node, sub-clusters can still achieve up to 85% accuracy in task allocation decisions based on locally cached data and state assessments, far exceeding the 50-60% accuracy of traditional pre-set solutions.

[0069] A timeliness assessment mechanism based on an exponential decay model successfully addresses the difficulty in quantifying data degradation during communication interruptions. This implementation accurately assesses the timeliness value of each piece of data, prioritizing the use of highly time-sensitive data for decision-making, and reduces the error rate of decisions based on outdated data by approximately 60%.

[0070] By introducing the concept of information entropy to assess task state stability, we can identify tasks with large state fluctuations and high uncertainty, reduce their probability of being assigned, and make task allocation more robust. Tests have shown that this mechanism improves task completion rates by approximately 25% in highly uncertain environments.

[0071] Distributed decision-making based on consistency and fault tolerance mechanisms eliminates the risk of single-point decision failure during communication interruptions. Even when 20% of drone data is unreliable or exhibits abnormal behavior, consistent, high-quality decisions can still be reached, demonstrating strong fault tolerance and collective intelligence.

[0072] See also Figure 2 The present invention also provides a federated reinforcement learning task allocation system for heterogeneous UAV clusters, the system comprising: The parameter acquisition module is used to collect the stability performance parameters of the UAV in real time, normalize the stability performance parameters based on the preset weight factors, and obtain the resource capability index of the UAV; A reinforcement learning module is used to train a reinforcement learning model based on the local data of the UAV, calculate weights according to the resource capability indicators, perform weighted fusion of model parameters to generate a global model, and distribute the global model to each UAV; The task allocation module is used to use the drone's sensors to collect dynamic change information in the task area in real time, use a multi-objective optimization algorithm to plan the drone's task allocation plan and execute it, and obtain the drone's task execution status; The feedback optimization module is used to calculate the performance value of the task allocation plan based on the task execution status and environmental data. When the performance value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

[0073] Specifically, this embodiment of a federated reinforcement learning task allocation system for heterogeneous drone swarms forms a complete closed-loop intelligent decision-making architecture through the organic collaboration of four core modules. This system implements full-process intelligent management, from accurate drone capability assessment, collaborative modeling through federated learning, intelligent allocation of multi-objective constraints, to adaptive optimization with dynamic feedback. This embodiment not only addresses the challenges of heterogeneous drones facing resource heterogeneity, environmental complexity, and communication instability, but also, through redundant communication mechanisms and multi-agent collaborative decision-making algorithms, enables the swarm to maintain efficient operation even in communication-constrained environments, improving task completion efficiency, resource utilization, and environmental adaptability.

[0074] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory communication interface and a bus: wherein the processor, memory and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement a federated reinforcement learning task allocation method for a heterogeneous drone cluster.

[0075] The present invention also discloses a computer-readable storage medium storing computer instructions that cause the computer to implement all or part of the steps of the method for allocating federated reinforcement learning tasks for a heterogeneous drone cluster described in an embodiment of the present invention. The storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A federated reinforcement learning task allocation method for heterogeneous UAV clusters, characterized by: The following steps are involved: Collect the stability performance parameters of the UAV in real time, normalize the stability performance parameters based on the preset weight factors, and obtain the resource capability index of the UAV; A reinforcement learning model is trained based on the local data of the drone, weights are calculated according to the resource capability indicators, model parameters are weightedly integrated to generate a global model, and the global model is distributed to each drone; The drone sensors are used to collect dynamic change information in the mission area in real time, and a multi-objective optimization algorithm is used to plan and execute the drone's task allocation plan to obtain the drone's task execution status; According to the task execution status and environmental data, the efficiency value of the task allocation plan is calculated. When the efficiency value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

2. The method for allocating federated reinforcement learning tasks for heterogeneous UAV clusters according to claim 1, wherein: The reinforcement learning model is trained based on the local data of the UAV, weights are calculated according to the resource capability indicators, model parameters are weightedly integrated to generate a global model, and the global model is distributed to each UAV, including: The raw local data collected by each drone is normalized, missing value filled, and noise filtered to obtain clean local data. The clean local data is trained using a policy gradient method with a regularization term to obtain local model parameters. After the encrypted local model parameters are transmitted to the coordination node, the coordination node performs fusion weighting and security processing on the local model parameters according to the resource capability indicators of each drone to obtain a global model, and outputs the global model to each drone.

3. The method for allocating federated reinforcement learning tasks for heterogeneous UAV clusters according to claim 1, wherein: The method utilizes the drone sensor to collect dynamic change information in the mission area in real time, adopts a multi-objective optimization algorithm to plan and execute the drone's mission allocation plan, and obtains the drone's mission execution status, including: Obstacle distribution data, climate change data, and interference signal data are extracted from the sensor data collected by each drone, and the obstacle distribution data, climate change data, and interference signal data are fused and processed to obtain environmental state parameters; A multi-objective optimization function is constructed. The optimization objectives include task completion time, task energy consumption and task safety risk. Based on the iterative solution algorithm, a task allocation plan that meets each optimization objective is solved while satisfying resource and environmental constraints.

4. The method for allocating federated reinforcement learning tasks for a heterogeneous UAV cluster according to claim 1, wherein: The method of calculating the performance value of the task allocation scheme according to the task execution status and environmental data, and dynamically optimizing the task allocation strategy when the performance value is lower than a preset threshold, to obtain an optimized task allocation strategy, includes: Obtaining the task execution status of each UAV for the task allocation strategy, extracting the task completion rate, average response delay, and resource utilization rate based on the task execution status, cleaning and fusing the task completion rate, average response delay, and resource utilization rate to obtain comprehensive feedback data, and calculating and updating the performance value of each UAV under the corresponding task allocation scheme based on the comprehensive feedback data; After detecting that the performance value is lower than the preset threshold, a retraining or fine-tuning instruction is sent to each drone through the coordination node. Each drone updates the reinforcement learning model based on the latest local data and global information to obtain updated global model parameters. The updated global model parameters are uploaded to the coordination node, and global model weighted fusion is performed to obtain an updated global model and optimized task allocation strategy. The updated global model and optimized task allocation strategy are output to each drone.

5. The method for allocating federated reinforcement learning tasks for a heterogeneous UAV cluster as claimed in claim 4, wherein: The calculation formula of the efficacy value is: ； in, The effectiveness value of the task allocation strategy, is the total number of drones, For the m The mission completion rate of UAVs performing missions, is the maximum mission completion rate of all UAVs, For the m The average response delay of the UAV in executing the task, is the maximum value of the average response delay of all UAVs performing tasks, For the m The resource utilization rate of the drone in performing the mission, is the maximum resource occupancy rate of all drones, For the m The dynamic adjustment coefficient of the UAV, For the m The resource occupation response threshold factor of the UAV, is the exponential aging coefficient of the time delay, is the standardization coefficient of the efficacy value, is the cluster state entropy function, It is the uncertainty distribution indicator of task completion rate, average response delay and resource occupancy rate.

6. The method for allocating federated reinforcement learning tasks for heterogeneous UAV clusters according to claim 1, wherein: The efficiency value of the task allocation scheme is calculated based on the task execution status and environmental data. When the efficiency value is lower than a preset threshold, the task allocation strategy is dynamically optimized. After the optimized task allocation strategy is obtained, the following steps are further included: Model parameters and resource data are synchronized through a redundant communication link algorithm. When communication delay or interruption is detected, the local cache data and task execution status of each drone are analyzed and processed through a multi-agent collaborative decision-making algorithm to generate a local temporary task allocation plan.

7. The method for allocating federated reinforcement learning tasks for heterogeneous UAV clusters according to claim 6, wherein: The redundant communication link algorithm is used to synchronize model parameters and resource data. When a communication delay or interruption is detected, the multi-agent collaborative decision-making algorithm is used to analyze and process the local cache data and task execution status of each drone to generate a local temporary task allocation plan, including: Under normal communication conditions, multi-path synchronization and caching of model parameters and resource data of each UAV are performed through redundant link algorithms; When communication delay or interruption is detected, the cached local data and task execution status are analyzed in real time through a multi-agent collaborative decision-making algorithm to dynamically generate a local temporary task allocation plan.

8. A federated reinforcement learning task allocation system for heterogeneous UAV clusters, configured to execute a federated reinforcement learning task allocation method for heterogeneous UAV clusters as claimed in any one of claims 1 to 7, characterized in that: The system comprises: The parameter acquisition module is used to collect the stability performance parameters of the UAV in real time, normalize the stability performance parameters based on the preset weight factors, and obtain the resource capability index of the UAV; A reinforcement learning module is used to train a reinforcement learning model based on the local data of the UAV, calculate weights according to the resource capability indicators, perform weighted fusion of model parameters to generate a global model, and distribute the global model to each UAV; The task allocation module is used to use the drone's sensors to collect dynamic change information in the task area in real time, use a multi-objective optimization algorithm to plan the drone's task allocation plan and execute it, and obtain the drone's task execution status; The feedback optimization module is used to calculate the performance value of the task allocation plan based on the task execution status and environmental data. When the performance value is lower than the preset threshold, the task allocation strategy is dynamically optimized to obtain the optimized task allocation strategy.

9. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; The processor, memory, and communication interface communicate with each other through the bus, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method according to any one of claims 1 to 7.

Images