A low-altitude intelligent networking-oriented federated learning security optimization method based on reinforcement learning

Abstract

The application provides a low-altitude intelligent networking-oriented federated learning security optimization method based on reinforcement learning, which constructs a system state based on a malicious node proportion, data contribution, model accuracy and system time delay, takes a selection factor and an exclusion ratio as a decision output, and realizes adaptive selection of a UAV node through deep reinforcement learning; a two-stage mechanism of informed selection and safe random selection is adopted to determine a UAV set participating in federated learning; the selected UAV completes model training based on a local data set and uploads parameters, a ground server adopts a federated average algorithm to complete global model aggregation, and strategy iteration optimization is completed according to model convergence performance, node participation balance and time delay cost. The application can effectively inhibit malicious nodes, improve the participation of benign nodes, reduce system time delay, and improve the security and training efficiency of federated learning in a low-altitude intelligent networking scene.

Description

Technical Field

[0001] This invention relates to a reinforcement learning-based federated learning security optimization method for low-altitude intelligent networks, belonging to the field of federated learning technology. Background Technology

[0002] In recent years, with the explosive development of the low-altitude economy and drone technology, a large number of drones, acting as edge nodes in low-altitude intelligent network systems, generate a significant amount of privacy-sensitive data when performing tasks such as inspection and monitoring. Limited air-to-ground communication bandwidth and stringent data security requirements have rendered traditional centralized data processing models unsuitable, necessitating the introduction of a distributed machine learning architecture. Federated learning, with its innovative paradigm of "data remains stationary while the model moves," eliminates the need for drones to transmit raw sensitive data back to ground base stations. Drones can participate in global aggregation simply by training their models locally and uploading gradient parameters, significantly alleviating network bandwidth pressure and effectively protecting data privacy. However, the efficiency and security of this distributed learning framework heavily rely on the client selection mechanism—the system's strategy for selecting and scheduling drone nodes participating in training. In real-world low-altitude scenarios, the high-speed movement of drones leads to highly unstable air-to-ground wireless channels and high communication latency. Furthermore, the high heterogeneity of sensors and computing power across nodes results in inconsistent overall data quality. Even more serious is that some hijacked or malicious drone nodes may disrupt the aggregation process of ground base stations through attack methods such as gradient poisoning or backdoor implantation, which can easily lead to the collapse of the global model performance.

[0003] Research on federated learning client selection mechanisms has diversified both domestically and internationally. In 2025, Li Xueqi et al. proposed an in-memory computing-PUF-TRNG integrated architecture framework. This framework uses memristor in-memory computing technology to dynamically adjust the client selection ratio based on the redundancy remainder system coding scheme, effectively reducing the computing and communication energy consumption of the end-side devices and enhancing the security of parameter generation at the physical level. In 2024, Chinese patent application CN119918695A proposed a federated learning optimization framework based on ternary clustering. Using the Shadowed Sets algorithm, it dynamically divides the client data distribution into core, edge, and irrelevant regions based on the membership degree of the cluster centers, thereby mitigating the weight divergence problem caused by data heterogeneity and improving the stability of local model aggregation.

[0004] In recent years, reinforcement learning has been widely adopted in the field of client selection optimization in federated learning due to its advantages of not requiring a precise mathematical model of the environment and being able to maximize long-term cumulative rewards through continuous interaction with the environment. This state-aware, dynamic sequential decision-making capability allows reinforcement learning to overcome the limitations of traditional static heuristic algorithms, making it particularly suitable for addressing the complex challenges of highly time-varying node states, unknown network availability, and dynamic trade-offs among multiple objectives in federated learning. Based on this, in 2022, Li et al. proposed a deep reinforcement learning federated optimization framework, which uses the Q-learning algorithm to dynamically adjust the selection strategy based on factors such as client computing power, data quality, and network state, effectively balancing network latency and differences in node computing power. In 2021, Chinese patent application publication number CN113704287A proposed a blockchain-based client trust evaluation mechanism framework, which uses distributed ledger technology to screen nodes based on the trustworthiness of the client's historical behavior, effectively filtering out nodes suspected of poisoning. However, existing federated learning client selection mechanisms based on reinforcement learning methods often focus on a single optimization objective and lack synergistic consideration of data quality, security defense, and system training efficiency. In particular, they are difficult to directly adapt to the complex and dynamic environment of low-altitude intelligent networks, where drones are highly heterogeneous, communication links are time-varying, and they are extremely vulnerable to network attacks. Summary of the Invention

[0005] The purpose of this invention is to optimize the drone selection strategy in federated learning using reinforcement learning methods, thereby alleviating problems such as high data heterogeneity, severe interference from malicious nodes, and high system latency in the drone selection process. To this end, a reinforcement learning-based security optimization method for low-altitude intelligent networks is proposed. This method can increase the participation of benign drones while suppressing the participation of malicious drones, thus improving the model accuracy and system security of federated learning.

[0006] The technical solution adopted by this invention to solve its technical problem is: a federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks, which includes the following steps:

[0007] Step 1: In the federated learning system, assume there is 1 ground server and N drones. In each round, M drones are selected to participate in model aggregation (M < N).

[0008] Step 2: Initialize global model parameters on the ground server The weight parameters of the Q network and discount factor and learning rate .

[0009] Step 3: Ground server build time slots state vector .

[0010] In step 3, the state vector is constructed. The specific process is as follows: in time slots The ground server calculates the proportion of malicious nodes among the drones selected in the previous time slot. The minimum training frequency among all benign drones, i.e., the minimum benign participation rate. And the highest training participation frequency among all malicious drones, i.e., the highest malicious participation rate. Based on each selected drone Size of uploaded local dataset and standard deviation and local training losses Calculate the data contribution of the current time slot : in Temperature regulation parameters are used for data quality calculations. Next, the classification accuracy of the global model from the previous time slot on the test set is evaluated, i.e., model precision. Finally, the drone is assigned a specific task by the ground server. uplink bandwidth drones Transmission power Channel gain and noise power Calculate the bit rate of the communication link : .

[0011] And based on drones Number of uploaded model parameters And the processing speed of ground servers Obtain communication delay : and computational latency : Thus, the maximum system delay is obtained. : ,in Overall maximum system delay The proportion of malicious nodes in drones Minimum positive participation Maximum malicious participation Data contribution of the current time slot and model accuracy The ground server constructs the state vector : .

[0012] Step 4: The ground server will construct the state vector. Inputting into the Q network, the Q network outputs the long-run discount expected benefit of the selection strategies of all participating nodes in the current state. And adopt Greedy strategy outputs optimal action strategy This action includes the specific parameter values ​​used to select the drone in this round, namely the overselection factor. Exclusion ratio .

[0013] Step 5: The ground server determines the data for each drone. Size of uploaded local dataset Local training loss and local training latency Combined with the set maximum training latency threshold calculate The overall utility score of each drone : .

[0014] The utility score takes into account both data quality and system speed. This is an indicator function that triggers a penalty when the actual latency exceeds a threshold. This is a hyperparameter used as a timeout penalty factor. After calculation, the ground server assigns all drones to a comprehensive utility score. Sort in descending order and exclude based on the output ratio of that time slot. Delete the last one in the ranking The drone with the lowest score. The drones that were not deleted constitute the initial candidate set. .

[0015] Step 6: For the preliminary candidate set Each drone in Pseudo-random mapping values ​​are generated using verifiable pseudo-random functions. To make it satisfy the range of values (in (This is a pre-defined upper bound for the maximum hash function in the system), and simultaneously generates a zero-knowledge proof to verify the validity of the computation process. Based on this, drones Based on the overselection factor output in this time slot Determine the generated random value locally. Does it satisfy the inequality condition? If the inequality constraint is satisfied, and its accompanying zero-knowledge proof is... Once verified in the system, the drone formally submits an interactive application to the ground server to participate in this round of model aggregation. Ultimately, all drones that successfully pass random verification and submit applications according to regulations will collectively form the application participation set for the second round. .

[0016] Step 7: The ground server determines the size of the application group based on the second round of applications. With the predetermined goal The size relationship is used to execute the corresponding truncation or compensation strategy. When At that time, the ground server used a random sampling algorithm to... Precise interception The final participants in the cost slots of unmanned mechanisms ;when At that time, the ground server directly... All drones in the process were identified as the final participants. ;when If this occurs, it indicates that the current screening criteria are too stringent, resulting in an insufficient number of qualified nodes. The ground server will then dynamically increase the over-selection factor according to the system's preset step size. Repeat step 6 until the size of the generated application participation set reaches the target requirement. .

[0017] Step 8: Selected drones Based on local dataset The global model issued by the ground server in the previous time slot Perform local model training. Minimize the local loss function using the gradient descent algorithm and update the local model parameters. The formula is: in For local learning rate, For drones The loss function. After local training is complete, the drone... Update the local model parameters The data is uploaded to the ground server via a communication link. The ground server then uses the FedAvg algorithm to perform a weighted summation based on the proportion of each drone's data to the total data volume in this round, completing the aggregation operation of the global model. This is combined with the final set of participants in this time slot. Local dataset size , Aggregate calculation of global model : .

[0018] Step 9: Ground server calculates rewards .

[0019] In step 9, the ground server uses the aggregated global model. The global model accuracy for this time slot was evaluated on the test set. Based on the results of this time slot selection, the minimum participation level of benign unmanned aerial vehicles (UAVs) was statistically obtained. and the maximum participation of malicious drones Combined with the calculated maximum system delay Build rewards for: .

[0020] Step 10: The ground server transmits the status of the current time slot. ,action ,award and the state of the next time slot observed after the interaction. Constructed as an experience sequence And store it in the experience replay buffer pool.

[0021] Step 11: Randomly sample from the experience replay buffer pool These experiences form batch processing samples.

[0022] Step 12: Ground server combines experience sequence All possible actions The Adam optimization algorithm is used to update the weight parameters of the evaluation Q-network based on the extracted mini-batch empirical sequences. To obtain new weight parameters : in, Discount factor updated for weight parameters.

[0023] Step 13: Repeat steps 3 to 12 to continuously iterate and optimize the UAV dynamic selection strategy and global model until the accuracy of the global model on the ground server is improved. and system rewards It converges to a stable value.

[0024] Beneficial effects:

[0025] 1. This invention uses a deep reinforcement learning method to adaptively adjust the client selection strategy based on the data contribution, system latency, participation, and global model accuracy of the UAV. While improving the model training accuracy and system security, it can effectively solve the resource problem of low-altitude edge networks.

[0026] 2. This invention addresses the significant heterogeneity of UAV nodes, time-varying air-to-ground communication links, and the vulnerability of malicious nodes to poisoning attacks in low-altitude scenarios. It constructs a system state based on the proportion of malicious nodes, data contribution, model accuracy, and system latency, using over-selection factors and exclusion ratios as decision outputs. Deep reinforcement learning is used to achieve adaptive selection of UAV nodes. A two-stage mechanism of informed selection and safe random selection is employed to determine the set of UAVs participating in federated learning. Selected UAVs complete model training based on local datasets and upload parameters. The ground server uses a federated averaging algorithm to aggregate the global model. The strategy is iteratively optimized based on model convergence performance, node participation balance, and latency overhead.

[0027] 3. This invention can effectively suppress malicious nodes, increase the participation of benign nodes, reduce system latency, and improve the security and training efficiency of federated learning in low-altitude intelligent network scenarios. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall architecture for optimizing the safety of federated learning drones.

[0029] Figure 2 This is a diagram illustrating the informed choice mechanism.

[0030] Figure 3 This is a flowchart of a secure random selection mechanism. Detailed Implementation

[0031] The invention will now be described in further detail with reference to the accompanying drawings.

[0032] like Figures 1-3 As shown, this invention provides a reinforcement learning-based federated learning security optimization method for low-altitude intelligent networks, which includes the following steps:

[0033] Step 1: In the federated learning system, there is one ground server and N=50 drones. In each round, M=20 drones are selected to participate in model aggregation.

[0034] Step 2: Initialize global model parameters on the ground server The weight parameters of the Q network and discount factor and learning rate .

[0035] Step 3: Ground server build time slots state vector Constructing the state vector The specific process is as follows: in time slots The ground server calculates the proportion of malicious nodes among the drones selected in the previous time slot. The minimum training frequency among all benign drones, i.e., the minimum benign participation rate. And the highest training participation frequency among all malicious drones, i.e., the highest malicious participation rate. Based on each selected drone Size of uploaded local dataset and standard deviation and local training losses Calculate data contribution : .

[0036] Temperature control parameters used for data quality calculations Next, the classification accuracy of the global model in the previous time slot on the test set is evaluated, i.e., the model precision. Finally, the drone is assigned a specific task by the ground server. uplink bandwidth drones Transmission power Channel gain and noise power Calculate the bit rate of the communication link : .

[0037] And based on drones Number of uploaded model parameters And the processing speed of ground servers Obtain communication delay : and computational latency : Thus, the maximum system delay is obtained. : ,in Overall maximum system delay The proportion of malicious nodes in drones Minimum positive participation Maximum malicious participation Data contribution of the current time slot and model accuracy The ground server constructs the state vector : .

[0038] Step 4: The ground server will construct the state vector. Inputting into the Q network, the Q network outputs the long-run discount expected benefit of the selection strategies of all participating nodes in the current state. And adopt Greedy strategy outputs optimal action strategy This action includes the specific parameter values ​​used to select the drone in this round, namely the overselection factor. Exclusion ratio .

[0039] Step 5: The ground server determines the data for each drone. Size of uploaded local dataset Local training loss and local training latency Combined with the set maximum training latency threshold Calculate the overall utility score of 50 drones. : .

[0040] The utility score takes into account both data quality and system speed. This is an indicator function that triggers a penalty when the actual delay exceeds a threshold; the timeout penalty factor. =1.2. After the calculation is completed, the ground server assigns all drones a comprehensive utility score. Sort in descending order and exclude based on the output ratio of that time slot. Delete the last one in the ranking The drone with the lowest score. The drones that were not deleted constitute the initial candidate set. .

[0041] Step 6: For the preliminary candidate set Each drone in The system's preset upper bound for the maximum hash. Pseudo-random mapping values ​​are generated using verifiable pseudo-random functions. To make it satisfy the range of values Simultaneously, it generates zero-knowledge proofs to verify the legality of the computation process. Based on this, drones Based on the overselection factor output in this time slot Determine the generated random value locally. Does it satisfy the inequality condition? If the inequality constraint is satisfied, and its accompanying zero-knowledge proof is... Once verified in the system, the drone formally submits an interactive application to the ground server to participate in this round of model aggregation. Ultimately, all drones that successfully pass random verification and submit applications according to regulations will collectively form the application participation set for the second round. .

[0042] Step 7: The ground server determines the size of the application group based on the second round of applications. With the predetermined goal The size relationship is used to execute the corresponding truncation or compensation strategy. When At that time, the ground server used a random sampling algorithm to... Precise interception The final participants in the cost slots of unmanned mechanisms ;when The ground server directly All drones in the process were identified as the final participants. ;when If this occurs, it indicates that the current screening criteria are too stringent, resulting in an insufficient number of qualified nodes. The ground server will then dynamically increase the over-selection factor according to the system's preset step size. Repeat step 6 until the size of the generated application participation set reaches [a certain value]. .

[0043] Step 8: Selected drones Based on local dataset The global model issued by the ground server in the previous time slot Local model training is performed by minimizing the local loss function using the gradient descent algorithm, and then updating the local model parameters. The formula is: .

[0044] Local learning rate =0.01, For drones The loss function. After local training is complete, the drone... Update the local model parameters The data is uploaded to the ground server via a communication link. The ground server then uses the FedAvg algorithm to perform a weighted summation based on the proportion of each drone's data volume to the total data volume of participants in this round, completing the aggregation operation of the global model and combining it with the final participant set for this time slot. Local dataset size , Aggregate calculation of global model The formula is: .

[0045] Step 9: Ground server calculates rewards The ground server, based on the aggregated global model The global model accuracy for this time slot was evaluated on the test set. Based on the results of this time slot selection, the minimum participation level of benign unmanned aerial vehicles (UAVs) was statistically obtained. and the maximum participation of malicious drones Combined with the calculated maximum system delay Build rewards for: .

[0046] Step 10: The ground server transmits the status of the current time slot. ,action ,award and the state of the next time slot observed after the interaction. Constructed as an experience sequence And store it in the experience recycling buffer pool.

[0047] Step 11: Randomly sample 32 experiences from the experience replay buffer to form a batch sample.

[0048] Step 12: Ground server combines experience sequence All possible actions The Adam optimization algorithm is used to update the weight parameters of the evaluation Q-network based on the extracted mini-batch empirical sequences. To obtain new weight parameters : in, =0.9 is the discount factor for updating the weight parameters.

[0049] Step 13: Repeat steps 3 to 12 to continuously iterate and optimize the UAV dynamic selection strategy and global model until the accuracy of the global model on the ground server is improved. and system rewards It converges to a stable value.

[0050] The method proposed in this invention is compared with the random selection strategy of the FedAvg algorithm, and the global model accuracy is improved while reducing system latency.

[0051] The present invention has been described in detail above, but the present invention is not limited to the above-described embodiments. Those skilled in the art can make various changes to the present invention based on their knowledge to achieve better results.

Claims

1. A reinforcement learning-based federated learning security optimization method for low-altitude intelligent networks, characterized in that, The method includes the following steps: Step 1: Construct a low-altitude intelligent network federated learning system. The low-altitude intelligent network federated learning system includes a ground server and N UAV nodes. In each round of global model aggregation, M UAVs are selected to participate, where M < N. Step 2: Initialize global model parameters on the ground server Q-network weight parameters Discount Factor and learning rate ; Step 3: Ground server base construction time slot state vector ; Step 4: The ground server inputs the state vector into the Q network, using... The greedy strategy outputs the optimal action strategy, which includes an overselection factor. Exclusion ratio ; Step 5: The ground server adjusts the size of the drone's local dataset. Local training loss With training latency Calculate the overall utility score of the drone. Sort by score in descending order and according to the exclusion ratio Eliminate low-resolution drones and generate a preliminary candidate set. ; Step 6: Preliminary Candidate Set The drones in the program use verifiable pseudo-random functions to generate pseudo-random values. With zero-knowledge proof Based on the selection factors, local legality verification is completed, and the drones that pass the verification constitute the second round of application participation set; Step 7: The ground server executes a truncation or dynamic compensation strategy based on the relationship between the size of the application participation set and the target number M to determine the final set of drones participating in this round of aggregation; Step 8: Selected drones Local model training is performed and model parameters are uploaded. The ground server uses a federated averaging algorithm to complete global model aggregation. Step 9: Ground server combines global model accuracy Minimum participation for benign drones Maximum participation of malicious drones With maximum system delay Calculate the system reward value ; Step 10: The ground server will transmit the current timeslot status. Action strategies System reward value With the next time slot state Constructed as an experience sequence Stored in the experience replay buffer pool; Step 11: Randomly sample from the experience replay buffer pool Based on batch experience, batch processing samples are formed; Step 12: The ground server updates the Q-network weight parameters based on the batch-processed samples using the Adam optimization algorithm. ; Step 13: Repeat steps 3 through 12 until the global model accuracy is achieved. With system reward value It converges to a stable interval.

2. The federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks according to claim 1, characterized in that, The specific process of constructing the state vector in step 3 is as follows: The ground server calculates the proportion of malicious nodes in the selected drones in the previous time slot. Minimum training frequency for benign drones Maximum frequency of malicious drones participating in training According to drones Local dataset size Data standard deviation Compared with local training loss Calculate data contribution ; Evaluate the classification accuracy of the global model in the previous time slot Finally, the drone is assigned by the ground server. uplink bandwidth drones Transmission power Channel gain and noise power Calculate the bit rate of the communication link : And based on drones Number of uploaded model parameters And the processing speed of ground servers Obtain communication delay : and computational latency : Thus, the maximum system delay is obtained. : in Based on the above indicators, the ground server constructs a state vector. : 。 3. The federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks according to claim 1, characterized in that, In step 4, the Q-network is used to output the long-term expected benefit of the choice strategy of each node in the current state. Optimal action strategy Overselection factor for dynamic adaptation to low-altitude scenarios Exclusion ratio .

4. The federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks according to claim 1, characterized in that, The comprehensive utility score in step 5 The utility score takes into account both data quality and system speed. in, The indicator function penalizes drones that exceed the training latency threshold; the ground server sorts the utility scores in descending order and excludes them according to the exclusion ratio. Remove the last part A preliminary candidate set of low-resolution drones was formed. .

5. A federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks according to claim 1, characterized in that, In step 6, the pseudo-random value Make it satisfy the range of values (in (The maximum hash upper bound preset by the system) is used to determine the generated random value. Does it satisfy the inequality condition? Complete local verification and generate zero-knowledge proof. Used to verify the legitimacy of the calculation; the drones that pass the verification initiate participation applications, collectively forming the second round of application participation sets. .

6. A federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks according to claim 1, characterized in that, The truncation or dynamic compensation strategy in step 7 specifically involves: applying to participate in the set. When the scale is greater than M, M drones are randomly selected as the final participant set. ; Apply to participate in the group When the size is equal to M, the set of applicants is directly used as the final set of participants. ; Application to participate in the scale When the value is less than M, the overselection factor is increased by a preset step size. The verification process is repeated until the size of the application participation set reaches M.

7. A federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks according to claim 1, characterized in that, In step 8, the drone Based on local dataset The global model issued by the ground server in the previous time slot Local model training is performed, and the local model update is completed using the gradient descent algorithm. The update formula is as follows: in For local learning rate, For drones The loss function is calculated by the ground server using a weighted aggregation based on the proportion of local data from each UAV, resulting in the global model for the new time step. The aggregation calculation formula is as follows: 。 8. A federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks according to claim 1, characterized in that, System reward value in step 9 From global model accuracy Minimum participation for benign drones Maximum participation of malicious drones Maximum system delay It was jointly decided that the reward would be: Global model accuracy and benign participation are positive incentives, while malicious participation and system latency are negative penalty terms.

9. A federated learning security optimization method based on reinforcement learning for low-altitude intelligent networks according to claim 1, characterized in that, In step 12, an empirical replay mechanism is used to minimize the temporal difference error using the Adam optimizer, thus completing the Q-network weight parameters. Iterative updates: in, Discount factor updated for weight parameters.

Images