Federal learning security aggregation method and system based on privacy protection reinforcement learning

Abstract

The invention discloses a federated learning security aggregation method and system based on privacy protection reinforcement learning, belongs to the technical field of information security, and solves the technical problems of low calculation efficiency and poor model precision in the prior art. The system comprises a central server, G group servers and N clients, the N clients are divided into G groups, each group server manages one client group, and a hierarchical management architecture is formed among the central server, the group servers and the clients; the central server distributes a global model to the client, the client carries out training to obtain a classification model, encrypts the classification model and uploads the classification model to the group server, the group server carries out aggregation and constructs a group state vector, and the aggregation result and the group state vector are uploaded to the central server; and the central server dynamically screens out the client groups participating in global aggregation in the round by using the reinforcement learning agent, aggregates to obtain the global model, synchronously executes the reinforcement learning training process, and sends the updated global model to the client, so that the calculation complexity is low, and the model precision is good.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and more specifically, to a secure aggregation method and system for federated learning based on privacy-preserving reinforcement learning. Background Technology

[0002] In recent years, driven by the explosive growth of data volume, continuous breakthroughs in deep learning algorithms, and a significant increase in computing power, artificial intelligence technology has ushered in a new wave of development. However, traditional machine learning methods typically rely on centralized data collection and processing, requiring the aggregation of raw data from various locations to a central server for model training. This approach not only poses significant privacy risks but also faces challenges in terms of legality and compliance as domestic and international data protection regulations such as the General Data Protection Regulation (GDPR) become increasingly comprehensive, further exacerbating the formation of "data silos." Meanwhile, models trained solely on local data from a single institution or user often suffer from poor performance and insufficient generalization ability due to limited sample size and homogeneous distribution.

[0003] To address these issues, federated learning emerged. This technology emphasizes the core idea of ​​"data remaining stationary while the model moves," allowing multiple clients to collaboratively train a unified global model without exchanging raw data, solely through interactive model parameter or gradient updates. This method effectively protects user data privacy while leveraging the overall characteristics of distributed data to improve model performance, thus demonstrating broad application prospects in numerous fields such as smart healthcare, financial risk control, edge intelligence, and recommender systems.

[0004] However, federated learning still faces significant challenges in practical deployment. First, client data often exhibits non-independent and identically distributed characteristics, leading to inconsistent local model update directions, which in turn causes slow model convergence, decreased accuracy, and even training failure. Second, privacy and security issues are becoming increasingly prominent: model inversion attacks can reconstruct training samples from shared parameters, gradient inversion attacks can infer the original data from gradient updates, and model poisoning attacks can maliciously tamper with local updates to destroy the global model.

[0005] Existing technologies have proposed various privacy protection schemes, but all have obvious limitations. For example, the "Data Protection System Based on Federated Learning and Homomorphic Encryption" disclosed in patent publication CN120768634A uses homomorphic encryption technology to encrypt and protect model gradients. Although it provides strong privacy protection, its encryption and decryption processes involve complex mathematical operations, resulting in extremely high computational complexity. This leads to huge computational overhead and communication latency on resource-constrained edge devices and medical data nodes, seriously affecting the actual deployment efficiency and scalability of federated learning.

[0006] On the other hand, the "Federated Learning Model Training Method with Differential Privacy Protection" proposed in patent publication CN113762525A achieves privacy protection by adding noise during gradient updates. However, this method applies an equal degree of perturbation to all model updates, failing to consider the differences in data distribution and contribution among different clients, leading to a significant decrease in model accuracy. Especially in non-independent and identically distributed scenarios, this uniform noise addition further exacerbates the client drift problem, making it difficult for the global model to converge to the optimal solution.

[0007] Therefore, how to design a federated learning framework that balances privacy and security, model performance and system robustness in a highly heterogeneous and untrusted environment, and overcome the limitations of existing technologies in terms of computational efficiency, model accuracy and protection effectiveness, has become a key issue of common concern to industry and academia. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art. The purpose of the present invention is to provide a federated learning secure aggregation system based on privacy-preserving reinforcement learning, which can effectively solve the problems of high computational complexity of existing homomorphic encryption schemes and model accuracy reduction caused by differential privacy methods, and achieve high-precision model training under the premise of strong privacy protection.

[0009] The second objective of this invention is to provide a secure aggregation method for federated learning based on privacy-preserving reinforcement learning.

[0010] To achieve the first objective mentioned above, the present invention provides a federated learning security aggregation system based on privacy-preserving reinforcement learning, comprising: a central server, G group servers and N clients, wherein the N clients are divided into G groups, and each group server manages a client group, and a hierarchical management architecture is formed between the central server, group servers and clients; The central server, acting as the top-level coordinator, is responsible for maintaining the global model and executing global aggregation and intelligent group selection decisions with reinforcement learning agents based on dual deep Q networks. The group server acts as an intermediate layer, responsible for managing the various clients within its group, performing secure aggregation of classification models within the group, and constructing group state vectors for reinforcement learning decisions. The client, as the underlying data holder, trains the model on the local dataset to obtain a classification model, and uploads the classification model using privacy protection technology.

[0011] As a further improvement, the client group is divided in the following ways: random division or intelligent division strategy based on data distribution.

[0012] To achieve the second objective mentioned above, this invention provides a secure aggregation method for federated learning based on privacy-preserving reinforcement learning, comprising the following steps: Step 1. Each client obtains a local training sample set and initializes local training parameters; Step 2. Server initialization: The central server distributes the public validation dataset to all the group servers and the global model to all the clients; at the same time, it initializes the system training parameters, sets the reinforcement learning parameters, and uses the proportion of the public validation set as the decay factor in the reputation value calculation. Step 3. Each client uses the parameters of the global model as the weight parameters of the local model, and iteratively trains the global model using the local training sample set to obtain N trained local classification models; Step 4. Each client encrypts the classification model using a double-mask secure aggregation mechanism and uploads it to its respective group server; Step 5. The group server aggregates the classification model to obtain effective intra-group safe aggregation results, and constructs a group state vector based on the intra-group safe aggregation results. The group server then uploads the group state vector, the model parameters of the intra-group safe aggregation results, and the model parameters to the central server. Step 6. The central server inputs the collected group state vectors into the reinforcement learning agent. The reinforcement learning agent calculates the Q-value of each group through the value network, generates group selection actions according to the ε-greedy policy, and dynamically selects the client groups that participate in global aggregation in this round, thereby achieving a balance between privacy protection and training efficiency. Step 7. Aggregate the group model parameters based on the security aggregation results of the selected client groups to obtain the global model, and calculate the accuracy of the current global model; Step 8. After the global model is updated, the central server synchronously executes the reinforcement learning training process, calculates the reputation value based on the group validation accuracy, stores the state-action-reward tuple in the experience replay buffer, samples batch data after each P round of training, updates the parameters of the dual deep Q network through gradient descent, and periodically aligns the parameters of the target network with those of the main network.

[0013] Further, in step 1, each client obtains a local dataset covering L categories by dividing the dataset using a Dirichlet distribution; at the same time, the local training parameters are initialized: the current iteration number e=0, the maximum local iteration number E; the batch size of the client's local training dataset is B; and the local model learning rate is r.

[0014] Furthermore, in step 2, the system training parameters include: aggregation rounds, maximum aggregation rounds, and global model; the reinforcement learning parameters include: batch size, target network update cycle, and learning rate.

[0015] Furthermore, step 4 includes the following steps: Step 41. Each client is based on a pre-allocated local seed. A mask vector is generated using a cryptographically secure pseudo-random generator. ; Step 42. All clients within the group establish a shared key with each other using the Diffie-Hellman key exchange protocol. Based on this shared key Generate satisfying Symmetric mask vector of characteristics; Step 43. Each client configures its local classification model. Perform encrypted calculations:

[0016] in, A sufficiently large modulus is used to prevent numerical overflow; Step 44. The client will encrypt the classification model. Upload to the group's server.

[0017] Furthermore, in step 5, based on the design characteristics of the double-mask mechanism, all pseudo-random masks and paired symmetric masks cancel each other out during the summation process, and the aggregation result is equivalent to:

[0018] in Represents the set of clients within the group; The group server constructs a four-dimensional state vector based on the aggregation results. The specific definition of a four-dimensional state vector is: PCA compression characteristics Used to aggregate group results Principal component analysis was performed, and the top 10 principal components were extracted as dimensionality-reduced feature representations. Precision improvement rate Used to calculate the relative accuracy improvement of this model set on the common validation set.

[0019] in Indicates the first Accuracy of the wheel group model on the validation set; Parameter Difference : Used to calculate the cosine similarity distance between the current group aggregation model and the global model.

[0020] Where K represents the set of all key values ​​of the model parameters. and These represent the parameters of the global model and the group aggregation model at the k-th layer, respectively. Select Count : Used to record the cumulative number of times this group has been selected in historical training rounds.

[0021] in This indicates that the group is in the [number]th [year]. An indicator function for whether the wheel is selected; The group server will construct the group state vector Aggregation results Uploaded to the central server, it provides state input for the decision-making of the reinforcement learning agent.

[0022] Furthermore, in step 6, the central server collects the group state vectors. The input reinforcement learning agent performs group selection using the following ε-greedy policy: Q-value calculation: Input the global state vector into the DDQN network and obtain the Q-value evaluation for each group.

[0023] Hybrid selection strategy: Employing the ε-greedy algorithm, in each round of selection... When there are multiple groups, for each selected position, the probability is... Randomly select from the remaining options, with probability Select the optimal group from the candidate group set to ensure that the final selection is successful. The selected group consists of 10 unique groups. ; Dynamic exploration rate: The exploration rate is updated after each round of selection.

[0024] Where the decay step size Ensure that the initial exploration rate is reached within the specified training rounds. Smoothly decline to the target exploration rate ,in, =0.9, =0.2.

[0025] Furthermore, in step 7, the central server bases the group selection action output by the reinforcement learning agent. For the selected group set The specific steps for model parameter aggregation and performance evaluation are as follows: Step 71. Collect the intra-group security aggregation results for all selected groups. ; Step 72. Global Model Calculation: Generate a new generation of global model using an average aggregation strategy.

[0026] Where |M| represents the number of groups selected in this round; Step 73. Model Performance Evaluation: Evaluate the current global model on the global test set. The performance includes: Loss function: ; Forward propagation calculation predicts labels: ; Calculate the loss value: ; Calculation accuracy: ; Step 74. Model Distribution: Distribute the updated global model Distribute to all clients.

[0027] Furthermore, in step 8, the reputation value is calculated based on the group verification accuracy. , The proportion of the public validation set; After the global model is updated, the central server synchronously executes the reinforcement learning training process, the specific steps of which are as follows: Step 81. Experience Storage: Store the state-action-reward tuple Stored in the experience replay buffer, where: For group state vectors, Select actions for the group. As a reward value, For the next state, This is a termination status identifier; Step 82. Calculate the Q-value loss: The Q-value loss for each selected group is given by the following formula:

[0028] Its goal is:

[0029] in It's the discount factor, the first item. Represents an immediate reward; When the number of samples in the experience replay buffer meets the training requirements, the following update process is executed: Batch sampling: Random sampling from the experience pool An empirical sample,

[0030] Q-value calculation: Main network: Target network: ; Target Q value calculation:

[0031]

[0032] Among the discount factors ; Multi-action Q-learning: For each sample and each selected group ,

[0033] Total loss calculation:

[0034] Update the main network parameters using the Adam optimizer, and synchronize the target network after every P rounds of training:

[0035] Step 83. Calculate the linear decay exploration rate: .

[0036] Beneficial effects Compared with the prior art, the advantages of this invention are as follows: This invention achieves a breakthrough balance between privacy protection and model performance by constructing a hierarchical secure aggregation architecture and a reinforcement learning-driven intelligent group selection mechanism. The system employs double-mask encryption to ensure client data privacy while accurately evaluating client group contributions through multi-dimensional state awareness (including PCA compression features, dynamic accuracy changes, model dissimilarity, and historical participation frequency). The dynamic selection strategy based on deep reinforcement learning effectively mitigates client drift caused by non-independent and identically distributed data, significantly improving model convergence speed and global accuracy. Furthermore, the system introduces an adaptive reputation mechanism and adversarial reward design, intelligently identifying and suppressing malicious client attacks, enhancing the system's robustness in untrusted environments. Through selective aggregation and distributed computing load, communication overhead and computational efficiency are significantly optimized while ensuring security, overcoming technical bottlenecks such as high computational complexity of traditional homomorphic encryption and performance degradation caused by differential privacy protection. This provides an innovative solution for deploying federated learning in real-world scenarios. Attached Figure Description

[0037] Figure 1 This is a diagram of the architecture of the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to specific embodiments shown in the accompanying drawings.

[0039] See Figure 1 A privacy-preserving reinforcement learning-based federated learning secure aggregation system includes: a central server (TOP Server), G group servers (Server 1-Server G), and N clients (Client 1-Client N). The N clients are divided into G groups, and each group server manages one client group. A hierarchical management architecture is formed between the central server, group servers, and clients. If the clients are divided equally, a client group contains M=N / G clients.

[0040] The nth client of the system is denoted as (1≤n≤N), the m-th group server is denoted as (1≤m≤G). Client Belongs to one and only one group server .

[0041] In this embodiment, G=10 and N=100.

[0042] The central server, acting as the top-level coordinator, is responsible for maintaining the global model. And a reinforcement learning agent based on a dual deep Q network (DDQN) performs global aggregation and intelligent group selection decisions.

[0043] The group server acts as an intermediate layer, responsible for managing the various clients (M clients) within its group, performing secure aggregation of classification models within the group, constructing group state vectors for reinforcement learning decisions, and then uploading them to the central server.

[0044] The client, as the underlying data holder, trains the model on the local dataset to obtain a classification model, and uploads the classification model to the group server using privacy protection technology.

[0045] The client groups are divided either by random partitioning or by intelligent partitioning based on data distribution.

[0046] The specific implementation process of the intelligent partitioning strategy based on data distribution is as follows: Each client first calculates the category distribution vector of its local data, which accurately reflects the quantity distribution of samples of different categories in the local dataset. To protect data privacy, each client adds Laplace noise that satisfies ε-differential privacy to the distribution vector for blurring, where the privacy budget parameter ε is set to 0.5, ensuring that the overall characteristics of the data distribution are preserved while providing strong privacy protection. After the privacy-protected distribution vector is securely uploaded to the central server, the central server uses the K-Means clustering algorithm to perform cluster analysis on the noisy distribution vectors of all clients, setting the number of clusters G=10 to be consistent with the number of group servers.

[0047] The core advantage of this intelligent partitioning mechanism lies in its ability to provide provable privacy protection for the local data distribution of each client through differential privacy technology, preventing the inference of original data information from the distribution vector. Simultaneously, since the expected value of Laplace noise is zero, the noise effects cancel each other out when a large number of clients participate, allowing the clustering algorithm to accurately identify the macroscopic patterns of the data distribution and group clients with similar data characteristics into the same group. This grouping method lays a solid foundation for intelligent group selection in subsequent reinforcement learning, ensuring that clients within each group have relatively consistent data distribution characteristics, effectively improving the training efficiency and model performance of federated learning in scenarios with non-independent and identically distributed data.

[0048] A secure aggregation method for federated learning based on privacy-preserving reinforcement learning includes the following steps 1 to 8: Step 1. Each client obtains its local training sample set and initializes its local training parameters.

[0049] Each client Through Dirichlet distribution (concentration parameter) The dataset is divided into L categories, where L ≥ 2. Local training parameters are initialized: current iteration number e = 0, maximum local iteration number E; client-side local training dataset batch size B; local model learning rate r.

[0050] The local training sample set used in this embodiment is the training set from the CIFAR-10 universal object color image dataset. This dataset has 10 target categories (L=10): airplane, car, bird, cat, deer, dog, frog, horse, boat, and truck, containing a total of 50,000 images. Each image has a pixel size of 32×32, with 5,000 images for each target category. First, 1 / 10 (5,000 images) is randomly selected from the complete dataset as a public validation set P. This validation set is not used for client data allocation but only for subsequent model validation and reinforcement learning state evaluation. The remaining 45,000 images are divided using a Dirichlet distribution (concentration parameter α=0.5 or α=0.8). First, the category distribution ratio between clients is generated according to the Dirichlet distribution. Then, images are allocated to each client according to this ratio, ensuring that different clients do not hold duplicate images, thus simulating a real-world non-independent and identically distributed data scenario. Here, B=10, E=5, and r=0.01.

[0051] Step 2. Server initialization: The central server distributes the common validation dataset P to all group servers and the global model to all clients; simultaneously, it initializes the system training parameters, sets the reinforcement learning parameters, and adjusts the proportion of the common validation set. As a decay factor in reputation score calculation.

[0052] System training parameters include: aggregation rounds Maximum aggregation round global model Reinforcement learning parameters include: batch size. Target network update cycle Learning rate .

[0053] The global test set is set to the CIFAR-10 test set, containing 10,000 images covering 10 categories. Each image is 32×32 pixels, with 1,000 images per category. The global model uses a CNN architecture, consisting of a first convolutional layer (3 input channels, 6 output channels, 5×5 kernel, stride 1), a 2×2 max-pooling layer, a second convolutional layer (6 input channels, 16 output channels, 5×5 kernel, stride 1), a 2×2 max-pooling layer, a first fully connected layer (400-dimensional input, 256-dimensional output), and a second fully connected layer (256-dimensional input, 10-dimensional output). During forward propagation, each convolutional layer is followed by a ReLU activation function and pooling operation, and the classification result is finally output through the fully connected layer. The reinforcement learning model used in this example employs a dual-deep Q-network (DDQN) with a self-attention mechanism, implemented by a feature extraction network and a Q-value output network. The feature extraction network contains a fully connected layer and a multi-head self-attention layer. The input dimension of the fully connected layer is the group state dimension. The output dimension is Self-attention layer maintains The network hides features and calculates inter-group dependencies; the Q-value output network will... 1D feature mapping to the dimension of number of groups The Q-value vector. Where T=300, =0.1, =50, =0.01, =10.

[0054] Step 3. Each client The parameters of the global model are used as the weight parameters of the local model, and the global model is iteratively trained using the local training sample set to obtain N trained local classification models.

[0055] Step 4. Each client The classification model is encrypted using a double-mask secure aggregation mechanism and then uploaded to the group server. That is, each client... After completing local model training, follow these steps to perform a secure upload with a double mask: Mask generation stage: Step 41. Pseudo-random mask generation, for each client Based on pre-allocated local seeds A mask vector is generated using a cryptographically secure pseudo-random generator. ; Step 42. Establish pairwise symmetric masks. All clients in the group establish shared keys with each other through the Diffie-Hellman key exchange protocol. Based on this shared key Generate satisfying Symmetric mask vector of characteristics; Encrypted upload stage: Step 43. Each client For local classification models Perform encrypted calculations:

[0056] in, A sufficiently large modulus is used to prevent numerical overflow; Step 44. The client will encrypt the classification model. Upload to the group's server.

[0057] During this process, the group server can only receive the encrypted classification model. Due to the existence of pseudo-random masks and paired symmetric masks, the group server cannot parse the real classification model of any single client. Through the threshold secret sharing mechanism, the correctness of the aggregation can still be guaranteed even if some clients go offline.

[0058] Step 5. Set up the server The classification model is aggregated to obtain effective intra-group safe aggregation results, and a group state vector is constructed based on the intra-group safe aggregation results. The group server uploads the group state vector, the model parameters of the intra-group safe aggregation results, and the group server to the central server.

[0059] Specifically, group server Encryption classification model for all clients within the receiving group Then, the received encrypted classification model Perform aggregation:

[0060] in Representation group The client collection within.

[0061] Based on the design characteristics of the double masking mechanism, all pseudo-random masks and paired symmetric masks cancel each other out during the summation process, and the aggregation result is equivalent to:

[0062] This allows for the acquisition of effective intra-group aggregation results while protecting the privacy of individual clients.

[0063] The group server constructs a four-dimensional state vector based on the aggregation results. The specific definition of a four-dimensional state vector is: PCA compression characteristics Used to aggregate group results Principal component analysis was performed, and the top 10 principal components were extracted as dimensionality-reduced feature representations.

[0064] Precision improvement rate Used to calculate the relative accuracy improvement of this model set on the common validation set.

[0065] in Indicates the first The accuracy of the wheel group model on the validation set.

[0066] Parameter Difference : Used to calculate the cosine similarity distance between the current group aggregation model and the global model.

[0067] Where K represents the set of all key values ​​of the model parameters. and These represent the parameters of the global model and the group aggregation model at the k-th layer, respectively.

[0068] Select Count : Used to record the cumulative number of times this group has been selected in historical training rounds.

[0069] in This indicates that the group is in the [number]th [year]. An indicator function for whether a wheel is selected.

[0070] The group server will construct the group state vector Aggregation results Uploaded to the central server, it provides state input for the decision-making of the reinforcement learning agent.

[0071] Step 6. The central server inputs the collected group state vectors into the reinforcement learning agent. The reinforcement learning agent calculates the Q-value of each group through the value network and generates group selection actions according to the ε-greedy policy. The system dynamically selects the client groups that participate in the global aggregation in this round, achieving a balance between privacy protection and training efficiency.

[0072] Specifically, the central server will collect group state vectors The input reinforcement learning agent performs group selection using the following ε-greedy policy: Q-value calculation: Input the global state vector into the DDQN network and obtain the Q-value evaluations for each group.

[0073] Hybrid selection strategy: Employing the ε-greedy algorithm, in each round of selection... When there are multiple groups, for each selected position, the probability is... Randomly select from the remaining options, with probability Select the optimal group from the candidate group set to ensure that the final selection is successful. The selected group consists of 10 unique groups. ; Dynamic exploration rate: The exploration rate is updated after each round of selection.

[0074] Where the decay step size Ensure that the initial exploration rate is reached within the specified training rounds. Smoothly decline to the target exploration rate ,in, =0.9, =0.2.

[0075] Step 7. Aggregate the group model parameters based on the security aggregation results of the selected client group M to obtain the global model. And calculate the accuracy of the current global model. .

[0076] Specifically, the central server uses the group selection action output by the reinforcement learning agent. For the selected group set The specific steps for model parameter aggregation and performance evaluation are as follows: Step 71. Collect the intra-group security aggregation results for all selected groups. ; Step 72. Global Model Calculation: A new generation of global model is generated using an average aggregation strategy (i.e., weighted average calculation).

[0077] Where |M| represents the number of groups selected in this round; Step 73. Model Performance Evaluation: Evaluate the current global model on the global test set. The performance includes: Loss function: ; Forward propagation calculation predicts labels: ; Calculate the loss value: ; Calculation accuracy: ; Step 74. Model Distribution: Distribute the updated global model Distribute to all clients .

[0078] Step 8. After the global model is updated, the central server synchronously executes the reinforcement learning training process, calculates the reputation value based on the group validation accuracy, stores the state-action-reward tuple in the experience replay buffer, samples batch data after each P round of training, updates the parameters of the dual deep Q network through gradient descent, and periodically aligns the parameters of the target network with those of the main network.

[0079] Specifically, reputation scores are calculated based on group validation accuracy. , The proportion of the public validation set.

[0080] After the global model is updated, the central server synchronously executes the reinforcement learning training process, with the following specific steps: Step 81. Experience Storage: Group the Status-Action-Reward Element (Complete Experience Element) Store in the experience replay buffer (capacity) ),in: For group state vectors, Select actions for the group. As a reward value, For the next state, This is a termination status identifier; Step 82. Calculate the Q-value loss: The Q-value loss for each selected group is given by the following formula:

[0081] Its goal is:

[0082] in It's the discount factor, the first item. The first term represents immediate rewards, while the second term captures expected future returns, enabling the agent to balance short-term gains with long-term global model improvements. When the number of samples in the experience replay buffer meets the training requirements, the following update process is executed: Batch sampling: Random sampling from the experience pool An empirical sample,

[0083] Q-value calculation: Main network: Target network: ; Target Q value calculation:

[0084]

[0085] Among the discount factors ; Multi-action Q-learning: For each sample and each selected group ,

[0086] Total loss calculation:

[0087] Using the Adam optimizer (learning rate) Update the main network parameters and synchronize the target network after every P rounds of training: ; Step 83. Calculate the linear decay exploration rate:

[0088] The reinforcement learning agent achieves an optimal balance between privacy protection and model performance by dynamically evaluating the contribution of client groups and intelligently selecting groups to participate in aggregation. Group states, selection actions, and validation accuracy rewards are stored in an experience pool to provide training samples for the dual-deep Q-network, enabling it to accurately learn the intrinsic relationship between the Q-values ​​of each group and the global model improvement.

[0089] By designing a multi-objective reward function focused on improving verification accuracy, the reinforcement learning agent gradually optimizes its selection strategy while balancing exploration and exploitation: prioritizing client groups that significantly improve the global model accuracy, while also considering system communication efficiency. This mechanism enables the agent to identify and favor groups with high-quality data distribution and stable update directions, while reducing the selection frequency of groups with lower performance contributions or potential anomalies.

[0090] The significance of this reinforcement learning mechanism lies in two aspects: firstly, it alleviates the client drift problem caused by non-independent and identically distributed data through intelligent group selection, accelerating model convergence; secondly, it reduces communication overhead and improves system scalability through selective aggregation. Ultimately, it achieves dynamic optimization of the entire federated learning process in a data-driven manner, significantly improving global model performance and training efficiency while ensuring privacy and security.

[0091] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention, and these will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A federated learning secure aggregation system based on privacy-preserving reinforcement learning, characterized in that, The application relates to a privacy-preserving federated learning method based on hierarchical management architecture. The application comprises a central server, G group servers and N clients, N clients are divided into G groups, each group server manages a client group, a hierarchical management architecture is formed among the central server, the group server and the client; The central server serves as a top coordinator, is responsible for maintaining a global model, and executes global aggregation and intelligent group selection decisions based on a double deep Q network reinforcement learning agent; The group server serves as an intermediate layer, is responsible for managing each client in the group, executing a safe classification model aggregation in the group, and constructing a group state vector for reinforcement learning decision-making; The client serves as a bottom data holder, performs model training on a local data set to obtain a classification model, and uploads the classification model by using a privacy protection technology.

2. The federated learning secure aggregation system based on privacy protection reinforcement learning according to claim 1, wherein, The client group is divided in the following manner: a random division or an intelligent division strategy based on data distribution is adopted.

3. A federated learning secure aggregation method based on privacy protection reinforcement learning, characterized in that, The application comprises the following steps: Step 1: each client obtains a local training sample set and initializes a local training parameter; Step 2: the server is initialized, the central server distributes a public verification data set to all group servers and distributes a global model to all clients; meanwhile, system training parameters are initialized, reinforcement learning parameters are set, and the proportion of the public verification set is taken as a decay factor in credit value calculation; Step 3: each client takes the parameters of the global model as weight parameters of a local model, and iteratively trains the global model through a local training sample set to obtain N trained local classification models; Step 4: each client uses a double-masked safe aggregation mechanism to encrypt the classification model and upload it to the group server; Step 5: the group server aggregates the classification models to obtain an effective group-in-safe aggregation result, constructs a group state vector based on the group-in-safe aggregation result, and uploads the group state vector and the model parameters of the group-in-safe aggregation result to the central server; Step 6: the central server inputs the collected group state vector into the reinforcement learning agent, the reinforcement learning agent calculates the Q value of each group through a value network, generates a group selection action according to an epsilon-greedy strategy, dynamically screens the client group participating in global aggregation in this round, and realizes the balance optimization of privacy protection and training efficiency; Step 7: the global model is obtained by aggregating the group-in-safe aggregation result of the screened client group, and the precision of the current global model is calculated; Step 8: after the global model is updated, the central server synchronously executes the reinforcement learning training process, calculates the credit value based on the group verification precision, stores the state-action-reward tuple into an experience replay buffer, samples batch data after completing P rounds of training, updates the double deep Q network parameters through gradient descent, and periodically aligns the target network and the main network parameters.

4. The federated learning secure aggregation method based on privacy protection reinforcement learning according to claim 3, characterized in that, In step 1, each client obtains a local data set covering L categories through Dirichlet distribution division; meanwhile, the local training parameters are initialized: the current iteration number e=0, and the maximum local iteration number E. The client local training dataset batch size is B; the local model learning rate is r.

5. The federated learning secure aggregation method based on privacy protection reinforcement learning according to claim 3, characterized in that, In step 2, the system training parameters include: aggregation round, maximum aggregation round, global model; the reinforcement learning parameters include: batch size, target network update period, learning rate.

6. The federated learning secure aggregation method based on privacy protection reinforcement learning according to claim 3, characterized in that, In step 4, the following steps are included: Step 41. Each client generates a local seed based on the pre-assigned local seed The mask vector is generated by a cryptographically secure pseudo-random generator ; Step 42. All clients in the group establish shared keys between each other through Diffie-Hellman key exchange protocol , based on which symmetric masking vectors are generated that satisfy properties Step 43. Each client performs encrypted computation on local classification model perform encrypted computation: wherein is a modulus large enough to prevent numerical overflow; Step 44. The client uploads the encrypted classification model to the group server to which it belongs. Step 44. The client uploads the encrypted classification model to the group server to which it belongs.

7. The federated learning secure aggregation method based on privacy protection reinforcement learning according to claim 3, characterized in that, In step 5, according to the design characteristics of the double mask mechanism, all pseudo-random masks and paired symmetric masks cancel each other out in the summation process, and the aggregation result is equivalent to: wherein represents a set of clients within a group; The group server constructs a four-dimensional state vector based on the aggregation results The specific definition of the four-dimensional state vector is: PCA compressed features : for principal component analysis on the group aggregation results , extract the first 10 principal components as the dimension reduction feature representation; Accuracy improvement rate : for calculating the relative accuracy improvement of the present model on the public validation set, wherein indicates the accuracy of the wheel group model on the validation set; Parameter difference degree : for calculating the cosine similarity distance between the current group aggregation model and the global model, where K denotes a set of all key values of model parameters, and denote the global model and group aggregation model parameters at the k-th layer, respectively; selection count : to record the cumulative number of times the set was selected in historical training rounds, wherein an indicator function that indicates whether the set of wheels is selected in the round. The group server will upload the constructed group state vector and group aggregation results to the central server to provide state input for the decision-making of the reinforcement learning intelligent agent.

8. The federated learning secure aggregation method based on privacy protection reinforcement learning according to claim 3, characterized in that, In step 6, the central server will collect the group state vectors The input reinforcement learning agent implements group selection by the following ε-greedy policy: Q value calculation: input the global state vector into the DDQN network to obtain the Q value evaluation of each group Mixed selection strategy: adopt ε-greedy algorithm, in each round selection from the remaining selectable groups, with a probability randomly selected from the remaining selectable groups, with a probability select the optimal group in the candidate group set, ensure that the final selected groups constitute the selected group set ; Dynamic exploration rate: update the exploration rate after each round of selection where the decay step ensures a smooth decrease from the initial exploration rate to the target exploration rate where = 0.9, = 0.

2.

9. The federated learning secure aggregation method based on privacy protection reinforcement learning according to claim 3, characterized in that, In step 7, the central server selects an action based on the group output of the reinforcement learning agent The selected group set Model parameter aggregation and performance evaluation are performed, and the specific steps are as follows: Step 71. Collect all the selected group's group-wise secure aggregation results ; Step 72. Global model calculation: generate a new generation of global model using the average aggregation strategy: Where |M| represents the number of selected groups in this round; Step 73. Model performance evaluation: Evaluate the performance of the current global model on the global test set, including: • The global test set is used to evaluate the performance of the current global model. Loss function: ; Forward propagation computes the predicted labels: ; Computing loss values: ; Computing accuracy: ; Step 74. Model distribution: The updated global model is distributed to all clients. Step 74. Model distribution: The updated global model is distributed to all clients.

10. The federated learning secure aggregation method based on privacy protection reinforcement learning according to claim 3, characterized in that, In step 8, the reputation value is calculated based on the group verification accuracy , is the public verification set ratio; After updating the global model, the central server synchronously executes the reinforcement learning training process, and the specific steps are as follows: Step 81. Experience storage: store the state-action-reward tuple into the experience replay buffer, where: is the group state vector, is the action selected for the group, is the reward value, is the next state, is the terminal state identifier; Step 82. Calculate Q value loss: the Q value loss of each selected group is given by the following formula: The goal is to: wherein is a discount factor, the first term represents an instant reward; When the number of samples in the experience replay buffer meets the training condition, the following update process is executed: Batch sampling: Random sampling from the experience pool of experience samples, Q value calculation: host network: ; target network: ; Target Q value calculation: where the discount factor ; Multi-action Q-learning: for each sample and each selected group , Total loss calculation: The main network parameters are updated using an Adam optimizer, and the target network is synchronized after every P rounds of training: Step 83. Calculate the linear decay exploration rate: 。

Images