Personalized federated learning method and system based on multi-dimensional adaptive aggregation and local dynamic representation alignment
By employing multi-dimensional adaptive aggregation and local dynamic representation alignment within the FedMAS algorithm framework, the adaptability problem of traditional federated learning in heterogeneous data environments is solved, enabling efficient fusion and accurate quantization of personalized models, thereby improving model accuracy and communication efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN UNIV OF SCI & TECH
- Filing Date
- 2026-01-20
- Publication Date
- 2026-05-29
AI Technical Summary
Traditional federated learning struggles to adapt to all clients in heterogeneous data environments, resulting in insufficient model performance. Personalized federated learning methods suffer from issues such as coarse aggregation granularity and unbalanced regularization constraints.
By adopting the FedMAS algorithm framework, we can coordinate the optimization of model initialization, training and evaluation through multi-dimensional adaptive aggregation and local dynamic representation alignment. We also introduce layer importance mask and prototype contrast loss to achieve accurate quantification and dynamic fusion of personalized models.
It improves the personalized accuracy and communication efficiency of the model in heterogeneous data environments, and significantly enhances model adaptability and convergence speed.
Smart Images
Figure CN122114080A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of federated learning, specifically involving a personalized federated learning framework based on multi-dimensional adaptive aggregation and local dynamic representation alignment. Background Technology
[0002] Federated learning, as an innovative distributed machine learning paradigm, allows multiple data holders to participate in the model training process without directly sharing their local data, thereby achieving the dual goals of data privacy protection and model performance optimization. Compared to traditional centralized machine learning methods, federated learning does not require uploading users' raw data to a central server; instead, it interacts with information through local model parameters or gradients. This mechanism reduces the risk of privacy breaches, making federated learning significantly advantageous in privacy-sensitive applications requiring multi-party collaboration, such as healthcare and smart terminals. However, traditional federated learning faces the challenge of inherent heterogeneity among clients' local data in practical applications. This leads to significant differences in local updates across clients, making it difficult to obtain a single global model that can adapt to all clients, thus limiting the effectiveness of federated learning.
[0003] To address the adaptability issues of traditional federated learning in heterogeneous data environments, personalized federated learning has emerged. Instead of aiming to build a single global model, personalized federated learning focuses on training a custom model for each client that matches its local data distribution. Building upon the privacy advantages of federated learning, this mechanism further enhances the model's performance and applicability in heterogeneous data environments through knowledge sharing and local parameter adaptation among clients.
[0004] Currently, typical implementations of personalized federated learning fall into three categories: global model fine-tuning, personalized (local) aggregation, and regularization-based methods. Specifically, global model fine-tuning involves first training a general global model and then fine-tuning it using local data to adapt to the personalized needs of each client. However, this process relies on the initial performance of the global model, and personalization is relatively passive. Personalized (local) aggregation methods aggregate the global model and the client's local model according to certain weights to generate a locally adapted personalized model. However, its aggregation granularity is coarse, lacking refined guidance strategies. Regularization-based methods introduce regularization constraints on the global model into the local training objective to maintain a certain degree of global consistency. However, uniformly applied regularization constraints ignore the differentiated contributions of different parameters in personalization and sharing. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a personalized federated learning method and system based on multi-dimensional adaptive aggregation and local dynamic representation alignment, employing an algorithmic model framework named FedMAS. For highly heterogeneous data scenarios, this framework achieves collaborative optimization across three stages: local model initialization, model training, and model evaluation. This ensures that clients obtain personalized models adapted to their specific data scenarios while improving the personalization accuracy and communication efficiency of the federated learning algorithm.
[0006] The technical solution provided by this invention is as follows:
[0007] (1) The server randomly selects some clients to participate in the current round of training and distributes the latest global model and round information to them to start the current round of personalized federated learning.
[0008] (2) After receiving the global model parameters, each client performs multi-dimensional adaptive aggregation of the global model and the local personalized model obtained in the previous round of training based on the layer importance mask obtained in the previous round of training, and generates the initialization model for local training in the current round.
[0009] (3) The client uses this initial model as a starting point to perform local training. The standard cross-entropy loss and the prototype contrast loss are calculated through forward propagation, and the total loss (local dynamic representation alignment) is obtained by weighting. The model parameters are then updated through backpropagation.
[0010] (4) After local training is completed, the client calculates the new round of parameter importance (parameter importance fair quantification) by comparing the changes in model parameters at the start and end of training, and aggregates and generates a new layer importance mask for use in the next round of federated training.
[0011] (5) The client uploads the local personalized model obtained from this round of training to the server. The server performs weighted aggregation on all received client models and updates the global model.
[0012] Furthermore, the multi-dimensional adaptive aggregation mechanism in step (2) includes the following steps:
[0013] 2-1) Based on step (1), the client calculates the mixed weights for the parameters of a certain layer in the current round. The formula is as follows:
[0014]
[0015] in, It's the Sigmoid function, used to map weights to... between. For historical local characteristics, the base weight values of the parameters of layer l are derived from the importance score of that layer. Decision, that is This is the round attenuation factor, which is related to the number of communication rounds t and increases with the number of rounds. The specific formula is as follows: To determine the structural characteristics of the model, different hybridization preferences are assigned to different types of layers, as specified in the formula:
[0016]
[0017] in, For model structure weight function, For feature extraction layers (convolutional layers), we tend to assign relatively high values (e.g., 1.2) to encourage personalization. For the normalization layer, we assign a lower value (such as 0.8) to promote generalization.
[0018] 2-2) Obtain the mixed weights Then, the global model and the local personalized model obtained from the previous training round are subjected to multi-dimensional adaptive aggregation, and the calculation formula is as follows:
[0019]
[0020] in This represents element-wise multiplication. It is a hybrid weight with the same dimension as the model parameters. This is the local personalized model obtained from the previous training round. This is the global model received in this round.
[0021] Furthermore, the standard cross-entropy loss and prototype comparison loss in step (3) are calculated as follows:
[0022]
[0023] in, C is the batch size, and C is the number of categories. It is the true label (one-hot form) of sample k. It is the probability that the model predicts belongs to category C. and Features and labels extracted for the current batch of samples. It is the cosine similarity function. This represents the temperature coefficient. The client maintains an updatable prototype dictionary. }, The feature prototype representing category c. To update the prototype using an exponential moving average strategy, the features extracted for the current batch of samples... and its tags The corresponding prototype update formula is:
[0024]
[0025] in, For fixed as The aim is to balance the stability of the prototype representation with its adaptability to local data distribution.
[0026] Furthermore, in step (3), the total local training loss is a weighted linear combination of the standard cross-entropy loss and the prototype contrast loss:
[0027]
[0028] in, It is a hyperparameter used to coordinate the two losses. The standard cross-entropy loss is used to measure the difference between the model's predicted distribution and the true label distribution, ensuring that the model completes the main classification task. It is a prototype contrast loss that improves feature discriminativeness.
[0029] Furthermore, the specific process of calculating the layer importance mask in step (4) includes the following steps:
[0030] 4-1) After one round of local training, the client compares the training starting points. and training endpoint Importance is calculated based on the changes in the parameters. For the j-th parameter in the model... Its importance The calculation is as follows:
[0031]
[0032] in It is a hyperparameter that balances the two terms. It is a very small constant used to prevent division by zero errors. The first term measures the absolute change of the parameter, and the second term measures its relative rate of change, overcoming the problem of large-value parameters dominating when only absolute values are used. To initialize the model at the start of training, This is the model at the end of training.
[0033] 4-2) Direct transmission and manipulation of parameters incur significant overhead; therefore, we aggregate importance at the layer level. For layer l, its importance score is... It is the sum of the importance of all its parameters:
[0034]
[0035] The layer importance mask is obtained by normalization for the next round. :
[0036]
[0037] The mask will be as The historical importance dimension input of the multi-dimensional adaptive aggregation mechanism in the round forms a closed-loop, self-evolving personalized federated learning system.
[0038] Compared with the prior art, the advantages of the present invention are as follows:
[0039] (1) The multi-dimensional and dynamic personalized federated learning framework of this invention achieves collaborative optimization in three stages: local training model initialization, model training and model evaluation. While ensuring that the client obtains a personalized model that is adapted to its own data scenario, it significantly improves the personalization accuracy and communication efficiency of the federated learning algorithm.
[0040] (2) The multi-dimensional adaptive aggregation mechanism of the present invention accurately quantifies the importance of parameters at each layer through a parameter fairness quantification mechanism, and dynamically generates fine-grained personalized training starting points by integrating the influence of multiple dimensions, thereby realizing the adaptive and fair fusion of the global model and the local models of each client.
[0041] (3) The local dynamic representation alignment mechanism of the present invention introduces a hybrid loss function by fusing prototype contrast learning, and enhances the discriminativeness and robustness of the model by explicitly constraining the feature space, thereby optimizing the personalized learning process.
[0042] (4) The effectiveness of the present invention was verified under various Non-IID experimental dataset partitioning methods based on the Dirichlet distribution. Experimental results show that the present invention can effectively improve model accuracy, accelerate model convergence, and achieve better results than the benchmark method. Attached Figure Description
[0043] Figure 1 This is the overall flowchart of the present invention.
[0044] Figure 2 This is a system framework diagram of the present invention. Detailed Implementation
[0045] The present invention will be further described below with reference to the embodiments and accompanying drawings. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0046] See Figure 1 , Figure 2This embodiment discloses a personalized federated learning method and system based on multi-dimensional adaptive aggregation and local dynamic representation alignment. The method includes the following steps:
[0047] Step 1: The server randomly selects some clients to participate in the current round of training and distributes the latest global model and round information to them to start this round of personalized federated learning.
[0048] Step 2: After receiving the global model parameters, each client performs multi-dimensional adaptive aggregation of the global model and the local personalized model obtained in the previous training round based on the layer importance mask obtained in the previous training round, and generates the initialization model for local training in the current round.
[0049] (1) The client calculates the mixed weights for the parameters of a certain layer in the current round. The formula is as follows:
[0050]
[0051] in, It's the Sigmoid function, used to map weights to... between. For historical local characteristics, the base weight values of the parameters of layer l are derived from the importance score of that layer. Decision, that is This is the round attenuation factor, which is related to the number of communication rounds t and increases with the number of rounds. The specific formula is as follows: To determine the structural characteristics of the model, different hybridization preferences are assigned to different types of layers, as specified in the formula:
[0052]
[0053] in, For model structure weight function, For feature extraction layers (convolutional layers), we tend to assign relatively high values (e.g., 1.2) to encourage personalization. For the normalization layer, we assign a lower value (such as 0.8) to promote generalization.
[0054] (2) Obtain the mixed weights Then, the global model and the local personalized model obtained from the previous training round are subjected to multi-dimensional adaptive aggregation, and the calculation formula is as follows:
[0055]
[0056] in This represents element-wise multiplication. It is a hybrid weight with the same dimension as the model parameters. This is the local personalized model obtained from the previous training round. This is the global model received in this round.
[0057] Step 3: The client uses this initialized model as a starting point for local training. The standard cross-entropy loss and prototype contrast loss are calculated via forward propagation, weighted to obtain the total loss (local dynamic representation alignment). The model parameters are then updated via backpropagation.
[0058] (1) During local training, calculate the standard cross-entropy loss:
[0059]
[0060] in, C is the batch size, and C is the number of categories. It is the true label (one-hot form) of sample k. It is the probability that the model predicts belonging to category C.
[0061] (2) The client maintains an updatable prototype dictionary. }, The feature prototype represents category c. Exponential moving average is a widely used technique for time series smoothing; it updates the state by combining historical values with current observations. We apply this mechanism to prototype updates, for features extracted from the current batch of samples. and its tags The corresponding prototype update formula is:
[0062]
[0063] in, For fixed as This approach aims to balance the stability of prototype representations with adaptability to local data distributions. The prototype contrastive loss is calculated as follows:
[0064]
[0065] in, C is the batch size, and C is the number of categories. and Features and labels extracted for the current batch of samples. It is the cosine similarity function. This is the temperature coefficient.
[0066] (3) The total loss during local training is a weighted linear combination of the standard cross-entropy loss and the prototype contrast loss:
[0067]
[0068] in, It is a hyperparameter used to coordinate the two losses. The standard cross-entropy loss is used to measure the difference between the model's predicted distribution and the true label distribution, ensuring that the model completes the main classification task. It is a prototype contrast loss that improves feature discriminativeness.
[0069] Step 4: After local training is completed, the client calculates the new round of parameter importance (parameter importance fairness quantification) by comparing the changes in model parameters at the start and end of training, and aggregates them to generate a new layer importance mask for use in the next round of federated training.
[0070] (1) After a round of local training, the client compares the training starting point. and training endpoint Importance is calculated based on the changes in the parameters. For the j-th parameter in the model... Its importance The calculation is as follows:
[0071]
[0072] in It is a hyperparameter that balances the two terms. It is a very small constant used to prevent division by zero errors. The first term measures the absolute change of the parameter, and the second term measures its relative rate of change, overcoming the problem of large-value parameters dominating when only absolute values are used. To initialize the model at the start of training, This is the model at the end of training.
[0073] (2) Direct transmission and operation of parameters incur significant overhead; therefore, we aggregate importance at the layer level. For layer l, its importance score is... It is the sum of the importance of all its parameters:
[0074]
[0075] The layer importance mask is obtained by normalization for the next round. :
[0076]
[0077] The mask will be as The historical importance dimension input of the multi-dimensional adaptive aggregation mechanism in the round.
[0078] Step 5: The client uploads the local personalized model obtained from this round of training to the server. The server performs weighted aggregation on all received client models and updates the global model.
[0079] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A personalized federated learning framework based on multi-dimensional adaptive aggregation and local dynamic representation alignment, characterized in that: Each round of federal training includes the following steps: Step 1: The server randomly selects some clients to participate in the current round of training and distributes the latest global model and round information to them to start this round of personalized federated learning; Step 2: After receiving the global model parameters, each client performs multi-dimensional adaptive aggregation of the global model and the local personalized model obtained in the previous training round based on the layer importance mask obtained in the previous training round, and generates the initialization model for the current round of local training. Step 3: The client starts with this initialized model and performs local training; the standard cross-entropy loss and prototype contrast loss are calculated through forward propagation, and the total loss (local dynamic representation alignment) is obtained by weighting; the model parameters are updated through backpropagation. Step 4: After local training is completed, the client calculates the new round of parameter importance (parameter importance fairness quantification) by comparing the changes in model parameters at the beginning and end of training, and aggregates and generates a new layer importance mask based on this, for use in the next round of federated training. Step 5: The client uploads the local personalized model obtained from this round of training to the server. The server performs weighted aggregation on all received client models and updates the global model.
2. The personalized federated learning framework according to claim 1, characterized in that: The process of the multi-dimensional adaptive aggregation mechanism in step 2 is as follows: 2-1) For the parameters of a certain layer, calculate the mixing weights. The formula is as follows: in, It's the Sigmoid function, used to map weights to... between; For historical local characteristics, for layers l The parameters, whose base weights are derived from the importance score of that layer. Decision, that is ; This is the round attenuation factor, which is related to the number of communication rounds t and increases with the number of rounds. The specific formula is as follows: ; To determine the structural characteristics of the model, different hybridization preferences are assigned to different types of layers, as specified in the formula: in, The model structure weight function; For feature extraction layers (convolutional layers), we tend to assign relatively high values (such as 1.2) to encourage personalization; For the normalization layer, we assign a lower value (such as 0.8) to promote generalization; 2-2) Obtain the mixed weights Then, the global model and the local personalized model obtained from the previous training round are subjected to multi-dimensional adaptive aggregation, and the calculation formula is as follows: in This represents element-wise multiplication; It is a hybrid weight with the same dimension as the model parameters; This is the local personalized model obtained from the previous round of training; This is the global model received in this round.
3. The personalized federated learning framework according to claim 1, characterized in that: The specific calculation formulas for the standard cross-entropy loss and the prototype comparison loss in step 3 are as follows: in, C is the batch size, and C is the number of categories. It is the true label (one-hot form) of sample k. It is the probability that the model predicts belonging to category C; and Features and labels extracted for the current batch of samples; It is the cosine similarity function. For temperature coefficients; the client maintains an updatable prototype dictionary. }, Representative category c The characteristic prototype; To update the prototype using an exponential moving average strategy; features extracted for the current batch of samples. and its tags The corresponding prototype update formula is: in, For fixed as The aim is to balance the stability of the prototype representation with its adaptability to local data distribution.
4. The personalized federated learning framework according to claim 1, characterized in that... In step 3, the total local training loss is a weighted linear combination of the standard cross-entropy loss and the prototype contrast loss. in, It is a hyperparameter used to reconcile the two losses; The standard cross-entropy loss is used to measure the difference between the model's predicted distribution and the true label distribution, ensuring that the model completes the main classification task; It is a prototype contrast loss that improves feature discriminativeness.
5. The personalized federated learning framework according to claim 1, characterized in that: The specific process for calculating the layer importance mask in step 4 is as follows: 5-1) After one round of local training, the client compares the training starting points. and training endpoint The importance is calculated based on the changes in the parameters; for the j-th parameter in the model... Its importance is calculated as follows: in It is a hyperparameter that balances the two terms. It is a very small constant used to prevent division by zero errors; the first term measures the absolute change of the parameter, and the second term measures its relative rate of change, overcoming the problem that large-value parameters dominate when only absolute values are used. This is the initialization model at the start of training; The model at the end of training; 5-2) Direct transmission and manipulation of parameter-level importance incur huge overhead, so we aggregate importance to the layer level; for layer l, its importance score is the sum of the importance of all its parameters: The layer importance mask for the next round is obtained by normalization: The mask will be as The historical importance dimension input of the multi-dimensional adaptive aggregation mechanism in the round forms a closed-loop, self-evolving personalized federated learning system.
6. A personalized federated learning method and system, characterized in that: Including the method described in any one of claims 1-5, for achieving the following objectives in a non-independent and identically distributed (Non-IID) data environment: A multi-dimensional adaptive aggregation mechanism is used to perform refined aggregation of the global model and the local personalized model; The feature space is explicitly constrained through a local dynamic representation alignment mechanism to improve feature discriminativeness; The importance of parameters is measured through a fair quantification mechanism.