Personalized federal learning method and system based on optimal transmission

By adopting the optimal transmission method and EMD distance in personalized federated learning, decoupling the client model structure and enhancing the feature capture capability, the problems of poor generalization and insufficient feature capture capability in the existing technology are solved, and more accurate and Safe model training.

CN118917444APending Publication Date: 2024-11-08ZHEJIANG UNIV

Patent Information

Application Number
CN202410952672.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing personalized federated learning methods lack generalization and feature capture capabilities when transmitting model parameters, and cannot effectively reduce the impact of data distribution differences, while ignoring the internal structural characteristics of the model parameters.

Method used

Using a personalized federated learning method based on optimal transmission, the cost matrix is ​​constructed by splitting the data into non-independent and homogeneous data sets, using EMD distance and sinkhorn methods, introducing entropy regular terms, decoupling the client model structure, and using two The dimensional matrix considers the model parameter relationship and enhances feature capture capabilities.

Benefits of technology

It improves the generalization and feature capture capabilities of the model, reduces the impact of data distribution differences, and ensures data privacy and the accuracy of model application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917444A_ABST
    Figure CN118917444A_ABST
Patent Text Reader

Abstract

The invention discloses a personalized federal learning method and system based on optimal transmission, and the method comprises the steps: determining the number of clients participating in learning, initializing global model parameters, transmitting the parameters to each client, receiving the parameters by each client, and transmitting the parameters to the client; a local global model is frozen, model training is performed by using local data to obtain new local model parameters, and a cost matrix is constructed by using an optimal transmission thought to reduce the distance between a local adapter and a global adapter, so that the local adapter learns global information in the global adapter. And finally, summing the generated parameters and uploading the summed parameters to a server, and adding the parameters by the server according to a sample proportion and updating a new round of global model parameters by aggregating hyper-parameters. According to the method, the model is decoupled at each client, the local generalization is improved, the data is not out of the local user, the data security is ensured, the internal relationship of the feature structure is considered by using the optimal transmission thought, and the feature capturing capability is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a personalized federated learning method and system based on optimal transmission. Background Art

[0002] Federated learning is a distributed machine learning method that aims to train machine learning models without collecting data from devices or locations to a central server. This is to avoid potential privacy risks that may be involved in traditional centralized machine learning. In recent years, due to concerns about user privacy, many countries have enacted data protection regulations to limit the direct transmission of private data.

[0003] Personalized federated learning is a branch of federated learning, which aims to solve the problems of personalized modeling and client local data heterogeneity. In some scenarios, different users or devices may have different preferences, characteristics or behavior patterns, and require customized models to meet their personalized needs. It can optimize and generate personalized local models using the data of all parties while protecting the data of all parties from being exposed.

[0004] The field of optimal transmission involves how to find the optimal mapping between different distributions to minimize the "distance" at a certain cost.

[0005] The local model currently used in the solution directly adds up and uploads the model parameters according to the proportion of data volume, which requires ensuring that the client model structure is the same and lacks generalization. Alternatively, the model parameters are added to the loss function using the common L1 and L2 paradigms as the metric distance, ignoring the structural characteristics within the parameters.

[0006] By personalizing the hierarchical structure and optimal transmission ideas in federated learning, the generalization of the model structure can be effectively improved, and the impact of differences in data distribution among parties can be reduced. At the same time, the characteristic of federated learning data not leaving the client is retained to protect the privacy of local data, which can improve the accuracy of the model application system. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention aims to provide a personalized federated learning method and system based on optimal transmission, and to provide a solution to the problems of poor generalization and poor feature capture capability existing in the above existing methods.

[0008] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0009] A personalized federated learning method based on optimal transmission includes the following steps:

[0010] S1: The server obtains the data source, calculates the data volume, and then splits the data into non-independent and identically distributed data sets and distributes them to each client;

[0011] S2: Determine the number of clients participating in learning, initialize the global model parameters Wg, and send the parameters to each client;

[0012] S3: Each client receives the parameters and uses local data and the local global model to perform client model training to obtain new local model parameters, and finally uploads the generated parameters to the server;

[0013] S4: The server uses the weighted merge parameters based on the proportion of data volume of each client, and updates the shared parameters using the global model update hyperparameter β. This step is performed multiple times until the termination condition is reached.

[0014] Preferably, in step S1, a Dirichlet distribution with a preset sharpness is used to perform non-independent and identically distributed division on the data label distribution.

[0015] Preferably, in step S2, the global model of the server can be split into a sample feature extractor and a feature adapter, the sample feature extractor is composed of a fully connected layer, a linear layer, and an activation layer, and the feature adapter includes a fully connected layer, which is used to identify sample features and generate decision results.

[0016] Preferably, the step S3 specifically includes:

[0017] S3.1: Each client receives the aggregated global model parameters from the server, updates the local global model parameters, uses local data for model training, and iteratively updates the local model parameters;

[0018] S3.2: Use the optimal transmission idea to construct a cost matrix to reduce the distance between the local adapter and the local global adapter. Use the EMD distance as the metric distance to reduce the local adapter model parameters and the local global adapter model parameters. Introduce entropy regularization and set auxiliary variables. Use the sinkhorn method to iteratively solve the transmission matrix P. Finally, upload the local model parameters to the server. Repeat this step until the number of iterations reaches the preset number.

[0019] As a further limitation of the technical solution of the embodiment of the present invention, the EMD distance needs to use local adapter parameters and global adapter parameters to construct a cost matrix C, and use the sinkhorn method to solve the optimal transmission matrix P.

[0020] In step S4, the termination condition is that the round of communication reaches a preset value T.

[0021] A personalized federated learning system based on optimal transmission, oriented to classification tasks, can combine the global model on the server side and the local model on the client side to jointly optimize the model in training, so as to achieve the personalized effect of each client node model. The system includes:

[0022] The data preprocessing module is used to collect and clean the data required for training, set data set parameters to divide the data set, and generate distributed non-independent and identically distributed data for system training.

[0023] The parameter initialization module is used to determine the number of clients participating in learning, initialize the global model parameters, and send the parameters to each client.

[0024] The client training module is used to receive the model sent by the server and assign it to the local corresponding model parameters, and calculate the corresponding loss value based on the EMD distance for the local model for back propagation during local model training. The server aggregation module is used to receive the model uploaded by each client after reaching the preset round of training, update the server model parameters according to the weight of each client data volume and hyperparameters, and re-send it to each client until the preset round is reached.

[0025] Compared to the requirement that all client model structures must be the same as the server model structure, the present invention decouples the local model, and the feature extraction structure of each client can be different, thereby improving the versatility of the local client model structure. Compared with directly measuring the model parameters in one dimension using a paradigm, the method uses a two-dimensional matrix to consider the structural relationship between model parameters, enhances the ability to capture model space features, and improves model performance.

[0026] The advantages of the present invention are: decoupling the model at each client, improving local generalization, ensuring data security by not leaving the user's local area, using the optimal transmission concept to consider the internal relationship of the feature structure, and enhancing the ability to capture features. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0028] Figure 1 Data flow diagram when training a local model for personalized federated learning based on optimal transmission.

[0029] Figure 2 Diagram of model parameter exchange between clients and servers for personalized federated learning based on optimal transmission.

[0030] Figure 3 Schematic diagram of the personalized federated learning system flow based on optimal transmission. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] Example 1

[0033] The local model currently used in the solution directly adds up and uploads the model parameters according to the proportion of data volume, which requires ensuring that the client model structure is the same and lacks generalization. Alternatively, the model parameters are added to the loss function using the common L1 and L2 paradigms as the measurement distance, and are measured from a one-dimensional perspective, ignoring the structural characteristics within the parameters.

[0034] Figure 1 The flow diagram of data during model training provided by an embodiment of the present invention is shown. Figure 2 The specific parameters uploaded and received by the client and server are shown.

[0035] Specifically, in a preferred embodiment of the present invention, a personalized federated learning method based on optimal transmission is provided, wherein the models in the method can be split into a sample feature extractor and a feature adapter, wherein the sample feature extractor is composed of a fully connected layer, a linear layer, and an activation layer, and the feature adapter is composed of a fully connected layer. The server includes a global model aggregated from each client; the client includes a global model and a local model, and the two cooperate to perform reasoning for training the local model.

[0036] Specifically, in a preferred embodiment of the present invention, a personalized federated learning method based on optimal transmission, the client receives the server parameter to set the global model feature extractor parameter φ g and the local feature extractor parameter φ i , through the function f(φ i , x i ) Obtain local and global feature representation information h i and h g , where the global feature extractor φ g It does not participate in back propagation updates and is only updated when the server aggregates parameters and then delegates them. The feature recognition layer is used to identify features for inference, and the loss is calculated through the cross entropy loss function and sample labels to solve W h,g and W h,i The optimal transmission matrix P and the minimum transmission distance C between * , update the feature identifier parameters W h,i , W h,g And the average of the two and φ iUpload to the server, the server aggregates the parameters and updates the global model parameters and sends the parameters to each client until the communication round is reached.

[0037] Figure 3 A schematic diagram of a system flow provided by an embodiment of the present invention is shown.

[0038] The present invention provides an image classification task method based on a personalized federated learning method for optimal transmission, which specifically includes the following steps:

[0039] Step S1: data preprocessing and generation;

[0040] Specifically, in the scenario of image classification tasks, the system can combine the global model on the server side and the local model on the client side to jointly optimize the model in training, so as to achieve the personalized effect of each client node model. The system obtains the data source, calculates the data volume, and then splits the data into non-independent and identically distributed data sets using Dirichlet distribution and distributes them to each client.

[0041] Step S2: The server initializes the global sample feature extractor parameter φ g , personalized feature extractor Φ = φ 1 ,φ 2 , ..., φ n , global feature adapter W h,g , Personalized Feature Adapter W h = {W h,1 , W h,2 , ..., W n,2}. Then the server sends initialization parameters to each client;

[0042] Step S3: After receiving the parameters, the client uses local data and W h,g and φ g To update the parameters, the client uses the local data to perform mini-batch gradient descent, using the dist(·, ·) function and the local feature adapter W h,i and the global feature adapter W h,g Construct the transfer cost matrix for each dimension feature where dist(a, b) = ||ab|| 2 ; Use the sinkhorn iteration method to construct auxiliary variables and calculate the transmission matrix P and the minimum transmission distance C * , minimum transmission distance C * Can be used as a loss function to train the model. After backpropagating the loss function, we get and And upload to the server;

[0043] Step S4: The server aggregates the client adapter parameters according to the proportion of each client data volume and client feature extractor parameters The server updates the global extractor parameters with hyperparameters α, β Update global adapter parameters And delegate it to the client. Repeat steps S3 to S4 until the communication round is reached.

[0044] Step S5: After the communication round is over, the client obtains the final optimized local model containing the global aggregation model module parameters. When processing a new image classification task, the user only needs to input the processed data into the trained model to process the task.

[0045] Example 2

[0046] This embodiment relates to a personalized federated learning system based on optimal transmission, which is oriented to classification tasks and can optimize the model in training by combining the global model on the server side and the local model on the client side to achieve the personalized effect of each client node model. The system includes: data preprocessing module, client training module, server aggregation module;

[0047] The data preprocessing module is used to collect and clean the data required for training, set data set parameters to divide the data set, and generate distributed non-independent and identically distributed data for system training.

[0048] The parameter initialization module is used to determine the number of clients participating in the learning, initialize the global model parameters, and send the parameters to each client.

[0049] The client training module is used to receive the model transmitted from the server and assign values ​​to the local corresponding model parameters, and calculate the corresponding loss value based on the EMD distance for the local model for back propagation during local model training.

[0050] The server aggregation module is used to receive the models uploaded by each client after reaching the preset round of training, update the server model parameters according to the data volume weight and hyperparameters of each client, and re-send them to each client until the preset round is reached.

[0051] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A personalized federated learning method based on optimal transmission, characterized in that: The following steps are involved: S1. The server obtains the data source, calculates the data volume, and then splits the data into non-independent and identically distributed data sets and delegates them to each client. S2, the server determines the number of clients participating in the learning, initializes the server's global model parameters Wg, and sends the parameters to each client; S3, each client receives the parameters and uses the local data and the local global model to perform client model training to obtain new local model parameters, and finally uploads the generated parameters to the server; S4. The server uses the proportion of data volume of each client as the weighted merge parameter, and uses the server's global model to update the hyperparameter β to update the shared parameter, and sends the parameter to each client for multiple rounds of training until the termination condition is reached.

2. The personalized federated learning method based on optimal transmission according to claim 1, characterized in that: In step S1, a Dirichlet distribution with a preset sharpness is used to divide the data label distribution into non-independent and identically distributed ones.

3. The personalized federated learning method based on optimal transmission according to claim 1, characterized in that: In step S2, the global model of the server can be split into a sample feature extractor and a feature adapter. The sample feature extractor is composed of a fully connected layer, a linear layer, and an activation layer. The feature adapter includes a fully connected layer for identifying sample features to generate decision results.

4. The personalized federated learning method based on optimal transmission according to claim 1, characterized in that: In step S3, the client receives the model parameters from the server and uses local data to perform model training to obtain new personalized model parameters, which specifically includes the following steps: S31, each client receives the aggregated model parameters from the server, updates the local global model parameters, uses the local data to perform model training, and iteratively updates the local model parameters; S32. Use the optimal transmission idea to construct a cost matrix to reduce the distance between the local adapter and the local global adapter. Use the EMD distance as the metric distance to reduce the local adapter model parameters and the local global adapter model parameters. Introduce entropy regularization and set auxiliary variables. Use the sinkhorn method to iteratively solve the transmission matrix P. Finally, upload the local model parameters to the server. Repeat this step until the number of iterations reaches the preset number.

5. The personalized federated learning method based on optimal transmission according to claim 4, characterized in that: The EMD distance requires constructing a cost matrix C using local adapter parameters and global adapter parameters, and solving the optimal transfer matrix P using the sinkhorn method.

6. The personalized federated learning method based on optimal transmission according to claim 1, characterized in that: In step S4, the termination condition is that the round of communication reaches a preset value T.

7. A personalized federated learning system based on optimal transmission, characterized by: include: The data preprocessing module is used to collect and clean the data required for training, set data set parameters to divide the data set, and generate distributed non-independent and identically distributed data for system training. The parameter initialization module is used to determine the number of clients participating in learning, initialize the global model parameters, and send the parameters to each client. The client training module is used to receive the model sent by the server and assign it to the local corresponding model parameters, and calculate the corresponding loss value based on the EMD distance for the local model for back propagation during local model training. The server aggregation module is used to receive the models uploaded by each client after reaching the preset round of training, update the server model parameters according to the data volume weight and hyperparameters of each client, and re-send them to each client until the preset round is reached.

Citation Information

Patent Citations

  • Federal learning method and system for data non-independent identically distributed scene

    CN114580663A

  • Image classification method based on personalized federal learning

    CN117253072A

  • Small sample target detection and identification method based on self-supervised ion separation space

    CN117876868A

  • Heterogeneous federated learning privacy protection method and system based on diffusion model

    CN118211268A

  • Multi-source transfer learning from pre-trained networks

    US20210319303A1

Cited By

  • Personalized global prototype federal learning method and system based on adaptive feature alignment

    CN120509507A