Personalized federated learning method based on hybrid expert model

Abstract

In order to overcome the defect that a private model in a large-scale stateless mobile federation environment is difficult to realize full training, the invention provides a personalized federation learning method based on a hybrid expert model, which comprises the following steps that: all clients join federation learning to jointly participate in the training of a global model to obtain a globalmodel parameter thetaG; the thetaG is downloaded from the server by each client, a feature extraction layer and a personalized classification layer in the client are initialized by utilizing the parameters, and personalization is performed by utilizing a fixed base method to obtain personalized classification layer parameters; at the moment, the client i has the thetaG including feature extraction layer parameters and global classification layer parameters and personalized classification layer parameters, and the feature extraction layer, the global classification layer and the personalized classification layer are initialized through the personalized classification layer parameters, the feature extraction layer parameters and the global classification layer parameters to jointly train agating model to obtain gating model parameters. And finally, the client obtains the parameters of the feature extraction layer, the global classification layer, the personalized classification layer and the gating model to complete personalized federated learning.

Description

technical field [0001] The present invention relates to the field of federated learning, and more specifically, relates to a personalized federated learning method based on a mixed expert model. Background technique [0002] In the field of deep learning, the quantity and quality of training data largely determine the training effect of deep neural network models. Under the premise of considering user privacy, user data can no longer be collected in the data center, but it is difficult to train an effective model only through the isolated data of each client. Federated learning where data is kept locally on the client is an effective solution. Federated learning uses data and computing resources scattered across each client to update the model, and aggregates the model updates of each client to obtain a global model to ensure user data privacy. At the same time, a global model that makes full use of global data is obtained through training. In personalized federated learni...

AI Technical Summary

Problems solved by technology

[0004] However, privacy-adaptive privacy federated learning requires all clients to participate synchronously, and it requires all clients to maintain an intermediate state during the federated training process, including the local gated model and private model state, which makes it limited to inter-agency Federated learning, it is difficult to achieve sufficient training for private models in large-scale stateless mobile federated environments

Since the gating model is a single-layer linear neural network, in federated training tasks based on high-dimensional data such as images, the gating model cannot directly use high-dimensional input data to effectively balance the private model and the global model.

Images