Federal mutual learning model training method for non-independent identically distributed data

Abstract

The invention provides a federal mutual learning model training method for non-independent identically distributed data, which comprises the following steps: S1, sending initial global model parameters to an intermediate client, generating intermediate client model parameters by the intermediate client, and S2, generating edge client model parameters by an edge client by using a local data set; S3, enabling the intermediate client and the edge client to update parameters by using a mutual learning method; s4, uploading a probability prediction value output by the intermediate client model to a server, and updating the global model and the intermediate client model by the server by utilizing a distillation technology; and S5, executing the steps S3-S4 until the models meet convergence conditions to obtain a final intermediate client model, a final edge client model and a final global model, and then broadcasting the final global model to all edge clients by the server. The problems of federated learning communication bandwidth limitation and model generation of the non-independent identically distributed data are solved through a grouping mutual learning and knowledge distillation method.

Description

technical field [0001] The invention relates to the technical field of federated learning, in particular to a federated mutual learning model training method for non-independent and identically distributed data. Background technique [0002] With the development of artificial intelligence and big data, information resources can achieve high-speed transmission in distributed networks, realizing the global management of physical information. Cloud computing and edge computing have promoted the development of deep learning. Deep learning models usually contain millions of parameters. Usually, expanding the scale of the neural network can effectively improve the accuracy of the model. However, in a distributed network, resource-constrained edge devices cannot When deploying a large-scale neural network model, there will be communication delays and the rejection of low-priority user terminal access. At this time, the powerful computing power and storage space of the central serve...

AI Technical Summary

Problems solved by technology

[0008] 1. The neural network will generate a large number of model parameters, and bandwidth limitations will affect the transmission of parameters and model training. Compared with cloud-based federated learning, the client-edge-cloud hierarchical federated learning system can reduce the number of clients by introducing an intermediate edge server. Direct communication with the cloud server reduces model training time, but still needs to transmit a large number of parameters, and there is still a lack of solutions that can reduce communication costs

[0009] 2. In recent years, knowledge distillation technology has been combined with federated learning, but when dealing with non-independent and identically distributed data, it is necessary to pre-generate a teacher model on a suitable proxy data set, but it is difficult to find a suitable data set to generate a teacher model. At present, there is still a lack of federated learning methods to solve the high communication cost and training performance loss encountered in non-independent and identically distributed data

Images