Federal learning-oriented dynamic personalized network construction method and device

Abstract

The invention discloses a federated learning-oriented dynamic personalized network construction method and device, and the method comprises the following steps: creating a neural network model, initializing the neural network model, and transmitting the neural network model to a client; performing dynamic calculation on a sample of the client according to an early-leaving strategy, and finishing updating of an early-leaving layer; selecting an early leaving strategy execution layer, and selectively sending parameters in the selected early leaving layer; after the server receives the nodes and the parameters uploaded by the client, aggregation of the weights of the nodes of the early leaving layer is completed according to an aggregation strategy; and receiving the aggregated parameters from the server, completing the updating operation of the model parameters in the client, and completing the construction of the personalized network by means of the updating of the early leaving layer. According to the dynamic personalized network construction method disclosed by the invention, the reasoning efficiency of the model and the attention of the difficult sample are improved, and the cost of federal learning in sample calculation and model reasoning is reduced while the personalization of the client model is ensured.

Description

technical field [0001] The invention relates to a method for constructing a personalized network, in particular to a method and device for constructing a dynamic personalized network oriented to federated learning. Background technique [0002] Deep neural networks have achieved great success in the fields of computer vision, natural language processing, etc. It has also witnessed the birth of efficient neural networks such as AlexNet, ResNet, and Transformer, and has also given many mobile deep learning development, such as mobile chipsets equipped with There are more and more specialized processing units to execute DNNs more efficiently. Therefore, people put forward higher requirements for device performance, energy consumption, and data privacy in model training on mobile devices. [0003] Typically, DNN models are trained on datasets and implement model generalization operations. However, since the data often appear in the form of non-IIDs in real life, the global mod...

AI Technical Summary

Problems solved by technology

It mainly includes: 1) increasing the user's context, but it is debatable whether there is a leakage of privacy due to context characterization; 2) federated migration learning, which has more or less catastrophic forgetting problems; 3) federated multi-task learning, for When each task generates a model, all clients must participate in each round of training; 4) Federal knowledge distillation, in which users are prone to overfitting for small-sample datasets; 5) Federal meta-learning, which can build a High-precision global model, but it will damage the personalization ability of the model

[0004] Although a lot of work has been done on the construction of federated learning personalization in recent years, the current construction of the personalization model is mainly based on the static network structure, which means that all samples consume the same amount of calculation

Choosing a small model may not get a good training, and choosing a larger model will bring inevitable redundant calculations

This limits the expressive power, reasoning efficiency and interpretability of the model to some extent

In addition, in the test phase, the sample reasoning process still cannot reasonably release computing resources, and the model involved in the prediction is relatively simple, which cannot achieve the prediction of the sample personalized network

[0005] In view of the shortcomings of the current personalized federated learning model, such as the difficulty of releasing capacity, the difficulty of allocating computing resources of equipment, and the single structure of training models, etc.

Images