Federal learning method for submodel extraction based on gradient in resource limited scene
By selecting the sub-model based on the gradient and performing round-rotation updates in resource-constrained scenarios, combining mask cross-entropy loss and static batch normalization technology, the high computational cost and model generalization ability of federated learning on edge devices is solved. The problem of insufficient results enables efficient model training and data privacy protection.
Patent Information
- Application Number
- CN202411020955.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
AI Technical Summary
In scenarios where data is not shared and client resources are limited, the existing federated learning methods are computationally costly and difficult to train large-scale models on edge devices. The distillation learning method adds the steps of model aggregation, affecting the generalization ability of the model .
By initializing the central server and client, selecting data with similar distributions of the public subdata set to the client's private data, constructing sub-models based on gradient information, and training them on the client, updating the global model round by round, using mask cross-entropy loss, Scaling module and static batch normalization technology optimize model training.
It reduces the demand for computing resources, improves the generalization ability of the model, makes resource-constrained federated learning close to the effect of ordinary federated learning, and solves data privacy issues through various ways of constructing public data sets.
Smart Images

Figure CN118966320A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a federated learning method for gradient-based sub-model extraction in resource-constrained scenarios. Background Art
[0002] With the acceleration of the big data era, training deep models based on huge amounts of data has achieved remarkable results, making people's work more efficient. In traditional deep learning, data is collected on a single computer or a cluster that can access each other for training models. This method is simple and efficient. However, given that certain specific data sets are highly sensitive (such as patient information in medical institutions and financial records in financial institutions), cross-subject data sharing has become a major problem. In this context, model training based solely on limited data sets is very likely to encounter performance ceilings and fall into the dilemma of overfitting, and the model will not be applicable to complex real-world problems. In order to ensure data privacy while mining data value, Google first proposed the concept of federated learning in 2016. This innovative technology, based on distributed optimization algorithms under multi-party collaboration, aims to cross data silos so that all participants can collaboratively improve model performance without directly exchanging data.
[0003] Although federated learning provides a feasible path to alleviate data privacy and distributed processing requirements, it also faces several unresolved problems. On the one hand, the computing cost is relatively high, especially when edge devices perform model training tasks. The current technical framework's large demand for computing resources increases the computing burden on the edge, which undoubtedly constitutes a major obstacle to training large-scale models for small organizations with scarce computing resources. On the other hand, the use of distillation learning methods to train smaller models that are different from the global model architecture on the client is not conducive to model aggregation and adds additional steps. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a sub-model similar to the server-side model architecture for client training under the premise that data is not shared and client resources are limited. The gradient-based selection makes the sub-model more in line with the client's private data distribution, thereby enhancing the generalization ability of the global model.
[0005] In order to achieve the above object, the present invention provides a federated learning method for gradient-based sub-model extraction in a resource-constrained scenario, comprising the following steps:
[0006] S1. Initialize the central server and client, and initialize the parameters of the global model on the server;
[0007] S2, the server selects the clients that need to participate in training in this round;
[0008] S3. Select public sub-datasets suitable for different clients from the public data set according to the historical models saved by different clients from the previous round. These public sub-datasets have similar data distribution to the private data of the corresponding clients.
[0009] S4, back-propagating the global model on the server according to the selected common sub-dataset, and saving the gradient of each neuron of the entire model;
[0010] S5. According to the gradient information, neurons are selected from the global model layer by layer to form a sub-model for client training, and the sub-model is sent to the corresponding client;
[0011] S6. The client receives the sub-model from the server, the sub-model is trained based on the local private data, and the trained sub-model is uploaded to the server;
[0012] S7, the server accepts the models uploaded by all clients in this round, saves these models, and then matches the positions of corresponding neurons for aggregation, and finally obtains the global model of this round;
[0013] S8. Steps S2 to S5 constitute a round of training, which is repeated each time until the model capability meets the requirements.
[0014] Preferably, the selecting of common sub-datasets applicable to different clients from the common data set according to the historical models saved by different clients in the previous round is specifically:
[0015] If this is the first round of training and there is no historical model on the server, a certain number of sub-datasets will be randomly selected. If this is not the first round, the model uploaded by the client last time will be used to predict on the public dataset, and the predicted data will be used to form the sub-datasets.
[0016] If the public dataset itself has labels, then use the historical model to make predictions on the entire public dataset, and use the correctly predicted data to form a sub-dataset; if the public dataset itself only has data, then make predictions on the entire dataset, and use the predicted labels as the labels of the sub-datasets.
[0017] Preferably, the public dataset is constructed using three methods, namely: directly uploading local data from the client, the client constructing similar data based on local data using a generative model, selecting an alternative dataset containing similar data types, or using local data labels as substitutes.
[0018] Preferably, the selecting neurons by layer from the global model according to the gradient information to form a sub-model for client training, and sending the sub-model to the corresponding client, is specifically as follows:
[0019] The number of neurons in each layer of the sub-model is allocated according to the computing power of the corresponding client, from shallow to deep, layer by layer;
[0020] In each layer, the absolute value of the gradient of the neurons in this layer is calculated, sorted by size, and selected from large to small until the number of selections reaches the maximum number of training that the client can bear.
[0021] Preferably, the server accepts the models uploaded by all clients in this round, saves these models, and then matches the positions of corresponding neurons for aggregation, and finally obtains the global model of this round, specifically:
[0022] Traverse the sub-models uploaded by the selected client, and record the specific position of each neuron in the global model in each sub-model, and obtain the number of occurrences and specific parameter values of the neuron at each position in the global model;
[0023] According to the number of occurrences of each neuron and the specific parameter value, the average parameter value of each neuron is calculated as the final value, and the entire aggregation process is completed.
[0024] Preferably, the gradient of each neuron is defined as g:
[0025]
[0026] In the classification task, using convolutional neural network, h l,i represents the activation value matrix of the i-th neuron in layer l, K and V represent the length and width of the feature map, and the definition of the neuron gradient is the sum of the absolute values of the gradient of the activation value of the entire feature map.
[0027] Preferably, the sub-model uses masked cross entropy loss, scaling module and static batch normalization during training based on local private data.
[0028] Preferably, the scaling module is inserted after the parameter layer and before the activation layer, and is used in the training phase. The input value is scaled by a factor of , where q is the offset rate of the global model parameters.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] 1. The present invention proposes three ways to construct public datasets, from simply providing them directly by customers to selecting other data to replace them to using generative models to construct virtual datasets. This solves the difficulty of collecting public datasets for federated learning. The three methods provide different degrees of solutions to the privacy issues that people are concerned about.
[0031] 2. The present invention proposes a method for constructing or selecting a public sub-dataset with a similar distribution to the relevant client private data set from a public sub-dataset. These two methods require the public data set to be labeled or unlabeled, respectively, which greatly improves the ability to solve practical problems.
[0032] 3. Compared with previous methods, the present invention proposes a more novel and reasonable way to solve the model extraction problem under resource-constrained conditions. The sub-model is selected by calculating the gradient of the global model, which ensures that the gradient update of the sub-model is closest to the gradient update of the global model, thereby making the resource-constrained federated learning effect closest to that of ordinary federated learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is an overall flow chart of the federated learning training based on the gradient extraction sub-model in the present invention;
[0034] Figure 2 To select the dataset and extract the overall picture of the sub-model;
[0035] Figure 3 Schematic diagram of two methods for selecting sub-datasets in a public dataset;
[0036] Figure 4 It is a specific schematic diagram of the gradient extraction model. DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] The present invention proposes a federated learning method for gradient-based sub-model extraction in a resource-constrained scenario, and the specific steps are as follows:
[0039] S1. Initialize the central server and client, and initialize the parameters of the global model on the server. Where f(w) is the input sample The predicted label and its true label The cross entropy loss of the difference between n (w) represents the average local loss function of all customers. n Indicates the number of data in the client n data set.
[0040] The goal of the federated learning algorithm is to minimize the size of the loss function and obtain a better model.
[0041]
[0042] S2. The server selects the clients that need to participate in the training in this round. In the actual execution process, 10% of all clients are generally selected. In the experiment, the number of participating clients is set to 100, so the number of clients participating in the training in each round is 10. These 10 clients are randomly selected each time to ensure that the probability of all clients being selected is the same.
[0043] S3. Select public sub-datasets suitable for different clients based on the historical models saved by different clients from the previous round on the public data set. These public sub-datasets have similar data distribution to the private data of the corresponding clients. First, construct the public data set. Use three methods to construct the data set:
[0044] The direct method is that the client uploads part of the local data to the server. Uploading a portion of the data up to 1% of the local data is considered sufficient. There is no requirement for these data labels.
[0045] Generate similar data. The client can use the generative model to construct similar data using local data, such as DDPM, GAN, Stable Diffusion and other generative algorithms. The generative model can learn the local data set distribution and then output similar data. This method can not only prevent privacy leakage, but also generate data that is very similar to the user.
[0046] Use similar data. Choose alternative datasets that contain similar data types or label local data as alternatives. For example, if the local dataset is CIFAR10, then ImageNet can be a viable alternative as a public dataset.
[0047] Determine whether it is the first round of training. If this is the first round of training, there is no historical model on the server. At this time, a certain number of sub-datasets are randomly selected; if it is not the first round, the model uploaded by the client last time is used to predict on the public data set, and the predicted data is used to form a sub-dataset.
[0048] Determine whether the public dataset has labels. If the public dataset itself has labels, use the historical model to make predictions on the entire public dataset, and use the correctly predicted data to form a sub-dataset; if the public dataset itself only has data, make predictions on the entire dataset, and use the predicted labels as the labels of the sub-datasets.
[0049] For public datasets, clients are required to upload data that accounts for 1% or even less of the local dataset. These data may be unlabeled, or the other two of the three methods provided by the present invention may be used. i ′ (represents the model of client i most recently saved on the server) in the public dataset D P Select the public sub-datasets suitable for different clients (represents a public sub-dataset with similar distribution to client i), these public sub-datasets have similar data distribution to the corresponding client private data, that is, like Figure 3 As shown, in method 1, M i ′ Output predicted value based on input (x,y) if Add (x,y) to the subdataset In method 2, M i ′ Output the predicted value based on the input (x) and will Add subdatasets Choose one of the methods to construct the common sub-dataset; if it is in the first round, that is, At this time, randomly select enough data sets to construct sub-datasets Finally, a common sub-dataset with similar data distribution to the client side is generated on the server side.
[0050] S4. Back-propagate the global model on the server based on the selected public sub-dataset and save the gradient of each neuron in the entire model. Figure 4 As shown, the gradient of each neuron is defined as g;
[0051]
[0052] In the specific classification task, convolutional neural network is used, h l,i represents the activation value matrix of the i-th neuron in layer l, K and V represent the length and width of the feature map, and the definition of the neuron gradient is the sum of the absolute values of the gradient of the activation value of the entire feature map.
[0053] S5. According to the gradient information, neurons are selected from the global model layer by layer to form a sub-model for client training, and the sub-model is sent to the corresponding client. globalFrom shallow to deep, from left to right, the number of neurons in each layer of the sub-model is allocated according to the computing power of the corresponding client. In each layer, the absolute value of the gradient of the neurons in this layer is calculated, sorted by size, and selected from large to small until the number of selected neurons reaches the maximum number of training that the client can bear. Figure 4 Indicates the specific allocation process in a certain layer, M global There are 5 neurons in a certain layer, corresponding to n clients. In the figure, client 0, client i, and client n have different computing capabilities. Client 0, client i, and client n have the ability to train 4, 3, and 2 neurons in this layer respectively. The gradients saved by back propagation for each client are {g 1 ,g 2 ,g 3 ,g 4}≥{g 0},{g 0 ,g 1 ,g 4}≥{g 2 ,g 3},{g 0 ,g 4}≥{g 1 ,g 2 ,g 3}, the corresponding selections on the three clients are the combinations of neurons {1,2,3,4}, {0,1,4}, and {0,4}. After each layer of selection is completed, a sub-model with the same depth and sparsity as the global model is formed based on the results in each layer.
[0054] S3~S5, Figure 2 It shows the overall process from S3 to S5, and the entire process is performed on the server. Figure 2 The first step is based on the historical model M i ′ Predict the public sub-dataset The second step uses the obtained common sub-dataset In the global model M global Perform back propagation to obtain the entire M global The gradient of the network is then selected layer by layer to form a sub-model. For the number of neurons selected on the client side, the client side is generally compared with the server side, and is generally classified into M global Capable This is conducive to the average distribution, which means that among 100 clients, the client capacity is There are 20 of them, and similarly there are 20 others. The computing power of each client is determined in advance.
[0055] S6. The client receives the sub-model from the server, trains the sub-model based on the local private data, and uploads the trained sub-model to the server. The training process of the client uses:
[0056] 1. Masked cross entropy loss. It has been shown that the failure of federated learning (FL) can be attributed to the weight differences between the parameters of local models trained locally in multiple iterations. This weight difference is mainly reflected in the final classification layer of the neural network. Therefore, unlike using a full cross entropy loss that includes all classes, the present invention proposes that the training of each local model is only related to their respective classes. This method allows each local model to focus on a specific subtask based on the available local label information. To achieve this, the present invention introduces a masked cross entropy loss, which involves masking the output of the model before applying the loss function.
[0057] 2. Scaling module. Since the goal of the present invention is to optimize the client model through multiple rounds, the parameters of the local model at different computational complexity levels may deviate and change in scale. This phenomenon is called offset. In the inference phase, in order to directly use the full model, an offset rate of q is used. This technology performs a scaling operation on the input value during the training phase. To solve this problem, the present invention introduces a scaling module, which is inserted immediately after the parameter layer and before the activation layer. The input value is scaled by a factor of . After global aggregation, the global model can be used directly.
[0058] 3. Static Batch Normalization. Batch Normalization (BN) is widely used in deep learning models. However, many mainstream algorithms avoid BN. An important issue related to BN is the need to continuously estimate the representation of each hidden layer. The process of passing these statistics to the server will increase communication costs and raise privacy concerns. Static Batch Normalization (sBN) is applied to optimize privacy-restricted heterogeneous models. During the training phase, sBN simply normalizes the batch data and does not track the running estimates. This method is suitable for the present invention because each communication round is independent. After completing the training process, the server queries the local client in turn.
[0059] S7. The server accepts the models uploaded by all clients in this round, saves these models, and then matches the positions of the corresponding neurons for aggregation, and finally obtains the global model of this round. Traverse the sub-models uploaded by the selected client, and record the specific position of each neuron in the global model in each sub-model, and obtain the number of occurrences and specific parameter values of neurons at each position in the global model. According to the number of occurrences and specific parameter values of each neuron, calculate the average parameter value of each neuron as the final value. The entire aggregation process is completed.
[0060] S8. Steps S2 to S5 constitute a round of training, which is repeated each time until the model capability meets the requirements.
[0061] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.
Claims
1. A federated learning method for gradient-based sub-model extraction in resource-constrained scenarios, characterized by: The following steps are involved: S1. Initialize the central server and client, and initialize the parameters of the global model on the server; S2, the server selects the clients that need to participate in training in this round; S3. Select public sub-datasets suitable for different clients from the public data set according to the historical models saved by different clients from the previous round. These public sub-datasets have similar data distribution to the private data of the corresponding clients. S4, back-propagating the global model on the server according to the selected common sub-dataset, and saving the gradient of each neuron of the entire model; S5. According to the gradient information, neurons are selected from the global model layer by layer to form a sub-model for client training, and the sub-model is sent to the corresponding client; S6. The client receives the sub-model from the server, the sub-model is trained based on the local private data, and the trained sub-model is uploaded to the server; S7, the server accepts the models uploaded by all clients in this round, saves these models, and then matches the positions of corresponding neurons for aggregation, and finally obtains the global model of this round; S8. Steps S2 to S5 constitute a round of training, which is repeated each time until the model capability meets the requirements.
2. The method for federated learning based on gradient sub-model extraction in resource-constrained scenarios according to claim 1, characterized in that: The method of selecting a public sub-dataset suitable for different clients from the public data set according to the historical model saved by different clients in the previous round is as follows: If this is the first round of training and there is no historical model on the server, a certain number of sub-datasets will be randomly selected. If this is not the first round, the model uploaded by the client last time will be used to predict on the public dataset, and the predicted data will be used to form the sub-datasets. If the public dataset itself has labels, then use the historical model to make predictions on the entire public dataset, and use the correctly predicted data to form a sub-dataset; if the public dataset itself only has data, then make predictions on the entire dataset, and use the predicted labels as the labels of the sub-datasets.
3. The method for federated learning based on gradient sub-model extraction in resource-constrained scenarios according to claim 2, characterized in that: The public dataset is constructed using three methods: directly uploading local data from the client, the client constructing similar data based on local data using a generative model, and selecting an alternative dataset containing similar data types or using local data labels as substitutes.
4. The method for federated learning based on gradient sub-model extraction in resource-constrained scenarios according to claim 1, characterized in that: The method selects neurons by layer from the global model according to the gradient information to form a sub-model for client training, and sends the sub-model to the corresponding client, specifically: The number of neurons in each layer of the sub-model is allocated according to the computing power of the corresponding client, from shallow to deep, layer by layer; In each layer, the absolute value of the gradient of the neurons in this layer is calculated, sorted by size, and selected from large to small until the number of selections reaches the maximum number of training that the client can bear.
5. The method for federated learning based on gradient sub-model extraction in resource-constrained scenarios according to claim 1, characterized in that: The server accepts the models uploaded by all clients in this round, saves these models, and then matches the positions of corresponding neurons for aggregation, and finally obtains the global model of this round, which is specifically: Traverse the sub-models uploaded by the selected client, and record the specific position of each neuron in the global model in each sub-model, and obtain the number of occurrences and specific parameter values of the neuron at each position in the global model; According to the number of occurrences of each neuron and the specific parameter value, the average parameter value of each neuron is calculated as the final value, and the entire aggregation process is completed.
6. The method for federated learning based on gradient sub-model extraction in resource-constrained scenarios according to claim 1, characterized in that: The gradient of each neuron is defined as g: In the classification task, using convolutional neural network, h l,i represents the activation value matrix of the i-th neuron in layer l, K and V represent the length and width of the feature map, and the definition of the neuron gradient is the sum of the absolute values of the gradient of the activation value of the entire feature map.
7. The method for federated learning based on gradient sub-model extraction in resource-constrained scenarios according to claim 1, characterized in that: The sub-model uses masked cross entropy loss, scaling module and static batch normalization during training based on local private data.
8. The method for federated learning based on gradient sub-model extraction in resource-constrained scenarios according to claim 7, characterized in that: The scaling module is inserted after the parameter layer and before the activation layer during the training phase. The input value is scaled by a factor of , where q is the offset rate of the global model parameters.
