An efficient personalized federated learning system and method based on lightweight model

Abstract

The present invention belongs to the field of federated learning, and discloses an efficient personalized federated learning system and method based on a lightweight model. In the local training process, a batch normalization layer (BN layer) is integrated into a convolutional neural network model, and some parameters of the layer are kept private so that the client retains the personalized characteristics of its local data. The model is compressed using a model pruning technology based on channel weights. The server comprehensively considers factors such as the amount of data, data diversity, and participation of the client, and adopts a comprehensive weight distribution principle based on reputation value to distribute client weights. Weighted aggregation is performed based on the weights, and a Nadam optimization algorithm combined with an adaptive learning rate adjustment mechanism and Nesterov momentum is used to update the global model. The present invention can effectively improve the speed and accuracy of model convergence, reduce communication costs, and enhance the fairness, robustness, and personalization capabilities of the model.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning, and specifically relates to proposing an efficient personalized federated learning system and method based on a lightweight model. Background Art

[0002] Nowadays, with the global emphasis on data privacy and security, the phenomenon of "data silos" is widely present in the fields of finance, medical care, government agencies, etc. However, machine learning technology relies on model training through massive data. In order to cope with the growing demand for data privacy protection, break down data silos and industry barriers, and improve the performance of machine learning models, Google has introduced a privacy-preserving machine learning framework - Federated Learning (FL). The core idea of ​​this framework is that terminal devices can use their own data to perform training tasks locally, and aggregate models through servers to obtain a global model with better performance.

[0003] However, in practical applications, each user may collect data from different sources and under different conditions, which will cause the client data to obey non-independent identical distribution (Non-IID), affecting the performance of the federated learning model. Based on this, traditional federated learning algorithms generally face two core challenges: First, data heterogeneity. The difference in data distribution will cause the local model trained by each node to gradually converge to the local optimum, deviating from the direction of the global optimum, thereby reducing the accuracy of the global model when the model is aggregated. Second, there is a lack of personalized solutions. Affected by data heterogeneity, a single global model is difficult to adapt to the unique data characteristics of all clients, affecting the model's adaptability and generalization ability to local data. Considering the differences in models, data distribution, and systems among clients, Personalized Federated Learning (PFL) uses partial private data for model training through collaboration between clients on the basis of privacy protection, allowing clients to achieve personalized model training by exchanging model parameters or gradients without sharing original data.

[0004] CN117592538A discloses an efficient federated learning system and method based on channel pruning. The invention removes redundant channels in the client model through the channel pruning method, uploads the compressed model to the server for weight distribution and weighted averaging, and reduces the communication cost. However, the invention does not consider the generalization ability of the model. In practical applications, especially when resources are unevenly distributed, the model performance on some clients may be much lower than that on other clients, limiting the scalability of the federated learning system.

[0005] CN118095412A discloses a personalized federated learning method based on discriminator knowledge distillation, which enables the global model to better generalize and integrate the characteristics of each model. The invention introduces a discriminator model on each client to evaluate the difference between the global model and the local model, and extracts the global model knowledge to the local model through knowledge distillation technology. However, the invention ignores that the introduction of the discriminator model and knowledge distillation may bring higher storage and communication requirements, and the integration of these technologies may also increase the complexity of the model. Summary of the invention

[0006] The purpose of the present invention is to provide an efficient personalized federated learning system and method based on a lightweight model to solve the above technical problems. The present invention personalizes the client model by introducing a private layer in the local model and keeping some parameters of the private layer private, and uses the Nadam optimization algorithm combined with adaptive learning rate and Nesterov momentum to perform global model updates to improve the accuracy and speed of model training, and uses a pruning method based on channel weights to further reduce storage space and communication overhead.

[0007] To achieve the above object, the technical solution adopted by the present invention is: an efficient personalized federated learning system based on a lightweight model, comprising a server and multiple clients;

[0008] The server performs weight allocation according to the reputation value of the client, performs weighted aggregation based on the weight after receiving the model update uploaded by the client, so as to update the global model. The updated global model is distributed to each client in the next round of training. The training ends when the global model converges or reaches the maximum number of iterations;

[0009] The client, each client has its own local data set, uses the local data set for training during local training, performs channel pruning during training, removes redundant channels of the local model, and updates the non-private layer parameters of the compressed model to the server, and the channel pruning is performed based on the weight of the channel;

[0010] Models and parameters can be uploaded and downloaded between the server and the client.

[0011] An efficient personalized federated learning method based on a lightweight model, the method is based on a server and multiple clients, and includes the following steps:

[0012] Step 1: The server initializes the global model and sends the initial global model to each client;

[0013] Step 2: The client downloads the global model sent by the server, performs personalized training and updates on the local data set, calculates channel weights based on the importance of different channels, performs pruning based on channel weights, removes redundant channels, and achieves model compression;

[0014] Step 3: The client retains the private layer update for the next training and uploads the pruned non-private layer model parameter update to the server;

[0015] Step 4: The server receives the model parameter update uploaded by the client, calculates and performs weighted aggregation based on the client weights, updates the global model parameters using the Nadam optimization algorithm, updates and saves the global model, and prepares for the next round of training;

[0016] Step 5: Repeat steps 2 to 4 until the global model converges or the maximum number of iterations is reached;

[0017] Step 6: The client downloads the final global model and completes the training of the local personalized model.

[0018] Preferably, in step 2, the client downloads the global model sent by the server, performs personalized training and updating on the local data set, calculates channel weights based on the importance of different channels, performs pruning according to the channel weights, removes redundant channels, and realizes model compression, specifically as follows:

[0019] Step 2.1: The client initializes a stochastic gradient descent (SGD) optimizer to update the local model parameters. The update formula of SGD is:

[0020] θ t+1 =θ t -η▽f i (θ t )

[0021] Among them, θ represents the model parameters, t represents the current iteration number, η represents the learning rate, and f i Represents the loss function, for the i-th data batch, ▽f i (θ t ) represents the parameter θ t Under the gradient, in each iteration, calculate the current parameter θ t ▽f i (θ t ), and then update the parameters in the opposite direction of the gradient to reduce the value of the loss function;

[0022] Step 2.2: During the local training process, the client introduces the batch normalization layer (BN layer) into the convolutional neural network model. The calculation formula of the BN layer is:

[0023]

[0024] Among them, BN(·) indicates the batch normalization operation, x i represents a mini-batch data input feature, μ represents the current input x i The mean value, σ 2 Represents the current input x i The variance of , ε represents a minimum value, which is used to ensure that the denominator is not zero, γ represents the scaling factor, β represents the offset, γ and β are two learnable reconstruction parameters, which are used to perform the scaling operation so that the network can learn to restore the feature distribution that the original network wants to learn;

[0025] Step 2.3: For each convolutional layer in the local model, evaluate the importance of the channel based on the L1 norm of its weights. The calculation formula of the L1 norm is:

[0026]

[0027] Among them, ||·||1 represents the L1 norm, W k Represents the weight matrix of the kth convolution kernel, W k,c,h,w Represents the weight value of the kth convolution kernel, the cth input channel, at position (h, w);

[0028] Step 2.4: Sort based on the L1 norm of the channel weights and prune some channels according to the pruning ratio ρ;

[0029] Step 2.5: Create a new convolutional layer based on the number of pruned channels, with the same parameters as the original convolutional layer, only changing the number of output channels, updating the weights and biases according to the pruned channel index, replacing the old convolutional layer, updating the number of input channels of the next layer according to the number of pruned weights, and reinitializing the weights and biases of the next layer;

[0030] Step 2.6: Pruning is completed and the pruned model is returned.

[0031] Preferably, in step 3, the client retains the private layer update for use in the next training, and uploads the pruned non-private layer model parameter update to the server, specifically:

[0032] During the training process, the trainable parameters γ and β of the BN layer are kept private, that is, each client has an independent scaling factor γ and offset β, and does not participate in the global aggregation process on the server side, so that the client retains the personalized characteristics of its data. At this time, the distribution changes of the input and output of the BN layer satisfy:

[0033]

[0034] in, represents the input of the client BN layer after the aggregation step. It follows a normal distribution, and the mean and variance are determined by the change in the mean and variance of the BN layer input before and after aggregation, i.e., Δμ i and Decide, represents the output of the client-side BN layer, which also follows a normal distribution, and its mean and variance take into account the client-private γ and β;

[0035] After training is completed, the client calculates the difference (pseudo gradient) between the local model parameters and the initial global model parameters, obtains the model parameter update, and uploads it to the server.

[0036] Preferably, in step 4, the server receives the model parameter update uploaded by the client, calculates and performs weighted aggregation based on the client weight, updates the global model parameters using the Nadam optimization algorithm, updates and saves the global model, and prepares for the next round of training, specifically:

[0037] Step 4.1: The server receives the model parameter update uploaded by the client and assigns weights based on the reputation value, which comprehensively evaluates multiple factors such as the client's data volume, data diversity, and participation. The specific calculation method is:

[0038] Weight allocation based on data volume can ensure that model training can fully utilize the data resources of each client and improve the efficiency and effectiveness of the entire federated learning system. Clients with large data volumes are generally more representative. The calculation formula for weight allocation based on data volume is:

[0039]

[0040] in, represents the weight assigned to client i based on the amount of data, n i represents the data volume of client i, N represents the total number of clients, represents the sum of all client data volumes. According to the formula, the larger the data volume of client i is, Also bigger;

[0041] Data diversity reflects the difference between client data and global data distribution. If the data of a client is significantly different from the global data distribution, then the data of the client may have a higher contribution to the generalization ability of the model. The calculation formula for weight allocation based on data diversity is:

[0042]

[0043] in, represents the weight assigned to client i based on data diversity, P irepresents the data distribution of client i, Q represents the global data distribution, KL(P i ||Q) means P i The KL divergence between and Q, represents the sum of the client KL divergences. According to the formula, when the data distribution of client i is more different from the global data distribution, KL(P i ||Q) is larger, the weight Also bigger;

[0044] The client's participation can reflect the client's computing power and network stability to a certain extent. Weight allocation based on client participation can be used as an incentive mechanism to encourage more extensive data sharing and more active model training participation. The calculation formula for weight allocation based on participation is:

[0045]

[0046] in, represents the weight assigned to client i based on participation, f i represents the number of participations of client i, Represents the total number of client participation. According to the formula, the higher the participation of client i, the higher the weight Also bigger;

[0047] The comprehensive weight of the client is calculated by the weighted average of the above three weights, and the calculation formula is:

[0048]

[0049] Among them, α, δ, λ are weight coefficients of the above three weights, and they satisfy α+δ+λ=1. The selection of weight coefficients can be adjusted according to actual needs and specific application scenarios. When the data size has a greater impact, set a larger α; when the data diversity has a greater impact on the model effect, set a larger δ; when it is necessary to encourage clients to participate in training frequently, set a larger λ;

[0050] Step 4.2: The server performs weighted aggregation on the model parameter updates uploaded by the client based on the client's weight to obtain the global pseudo gradient. The weighted aggregation calculation method of the pseudo gradient is:

[0051]

[0052] Among them, g i represents the pseudo gradient of client i, ω i represents the weight corresponding to client i;

[0053] Step 4.3: The server uses the Nadam optimization algorithm to update the global model parameters based on the aggregated pseudo gradients and save the updated global model parameters. The specific update rules of Nadam are:

[0054] Step 4.3.1: Calculate the global model gradient at time step t as follows:

[0055]

[0056] Among them, θ t represents the model parameters at time step t, and J represents the objective function;

[0057] Step 4.3.2: Update the first-order moment estimate, calculated as follows:

[0058] m t =β1·m t-1 +(1-β1)·g t

[0059] Among them, m t represents the first-order moment estimate at time step t, β1 represents the exponential decay rate of the first-order moment estimate;

[0060] Step 4.3.3: Update the second-order moment estimate, calculated as follows:

[0061]

[0062] Among them, v t represents the second-order moment estimate at time step t, β2 represents the exponential decay rate of the second-order moment estimate;

[0063] Step 4.3.4: Calculate the bias-corrected first-order moment estimates and second-order moment estimates to remove the bias at the initial time step as follows:

[0064]

[0065] Step 4.3.5: Use Nesterov momentum to adjust the gradient. By considering the update direction of the parameters in advance, the algorithm is smoother during the gradient descent process and further accelerates convergence. The calculation method is as follows:

[0066]

[0067] Where τ represents the coefficient of Nesterov momentum, which is usually the same as β1;

[0068] Step 4.3.6: Update the global model parameters, calculated as follows:

[0069]

[0070] Among them, θ t+1 represents the updated global model parameters, η represents the global learning rate, and ε represents a constant for numerical stability.

[0071] Compared with the prior art, the present invention has the following improvements and advantages:

[0072] 1. The BN layer is introduced into the local training model. By normalizing the input of the layer, the gradient distribution in the training process is made more stable, which accelerates the convergence of the model. At the same time, by retaining some parameters of the BN layer and not uploading them in each round of iteration, the client retains the personalized characteristics of its local data, thus realizing personalized training of the local model.

[0073] 2. For the convolutional layer channels of the local model, a pruning scheme is formulated based on the L1 norm of its weight to remove redundant channels, compress the model size uploaded by the client to the server during each iteration, reduce communication overhead, and shorten the convergence time of the global model.

[0074] 3. A client weight allocation method based on reputation is proposed. By comprehensively considering the client's data volume, data diversity and participation, it helps to optimize resource utilization in the federated learning process, reduce model bias and improve data quality, while enhancing the robustness and fairness of the model.

[0075] 4. The Nadam optimization algorithm is used to update the global model. It inherits the adaptive learning rate adjustment mechanism of the Adam algorithm and adds the Nesterov momentum to achieve faster convergence speed and improve the stability and efficiency of the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a structural schematic diagram of an efficient personalized federated learning system based on a lightweight model of the present invention.

[0077] Figure 2 It is a flow chart of an efficient personalized federated learning method based on a lightweight model of the present invention.

[0078] Figure 3 It is a local model schematic diagram of an efficient personalized federated learning system and method based on a lightweight model of the present invention.

[0079] Figure 4 This is a comparison chart of the effects of the efficient personalized federated learning method based on a lightweight model and other federated learning methods. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0081] It should be noted here that in order to avoid unnecessary details that would obscure important parts of the present invention, the drawings only show structures or processing steps that are closely related to the present invention, while ignoring other details that are not very relevant to the present invention.

[0082] Combination Figure 1 ,The present invention provides an efficient personalized federated learning system based on a lightweight model, comprising a server and multiple clients;

[0083] The server performs weight allocation according to the reputation value of the client, performs weighted aggregation based on the weight after receiving the model update uploaded by the client, so as to update the global model. The updated global model is distributed to each client in the next round of training. The training ends when the global model converges or reaches the maximum number of iterations;

[0084] The client, each client has its own local data set, uses the local data set for training during local training, performs channel pruning during training, removes redundant channels of the local model, and updates the non-private layer parameters of the compressed model to the server, and the channel pruning is performed based on the weight of the channel;

[0085] Models and parameters can be uploaded and downloaded between the server and the client.

[0086] Combination Figure 2 The present invention provides an efficient personalized federated learning method based on a lightweight model, which specifically includes the following steps:

[0087] Step 1: The server initializes the global model and sends the initial global model to each client;

[0088] Step 2: The client downloads the global model sent by the server, performs personalized training and updates on the local data set, calculates channel weights based on the importance of different channels, performs pruning based on channel weights, removes redundant channels, and achieves model compression;

[0089] Specifically:

[0090] Step 2.1: The client initializes a stochastic gradient descent (SGD) optimizer to update the local model parameters. The update formula of SGD is:

[0091] θ t+1 =θ t-η▽f i (θ t )

[0092] Among them, θ represents the model parameters, t represents the current iteration number, η represents the learning rate, and f i Represents the loss function, for the i-th data batch, ▽f i (θ t ) represents the parameter θ t Under the gradient, in each iteration, calculate the current parameter θ t ▽f i (θ t ), and then update the parameters in the opposite direction of the gradient to reduce the value of the loss function;

[0093] Step 2.2: During the local training process, the client introduces the batch normalization layer (BN layer) into the convolutional neural network model (for details, see Figure 3 , Figure 3 , which is a local model schematic diagram of an efficient personalized federated learning system and method based on a lightweight model of the present invention), the calculation formula of the convolutional layer is:

[0094] Conv(x)=σ(W*x+b)

[0095] Where Conv(x) represents the output of the convolution operation applied to the input data x, σ represents the activation function, * represents the convolution operation, W represents the weight matrix of the convolution kernel, and b represents the bias term, which is used to provide additional flexibility to adjust the baseline of the convolution output;

[0096] The calculation formula of the BN layer is:

[0097]

[0098] Among them, BN(·) indicates the batch normalization operation, x i represents a mini-batch data input feature, μ represents the current input x i The mean value, σ 2 Represents the current input x i The variance of , ε represents a minimum value, which is used to ensure that the denominator is not zero, γ represents the scaling factor, β represents the offset, γ and β are two learnable reconstruction parameters, which are used to perform the scaling operation so that the network can learn to restore the feature distribution that the original network wants to learn;

[0099] The goal of traditional federated learning is to minimize the following objective function:

[0100]

[0101] Among them, F FL represents the global loss function, N represents the total number of clients, ωi represents the weight of the i-th client, represents the loss function on the i-th client, and Ω represents the set of global model parameters;

[0102] The specific method to achieve personalized local training is to use the BN layer as a private layer and keep some parameters of this layer private. After adding the BN layer to the original convolutional neural network model, the objective function of the federated learning objective minimization becomes:

[0103]

[0104] Among them, M i represents the patch model on client i, which is composed of the federated model parameters Ω1,...,Ω of client i j and the patch layer parameters unique to client i Composition, j represents the total number of federated layers, and m represents the total number of local patches;

[0105] Step 2.3: For each convolutional layer in the local model, evaluate the importance of the channel based on the L1 norm of its weights. The calculation formula of the L1 norm is:

[0106]

[0107] Among them, ||·||1 represents the L1 norm, W k Represents the weight matrix of the kth convolution kernel, W k,c,h,w Represents the weight value of the kth convolution kernel, the cth input channel, at position (h, w);

[0108] Step 2.4: Sort based on the L1 norm of the channel weights. The larger the L1 norm, the higher the contribution of the channel. Prune some channels according to the pruning ratio ρ.

[0109] Step 2.5: Create a new convolutional layer based on the number of pruned channels, with the same parameters as the original convolutional layer, only changing the number of output channels, updating the weights and biases according to the pruned channel index, replacing the old convolutional layer, updating the number of input channels of the next layer according to the number of pruned weights, and reinitializing the weights and biases of the next layer;

[0110] Step 2.6: Pruning is completed and the pruned model is returned.

[0111] Step 3: The client retains the private layer update for the next training and uploads the pruned non-private layer model parameter update to the server;

[0112] Specifically:

[0113] During the training process, the trainable parameters γ and β of the BN layer are kept private, that is, each client has an independent scaling factor γ and offset β, and does not participate in the global aggregation process on the server side, so that the client retains the personalized characteristics of its data. At this time, the distribution changes of the input and output of the BN layer satisfy:

[0114]

[0115] in, represents the input of the client BN layer after the aggregation step. It follows a normal distribution, and the mean and variance are determined by the change in the mean and variance of the BN layer input before and after aggregation, i.e., Δμ i and Decide, represents the output of the client-side BN layer, which also follows a normal distribution, and its mean and variance take into account the client-private γ and β;

[0116] After training is completed, the client calculates the difference (pseudo gradient) between the local model parameters and the initial global model parameters, obtains the model parameter update, and uploads it to the server.

[0117] Step 4: The server receives the model parameter update uploaded by the client, calculates and performs weighted aggregation based on the client weights, updates the global model parameters using the Nadam optimization algorithm, updates and saves the global model, and prepares for the next round of training;

[0118] Specifically:

[0119] Step 4.1: The server receives the model parameter update uploaded by the client and assigns weights based on the reputation value, which comprehensively evaluates multiple factors such as the client's data volume, data diversity, and participation. The specific calculation method is:

[0120] Weight allocation based on data volume can ensure that model training can fully utilize the data resources of each client and improve the efficiency and effectiveness of the entire federated learning system. Clients with large data volumes are generally more representative. The calculation formula for weight allocation based on data volume is:

[0121]

[0122] in, represents the weight assigned to client i based on the amount of data, n i represents the amount of data from client i, represents the sum of all client data volumes. According to the formula, the larger the data volume of client i is, Also bigger;

[0123] Data diversity reflects the difference between client data and global data distribution. If the data of a client is significantly different from the global data distribution, then the data of the client may have a higher contribution to the generalization ability of the model. The calculation formula for weight allocation based on data diversity is:

[0124]

[0125] in, represents the weight assigned to client i based on data diversity, P i represents the data distribution of client i, Q represents the global data distribution, KL(P i ||Q) means P i The KL divergence between and Q, represents the sum of the client KL divergences. According to the formula, when the data distribution of client i is more different from the global data distribution, KL(P i ||Q) is larger, the weight Also bigger;

[0126] The client's participation can reflect the client's computing power and network stability to a certain extent. Weight allocation based on client participation can be used as an incentive mechanism to encourage more extensive data sharing and more active model training participation. The calculation formula for weight allocation based on participation is:

[0127]

[0128] in, represents the weight assigned to client i based on participation, f i represents the number of participations of client i, Represents the total number of client participation. According to the formula, the higher the participation of client i, the higher the weight Also bigger;

[0129] The comprehensive weight of the client is calculated by the weighted average of the above three weights, and the calculation formula is:

[0130]

[0131] Among them, α, δ, λ are weight coefficients of the above three weights, and satisfy α+δ+λ=1. The selection of weight coefficients can be adjusted according to actual needs and specific application scenarios. When the data size has a greater impact, set a larger α; when the data diversity has a greater impact on the model effect, set a larger δ; when it is necessary to encourage clients to participate in training frequently, set a larger λ. By default, α=1, δ=0, λ=0, that is, client weights are allocated only according to the size of the client's data;

[0132] Step 4.2: The server performs weighted aggregation on the model parameter updates uploaded by the client based on the client's weight to obtain the global pseudo gradient. The weighted aggregation calculation method of the pseudo gradient is:

[0133]

[0134] Among them, g i represents the pseudo gradient of client i, ω i represents the weight corresponding to client i;

[0135] Step 4.3: The server uses the Nadam optimization algorithm to update the global model parameters based on the aggregated pseudo gradients and save the updated global model parameters. The specific update rules of Nadam are:

[0136] Step 4.3.1: Calculate the global model gradient at time step t as follows:

[0137]

[0138] Among them, θ t represents the model parameters at time step t, and J represents the objective function;

[0139] Step 4.3.2: Update the first-order moment estimate, calculated as follows:

[0140] m t =β1·m t-1 +(1-β1)·g t

[0141] Among them, m t represents the first-order moment estimate at time step t, β1 represents the exponential decay rate of the first-order moment estimate, which is 0.9;

[0142] Step 4.3.3: Update the second-order moment estimate, calculated as follows:

[0143]

[0144] Among them, v t represents the second-order moment estimate at time step t, β2 represents the exponential decay rate of the second-order moment estimate, which is 0.999;

[0145] Step 4.3.4: Calculate the bias-corrected first-order moment estimates and second-order moment estimates to remove the bias at the initial time step as follows:

[0146]

[0147] Step 4.3.5: Use Nesterov momentum to adjust the gradient. By considering the update direction of the parameters in advance, the algorithm is smoother during the gradient descent process and further accelerates convergence. The calculation method is as follows:

[0148]

[0149] Where τ represents the coefficient of Nesterov momentum, which is the same as β1;

[0150] Step 4.3.6: Update the global model parameters, calculated as follows:

[0151]

[0152] Among them, θ t+1 represents the updated global model parameters, η represents the global learning rate, and ε represents a constant for numerical stability.

[0153] Step 5: Repeat steps 2 to 4 until the global model converges or the maximum number of iterations is reached;

[0154] Step 6: The client downloads the final global model and completes the training of the local personalized model.

[0155] At this point, an efficient and personalized federated learning method based on a lightweight model can be realized.

[0156] In order to verify the effectiveness of the proposed method, the present invention is compared with other federated learning algorithms in an experiment:

[0157] 1) FedAvg: A classic federated learning algorithm for training deep neural networks by aggregating model updates from clients.

[0158] 2) PFedMe: A personalized federated learning algorithm based on a two-layer optimization framework and Moreau envelopes regularization.

[0159] 3) FedNadam: The present invention discloses an efficient personalized federated learning method based on a lightweight model.

[0160] Figure 4The experimental results comparing the above federated learning algorithms on the MNIST public dataset are shown. In the experiment, we set various parameters uniformly and adjusted the hyperparameters for each algorithm to achieve the highest user model accuracy after 500 iterations. User model accuracy is used to measure the model performance of a single client. It is calculated by evaluating the accuracy of the model on each client's local test set, which has similar distribution characteristics to the client's training data. As can be seen from the figure, the method of the present invention can stably reach a convergence state, converge faster, and achieve higher user model accuracy.

[0161] Furthermore, the present invention verifies the effect of model pruning. When other experimental parameters are set the same, the model pruning technology is applied to FedAvg, PFedMe and FedNadam, and the time for 500 iterations is reduced by about 35%, about 31% and about 27% respectively, and the change in user model accuracy is less than 0.02, which meets the requirement of shortening the training time with a small loss of accuracy.

[0162] The present invention realizes the personalization of the client model by combining the BN layer with the convolutional neural network model and keeping some parameters of the BN layer private. It performs global model update by combining the adaptive learning rate adjustment mechanism and Nesterov momentum to improve the accuracy and speed of model training, and further reduces the storage space and communication overhead based on channel weight pruning.

[0163] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. An efficient personalized federated learning method based on a lightweight model, characterized by: The method is based on a server and multiple clients and includes the following steps: Step 1: The server initializes the global model and sends the initial global model to each client; Step 2: The client downloads the global model sent by the server, performs personalized training and updates on the local data set, calculates channel weights based on the importance of different channels, performs pruning based on channel weights, removes redundant channels, and achieves model compression; Step 3: The client retains the private layer update for the next training and uploads the pruned non-private layer model parameter update to the server; In step 3, the client retains the private layer update for the next training, and uploads the pruned non-private layer model parameter update to the server, specifically: during the training process, the BN layer trainable parameters γ and β are kept private, that is, each client has an independent scaling factor γ and offset β, and does not participate in the global aggregation process on the server side, so that the client retains the personalized characteristics of its data; After training is completed, the client calculates the difference between the local model parameters and the initial global model parameters, i.e., pseudo gradients, obtains the model parameter updates, and uploads them to the server; Step 4: The server receives the model parameter update uploaded by the client, calculates and performs weighted aggregation based on the client weight, updates the global model parameters using the Nadam optimization algorithm, updates and saves the global model, and prepares for the next round of training; the client weight is calculated in the step 4, specifically: weight allocation is performed based on the reputation value, and the reputation value comprehensively evaluates the client's data volume, data diversity and participation, and the specific calculation method is: the data volume weight of client i is obtained based on the ratio of the data volume of client i to the sum of the data volume of all clients; the KL divergence between the data distribution of client i and the global data distribution represents the KL divergence of client i, and the data diversity weight of client i is obtained based on the ratio of the KL divergence of client i to the sum of the KL divergence of all clients; the participation weight of client i is obtained based on the ratio of the number of participations of client i to the total number of participations of all clients; the comprehensive weight ω of client i i It is calculated by taking the weighted average of the above three weights; Step 5: Repeat steps 2 to 4 until the global model converges or the maximum number of iterations is reached; Step 6: The client downloads the final global model and completes the training of the local personalized model.

2. According to claim 1, an efficient personalized federated learning method based on a lightweight model is characterized in that: In step 4, the server receives the model parameter update uploaded by the client, calculates and performs weighted aggregation based on the client weight, updates the global model parameters using the Nadam optimization algorithm, updates and saves the global model, and prepares for the next round of training, specifically: Step 4.1: The server performs weighted aggregation on the model parameter updates uploaded by the client based on the client's weight to obtain the global pseudo gradient. The weighted aggregation calculation method of the global pseudo gradient is: Among them, g i represents the pseudo gradient of client i, ω i represents the weight corresponding to client i; Step 4.2: The server uses the Nadam optimization algorithm to update the global model parameters based on the aggregated pseudo gradients and save the updated global model parameters. The specific update rules of Nadam are: Step 4.2.1: Calculate the global model gradient at time step t as follows: Among them, θ t represents the model parameters at time step t, and J represents the objective function; Step 4.2.2: Update the first-order moment estimate, calculated as follows: m t =β1·m t-1 +(1-β1)·g t Among them, m t represents the first-order moment estimate at time step t, β1 represents the exponential decay rate of the first-order moment estimate; Step 4.2.3: Update the second-order moment estimate, calculated as follows: Among them, v t represents the second-order moment estimate at time step t, β2 represents the exponential decay rate of the second-order moment estimate; Step 4.2.4: Calculate the bias-corrected first-order moment estimates and second-order moment estimates to remove the bias at the initial time step as follows: Step 4.2.5: Use Nesterov momentum to adjust the gradient. By considering the update direction of the parameters in advance, the algorithm is smoother during the gradient descent process and further accelerates convergence. The calculation method is as follows: Where τ represents the coefficient of Nesterov momentum; Step 4.2.6: Update the global model parameters, calculated as follows: Among them, θ t+1 represents the updated global model parameters, η represents the global learning rate, and ε represents a constant for numerical stability.

3. An efficient personalized federated learning system based on a lightweight model, the system is used to implement the efficient personalized federated learning method based on a lightweight model as described in claim 1, characterized in that: Includes a server and multiple clients; The server performs weight allocation according to the reputation value of the client, performs weighted aggregation based on the weight after receiving the model update uploaded by the client, so as to update the global model. The updated global model is distributed to each client in the next round of training. The training ends when the global model converges or reaches the maximum number of iterations; The client, each client has its own local data set, uses the local data set for training during local training, performs channel pruning during training, removes redundant channels of the local model, and updates the non-private layer parameters of the compressed model to the server, and the channel pruning is performed based on the weight of the channel; Models and parameters can be uploaded and downloaded between the server and the client.

Images