Federated Learning Client Scheduling Method Based on Weight Divergence and Update Increment

Abstract

The present invention relates to the technical field of client scheduling, and discloses a federated learning client scheduling method based on weight divergence and update increment. The client expands and balances the local dataset through image enhancement technology, so that the data distributions of the clients reach global balance; the clients then use their respective local datasets as inputs to train the global model respectively. After a certain number of rounds of iterative training, the updated parameters of the local model are obtained; after each client is trained, the weight divergence, update increment, and training loss of the local model parameters are calculated, and the respective calculation data are sent to the server; according to the calculation data sent by the clients, the server selects the clients with small model weight divergence, small update increment, and large model training loss; the server sends a request to these clients to upload model parameters; the selected clients send the corresponding model parameters to the server, and the server aggregates according to the uploaded model parameters to obtain new global model parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of client scheduling, and specifically to a federated learning client scheduling method based on weight divergence and update increment. Background Art

[0002] With the rise of deep learning in the industry, machine learning, as a popular technology, has been widely applied in fields such as the Internet of Things, healthcare, and edge computing. The models in these fields are all trained based on various data. In the medical field, medical image classification based on a model trained by deep learning is an efficient and low-cost method. The deep learning algorithm for medical image analysis requires a large amount of data for training. To make full use of distributed data, traditional machine learning methods need to centrally collect medical institution data on the server to perform centralized learning and train models. This will lead to privacy issues because medical institutions tend to protect the personal data privacy of patients and prohibit sharing any data information of patients. With the strengthening of the protection of personal privacy in various countries, such as the introduction of the General Data Protection Regulation in the European Union and the California Consumer Privacy Act (CCPA) in the United States, it shows that the protection of client data privacy will become increasingly strict. It is unrealistic to rely solely on traditional machine learning solutions to train models. To avoid privacy leakage problems of data owners and promote cooperation in deep learning among distributed institutions, federated learning has been introduced, a method that enables multiple parties to jointly learn a machine learning model without exchanging their respective local data while maintaining the decentralization of training data.

[0003] In federated learning, the server randomly selects a part of the clients as participants and broadcasts a global model to these clients. The clients update the model according to their local data. After the update is completed, the model is uploaded to the server for aggregation to update the global model. These steps are repeated until the model accuracy meets the requirements. During the process of model learning, the client data does not leave the local area, protecting the privacy of the clients.

[0004] Federated learning is a high-performance machine learning method for solving privacy problems, and its inherent privacy protection characteristics make it have important applications in the healthcare field. There are two main challenges in federated learning: (1) Statistical heterogeneity. (2) Communication overhead. In the real world, data may be unevenly distributed among parties. Due to differences in computing power and storage among each client, there are heterogeneous problems in the data distribution among different clients. When the data distribution of the selected clients is unbalanced, this may reduce performance. When each client updates its local model, its local objective may deviate from the global objective. Therefore, in order to achieve the desired model accuracy, clients with approximately independent and identically distributed (IID) data should be selected to participate in federated learning training. Federated learning clients need to communicate with the server frequently to exchange and update the model. Its deployment is limited by the communication bandwidth, the communication capacity is limited, and the number of clients is increasing continuously, which will become a communication bottleneck. The problem of statistical heterogeneity in federated learning is a popular research direction. Some studies have tried to solve the heterogeneity problem in federated learning by uploading additional update parameters in the clients or broadcasting a part of IID data to the clients for model training. However, these efforts increase additional communication costs or pose privacy threats to the clients, making it difficult to solve the heterogeneity problem.

[0005] The existing technologies are basically divided into the following three categories: solutions for solving statistical heterogeneity in federated learning, medical image classification solutions based on federated learning technology, and federated learning technology based on client selection.

[0006] The following disadvantages exist in the above solutions:

[0007] The client selection work focuses on selecting clients in terms of computing power and heterogeneity, ignoring the imbalance between local client data.

[0008] The commonly used federated learning method usually randomly selects clients in the current round, and this random selection strategy will cause the decline of model training accuracy under non-IID.

[0009] The communication overhead is large. The communication overhead in federated learning mainly exists in the frequent communication between the client and the server, and the number of training rounds required for the model to reach the specified accuracy is too high. In summary, we propose a federated learning client scheduling method based on weight divergence and update increment. Summary of the Invention

[0010] (1) Technical problems to be solved

[0011] Aiming at the deficiencies of the existing technology, the present invention provides a federated learning client scheduling method based on weight divergence and update increment, which solves the above problems.

[0012] (2) Technical solutions

[0013] To achieve the above object, the present invention provides the following technical solutions: A federated learning client scheduling method based on weight divergence and update increment, comprising the following steps:

[0014] The first step: After model evaluation, deploy the model to the server and identify it as the global model. The server broadcasts the global model to all clients participating in federated learning. At the same time, before client training, collect the local data distribution of the clients and send it to the clients together with the global model;

[0015] The second step: The client expands and balances the local dataset through image enhancement technology to make the data distributions of the clients reach global balance;

[0016] The third step: The client then uses its respective local dataset as input to train the global model. After a certain number of rounds of iterative training, obtain the local model update parameters;

[0017] The fourth step: After each client is trained, calculate the weight divergence, update increment, and training loss of the local model parameters, and send the respective calculation data to the server;

[0018] The fifth step: The server selects the clients with small model weight divergence, small update increment, and large model training loss according to the calculation data sent by the clients as the final selected client set for uploading model parameters;

[0019] The sixth step: The server sends a request to these clients to upload model parameters;

[0020] The seventh step: The selected clients send the corresponding model parameters to the server. The server aggregates according to the uploaded model parameters to obtain new global model parameters, which are used to update the global model and iterate the next round of training until the global model converges or meets the requirements.

[0021] Preferably, the client expands and balances the local dataset through image enhancement technology in the second step, including the following contents:

[0022] S1: The server initializes the global model w 0 according to its own service requirements and broadcasts it to the clients;

[0023] S2: During the federated learning training, the server calculates the maximum number of images in each label for the clients and sends this data to the clients;

[0024] S3: Use 28 different image enhancement methods from the Albumentations image enhancement library to enhance the original images;

[0025] S4: The client generates images by combining different image enhancement methods, transforms and expands the original labeled images of its respective dataset according to the maximum number of label samples until the maximum number of samples required for each label is reached.

[0026] Preferably, the weight divergence and update increment calculation methods in the fourth step are as follows:

[0027] Weight divergence: Obtain the local model parameter w k (t). After that, each client calculates its own model weight divergence d k (t) based on its respective local model parameter w k (t);

[0028] Calculate the divergence between the local model parameter w k (t) and the global model parameter w g (t - 1), and select the model parameters with appropriate model weight divergence for uploading;

[0029] Calculate the weight divergence d k (t) of the local model w g (t) of client k in the t-th round and the global model w k (t - 1). The formula is: where w k (t) represents the local model parameter of client k in the t-th round, w g (t - 1) represents the global model parameter of the (t - 1)-th round, and ||·|| represents the L2 norm calculation;

[0030] Update increment: Calculate the model update increment I k (t) of the local model w k (t) of client k in the t-th round. The formula is as follows: where w k (t) represents the local model parameter of client k in the t-th round, w k (t - 1) represents the local model parameter of client k in the (t - 1)-th round.

[0031] Preferably, in the fourth step, in addition to considering the influence of weight divergence and update increment, the influence of loss also needs to be considered. Compared with the method of randomly selecting clients, selecting the model parameters with larger loss for uploading can improve the convergence speed. Therefore, client scheduling is required.

[0032] Preferably, the client scheduling includes the following:

[0033] S1: After the client calculates its own model weight divergence d k (t), update increment I k (t), and model training loss lossk After (t), upload the data to the server;

[0034] S2: Based on the calculation data uploaded by the client, the server selects the clients with larger model training losses, smaller model weight divergence, and smaller update increments as the candidate client set to improve the model accuracy, and then requests these candidate clients to send parameters to the server;

[0035] S3: The formula for selecting clients to participate in aggregation is: U k (t) = α·d k (t) + β·loss k (t) + δ·I k (t). Where U k (t) represents the priority of client k being selected by the server to upload model parameter aggregation in the t-th round. α, β, and δ are adjustable parameters, representing the weights of model weight divergence, update increment, and model training loss in client selection respectively.

[0036] (III) Beneficial effects

[0037] Compared with the prior art, the present invention provides a federated learning client scheduling method based on weight divergence and update increment, which has the following beneficial effects:

[0038] 1. This federated learning client scheduling method based on weight divergence and update increment uses the proposed client scheduling strategy to replace the general random selection method. With model weight divergence, update increment, and training loss as references, it selects appropriate client model parameters to upload to the server for aggregation, improving the accuracy of the training model to alleviate the impact of accuracy decline caused by the non-IID problem.

[0039] 2. This federated learning client scheduling method based on weight divergence and update increment uses a variety of image enhancement methods to balance the local data of clients, reducing the impact of model accuracy decline caused by data imbalance, and making the data of each client in a globally balanced state before the start of training.

[0040] 3. This federated learning client scheduling method based on weight divergence and update increment constructs a federated learning client scheduling scheme based on weight divergence, update increment, and training loss. This scheme adopts a client scheduling strategy. With model weight divergence, update increment, and training loss as references, it selects appropriate client model parameters to upload to the server for aggregation, improving the accuracy of the model. It expands and balances the local data of clients through various image enhancement technologies to reduce the impact of accuracy decline caused by Non-IID. Description of the drawings

[0041] Figure 1This is a schematic diagram of the system model of the present invention. Specific embodiments

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0043] A federated learning client scheduling method based on weight divergence and update increment includes the following steps:

[0044] S1: Download the global model: The server determines the model to be used in federated learning. After model evaluation, the model is deployed to the server and identified as the global model. The server broadcasts the global model to all clients participating in federated learning for training using their local data. Meanwhile, before client training, the local data distribution of the clients is collected and sent to the clients together with the global model.

[0045] S2: Model training: After the client receives the global model and distribution data sent by the server, the client expands and balances the local dataset through image enhancement technology to make the data distributions of the clients reach global balance.

[0046] The client then uses its respective local dataset as input to train the global model. After a certain number of rounds of iterative training, the local model update parameters are obtained.

[0047] Before training on the local client, the client generates images through different combinations of image enhancement methods, and transforms and expands the original labeled images of its respective dataset according to the maximum number of samples with the same label. By the method of balancing the dataset through image enhancement, the label distributions of the datasets of each client can be balanced, reducing the impact of uneven data distribution on accuracy. Preprocessing for balancing the client dataset based on image enhancement:

[0048] In a certain round of iteration of model training, the server initializes the global model w according to its own service requirements 0, and broadcast it to the clients. To mitigate the impact of Non-IID, before training the local clients, the original images are enhanced using 28 different image enhancement methods from the Albumentations image enhancement library, such as median blur, sharpening, and perspective change. Compared with other image enhancement libraries, Albumentations has more image enhancement functions and faster processing speed to improve the model performance. During the federated learning training, the server calculates the maximum number of images in each label for the clients and sends this data to the clients. The clients generate images by combining different image enhancement methods, transform and expand the original labeled images of their respective datasets according to the maximum number of label samples until the maximum number of samples required for each label is reached.

[0049] S3: Calculate data related to model parameters: After each client is trained, calculate the weight divergence, update increment, and training loss of the local model parameters.

[0050] Model weight divergence and model update increment:

[0051] After each local client goes through the fixed-round random gradient descent process, the local model parameter w k (t) is obtained. After that, each client calculates its own model weight divergence d k (t) based on its own local model parameter w k (t). The accuracy of the model is affected by the distribution of the local dataset. When the data is more imbalanced, the problem of weight divergence becomes more serious. Due to the statistical heterogeneity of the data, there are significant differences in the model parameters of different clients. By calculating the divergence between the local model parameter w k (t) and the global model parameter w g (t - 1), the model parameters with appropriate model weight divergence are selected for uploading. Calculate the weight divergence d k (t) of the local model w g (t) of client k in the t-th round and the global model w k (t - 1) using the formula: where w k (t) represents the local model parameter of client k in the t-th round, w g (t - 1) represents the global model parameter of the (t - 1)-th round, and ||·|| represents the L2 norm calculation.

[0052] Due to the impact of data imbalance, the update increments of model training will also vary. The update increment is the ratio of the difference in the size of the updated parameters between two consecutive iterations of the client to the size of the parameters in the previous round. The update increment of the local model trained on the Non-IID dataset is much larger than that of the model trained on the IID dataset. As the data becomes more imbalanced, the model update increment will increase. By calculating the model increment of each client, the client model parameters with appropriate model update increments are selected for uploading. Calculate the model update increment I k (t) of the local model w k (t) of client k at the t-th round as follows: where w k (t) represents the local model parameters of client k at the t-th round, and w k (t - 1) represents the local model parameters of client k at the (t - 1)-th round.

[0053] They send their respective calculation data to the server.

[0054] The server selects clients with small model weight divergence, small update increment, and large model training loss based on the calculation data sent by the clients as the final set of clients selected to upload model parameters.

[0055] The server sends requests to these clients to upload model parameters.

[0056] The current implementation of federated learning randomly selects only a part of the clients to participate in each round of training. However, due to the existence of a large number of data-imbalanced clients in the federated learning environment, this random selection of clients will exacerbate the adverse effects of data heterogeneity. Analyzing the training results of model parameters under IID and Non-IID data, there will be differences in their weight divergence, update increment, and training loss. By selecting model parameters with a high degree of IID and uploading them to the server for aggregation, the accuracy and convergence speed of the aggregated model can be improved, and the same accuracy can be achieved with a lower number of communication rounds as other federated learning methods.

[0057] Client scheduling scheme based on model weight divergence, update increment, and loss:

[0058] In addition to considering the impact of weight divergence and update increment, the impact of loss also needs to be considered. Compared with the method of randomly selecting clients, selecting model parameters with larger losses for uploading can improve the convergence speed. When the clients calculate their respective model weight divergences d k (t), update increments I k (t), and model training losses loss kAfter (t), the data is uploaded to the server. Based on the calculation data uploaded by the client, the server selects the clients with larger model training losses, smaller model weight divergence, and smaller update increments as the candidate client set to improve the model accuracy, and then requests these candidate clients to send parameters to the server. The calculation formula for selecting clients to participate in aggregation is: U k (t) = α·d k (t) + β·loss k (t) + δ·I k (t). Where U k (t) represents the priority of client k being selected by the server to upload model parameter aggregation in the t-th round. α, β, and δ are adjustable parameters, representing the weights of model weight divergence, update increment, and model training loss in client selection respectively.

[0059] S4: Aggregate model parameters: The selected clients send the corresponding model parameters to the server. The server aggregates based on the uploaded model parameters to obtain new global model parameters for updating the global model and iterating the next round of training until the global model converges or meets the requirements.

[0060] Image classification process:

[0061] The training participants of the image classification model are a central server and multiple clients. The images are stored in the clients and are unevenly distributed. The server initializes the global model according to the classification requirements and sends it to each client. Before model training, the client sends the data distribution information to the server. The server sets the number of image expansions for the client according to the maximum number of images in each label. The client uses a variety of image enhancement methods to balance the expansion of the original images until the maximum value of each label image is reached. The client uses the local images as the input of the model. The classification model converts the input image into a set of pixel values, and the pixel values vary according to the size and resolution of the image, forming a corresponding pixel array. The numbers in the array correspond to the pixel grayscales at that point. The client sends the image pixel values to the convolutional neural network. In the convolutional layer, the convolutional kernel is used to extract the features of each small part of the image, and the activation map of complex features is obtained through the outputs of multiple convolutional layers. The high values in the activation map represent the advanced features of the predicted classification results. The activation map in the fully connected layer determines which image classifications these features are most compatible with. The fully connected layer assigns more weights to these more relevant features and outputs a four-dimensional vector representing the probability of a specific class. By aggregating the features of each component through the fully connected layer, a classifier is generated. The classifier classifies the input image and produces a classification result. The final output describes the probability value that the image belongs to a certain specific image classification. The maximum classification value is used as the output of the image classification result.

[0062] Compared with the prior art, the proposed solution has the advantage that our client scheduling solution can achieve higher classification accuracy for the training model in a classification environment. At the same time, it can use the difference in model parameters under IID and Non-IID data as the criterion for client selection, replacing the commonly used random client selection scheme. Through this client selection method, it can effectively identify the client model parameters that are beneficial to model training, which is more effective than the existing methods of analyzing model update directions and model training cut-off times, with higher training model accuracy and reduced communication overhead.

[0063] In addition, a client dataset balancing preprocessing method based on image enhancement is adopted. Before training the model on the local client, a combination of various different image enhancement methods is used to expand and balance the number of labels in the local dataset. The server calculates the maximum number of samples of each label on the client. The client generates images through different combinations of image enhancement methods and transforms and expands the original labeled images of its own dataset according to the maximum number of samples of the label until the maximum number of samples required for each label is reached, making the distribution of the global clients tend to be balanced, reducing the impact of uneven data distribution on accuracy degradation, and improving model accuracy.

[0064] The federated learning framework mainly consists of two entity parts: the server and the client. As Figure 1 shown, the data is distributed among the clients, and the local data of the clients is Non-IID. To improve the accuracy of the model, after the server broadcasts the global model to the clients and the clients use the local data to train and obtain the model update parameters, the weight divergence, update increment, and training loss of each local client are calculated and used as the basis for client selection. The client scheduling solution FedUC proposed by us can continuously select suitable client model parameters from the clients during the training process. The server records the selected candidate clients and requests them to upload the updated model parameters to the server for aggregation.

[0065] Client: The client is responsible for receiving the global model broadcast by the server, training the global model according to its own local data to obtain local model parameters for uploading to the server for aggregation to obtain a new global model. At the same time, the client is responsible for performing relevant parameter calculations and image enhancement operations on the model parameters.

[0066] Server: The server is responsible for initializing the global model according to the model design requirements and presetting the training parameters required for federated learning training, and sending them to all clients participating in federated learning training for local training. When the local training is completed, it collects the local model parameters of each client back to the server for aggregation to obtain a new global model. At the same time, it is responsible for selecting the clients participating in each round of federated learning and calculating and broadcasting the relevant numerical values of the model parameters.

[0067] Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A federated learning client scheduling method based on weight divergence and update increment, characterized in that, it includes the following steps: The first step: After model evaluation, deploy the model to the server and identify it as the global model. The server broadcasts the global model to all clients participating in federated learning. At the same time, before client training, collect the local data distribution of the clients and send it to the clients together with the global model; The second step: The client expands and balances the local dataset through image enhancement technology to make the data distributions of the clients reach global balance; The third step: The client then uses its respective local dataset as input to train the global model. After a certain number of rounds of iterative training, obtain the local model update parameters; The fourth step: After each client is trained, calculate the weight divergence, update increment and training loss of the local model parameters, and send their respective calculation data to the server; The fifth step: The server selects the clients with small model weight divergence, small update increment and large model training loss according to the calculation data sent by the clients as the final selected client set for uploading model parameters; The sixth step: The server sends a request to these clients to upload model parameters; The seventh step: The selected clients send the corresponding model parameters to the server. The server aggregates according to the uploaded model parameters to obtain new global model parameters for updating the global model and iterating the next round of training until the global model converges or meets the requirements.

2. The federated learning client scheduling method based on weight divergence and update increment according to claim 1, characterized in that: The expansion and balance of the local dataset by the client through image enhancement technology in the second step includes the following: S1: The server initializes the global model w according to its own service requirements 0 and broadcasts it to the clients; S2: During the federated learning training, the server calculates the maximum number of images in each label for the client and sends this data to the client; S3: Use 28 different image enhancement methods from the Albumentations image enhancement library to enhance the original images; S4: The client generates images by combining different image enhancement methods, transforms and expands the original label images of its respective dataset according to the maximum number of label samples until the maximum number of samples required for each label is reached.

3. The federated learning client scheduling method based on weight divergence and update increment according to claim 1, characterized in that: The weight divergence in the fourth step is as follows: Obtain the local model parameter w k (t). After that, each client calculates its respective model weight divergence d k (t) based on its respective local model parameter w k (t); Calculate the local model parameter w k (t) and the global model parameter w g (t - 1), and select the model parameters with appropriate model weight divergence for uploading; Calculate the local model \(w^{(t)}\) of client \(k\) in the \(t\)-th round k and the weight divergence \(d^{(t)}\) between the global model \(w^{(t)}\) g and the global model \(w^{(t - 1)}\) in the \((t - 1)\)-th round k is given by the formula: where \(w^{(t)}\) k represents the local model parameters of client \(k\) in the \(t\)-th round, and \(w^{(t - 1)}\) g represents the global model parameters in the \((t - 1)\)-th round, and \(\|\cdot\|\) represents the L2 norm calculation.

4. The federated learning client scheduling method based on weight divergence and update increment according to claim 1, characterized in that: The update increment in the fourth step is as follows: Calculate the local model \(w\) of client \(k\) at the \(t\)-th round k The model update increment \(I\) of \((t)\) k k The formula of \((t)\) is as follows: where \(w\) k k (t) represents the local model parameters of client \(k\) at the \(t\)-th round, and \(w\) k k (t - 1) represents the local model parameters of client \(k\) at the \((t - 1)\)-th round.

5. The federated learning client scheduling method based on weight divergence and update increment according to claim 1, characterized in that: In the fourth step, in addition to considering the influence of weight divergence and update increment, the influence of loss also needs to be considered. Compared with the method of randomly selecting clients, selecting model parameters with larger losses for uploading can improve the convergence speed. Therefore, client scheduling is required.

6. The federated learning client scheduling method based on weight divergence and update increment according to claim 5, characterized in that: The client scheduling includes the following: S1: After the client calculates the model weight divergence d k (t), the update increment I k (t) and the model training loss loss k (t), it uploads the data to the server; S2: Based on the calculation data uploaded by the client, the server selects clients with a relatively large model training loss, a relatively small model weight divergence, and a relatively small update increment as a candidate client set to improve the model accuracy, and then requests these candidate clients to send parameters to the server; S3: The formula for selecting clients to participate in aggregation is: U k (t) = α·d k (t) + β·loss k (t) + δ·I k (t), where U k (t) represents the priority of client k being selected by the server to upload model parameter aggregation in the t-th round. α, β, and δ are adjustable parameters, representing the divergence of model weights, update increment, and the weight of model training loss in client selection, respectively.

Images