A method and system for federated learning of Internet of Things based on dual dynamic sparse training

Abstract

The present invention discloses a method and system for federated learning of the Internet of Things based on dual dynamic sparse training. The present invention is oriented to the Internet of Things scenario, adopts dynamic sparse training, and only uses local data to train the model, which can achieve low communication overhead and personalization. Then only the model parameters are transmitted to the server for aggregation, and a personalized aggregation method is proposed to ensure that the model can retain the personalization of the model without loss after uploading the parameters. By transmitting lightweight sparse models, the problems of uneven device resources and excessive communication overhead in the Internet of Things scenario are effectively solved, and personalized aggregation is proposed to aggregate sparse models on different devices to improve performance.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things federated learning and neural network pruning, and in particular to an Internet of Things federated learning method and system based on dual dynamic sparse training. Background Art

[0002] With the surge in IoT devices, smart mobile devices such as mobile phones and wearable devices have generated a wealth of data. In order to utilize this data without leaking customer privacy, federated learning has emerged. However, the communication overhead in federated learning is huge, and most of the devices involved are resource-constrained IoT devices, which limits the training of the model.

[0003] In addition, the data held by IoT devices are mostly non-independent and identically distributed, and the data distribution is unbalanced, which is called statistical heterogeneity, resulting in inconsistent directions of client model update optimization and model non-convergence. These problems restrict the application of federated learning on resource-constrained IoT devices. Therefore, federated learning urgently needs to be optimized for the IoT and provide a new method to improve the problems of excessive communication overhead and statistical heterogeneity in IoT scenarios.

[0004] Neural networks used in federated learning are usually over-parameterized and dense, which requires the client to pay additional computational and communication overhead. For resource-constrained IoT devices, using sparse models is expected to save a lot of costs. Sparse models are easier to store and can also reduce the amount of computation.

[0005] Pruning is an effective method to transform a dense model into a sparse model, such as iterative pruning, which iterates the model during training and prunes away neurons that contribute little to model performance to reduce computation and model complexity. Dynamic sparse training is a special pruning method that sets thresholds to explore and re-grow model parameters after pruning the model. Unlike iterative pruning, dynamic sparse training allows the network structure to evolve during training. This flexibility enables the network to adaptively identify and maintain important connections, potentially finding more efficient and effective parameter configurations to make better use of available parameters. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides an Internet of Things federated learning method and system based on dual dynamic sparse training.

[0007] The technical solution adopted by the present invention is:

[0008] A first aspect of the present invention provides an Internet of Things federated learning method based on dual dynamic sparse training, the method comprising the following steps:

[0009] S1, system parameter initialization: the server randomly initializes the global model and sends the global model to each IoT device, i.e., the client;

[0010] S2, Client Pruning: After receiving the global model, the client performs a one-time pruning of the global model according to its own computing resource budget to obtain a sparse model suitable for the computing resources of the local device;

[0011] S3, client local training and upload: The client performs local sparse model training and dynamic sparse training based on local data, and then uploads the local sparse model parameters to the server for aggregation;

[0012] S4, server aggregation: after receiving the local sparse model parameters of the client, the server performs a personalized weighted aggregation algorithm update according to the local sparse model parameters of the client;

[0013] S5, server dynamic sparse training: After the server aggregates and obtains the global model, if it is a global adjustment round, it performs global dynamic sparse training on the global model; then sends the obtained global sparse model to each client;

[0014] S6. Repeat steps S1 to S5 until a predetermined communication round is reached.

[0015] A second aspect of the present invention provides an Internet of Things federated learning system based on dual dynamic sparse training, which is applied to an Internet of Things federated learning method based on dual dynamic sparse training, including a server and a client, wherein the server is connected to the client;

[0016] The server is used to randomly initialize the global model and send the global model to each IoT device, i.e., the client; after receiving the local sparse model parameters of the client, perform personalized weighted aggregation algorithm update according to the local sparse model parameters of the client; and after the global model is obtained by aggregation, if it is a global adjustment round, perform global dynamic sparse training on the global model; and then send the obtained global sparse model to each client;

[0017] The client is used to perform one-time pruning of the global model according to its own computing resource budget after receiving the global model to obtain a sparse model suitable for the computing resources of the local device; it is also used to perform local sparse model training and dynamic sparse training based on local data, and then upload the local sparse model parameters to the server for aggregation.

[0018] The beneficial effects of the present invention are:

[0019] The present invention adopts dynamic sparse training to ensure that the user model is sparse and performs well. In addition, the present invention adopts a personalized aggregation method, which ensures that the personalization of the local model is preserved by a good aggregation of the local model in the global.

[0020] The present invention realizes efficient federated learning in the Internet of Things scenario, ensures reduced communication overhead and achieves model personalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A method flow chart of the system of the embodiment of the present application;

[0022] Figure 2 This is a system framework diagram of an embodiment of the present application;

[0023] Figure 3 Personalize the aggregate process map for the server;

[0024] Figure 4 This is a comparison of the accuracy of the embodiment of the present application with other methods on the CIFAR-10 dataset. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The accompanying drawings are only used for exemplary description and should not be construed as limiting the present invention.

[0026] like Figure 1 As shown, an IoT federated learning method based on dual dynamic sparse training provided in an embodiment of the present application includes the following steps:

[0027] S1. System parameter initialization: The server randomly initializes the global model and sends the global model to each IoT device, i.e., the client.

[0028] S2. Client pruning: After receiving the global model, the client performs a one-time pruning of the global model according to its own computing resource budget to obtain a local sparse model suitable for the computing resources of the local device. After pruning, the size of the local sparse model becomes smaller and the communication overhead during transmission will also be reduced. This process is only performed in the first round.

[0029] S3, client local training and upload: After pruning, the client will perform local sparse model training and dynamic sparse training based on local data, and then upload the model parameters to the server for aggregation.

[0030] S4, server aggregation: After receiving the model parameters from the client, the server updates the personalized weighted aggregation algorithm based on the model parameters uploaded by the client.

[0031] S5. Server dynamic sparse training: After the server aggregates and obtains the global model, if it is a global adjustment round, it will perform global dynamic sparse training on the model. Then the obtained global sparse model will be sent to each client.

[0032] The model obtained after dynamic sparse training on the server contains global insights, while the model obtained after dynamic sparse training on the client only contains local insights of the client. The reason for dual dynamic sparse training is to use global insights to guide local insights, thereby achieving a balance between personalization and generalization, and enhancing the stability and consistency of the framework.

[0033] S6. Repeat steps S1 to S5 until a predetermined communication round is reached.

[0034] In a preferred embodiment, in step S2, the client prunes the global model, specifically including:

[0035] Client After receiving the global model, prune the model according to the local data distribution:

[0036] Send a small batch of data into the global model to calculate each global model parameter The corresponding gradient ,in is the loss function, is the index of the model parameter in the global model;

[0037] The client then calculates each global model parameter The pruning score ,reserve Highest The sparse model consists of 10 parameters.

[0038] After pruning, the model architecture will match the local data distribution, thereby improving the model accuracy.

[0039] In a preferred embodiment, in step S3, the client performs dynamic sparse training based on local data, and then uploads the local sparse model parameters to the server, specifically including:

[0040] S31, Client After local training, according to the local sparse model The sparsity target Further prune the local model and remove parameters until the sparsity target is reached ,in Given Indexes and their corresponding values .

[0041] Furthermore, a local mask corresponding to the local sparse model is also generated in this process Used to represent local sparse models Whether the parameters are retained. The value of will gradually decrease as the number of rounds increases. In this example, it is set to ,in For the current round, is the preset number of rounds; until the final convergence.

[0042] S32. After pruning, the client calculates the importance score of the instantaneous gradient. For local sparse models For growth parameters, select The parameters are updated, and in this process the local sparse model and local mask are updated to and .

[0043] S33, when uploading the local sparse model parameters to the server, the client will determine whether it is a global adjustment round. If not, the local sparse model and its corresponding local mask Upload to the server for aggregation. If it is a global adjustment round, not only the local sparse model and local mask Upload and update the gradient Also upload to the server for aggregation.

[0044] In a preferred embodiment, in step S4, in order to ensure that the personalized part of each client is retained, a personalized aggregation operation is taken, specifically:

[0045] The server receives the local sparse model of each client. and local mask Afterwards, the local mask will be used to determine whether the parameters in all uploaded client local sparse models are uniquely overlapped. If so, they will be weighted aggregated and updated. If not, they will be discarded and not updated.

[0046] After the aggregation is completed, the server will obtain a global model that only contains the common parameters. ,in is the total amount of data from all clients. Is the client The amount of data on Is the client The common part parameters on .

[0047] In a preferred embodiment, in step S5, the server performs a global model Perform dynamic sparse training:

[0048] S51, the server according to the global model The parameter size of each layer is pruned and selected Parameters are discarded.

[0049] S52: After receiving the gradient update from the client After that, the server aggregates it to get the total gradient metric , according to the total gradient metric For the global sparse model To re-grow, select The gradient is updated.

[0050] S53, sending the global sparse model When sending to a local client, the client's mask Select the part that the client needs and send it to the client. The global sparse model part of ,in It's Hadamard.

[0051] Based on the same concept as the above method, an embodiment of the present application also provides an Internet of Things federated learning system based on dual dynamic sparse training, which is applied to an Internet of Things federated learning method based on dual dynamic sparse training, including a server and a client, and the server is connected to the client.

[0052] The server is used to randomly initialize the global model and send the global model to each IoT device, i.e., the client; after receiving the local sparse model parameters of the client, it is used to update the personalized weighted aggregation algorithm according to the local sparse model parameters of the client; it is also used to perform global dynamic sparse training on the global model after the global model is obtained by aggregation, if it is a global adjustment round; and then send the obtained global sparse model to each client.

[0053] The client is used to perform one-time pruning of the global model according to its own computing resource budget after receiving the global model to obtain a sparse model suitable for the computing resources of the local device; it is also used to perform local sparse model training and dynamic sparse training based on local data, and then upload the local sparse model parameters to the server for aggregation.

[0054] Example: Consider a classic federated learning framework: the global model uses the Lenet-5 model, which is a neural network model containing two convolutional layers, two pooling layers and three fully connected layers. There are 30 local clients, and 10 clients are selected to participate in training in each round. Each local client uses the SGD optimizer to iterate on the local data set, and the number of communication rounds is 100.

[0055] This embodiment uses the public CIFAR-10 dataset, which contains 60,000 32x32 pixel color images and is divided into 10 categories, each of which contains 6,000 images. The dataset is divided into a training set of 50,000 images and a test set of 10,000 images. The following steps are used as the basis for one training cycle:

[0056] System parameter initialization: The server randomly initializes the global model and sends the global model to each IoT device, i.e., the client.

[0057] Client pruning: After receiving the global model, the client performs a one-time pruning of the global model according to its own computing resource budget to obtain a sparse model suitable for the computing resources of the local device. After pruning, the size of the model becomes smaller and the communication overhead during transmission will also be reduced. This process is only performed in the first round. After receiving the global model, the model will be pruned according to the local data distribution; here the mini-batch size is set to 32. Client-side pruning is reflected in Figure 2 In the one-time pruning in , a sparse model will be obtained after pruning.

[0058] Client local training and upload: After pruning, the client will perform local sparse model training and dynamic sparse training based on local data. The preset learning rate during training = 0.01, the number of local iterations is 10. This is reflected in Figure 2 In the client training and dynamic sparse training of , in the figure, each model will train the model according to its own local data, and then perform dynamic sparse training and separate the corresponding mask. The use of dynamic sparse training technology can ensure that the client obtains a sparse model and the model performs well.

[0059] Client After local training, according to the local sparse model The sparsity target Further prune the local model and remove parameters until the sparsity target is reached ,in Given Indexes and their corresponding values In this process, a local mask corresponding to the local sparse model is also generated Used to represent local sparse models Whether the parameters are retained. The value of will gradually decrease with the increase of rounds, and is set to ,in For the current round, For the preset rounds, the preset rounds here to 100 until convergence.

[0060] After pruning, the client will use the instantaneous gradient importance score For local sparse models For growth parameters, select The parameters are updated, and in this process the local sparse model and local mask are updated to and .

[0061] When uploading the local sparse model parameters to the server, the client will determine whether it is a global adjustment round. If not, the local sparse model and its corresponding local mask Upload to the server for aggregation. If it is a global adjustment round, not only the local sparse model and local mask Upload and update the gradient It is also uploaded to the server for aggregation. This part is reflected in Figure 2 In the model and mask aggregation, each device uploads its sparse model and mask to the server, and completes condition judgment and aggregation on the central server.

[0062] The server receives the local sparse model of each client. and local mask After that, the local mask will be used to determine whether the parameters in the model are uniquely overlapped. If so, they will be weighted and aggregated for update. If not, they will be discarded and not updated. After the aggregation is completed, the server will obtain a global model containing only the common parameters. ,in is the total amount of data from all clients. Is the client The amount of data on Is the client After aggregation, a global model is obtained.

[0063] Figure 3The figure describes the personalized weighted aggregation algorithm. The left half is a simplified diagram of each client and its model before aggregation, the middle is the server, and the right half is the situation after aggregation. Nodes with a value of 0 are pruned nodes, and non-zero values ​​are model parameters. From the figure, we can see that client 1 and client 2 have the same personalized parts, and client 3 has a part of the same part. These three same parts will be weighted averaged and aggregated. The dark part in client 3 will be copied as is to the client 3 model after aggregation as the personalized part of the client 3 model.

[0064] After the server aggregates and obtains the global model, if it is a global adjustment round, it will perform global dynamic sparse training on the model. The parameter size of each layer is pruned and selected Discard.

[0065] After receiving the gradient update from the client After that, the server aggregates it to get the total gradient metric , according to the total gradient metric For the global sparse model To re-grow, select Update the sparse model and send it to each client.

[0066] Sending the global sparse model When sending to a local client, the client's mask Select the part that the client needs and send it to the client. The global sparse model part of ,in It's Hadamard.

[0067] This example follows the above steps and the same settings to perform the CIFAR-10 classification prediction task. In the experiment, 30 IoT devices, i.e., clients, are set up, 10 clients are selected each time to participate in the training, and each client uses the SGD optimizer to iterate 10 times on the local dataset, with a total of 100 rounds of communication. After the experiment, the prediction is performed, and the accuracy of the result is as follows: Figure 4 As shown, it can be seen that compared with FedAvg, FedDST, PruneFL and FedDIP, the model performance obtained by training the IoT federated learning method based on dual dynamic sparse training proposed in this embodiment is better, and better accuracy can be obtained with fewer rounds.

[0068] The present invention uses sparse training to train the model, reducing the communication overhead of the client. At the same time, the present invention realizes personalized aggregation of sparse models and improves the performance of the model. The present invention realizes efficient and secure federated learning in the Internet of Things scenario, reduces the communication overhead on resource-constrained devices, and achieves personalization to cope with statistical heterogeneity.

[0069] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A federated learning method for IoT based on dual dynamic sparse training, characterized in that The method comprises the following steps: S1, system parameter initialization: the server randomly initializes the global model and sends the global model to each IoT device, i.e., the client; S2, Client Pruning: After receiving the global model, the client performs a one-time pruning of the global model according to its own computing resource budget to obtain a sparse model suitable for the computing resources of the local device; S3, client local training and upload: The client performs local sparse model training and dynamic sparse training based on local data, and then uploads the local sparse model parameters to the server for aggregation; S4, server aggregation: after receiving the local sparse model parameters of the client, the server performs a personalized weighted aggregation algorithm update according to the local sparse model parameters of the client; S5, server dynamic sparse training: After the server aggregates and obtains the global model, if it is a global adjustment round, it performs global dynamic sparse training on the global model; then sends the obtained global sparse model to each client; S6, repeating steps S1 to S5 until a predetermined communication round is reached; In step S2, after receiving the global model, the client prunes the global model according to the local data distribution so that the global model architecture is consistent with the local data distribution. The specific process of pruning is: Send a batch of data to the global model for calculation to obtain the gradient corresponding to each global model parameter; Calculate the pruning score of each global model parameter and retain the sparse model consisting of several parameters with the highest pruning scores; In step S3, the client performs dynamic sparse training based on local data, and then uploads the local sparse model parameters to the server, specifically including: S31. After the local training is completed, the client prunes the local model according to the sparsity target of the sparse model, and removes several parameters until the sparsity target is reached; S32, the client increases the parameters of the local sparse model according to the instantaneous gradient importance score and selects several parameters to update; In step S4, the personalized weighted aggregation algorithm is updated specifically as follows: After receiving the local sparse model and the corresponding mask of each client, the server determines whether the parameters in the uploaded local sparse models of all clients are uniquely overlapped based on the mask; If it is the only overlap, a weighted aggregate update is performed on it; If it is not the only overlap, it will be discarded and not updated; After the aggregation is completed, the server obtains a global model that only contains the common parameters; In step S5, the server performs dynamic sparse training on the global model, specifically including: S51, the server prunes according to the parameter size of each layer on the global model, and selects some parameters to be discarded; S52. After receiving the gradient update from the client, the server aggregates it to obtain the total gradient metric, re-grows the global sparse model according to the total gradient metric, and selects some gradients for update.

2. The method for federated learning of the Internet of Things based on dual dynamic sparse training according to claim 1, characterized in that: In step S31, a mask corresponding to the local sparse model is also generated to indicate whether the parameters in the local sparse model are retained.

3. The method for federated learning of the Internet of Things based on dual dynamic sparse training according to claim 2, characterized in that: The step S33 is also included, when uploading the local sparse model parameters to the server, the client determines whether it is a global adjustment round; If it is not a global adjustment round, the local sparse model and its corresponding mask are uploaded to the server for aggregation; If it is a global adjustment round, not only the local sparse model and its corresponding mask are uploaded, but also the gradient update is uploaded to the server for aggregation.

4. An IoT federated learning system based on dual dynamic sparse training, applied to an IoT federated learning method based on dual dynamic sparse training according to any one of claims 1 to 3, comprising a server and a client, wherein the server is connected to the client; characterized in that: The server is used to randomly initialize the global model and send the global model to each IoT device, i.e., the client; after receiving the local sparse model parameters of the client, perform personalized weighted aggregation algorithm update according to the local sparse model parameters of the client; and after the global model is obtained by aggregation, if it is a global adjustment round, perform global dynamic sparse training on the global model; and then send the obtained global sparse model to each client; The client is used to perform one-time pruning of the global model according to its own computing resource budget after receiving the global model to obtain a sparse model suitable for the computing resources of the local device; it is also used to perform local sparse model training and dynamic sparse training based on local data, and then upload the local sparse model parameters to the server for aggregation.

Images