Progressive federated learning method and system based on model pruning
By adopting model pruning and gradual learning methods in federated learning, the problems of high computational pressure and low training efficiency and accuracy in federated learning are solved, and more efficient communication and computing resource utilization are achieved.
Patent Information
- Application Number
- CN202411385451.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-02-18
AI Technical Summary
The computational pressure of training models in federated learning is high, and the training efficiency and accuracy are low, resulting in high communication costs and model training bottlenecks.
A progressive federated learning method based on model pruning is adopted, and the local model is pruned through hierarchical pruning technology to reduce traffic, and reduce the server parallel computing volume by gradually expanding the global model.
It effectively reduces the amount of communication data between end users and cloud centers, reduces the computational pressure and delay of the training model, and maintains the accuracy of the training model.
Smart Images
Figure CN119476413A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of federated learning, and specifically relates to a progressive federated learning method and system based on model pruning. Background Art
[0002] As an important branch of artificial intelligence, machine learning has been successfully and widely used in various fields such as pattern recognition, data mining, and computer vision. Due to the limited computing resources of terminal devices, the training of machine learning models is usually based on cloud center servers. In this way, the data collected by terminal devices, such as pictures, videos, or personal location information, must be uploaded to the central server to complete the model training. However, uploading real user data will leak their privacy. For the sake of privacy protection, terminal users are reluctant to share their private data, forming "data islands". In the long run, this seriously hinders the development and application of machine learning technology. In order to solve the problem of data islands, federated learning was proposed and attracted great attention. Federated learning provides a privacy-aware learning paradigm without the need to share original data, allowing participants to collaborate to build better global models by sharing only local model updates. The main idea is to train a local model on each client, the central server collects all local models, executes the federated averaging algorithm to complete model aggregation, and distributes the aggregated model to each client for the next round of local training. In this process, sensitive original data will not be exchanged on each client, thereby protecting user privacy. In recent years, the deployment of federated learning in practice has been widely used, such as mobile edge computing, IoT artificial intelligence, and autonomous driving.
[0003] In a federated learning environment, multiple rounds of interaction are required between the end user and the central server to obtain a global model of target accuracy. For complex model training, such as deep learning model training, each model update may contain millions of parameters. The high dimensionality of the model update will consume a lot of communication costs and even become a model training bottleneck. In addition, due to the heterogeneity of end users / devices, the unreliability of the network status of each device, and the asymmetry of the Internet connection speed, if the download speed is greater than the upload speed, resulting in a delay in the end user uploading the updated parameters, the model training bottleneck will be further aggravated.
[0004] In order to solve the problem of high communication cost in federated learning, domestic and foreign researchers have conducted extensive research and proposed many effective communication optimization methods. The mainstream methods to reduce communication volume usually compress, lightweight, distill and sparsify local model updates to make the uploaded model more compact, thereby achieving the purpose of reducing communication volume. However, they are usually accompanied by other shortcomings, such as consuming more local computing resources or seriously reducing the accuracy of the training model. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a progressive federated learning method and system based on model pruning, which solves the problems of high computational pressure, low training efficiency and accuracy of federated learning training models in the prior art.
[0006] The present invention adopts the following technical solutions to solve the above technical problems:
[0007] The progressive federated learning method based on model pruning includes the following steps:
[0008] Step S1, the central server is initialized, the central server builds a training model, and initializes the complete global model, progressive sub-models and global variables;
[0009] Step S2: local model training: each client in the client set C performs local model training in parallel based on its local data to obtain a local model update;
[0010] Step S3: The client uploads the local model, and each client prunes the local model layer by layer and sends it to the server;
[0011] Step S4: The central server updates the global model. The central server aggregates the uploaded local updates, calculates the aggregated global model updates and the global model, and the model training enters the next round of iteration.
[0012] Step S5, global model expansion, after repeating steps S1 to S4 for a certain number of iterations, the progressive sub-model is expanded to obtain a new progressive sub-model, which is used as a new global model;
[0013] Step S6: Repeat steps S1 to S5 until the global model converges and the model training is completed.
[0014] The specific process of step 1 is as follows:
[0015] Step S1.1, construct a training model and design the number of layers and neurons in the hidden layer and output layer;
[0016] Step S1.2, initialize global variables, including client set C = {C1, C2, ..., C k}, number of communication rounds R, progressive training phase J;
[0017] Step S1.3, initialize the progressive sub-model as the global model, including global model parameters W0, and global model update G0.
[0018] The specific process of local model training in step 2 is as follows:
[0019] Step S2.1, obtain the global model parameter W of the rth iteration from the central serverr ;
[0020] Step S2.2: Taking the kth client as an example, the local model training is performed in parallel based on its local data to obtain the local model L of the rth iteration. k,r .
[0021] The specific process of step 3 is as follows:
[0022] Step S3.1: The client performs a local model L according to the retention probability of each hidden layer. k,r Perform layered pruning to obtain the local pruning model L k,r ;
[0023] Step S3.2: The client takes the local pruned model L k,r Send to the central server;
[0024] Step S3.3, repeat step S3.1 to obtain the rth iteration local pruning model of all clients
[0025]
[0026] The process of layered pruning is as follows:
[0027] According to the formula The layer retention flag is calculated, and the layer parameters with the flag 1 are retained, and the layer parameters with the flag 0 are pruned; Retention probability for the layer, a global parameter set before training.
[0028] The specific process of expanding the progressive sub-model in step 5 is as follows:
[0029] Step S5.1, after completing a certain number of global iterations, the central server determines whether the progressive sub-model is expanded to the complete model. If so, the expansion is stopped and the federated learning training is continued until the model converges. If not, a new progressive sub-model with new hidden layers and output layers is constructed;
[0030] In step S5.2, the central server transfers the corresponding parameters of the old sub-model to the new progressive sub-model as the global model of the next stage.
[0031] The progressive federated learning system based on model pruning includes a central server and several local clients;
[0032] The central server aggregates the uploaded local updates, calculates the aggregated global model updates and expands the global model to obtain a new global model;
[0033] The local client trains the local model in parallel on the local data, prunes the local model in layers, and uploads it to the central server.
[0034] The client includes any electronic device with display and control functions.
[0035] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, all or part of the steps of the method are called.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] 1. This method is suitable for improving communication efficiency and reducing computing pressure. Inspired by progressive learning, we consider gradually expanding the global model to reduce communication overhead and server parallel computing, thereby reducing training delay.
[0038] 2. Use layered pruning techniques to prune each client’s local updated model before sending it to the central server.
[0039] 3. The proposed method and system can not only reduce the amount of communication data between the end users and the cloud center, but also minimize the accuracy of the training model. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 The present invention provides an overall method flow chart for an embodiment of the present invention.
[0041] Figure 2 A flowchart of specific steps provided for an embodiment of the present invention.
[0042] Figure 3 A schematic diagram of the overall principle provided for an embodiment of the present invention. DETAILED DESCRIPTION
[0043] The structure and working process of the present invention will be further described below in conjunction with the accompanying drawings.
[0044] In order to improve communication efficiency and reduce computing pressure while ensuring the accuracy of the training model, the present invention proposes a progressive federated learning method based on hierarchical pruning. This method is suitable for improving communication efficiency and reducing computing pressure. Inspired by progressive learning, we consider gradually expanding the global model to reduce communication overhead and reduce the amount of parallel computing on the server, thereby reducing training delays. At the same time, we use hierarchical pruning technology to prune the local update model of each client before sending it to the central server. The method and system proposed in the present invention can not only reduce the amount of communication data between the end user and the cloud center, but also minimize the accuracy of the training model.
[0045] The progressive federated learning method based on model pruning includes the following steps:
[0046] Step S1, the central server is initialized, the central server builds a training model, and initializes the complete global model, progressive sub-models and global variables;
[0047] Step S2: local model training: each client in the client set C performs local model training in parallel based on its local data to obtain a local model update;
[0048] Step S3: The client uploads the local model, and each client prunes the local model layer by layer and sends it to the server;
[0049] Step S4: The central server updates the global model. The central server aggregates the uploaded local updates, calculates the aggregated global model updates and the global model, and the model training enters the next round of iteration.
[0050] Step S5, global model expansion, after repeating steps S1 to S4 for a certain number of iterations, the progressive sub-model is expanded to obtain a new progressive sub-model, which is used as a new global model;
[0051] Step S6: Repeat steps S1 to S5 until the global model converges and the model training is completed.
[0052] The specific process is as follows:
[0053] Step S1, central server initialization. The central server builds a training model, initializes the progressive sub-model and global variables, including the following sub-steps:
[0054] Step S1.1, construct a training model and design the number of layers and neurons in the hidden layer and output layer.
[0055] Step S1.2, initialize global variables, including client set C = {C1, C2, ..., C k}, number of communication rounds R, progressive training stage J.
[0056] Step S1.3, initialize the progressive sub-model as the global model, including global model parameters W0, and global model update G0.
[0057] Step S2: local model training. Each client in the client set C performs local model training in parallel based on its local data to obtain a local model update, including the following sub-steps:
[0058] Step S2.1, obtain the global model parameter W of the rth iteration from the central server r .
[0059] Step S2.2: Taking the kth client as an example, the local model training is performed in parallel based on its local data to obtain the local model L of the rth iteration. k,r .
[0060] Step S3: The client uploads the local model. Each client prunes the local model and sends it to the server, including the following sub-steps:
[0061] Step S3.1, client layer retention probability Calculate the retention flag for each hidden layer and calculate the retention flag for the local model L k,r Prune to get the local pruned model
[0062] Step S3.2: The client pruned the local model Sent to the central server.
[0063] Repeat steps 2 and 3 to obtain the local pruned model of the rth iteration for all clients.
[0064] Step S4: The central server updates the global model. The central server aggregates the uploaded local updates, calculates the aggregated global model updates and the global model, and the model training enters the next round of iteration.
[0065] Step S5, global model expansion. After a certain number of iterations, the progressive sub-model is expanded to obtain a new progressive sub-model, and steps S1 to S4 are repeated as a new global model. The following sub-steps are included:
[0066] Step S5.1, after completing a certain number of global iterations, the central server determines whether the progressive sub-model is extended to the complete model. If not, a new progressive sub-model is constructed with new hidden layers and output layers added.
[0067] In step S5.2, the central server transfers the corresponding parameters of the old sub-model to the new progressive sub-model as the global model of the next stage.
[0068] Step S6, repeat steps S1 to S5 until the global model converges and the model training is completed.
[0069] Specific embodiments, such as Figures 1 to 3 As shown,
[0070] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings and embodiments, so as to fully understand the purpose, characteristics and effects of the present invention.
[0071] The following takes the example of 100 users jointly training a convolutional neural network to illustrate the specific implementation steps of the present invention.
[0072] The method provided by the technical solution of the present invention can use computer software technology to realize automatic operation process. Figure 1 is a flow chart of the overall method of an embodiment of the present invention, combined with Figure 2The specific steps of the embodiment of the present invention are as follows:
[0073] Step S1, central server initialization. The central server builds a training model, initializes the progressive sub-model and global variables, including the following sub-steps:
[0074] Step S1.1, construct a training model and design the number of layers and neurons in the hidden layer and output layer.
[0075] In the embodiment, a convolutional neural network with 6 Conv layers is simulated.
[0076] Step S1.2, initialize global variables, including client set C = {C1, C2, ..., C k}, number of communication rounds R, progressive training stage J.
[0077] In the embodiment, the initialization client set C = {C1, C2, ..., C k}, the number of communication rounds R = 0, and the progressive training stage J = 0.
[0078] Step S1.3, initialize the progressive sub-model as the global model M, including global model parameters W0, and global model update G0.
[0079] In the embodiment, the global model parameter W0 is initialized and the global model is updated G0.
[0080] Step S2: local model training. Each client in the client set C performs local model training in parallel based on its local data to obtain a local model update, including the following sub-steps:
[0081] Step S2.1, obtain the global model parameter W of the rth iteration from the central server r .
[0082] In the embodiment, for example, if the current iteration round r=5, the global model parameter W5 of the fifth iteration round is obtained from the central server.
[0083] Step S2.2: Taking the kth client as an example, the local model training is performed in parallel based on its local data to obtain the local model L of the rth iteration. k,r .
[0084] In the embodiment, taking the 10th client as an example, the local model training is performed in parallel based on its local data to obtain the local model L of the 5th iteration. 10,5 .
[0085] Step S3: The client uploads the local model. Each client prunes the local model and sends it to the server, including the following sub-steps:
[0086] Step S3.1: The client reserves the probability of Calculate the retention flag for each hidden layer and calculate the retention flag for the local model L k,r Prune to get the local pruned model
[0087] In the embodiment, taking the 10th client as an example, the layer retention probability Set to 0.7, calculate the retention flag of each hidden layer, and calculate the retention flag of the local model L 10，5 Prune to get the local pruned model
[0088] Step S3.2: The client pruned the local model Sent to the central server.
[0089] In the embodiment, taking the 10th client as an example, the local pruning model Sent to the central server.
[0090] Repeat steps 2 and 3 to obtain the r-th iteration local pruning model L = {L k,1 ,L k,2 ,...,L k,r}
[0091] Step S4: The central server updates the global model. The central server aggregates the uploaded local updates, calculates the aggregated global model updates and the global model, and the model training enters the next round of iteration.
[0092] In the embodiment, the central server aggregates the uploaded local updates, calculates the aggregated global model update G5 and the global model M, and the model training enters the next round of iteration.
[0093] Step S5, progressive sub-model expansion: After a certain number of iterations, the progressive sub-model is expanded to obtain a new progressive sub-model, and steps S1 to S4 are repeated as a new global model.
[0094] Step S5.1, after completing a certain number of global iterations, the central server determines whether the progressive sub-model is extended to the complete model. If not, a new progressive sub-model is constructed with new hidden layers and output layers added.
[0095] In the embodiment, taking the progressive training stage J=2 as an example, the progressive sub-model is not expanded to be complete, and a new hidden layer E2 and output layer F2 are added.
[0096] In step S5.2, the central server transfers the corresponding parameters of the old sub-model to the new progressive sub-model as the global model of the next stage.
[0097] In the embodiment, taking the progressive training stage J=2 as an example, the central server transmits the parameters of the global model hidden layer E1 of the progressive stage J=1 to the global model hidden layer E1 of the progressive stage J=2.
[0098] Repeat steps S1 to S5 until the global model converges and the model training is completed.
[0099] The progressive federated learning system based on model pruning includes a central server and several local clients;
[0100] The central server aggregates the uploaded local updates, calculates the aggregated global model updates and expands the global model to obtain a new global model;
[0101] The local client trains the local model in parallel on the local data, prunes the local model in layers, and uploads it to the central server.
[0102] The client includes any electronic device with display and control functions, such as a mobile phone, a computer, a flat-panel display, etc.
[0103] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, all or part of the steps of the method are called.
[0104] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.
[0105] The present invention provides a technical solution that can be implemented by those skilled in the art. The above embodiments are only used to illustrate the present invention, but not to limit the present invention.
[0106] It should be understood that the present solution is not limited to the above-mentioned specific implementation methods, and the devices and structures not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present solution without departing from the scope of the technical solution of the present solution, or modify it into an equivalent embodiment with equivalent changes, which does not affect the essential content of the present solution. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present solution without departing from the content of the technical solution of the present solution still falls within the scope of protection of the technical solution of the present solution.
Claims
1. A progressive federated learning method based on model pruning, characterized by: The steps include: Step S1, the central server is initialized, the central server builds a training model, and initializes the complete global model, progressive sub-models and global variables; Step S2: local model training: each client in the client set C performs local model training in parallel based on its local data to obtain a local model update; Step S3: The client uploads the local model, and each client prunes the local model layer by layer and sends it to the server; Step S4: The central server updates the global model. The central server aggregates the uploaded local updates, calculates the aggregated global model updates and the global model, and the model training enters the next round of iteration. Step S5, global model expansion, after repeating steps S1 to S4 for a certain number of iterations, the progressive sub-model is expanded to obtain a new progressive sub-model, which is used as a new global model; Step S6: Repeat steps S1 to S5 until the global model converges and the model training is completed.
2. The progressive federated learning method based on model pruning according to claim 1, characterized in that: The specific process of step 1 is as follows: Step S1.1, construct a training model and design the number of layers and neurons in the hidden layer and output layer; Step S1.2, initialize global variables, including client set C = {C1, C2, ..., C k }, number of communication rounds R, progressive training phase J; Step S1.3, initialize the progressive sub-model as the global model, including global model parameters W0, and global model update G0.
3. The progressive federated learning method based on model pruning according to claim 2, characterized in that: The specific process of local model training in step 2 is as follows: Step S2.1, obtain the global model parameter W of the rth iteration from the central server r ; Step S2.2: Taking the kth client as an example, perform local model training based on its local data to obtain the local model L of the rth iteration. k,r .
4. The progressive federated learning method based on model pruning according to claim 3, characterized in that: The specific process of step 3 is as follows: Step S3.1: The client performs a local model L according to the retention probability of each hidden layer. k,r Perform layered pruning to obtain the local pruning model L k,r ; Step S3.2: The client takes the local pruned model L k,r Send to the central server; Step S3.3, repeat step S3.1 to obtain the rth iteration local pruning model of all clients 5. The progressive federated learning method based on model pruning according to claim 4, characterized in that: The process of layered pruning is as follows: According to the formula The layer retention flag is calculated, and the layer parameters with the flag 1 are retained, and the layer parameters with the flag 0 are pruned; Retention probability for the layer, a global parameter set before training.
6. The progressive federated learning method based on model pruning according to claim 4, characterized in that: The specific process of expanding the progressive sub-model in step 5 is as follows: Step S5.1, after completing a certain number of global iterations, the central server determines whether the progressive sub-model is expanded to the complete model. If so, the expansion is stopped and the federated learning training is continued until the model converges. If not, a new progressive sub-model with new hidden layers and output layers is constructed; In step S5.2, the central server transfers the corresponding parameters of the old sub-model to the new progressive sub-model as the global model of the next stage.
7. A progressive federated learning system based on model pruning, characterized by: It includes a central server and several local clients; among them, The central server aggregates the uploaded local updates, calculates the aggregated global model updates and expands the global model to obtain a new global model; The local client trains the local model in parallel on the local data, prunes the local model in layers, and uploads it to the central server.
8. The progressive federated learning system based on model pruning according to claim 7, characterized in that: The client includes any electronic device with display and control functions.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, all or part of the steps of the method described in any one of claims 1 to 6 are called.