Federal edge learning model training method, system, device and medium
By adopting lightweight models and pseudo-vector transmission methods in edge computing, the problem of high federated learning energy consumption is solved, and the global model is efficiently trained on edge devices while reducing energy consumption and maintaining performance.
Patent Information
- Application Number
- CN202510411850.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-02
- Publication Date
- 2025-07-18
AI Technical Summary
In the edge computing scenario, how to maintain federated learning benefits while reducing energy consumption, especially for limited energy budgets for mobile devices, has become an urgent problem.
The lightweight model is used for local training, pseudo-vectors are generated and transmitted to the edge server for global model construction, combining data vectorization compression and model pruning, the global model is trained through the edge server, and some parameters are distributed to the client, and the model is fused using the mask matrix.
While reducing energy consumption, the performance of the model is maintained. By compressing pseudo-vector transmission and model pruning, the accuracy of the global model is improved and communication and computing overhead is reduced.
Smart Images
Figure CN120338051A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of federated learning, and particularly relates to a method, system, device and medium for training a federated edge learning model. Background Art
[0002] With the rapid development of network infrastructure and mobile devices (such as autonomous vehicles and mobile phones), the demand for edge computing is also increasing. As a representative technology, federated learning allows multiple edge devices to participate in model training while protecting their privacy. This greatly promotes information sharing and data utilization among users. For example, autonomous vehicles have many sensors that can collect a large amount of private environmental data. This data can be fully utilized through federated learning. However, compared with traditional wired infrastructure, devices in edge computing usually have limited energy budgets. Energy is crucial for such devices. Federated learning requires a large amount of communication and computational overhead, making it unsuitable for mobile devices. Therefore, how to reduce energy consumption while maintaining the benefits of federated learning has become the core challenge in promoting this technology in edge computing scenarios. Adopting small neural network models is a viable coping strategy. The parameter scale of such models is much smaller than that of conventional models, thus achieving significant savings in communication and energy consumption. However, it should be noted that small models often struggle to reach the desired performance level. How to find an appropriate balance between energy conservation and performance is an urgent problem to be solved in current research. Summary of the Invention
[0003] The purpose of the present invention is to provide a method, system, device and medium for training a federated edge learning model to solve the problems existing in the above-mentioned prior art.
[0004] To achieve the above purpose, the present invention provides a method for training a federated edge learning model, including:
[0005] Enabling a plurality of clients to train local models and generate pseudo-vectors, where the local models are lightweight models;
[0006] Transmitting the pseudo-vectors generated by each client and the model parameters of the local models to an edge server;
[0007] Constructing and training a global model in the edge server based on the pseudo-vectors and model parameters of each client, and distributing partial parameters of the trained global model to each client based on a preset partitioning rule.
[0008] Optionally, the generation process of the pseudo-vector specifically includes:
[0009] Dividing the local data set of each client into several categories of data according to categories, converting the data of each category into vectors based on an embedding function, and linearly combining the embedding vectors of different categories to obtain a pseudo-vector.
[0010] Optionally, transmit the pseudo-vectors generated by each client and the model parameters of the local model to the edge server, specifically including:
[0011] Define a mask matrix to perform model pruning on the local model, and dynamically update the mask matrix during the pruning process until the target pruning rate is met, obtaining a pruned local model;
[0012] Transmit the pseudo-vectors corresponding to each client and the model parameters of the pruned local model to the edge server.
[0013] Optionally, the construction and training process of the global model specifically includes:
[0014] Aggregate the model parameters uploaded by each client as the first part of the model parameters of the global model, randomly initialize and generate the second part of the model parameters, and construct the global model based on the first part of the model parameters and the second part of the model parameters;
[0015] Input the pseudo-vectors of each client into the global model, and train based on the target loss function to obtain a trained global model;
[0016] Distribute some parameters of the trained global model to each client based on a preset partitioning rule.
[0017] Optionally, the distributing some parameters of the trained global model to each client based on a preset partitioning rule specifically includes:
[0018] Distribute the updated first part of the model parameters in the trained global model to each client, and keep the second part of the model parameters in the edge server;
[0019] After a preset number of rounds of communication, switch the mask matrix of the client to a semi-mask matrix. After the client receives the distributed model parameters, fuse the local model parameters and the distributed model parameters through the semi-mask matrix.
[0020] A federated edge learning model training system includes:
[0021] A pseudo-vector generation module, which is used to enable several clients to train local models and generate pseudo-vectors, and the local model is a lightweight model;
[0022] A global model training module, which transmits the pseudo-vectors generated by each client and the model parameters of the local model to the edge server; constructs and trains a global model in the edge server based on the pseudo-vectors and model parameters of each client, and distributes some parameters of the trained global model to each client based on a preset partitioning rule.
[0023] An electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the described method for training a federated edge learning model.
[0024] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the described method for training a federated edge learning model.
[0025] The technical effect of the present invention is as follows:
[0026] The framework provided by the present invention has a large global model on the server and lightweight sub-models on the clients for training. It balances the energy cost and model performance of mobile devices. This application also proposes a method for generating compressed pseudo-vectors for client data to support the training of the global model on the server side. The compressed pseudo-vectors are uploaded to the edge server to improve the accuracy of the global model while reducing energy consumption. In addition, we further prune and fine-tune the lightweight model on the client side. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0028] The drawings forming a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:
[0029] Figure 1 It is a schematic diagram of the federated edge learning framework in the embodiments of the present invention;
[0030] Figure 2 It is the test accuracy at different communication rounds on MNIST in the embodiments of the present invention;
[0031] Figure 3 It is the test accuracy at different communication rounds on EMNIST in the embodiments of the present invention;
[0032] Figure 4 It is the test accuracy at different communication rounds on CIFAR-10 in the embodiments of the present invention;
[0033] Figure 5 It is a schematic diagram of the local model personalization process in the embodiments of the present invention;
[0034] Figure 6 It is the local data compression flowchart in the embodiment of the present invention;
[0035] Figure 7 It is the flowchart of federated edge learning model training in the embodiment of the present invention. Detailed implementation manners
[0036] Now, various exemplary implementation manners of the present invention will be described in detail. This detailed description should not be regarded as a limitation of the present invention, but should be understood as a more detailed description of certain aspects, features, and implementation schemes of the present invention.
[0037] It should be understood that the terms described in the present invention are only for describing specific implementation manners and are not used to limit the present invention. In addition, for the numerical ranges in the present invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Each intermediate value within any stated value or stated range and each smaller range between any other stated value or intermediate value within the stated range are also included in the present invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.
[0038] Without departing from the scope or spirit of the present invention, various improvements and changes can be made to the specific implementation manners of the specification of the present invention, which are obvious to those skilled in the art. Other implementation manners obtained from the specification of the present invention are obvious to those skilled in the art. The specification and embodiments of this application are only exemplary.
[0039] Regarding the terms "comprising", "including", "having", "containing", etc. used herein, they are all open-ended terms, that is, they are meant to include but not limited to.
[0040] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the drawings and combine the embodiments to detail this application.
[0041] As Figure 1 - Figure 7 shown, in this embodiment, a method for training a federated edge learning model is provided, including: enabling a plurality of clients to train local models and generate pseudo vectors, where the local models are lightweight models; transmitting the pseudo vectors and model parameters of the local models generated by each client to an edge server; constructing and training a global model based on the pseudo vectors and model parameters of each client in the edge server, and distributing partial parameters of the trained global model to each client based on a preset partitioning rule.
[0042] Edge devices in federated learning require a lightweight model to save energy with limited communication and computational overhead. On the contrary, federated learning tasks require the largest possible model to ensure high performance. To overcome this difficulty, this embodiment designs a new training framework for edge federated learning. Each device only maintains a lightweight sub-model to reduce overhead, while the server maintains a complete large model to resist the performance degradation caused by the lightweight model. Compared with training a complete model in existing federated learning, in the framework of this embodiment, mobile devices only need to train and update the lightweight model, which greatly reduces the computational and communication overhead. In addition, this embodiment adopts a data vectorization compression method, which greatly reduces the amount of data transmitted.
[0043] This embodiment proposes a new federated edge learning framework, which has a large global model on the server and lightweight sub-models on the clients for training. It balances the energy cost and model performance of mobile devices. This embodiment also proposes a method for generating compressed pseudo-vectors for client data to support the global model training on the server side. The compressed pseudo-vectors are uploaded to the edge server to improve the accuracy of the global model while reducing energy consumption. In addition, this embodiment further prunes and fine-tunes the lightweight model on the client side.
[0044] The federated edge learning framework is as Figure 1 shown. For the data category k in the dataset D of any client i i in, the data This embodiment uses the embedding function f(.) to convert the data into a vector. Before each round of global training, client i randomly extracts m data from each data category k Uploading data may lead to privacy leakage. However, the best model parameters of each client are not the same. This means that gradient-based aggregation cannot fully provide useful information for each client. One solution is to embed and combine the data into new pseudo-vectors according to the class to hide the real data. The obtained pseudo-vectors will also carry useful information and be uploaded to the edge server. This process can be expressed as:
[0045]
[0046] where, a l is the weight of the vector and the weights of different vectors are random.
[0047] To ensure performance, this embodiment prunes the networks of the clients while maintaining a small network structure and facilitating transmission. The gradients of the global model at the edge server are aggregated from each client, and the calculation formula is:
[0048]
[0049] Among them, represents the number of samples in all datasets, |D i | represents the number of dataset samples of client D i . w A are the parameters of the client lightweight model.
[0050] Pruning of the model will be performed locally for each client because the pruned model is more personalized. The pruning loss of the local model network weights w i of client i is expressed as follows:
[0051]
[0052] The final total loss of the global model is expressed as the weighted sum of the losses of the client lightweight model and the global large model, and the total loss is calculated as follows:
[0053]
[0054] To facilitate the aggregation of client local models on the edge server, in this embodiment, a mask matrix is designed for each client i network in the t-th round of communication. Its main function is to eliminate the influence of pruned parameters while keeping the network structure unchanged. The mask value corresponding to the pruned element is 0, otherwise it is 1. Before reaching the given pruning rate, the local model will continue to be pruned.
[0055] To maintain personalization, in this embodiment, after the t-th round of aggregation, the mask matrix of client i is changed to a semi-mask matrix This increases the personalization of the local model of client i while retaining the global model information. After the global model is transmitted to the client, it will be multiplied by the semi-mask matrix The formula is as follows:
[0056]
[0057] Among them, is part A of the global network, is the local network obtained by client i after the t-th round of aggregation.
[0058] Figures 2 to 4 Shows the accuracy of the algorithm of this embodiment and other methods on three datasets.
[0059] The method LSFel of this embodiment has achieved the best results on three datasets, MNIST, EMNIST, and CIFAR-10. As the number of communication rounds increases, the accuracy of the algorithm in this embodiment gradually improves. More importantly, the algorithm in this embodiment requires very few communication rounds to obtain significant results.
[0060] The accuracies of the lightweight local model and the global model under global models of different sizes are shown in Table 1. Table 1 shows the accuracy of the global network after 60 communication rounds. The global network is of the ResNet architecture, with 20 to 60 layers, 80 layers, 100 layers, and 120 layers respectively. The number of layers in the lightweight network is set to not exceed half of the number of layers in the global network. It can be observed that as the size of the global model increases, the accuracy of the lightweight local model is still very close to that of the global model. This means that with appropriate adjustment, the lightweight model also has good usability and stability.
[0061] Table 1 Model accuracies under different global model sizes
[0062] Dateset-Model 20-layer 30-layer 40-layer 50-layer 60-layer 80-layer 100-layer 120-layer Mnist-A 90.25±0.18 92.05±0.21 92.78±0.19 93.24±0.21 3.73±0.20 94.05±0.19 94.54±0.17 4.82±0.18 Mnist-AB 91.34±0.20 93.13±0.22 93.11±0.18 93.65±0.20 94.12±0.17 94.53±0.17 94.81±0.17 95.34±0.15 EMnist-A 98.12±0.07 98.51±0.08 98.66±0.04 98.72±0.04 98.77±0.03 98.77±0.04 98.78±0.04 98.79±0.03 EMnist-AB 98.33±0.11 98.64±0.09 98.71±0.03 98.76±0.04 98.78±0.03 98.80±0.04 98.82±0.04 98.83±0.03 CIFAR-10-A 47.86±0.31 51.85±0.22 52.24±0.27 53.13±0.28 54.98±0.25 55.18±0.21 55.73±0.26 56.33±0.24 CIFAR-10-AB 49.25±0.43 52.23±0.27 53.55±0.33 54.25±0.34 55.12±0.23 55.34±0.22 55.89±0.23 56.49±0.21
[0063] The specific implementation process of this embodiment includes:
[0064] (1) The client trains the local model and generates pseudo vectors. The local model is set as a lightweight model.
[0065] (2) A small amount of data and the lightweight local model are transmitted to the edge server.
[0066] (3) The edge server uses the lightweight model as the first half of the global model to construct a larger new global model. The pseudo vector data uploaded by the client is used for training to obtain a more powerful global model. This global model can be regarded as consisting of part A and part B, where part A is the lightweight local model obtained from the client.
[0067] (4) After multiple rounds of training of the global model, the lightweight model part A is distributed to each client, while part B remains on the edge server.
[0068] Implementably, this embodiment provides a federated edge learning model training system, including:
[0069] A pseudo vector generation module, configured to enable a plurality of clients to train local models and generate pseudo vectors, where the local models are lightweight models;
[0070] The global model training module transmits the pseudo vectors generated by each client and the model parameters of the local model to the edge server; constructs and trains a global model based on the pseudo vectors and model parameters of each client in the edge server, and distributes some parameters of the trained global model to each client based on a preset partitioning rule.
[0071] Implementable, this embodiment provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the described federated edge learning model training method.
[0072] Implementable, this embodiment provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the described federated edge learning model training method.
[0073] As described above, only the preferred specific implementation manner of this application is provided, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by this application should be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.
Claims
1. A method for training a federated edge learning model, characterized in that Including: Enabling a number of clients to train local models and generate pseudo-vectors, where the local models are lightweight models; Transmitting the pseudo-vectors generated by each client and the model parameters of the local models to the edge server; Building and training a global model in the edge server based on the pseudo-vectors and model parameters of each client, and distributing some parameters of the trained global model to each client based on a preset partitioning rule.
2. The method for training a federated edge learning model according to claim 1, wherein The process of generating the pseudo-vectors specifically includes: For the local dataset of each client, dividing a number of classes of data by category, converting the data of each category into vectors based on an embedding function, and linearly combining the embedding vectors of different categories to obtain pseudo-vectors.
3. The method for training a federated edge learning model according to claim 1, wherein Transmitting the pseudo-vectors generated by each client and the model parameters of the local models to the edge server specifically includes: Defining a mask matrix to perform model pruning on the local model, and dynamically updating the mask matrix during the pruning process until the target pruning rate is met, to obtain the pruned local model; Transmitting the pseudo-vectors corresponding to each client and the model parameters of the pruned local model to the edge server.
4. A method for training a federated edge learning model according to claim 1, wherein The process of building and training the global model specifically includes: Aggregating the model parameters uploaded by each client as the first part of the model parameters of the global model, randomly initializing to generate the second part of the model parameters, and building the global model based on the first part of the model parameters and the second part of the model parameters; Inputting the pseudo-vectors of each client into the global model and training based on the target loss function to obtain the trained global model; Distributing some parameters of the trained global model to each client based on a preset partitioning rule.
5. The method for training a federated edge learning model according to claim 4, wherein, The distributing some parameters of the trained global model to each client based on a preset partitioning rule specifically includes: Distributing the updated first part of the model parameters in the trained global model to each client, and retaining the second part of the model parameters in the edge server; After a preset number of rounds of communication, switching the mask matrix of the client to a semi-mask matrix. After the client receives the distributed model parameters, fusing the local model parameters and the distributed model parameters through the semi-mask matrix.
6. A federated edge learning model training system, characterized in that Including: A pseudo-vector generation module for enabling a number of clients to train local models and generate pseudo-vectors, where the local models are lightweight models; A global model training module that transmits the pseudo-vectors generated by each client and the model parameters of the local models to the edge server; builds and trains a global model in the edge server based on the pseudo-vectors and model parameters of each client, and distributes some parameters of the trained global model to each client based on a preset partitioning rule.
7. An electronic device, characterized in that, Including a memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a federated edge learning model training method according to any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by the processor, it implements a federated edge learning model training method according to any one of claims 1-5.