A federated learning system and method based on model pruning and transmission compression optimization

Abstract

The application discloses a kind of federal learning system and method based on model pruning and transmission compression combined optimization, the system includes server end and multiple clients;Method is: first, client washes data and is normalized into structured data;Then, server end prunes original global model into sparse global model, and is issued to client;Client calculates sparse global model weight using local data set, and uploads to server end after sparse global model weight data is compressed;Server end uses sparse global model weight data to update sparse global model, then is issued to client again and carries out local training;Sparse global model and sparse global model weight data update are repeatedly carried out until sparse global model converges, original global model is trained using final data, and final global prediction model trained according to the data of each client is obtained.The application reduces the complexity of client training model, and reduces communication cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to a federated learning system and method based on model pruning and transmission compression optimization. BACKGROUND

[0002] In recent years, machine learning technology is increasingly applied in daily life, such as smart home, smart retail, autonomous driving, etc. However, advanced artificial intelligence systems usually require large labeled data sets, a large amount of computing resources, numerous experts, etc. for training and reasoning. Due to data privacy regulations and communication bandwidth limitations, it is often not feasible to transmit and store all training data at a central location. Therefore, federated learning has become a new distributed learning paradigm, which enables multiple clients to collaboratively learn a global model without sharing local data. Although federated learning can keep data locally, it alleviates the heavy burden of transmitting training data in the traditional paradigm, but the client devices in federated learning are usually resource-constrained, such as computing power, communication bandwidth, memory, and storage size, etc. Therefore, it is necessary to train a usable federated learning model to adapt to various situations, such as smart healthcare and service recommendation scenarios, through collaborative resource-limited edge devices.

[0003] Therefore, researchers have proposed model gradient compression techniques that do not upload models with complete parameter values, but extract a small model from the original model for transmission during the fusion stage. Model gradient compression techniques include quantization techniques and sparsification techniques. The commonly used method of quantization technique is the bucket quantization method with various modes, i.e. random quantization, count sketch, and quantile bucket quantization. Quantization techniques have a better trade-off between communication efficiency and learning performance. Sparsification techniques are classic methods to reduce the number of bits transmitted by client devices, which only transmit a small part of the important elements of the upload vector. However, these methods can only reduce communication overhead and cannot generate small models for efficient computation.

[0004] The sparsity of deep neural networks (DNN), such as model pruning, is another option to improve communication overhead and computational complexity. Researchers have proposed an improved sparse evolutionary training algorithm to reduce the connections of neural networks in federated learning, thereby indirectly reducing the number of model parameters to be sent to the server. However, this sparsification method can only be applied to fully connected layers, and the application range is limited. Because the pruned model of a single client depends on the data distribution of the client, it cannot be applied to clients with heterogeneous data statistics. Existing work aims to design a personalized and communication-efficient federated learning framework by exploiting the lottery ticket hypothesis framework. Therefore, how to reduce the model size on the server side to achieve the most efficient training and further compress the transmission of client devices is still a problem that needs to be solved in federated learning. SUMMARY

[0005] The purpose of this invention is to provide a federated learning system and method based on model pruning and transmission compression optimization, which has low client-side training model complexity and low communication cost.

[0006] The technical solution to achieve the purpose of this invention is: a federated learning system based on model pruning and transport compression optimization, including a server and multiple clients;

[0007] The server is used to perform basic operations, establish and initialize the global model, and accept global model parameters uploaded by various user terminals.

[0008] The client is used for data cleaning and normalization, model weight data calculation, model weight data compression, and model parameter reception and uploading.

[0009] A federated learning method based on model pruning and transport compression optimization includes the following steps:

[0010] Step 1: The client cleans the data and normalizes it into structured data;

[0011] Step 2: The server-side trims the original global model to obtain a sparse global model;

[0012] Step 3: The server distributes the sparse global model to the client;

[0013] Step 4: After receiving the sparse global model, the client runs the optimization method on the local dataset. After several local iterations, it obtains the locally updated sparse global model weights. The client then applies transport quantization and transport sparsification to compress the sparse global model weight data before uploading it to the server.

[0014] Step 5: The server compresses and aggregates the received sparse global model weight data, updates the sparse global model, and then sends it to the client for local training.

[0015] Step 6: Repeat steps 4 and 5 until the sparse global model converges. Use the final data to train the original global model to obtain the final global prediction model trained based on the data of each client.

[0016] Compared with the prior art, the significant advantages of this invention are: (1) trimming the unimportant model parameters in the original global model on the server side, reducing the complexity of training the model on the client side; (2) compressing the sparse global model weight data using transmission quantization and transmission sparsification, reducing communication costs. Attached Figure Description

[0017] Fig. 1This is a flowchart illustrating the federated learning method based on model pruning and transport compression optimization of this invention.

[0018] Fig. 2 This is a system schematic diagram of the model training process in an embodiment of the present invention.

[0019] Fig. 3 This is a performance comparison curve between the algorithm of the present invention and the traditional algorithm in the embodiments of the present invention. Detailed Implementation

[0020] Combination Figs. 1-2 This invention discloses a federated learning system based on model pruning and transport compression optimization, comprising a server and multiple clients;

[0021] The server is used to perform basic operations, establish and initialize the global model, and accept global model parameters uploaded by various user terminals.

[0022] The client is used for data cleaning and normalization, model weight data calculation, model weight data compression, and model parameter reception and uploading.

[0023] Furthermore, in the federated learning system based on model pruning and transport compression optimization, the number of clients is two or more.

[0024] This invention discloses a federated learning method based on model pruning and transport compression optimization, comprising the following steps:

[0025] Step 1: The client cleans the data and normalizes it into structured data;

[0026] Step 2: The server-side trims the original global model to obtain a sparse global model;

[0027] Step 3: The server distributes the sparse global model to the client;

[0028] Step 4: After receiving the sparse global model, the client runs the optimization method on the local dataset. After several local iterations, it obtains the locally updated sparse global model weights. The client then applies transport quantization and transport sparsification to compress the sparse global model weight data before uploading it to the server.

[0029] Step 5: The server compresses and aggregates the received sparse global model weight data, updates the sparse global model, and then sends it to the client for local training.

[0030] Step 6: Repeat steps 4 and 5 until the sparse global model converges. Use the final data to train the original global model to obtain the final global prediction model trained based on the data of each client.

[0031] Furthermore, the client-side data cleaning and normalization into structured data described in step 1 is as follows:

[0032] Each client uses natural language processing and image feature extraction techniques to clean the collected data and normalize it to form structured data.

[0033] Furthermore, in step 2, the server-side trims the original global model to obtain a sparse global model, as detailed below:

[0034] Applying the Taylor expansion to the objective function of the original global model, we obtain:

[0035]

[0036] Where F() represents the loss function, This represents the loss function F(w) using the reshaped model weights. t () represents the loss function of the original model, ignoring higher-order terms in the expansion:

[0037]

[0038] Use cumulative updates to estimate the momentum of the global gradient:

[0039] Δu t+1 =βΔu t +(1-β)Δw t+1

[0040] Where β is the momentum rate, Δu 0 It is the zero vector, Δu t+1 This represents the cumulative update vector from round 0 to round t+1, from which the removal of [w] is obtained. t ] j Utility cost:

[0041] [T(w t ,Δu t )] j =|[(Δu t ) T w t ] j |

[0042] Among them, [w t ] j This represents the weight of the j-th parameter in the model; [T(w t ,Δu t )] j This indicates the removal of parameter [w] t ] j The utility cost, therefore retaining [T(w t ,Δu t )]j [w] greater than the set value t ] j The impact of updating the setpoint (Δu) on model performance is less than that of updating the setpoint. t ) T Reset to zero.

[0043] Furthermore, the sparse global model described in step 2 is as follows:

[0044] Define a mask function to generate a binary mask vector:

[0045]

[0046] Where T s (w t ,Δu t ) is the threshold controlling the construction rate p. The sparse training process on the client side is represented as:

[0047]

[0048] Where i∈{1,2,…,U} Indicates the local iteration time. This indicates that the i-th client has completed the t-th communication. The model weights are obtained from the first local iteration, where η represents the learning rate. It is the average gradient vector of a batch of samples, ⊙ represents the element-wise multiplication process, and the local update is represented as:

[0049]

[0050] in, This represents the model weights in the local iteration τ. This represents the model weights that were not trained locally.

[0051] Furthermore, the transmission quantization described in step 4 is to perform non-uniform quantization on the unpruned weights of all layers and all channels of the sparse global model, and all parameters to be uploaded share a common quantization strategy and quantization assignment in the same round of communication.

[0052] Will As a cluster, v i,u As a parameter Falling The quantization value in the code is represented as:

[0053]

[0054] in, Indicates the quantification strategy, This represents the communication choice made by the i-th user in the t-th round. A cluster of quantification strategies;

[0055] Select v i,u Value and Clustering, construct the following optimization problem:

[0056]

[0057]

[0058]

[0059]

[0060]

[0061] The compressed value uses 32-bit floating-point numbers. This represents the cluster of communication for the i-th user in the t-th round. This represents the quantized value that falls within the cluster. It is a transmission capacity constraint, determined by the number of quantization strategies. and the original model size Control, this optimization is a mixed-integer nonlinear programming problem, and a suboptimal solution is achieved through the following algorithm:

[0062]

[0063] in, This represents the optimal solution. This represents the quantized value of the communication of the i-th user in the t-th round that falls within u′.

[0064] Furthermore, the transmission sparsity described in step 4 is to reduce the number of transmission bits by retaining only a portion of the transmission values, specifically by using differential coding to further reduce communication overhead.

[0065] Furthermore, the differential coding is specifically as follows:

[0066] The transmission cost caused by transmission sparsity is expressed as:

[0067]

[0068] in yes The set of keys that retain values ​​in r s It is an index of the reserved value;

[0069] Analysis of the sparse update data distribution reveals three characteristics:

[0070] First, the keys are unique;

[0071] Secondly, the keys are arranged in ascending order;

[0072] Third, the difference between two adjacent bonds is smaller than the bond itself;

[0073] Therefore, calculating the difference between two adjacent keys and storing only the key increment to reduce transmission costs is called differential coding;

[0074] The transmission cost using differential coding is expressed as in This represents the maximum differential key, i.e.:

[0075]

[0076] in This represents the s-th difference key, where r0 = 0 indicates that the index of no retained value is 0;

[0077] Based on differential coding, construct the following optimization problem for transmission sparsity:

[0078]

[0079]

[0080] In this optimization problem, based on loss reduction Select the retain set Then recalculate the maximum difference key and the tolerable size of the retention set with transmission capacity constraints; perform these two steps in sequence until the maximum difference key remains unchanged.

[0081] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0082] Example

[0083] This embodiment uses a federated learning method based on model pruning and transport compression optimization to train a model for the data silo problem in the medical field.

[0084] The problem of data silos in the medical field is a challenging yet highly valuable one. Federated learning stipulates that all hospitals have consistent feature inputs, constructing a disease label set and feature set based on the same data standard. A unified model is then built upon this feature standard. This technology can integrate data from multiple hospitals for joint training without data leakage and can be applied to the prediction of the incidence of major chronic diseases. The specific process is as follows:

[0085] Step 1: Each participating hospital uses artificial intelligence technologies such as natural language processing and image feature extraction to clean the patient data collected from the hospital and normalize it to form structured medical data.

[0086] Step 2: The server randomly initializes the global model parameters and prunes the unimportant parameters in the model. Specifically, the Terra expansion is applied to the model, the higher-order terms in the expansion are ignored, and the cumulative update is used to estimate the global gradient of the client dataset. Through this estimation, parameters that have little impact on model performance are removed, while parameters that have a high utility cost to the model are retained.

[0087] Step 3: The server distributes the model to each participating hospital;

[0088] Step 4: Each hospital's client continuously updates the model's accuracy using a local dataset and a local stochastic gradient descent optimization method. When the predetermined number of local training iterations is reached, the model's parameters are compressed. Specifically, non-uniform quantization and differential encoding are performed on the unpruned weights of all layers and all channels of the model: all channels of a layer share a common quantization strategy and quantized bins / centers during communication; the difference between two adjacent keys is calculated, and only the key increment is stored to reduce transmission costs.

[0089] Step 5: The server receives the model uploaded by each hospital, compresses it, aggregates and updates the global model, and then sends it back to the client for local training.

[0090] Step 6: Repeat steps 4 and 5 until the global model converges, resulting in a disease prediction model trained based on data from each participating hospital.

[0091] Fig. 3 This figure shows the performance of a federated learning framework based on model pruning and transport compression optimization. As can be seen from the figure, traditional federated learning frameworks suffer from complex structures and meaningless layers in the global model, wasting significant computational power and incurring huge communication overhead during transmission. This invention proposes a federated learning training method based on model pruning and transport compression optimization. This method prunes unimportant model parameters and compresses the transmission process of model parameters. The joint optimization of these two methods reduces computational power consumption and communication overhead while maintaining model performance, thus improving the efficiency of federated learning.

Claims

1. A federated learning method based on model pruning and transport compression optimization, characterized in that, Includes the following steps: Step 1: The client cleans the data and normalizes it into structured data; Step 2: The server-side trims the original global model to obtain a sparse global model; Step 3: The server distributes the sparse global model to the client; Step 4: After receiving the sparse global model, the client runs the optimization method on the local dataset. After several local iterations, it obtains the locally updated sparse global model weights. The client then applies transport quantization and transport sparsification to compress the sparse global model weight data before uploading it to the server. The aforementioned transmission quantization involves non-uniform quantization of the unpruned weights of all layers and channels in the sparse global model. All parameters to be uploaded share a common quantization strategy and quantization assignment during the same round of communication. Will As a cluster, As a parameter Falling The quantization value in the code is represented as: in, Indicates the quantification strategy, This represents the communication choice made by the i-th user in the t-th round. A cluster of quantization strategies of -1; choose Value and Clustering, construct the following optimization problem: The compressed value uses 32-bit floating-point numbers. This represents the cluster of communication for the i-th user in the t-th round. This represents the quantized value that falls within the cluster. It is a transmission capacity constraint, determined by the number of quantization strategies. and the original model size Control, this optimization is a mixed-integer nonlinear programming problem, and a suboptimal solution is achieved through the following algorithm: in, This represents the optimal solution. This indicates that the communication of the i-th user in the t-th round falls on Quantization value in; Step 5: The server compresses and aggregates the received sparse global model weight data, updates the sparse global model, and then sends it to the client for local training. Step 6: Repeat steps 4 and 5 until the sparse global model converges. Use the final data to train the original global model to obtain the final global prediction model trained based on the data of each client.

2. The federated learning method based on model pruning and transport compression optimization according to claim 1, characterized in that, Step 1 involves cleaning and normalizing the client-side data into structured data, as detailed below: Each client uses natural language processing and image feature extraction techniques to clean the collected data and normalize it to form structured data.

3. The federated learning method based on model pruning and transport compression optimization according to claim 1, characterized in that, Step 2 describes the server-side pruning of the original global model to obtain a sparse global model, as detailed below: Applying the Taylor expansion to the objective function of the original global model, we obtain: Where F() represents the loss function, This indicates the loss function used with the reshaped model weights. This represents the loss function of the original model, ignoring higher-order terms in the expansion: Use cumulative updates to estimate the momentum of the global gradient: in, It is momentum rate. It is the zero vector. This represents the cumulative update vector from round 0 to round t+1, from which the removal of [ ] j Utility cost: in, This represents the weight of the j-th parameter in the model; Indicates the removal of parameters The utility cost, therefore retention makes Greater than the set value Updates with less impact on model performance than the set value Reset to zero.

4. The federated learning method based on model pruning and transport compression optimization according to claim 1, characterized in that, The sparse global model described in step 2 is as follows: Define a mask function to generate a binary mask vector: in This is the threshold for controlling the construction rate p. The sparse training process on the client side is represented as: in, , Indicates the local iteration time. This indicates that the i-th client has completed the t-th communication. The model weights obtained from the next local iteration. Indicates the learning rate. It is the average gradient vector of a batch of samples. The element-wise multiplication process is represented as follows: in, Indicates local iteration Model weights of the order This represents the model weights that were not trained locally.

5. The federated learning method based on model pruning and transport compression optimization according to claim 1, characterized in that, The transmission sparsity described in step 4 is to reduce the number of transmitted bits by retaining only a portion of the transmitted values, specifically by using differential coding to further reduce communication overhead.

6. The federated learning method based on model pruning and transport compression optimization according to claim 5, characterized in that, The differential encoding is as follows: The transmission cost caused by transmission sparsity is expressed as: in yes The set of keys that retain values ​​in the middle. It is an index of the reserved value; Analysis of the sparse update data distribution reveals three characteristics: First, the keys are unique; Secondly, the keys are arranged in ascending order; Third, the difference between two adjacent bonds is smaller than the bond itself; Therefore, calculating the difference between two adjacent keys and storing only the key increment to reduce transmission costs is called differential coding; The transmission cost using differential coding is expressed as ,in This represents the maximum differential key, i.e.: in Indicates the first One difference key, The index indicating no retained values ​​is 0; Based on differential coding, construct the following optimization problem for transmission sparsity: In this optimization problem, based on loss reduction Select the retain set Then recalculate the maximum difference key and the tolerable size of the retention set with transmission capacity constraints; perform these two steps in sequence until the maximum difference key remains unchanged.

7. A federated learning system based on model pruning and transport compression optimization, characterized in that, The system is used to implement the federated learning method based on model pruning and transport compression optimization as described in any one of claims 1 to 6, and the system includes a server and multiple clients; The server is used to perform basic operations, establish and initialize the global model, and accept global model parameters uploaded by various user terminals. The client is used for data cleaning and normalization, model weight data calculation, model weight data compression, and model parameter reception and uploading.

8. The federated learning system based on model pruning and transport compression optimization according to claim 7, characterized in that, The number of clients is two or more.

Images