A dynamic gradient compression learning method of a federated learning system

Abstract

The application relates to a dynamic gradient compression learning method of a federal learning system and belongs to the field of artificial intelligence. The method comprises the following steps: constructing a federal learning model, a server distributing different sampling communication bandwidths to clients, and the clients obtaining local models after training; considering resource constraints and gradient states of each client local model, calculating a pruning rate and a quantization level of a dynamic gradient compression strategy in this round; according to the pruning rate, each client performs a sparse operation on the gradient of the client to obtain a sparse gradient; according to the quantization level, each client performs a quantization and encoding operation on the sparse gradient to obtain compressed gradient data; each client uploads the compressed gradient data to the server, the server decompresses the gradient data to obtain decompressed gradient data; and the server aggregates the decompressed gradient data according to elements and updates a global model until the model converges. The application realizes efficient communication federal learning with better performance in a resource-limited scene.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a dynamic gradient compression learning method for federated learning systems. Background Technology

[0002] Existing research has demonstrated that gradient compression methods can overcome the deployment challenges of federated learning in resource-constrained scenarios by reducing the number of model parameters and lowering the computational complexity of model training. However, most existing studies employ static gradient compression methods, simply applying a fixed gradient compression strategy to different clients across all global model training epochs. This traditional approach has significant drawbacks:

[0003] 1. Static compression methods cannot adapt to dynamic changes in model performance: Static gradient compression methods cannot adapt to the needs of dynamic changes in the model during training, which can easily lead to excessive gradient compression in the early stage, which damages the model's convergence or redundant training parameters in the later stage, thus wasting training resources.

[0004] 2. The heterogeneity of device resources was ignored: The gradient compression strategy corresponding to the performance requirements of different client devices with different resource differences has the ability to be dynamically adjusted. Summary of the Invention

[0005] This application provides a dynamic gradient compression learning method for federated learning systems, which can solve the problem that static compression cannot adapt to dynamic changes in training and differences in client resources, and achieves more efficient and flexible gradient compression.

[0006] This application discloses a dynamic gradient compression learning method for a federated learning system, wherein the federated learning system includes a server and... Each client has a local model shared by all clients, which is aggregated by the server, and different communication bandwidths are allocated to different clients. Each client communicates with the server via a wireless channel. The method includes the following steps:

[0007] S1. Construct the learning model of the federated learning system. The server allocates different sampling communication bandwidths to each client. The client obtains a local model after training based on the built-in local dataset.

[0008] S2. Considering the resource constraints and model gradient state of each client's local model, determine and calculate the pruning rate and quantization level of the dynamic gradient compression strategy in this round.

[0009] S3. Based on the pruning rate obtained in S2, each client performs a sparsification operation on its own gradient to obtain the sparsified gradient.

[0010] S4. Based on the quantization level obtained in S2, each client performs quantization and encoding operations on the sparsified gradient obtained in S3 to obtain compressed gradient data.

[0011] S5. Each client uploads the compressed gradient data to the server. The server decompresses the compressed gradient data to obtain the decompressed gradient data.

[0012] S6. The server aggregates the decompressed gradient data element by element and updates the global model;

[0013] S7. Repeat S1 to S6 until the global model converges.

[0014] Furthermore, S1 includes:

[0015] Build a federated learning model, with the server aggregating all clients. Shared local model parameters, in Independent random sampling within the bandwidth range and allocation of different sampling communication bandwidths For each client Following a continuous uniform distribution ;

[0016] The client It has a built-in local dataset and communicates with the server via wireless channels;

[0017] In each round of global model training, each client The local model is trained based on a pre-set local dataset, and the local model is obtained.

[0018] Furthermore, S2 includes:

[0019] S21. Calculate the average of the absolute values ​​of non-zero elements in the gradients of each layer of the local model. ;

[0020] S22, The average value described in S21 Convert to sparsity index ,in For the first The number of non-zero elements in the layer gradient. For the first The total number of elements in the layer gradient;

[0021] S23, The sparsity index described in S21 Normalization yields importance indicators ;

[0022] S24. Dynamic decision gradient compression strategy for the aforementioned importance indicators. The median value of the pruning rate and the median value of the quantification level are dynamically adjusted by mapping;

[0023] S25, Consider each client If the resource and energy consumption and gradient propagation delay meet the preset constraints, adjust the gradient adjustment strategy, and finally obtain the pruning rate. and quantification level .

[0024] Furthermore, in the gradient adjustment strategy, the quantization level has a higher priority than the pruning rate. When the constraint requirements are not met, the quantization level is reduced first to reduce communication overhead, and then the pruning rate is increased to reduce the amount of computation. Only when the quantization level reaches the minimum is the pruning rate increased.

[0025] The preset constraint requirement is as follows:

[0026]

[0027]

[0028] in To calculate energy consumption, For communication energy consumption, To calculate the delay, Due to communication delay, For the client Preset maximum energy consumption limit The maximum tolerable latency for each round of global model training is preset for federated learning.

[0029] Furthermore, S3 includes:

[0030] S31, Client After completing the local model training, obtain the parameter weights of the convolutional layers of the local model. and local model gradient increment ;

[0031] S32. Calculate the L2 norm of the convolution kernels of each layer in the local model. The L2 norm Expressed in matrix form, the norm matrix is ​​obtained. ;

[0032] S33. Select and set the pruning rate using probability-driven kernel selection. The range of pruning thresholds is obtained ;

[0033] S34. The L2 norm described in S32 The pruning threshold described in S33 Compare and generate a mask matrix. ;

[0034] S35, the mask matrix With the weight matrix of the original convolutional layer Multiply the elements one by one and retain the mask matrix. The value above corresponds to the original convolutional layer weight matrix. The elements of the mask matrix are removed. The value above 0 corresponds to the original convolutional layer weight matrix. The elements are used to obtain the pruned weight matrix. ;

[0035] S36. The pruned weight matrix retains non-zero convolution kernels and removes zero convolution kernels, finally obtaining a sparse weight matrix. .

[0036] Furthermore, based on the L2 norm in each convolutional kernel of the convolutional layer... The values ​​are set such that the mask positions corresponding to convolution kernels with values ​​not less than a threshold are set to 1, and those with values ​​less than a threshold are set to 0. All masks are then merged to generate a mask matrix. mask matrix Expressed as a formula .

[0037] Furthermore, S4 includes:

[0038] S41. Save the sparsified weight matrices respectively. The sign matrix is ​​obtained by combining the sign information and value of each weight value. sum numerical matrix The symbol matrix and the numerical matrix Expressed as a formula:

[0039]

[0040]

[0041] For the symbol matrix For elements less than 0, decrement by 1; for elements greater than or equal to 0, decrement by 1. For the numerical matrix... The maximum value of the numerical matrix is ​​obtained by taking the absolute value of all elements within it. Minimum value of the sum of numerical matrices ;

[0042] S42. The maximum value of the numerical matrix as described in S41 Minimum value of the sum of numerical matrices For numerical matrices The weight matrix is ​​divided into several quantization levels, each corresponding to a quantization interval, and the sparsed weight matrix is... Each weight value in the algorithm is assigned to a corresponding quantization level based on its numerical value.

[0043] S43. For the quantization interval described in S42, random quantization is performed using the following quantization strategy to obtain the quantized index matrix. :

[0044]

[0045]

[0046] in For the first The size of each interval in the layer, For the weight value to fall within The probability within the interval, said probability is based on It is calculated based on the distance from the boundary of the quantization interval;

[0047] S44, the index matrix after quantization in S43 The frequency of occurrence of all characters is statistically analyzed to evaluate the relative importance of each index. The statistical results are used to construct a Huffman tree. Each index is assigned a corresponding binary code according to its frequency, resulting in Huffman coding and Huffman code table.

[0048] S45. Replace the quantized index matrix with the Huffman coding. The encoded index matrix is ​​obtained. ;

[0049] The compressed gradient data is an encoded index matrix. Other auxiliary decompression data, including the minimum value of the numerical matrix. Symbol matrix Mask matrix and Huffman code table.

[0050] Furthermore, S5 includes:

[0051] S51. Encoded index matrix corresponding to the client device Huffman decoding is performed using a Huffman code table to obtain the decoded index matrix. The decoded index matrix With the quantized index matrix They are the same matrix;

[0052] S52, the decoded index matrix The weight values ​​are mapped to the corresponding quantization intervals through the symbol matrix. Obtain the weight matrix for recovering symbolic information The weight matrix for recovering symbolic information With the sparse weight matrix They are the same matrix;

[0053] S53, the position of the zero-value convolution kernel is determined by the mask matrix. Restore the parameters to zero, retaining only the positions of non-zero convolution kernels, to obtain the increments of model parameters after quantization sparsity. ;

[0054] The server decompresses the compressed data uploaded by all client devices using the above process to obtain decompressed data, which consists of the quantized and sparsed model parameter increments from all clients. ,in This represents the number of clients.

[0055] Furthermore, the server receives compressed incremental data, which includes quantization error and encoding noise, and is processed through a mask matrix. Explicitly restoring to zero values ​​ensures parameter increments after decompression. It is completely consistent with the client's original pruning structure, avoiding the occurrence of tiny values ​​in positions that should be zero due to potential errors in the transmission or calculation process, such as floating-point rounding.

[0056] Furthermore, step S6 includes:

[0057] make For the client The uploaded weight increments after sparse quantization, where The first convolutional layer The increment of each convolutional kernel weight, after aggregation, is expressed as the global model update gradient increment. To obtain the global model after the current round of updates. Update the global model parameters until the global model converges. During aggregation, only non-zero convolution kernels are considered.

[0058] Compared to existing technologies, the advantages of this application are:

[0059] This application requires measuring the sparsity of gradient data generated by different clients in each global model training round, constructing a gradient importance index from the gradient sparsity index to decide on different gradient compression strategies. Through this more refined and adaptive gradient compression strategy, communication overhead can be further reduced, and the gradient compression strategies of different clients can be adaptively adjusted to ensure the final model training effect, ultimately achieving better performance and more efficient federated learning in resource-constrained scenarios. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0061] Figure 1 Here is a flowchart of a federated learning method with dynamic gradient compression according to this application;

[0062] Figure 2 A model diagram of the federated learning gradient compression system provided in this application;

[0063] Figure 3 This is a schematic diagram of weight sparsity provided in this application;

[0064] Figure 4 This is a schematic diagram of weight quantization provided in this application;

[0065] Figure 5 This is a schematic diagram illustrating the aggregation of models provided in this application;

[0066] Figure 6 The flowchart for implementing gradient dynamic compression federated learning provided in this application is as follows;

[0067] Figure 7 The simulation performance comparison chart is provided in this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] The gradient compression strategy involved in this application includes sparsity, quantization, and encoding. In each global model training round, the gradient data (also called "weights") trained on the local model is obtained. First, the gradient sparsity index is obtained by calculating the proportion of zero values ​​in the gradients of each client's local model. The gradient importance index is then calculated using the gradient sparsity index, with a larger value indicating a more important gradient. Then, the current gradient compression strategy is dynamically adjusted based on the gradient importance index, including: sparsity by setting some weights to zero to reduce the number of non-zero weights, where a low pruning rate is used for gradients with high gradient importance and a high pruning rate is used for gradients with low gradient importance; quantization converts weights and activation values ​​from high-precision floating-point numbers to low-precision format, further reducing the storage space required for each weight and accelerating computation, where high-precision quantization is used for gradients with high gradient importance and low-precision quantization is used for gradients with low gradient importance; finally, encoding further compresses the sparsified and quantized data using a specific compression algorithm based on data redundancy. In this process, sparsification, quantization, and encoding are performed sequentially. This dynamic compression strategy, while ensuring the integrity of key gradient information, minimizes the communication overhead of the entire federated learning process and maintains the performance of the trained model. Figure 1 As shown, there are a total of 7 steps, and the specific steps are as follows:

[0070] S1. Construct a system model for federated learning, such as Figure 2 As shown, it includes a server and Each client executes parameters of a local model shared by all clients, aggregated by the server, and independently and randomly samples within a bandwidth range, allocating different sampling communication bandwidths to different clients, following a continuous uniform distribution. The local dataset consists entirely of image data, for each client. Having its own local dataset Each client trains its own local model and communicates with the server via ordinary wireless channels; in each round of global model training, each client... Requires based on your own local dataset Complete the local model training and obtain the local model.

[0071] S2. To determine the key parameters of this round of gradient compression strategy (such as pruning rate and quantization level), it is necessary to consider each client. Resource constraints and model gradient state (gradient sparsity and magnitude of non-zero elements) are considered. Based on these conditions, the pruning rate and quantization level are calculated and adjusted. Specifically, the following steps are included:

[0072] S21. Calculate the average of the absolute values ​​of the non-zero elements in the gradient: to measure the average value of each client. To determine the gradient sparsity of each layer in the local model, it is necessary to first calculate the average of the absolute values ​​of the non-zero elements in the gradients of each layer of the local model. :

[0073]

[0074] in, Indicates the first The number of non-zero elements in the layer gradient. Indicates the first The gradient of the first layer One non-zero element.

[0075] S22. Calculate the gradient sparsity index: For each client In the local model, for each layer's gradient, besides considering the number of non-zero elements, it's also necessary to pay attention to the magnitude of those non-zero elements to more accurately characterize their impact on the overall gradient update. Therefore, considering the influence of gradient sparsity and the magnitude of non-zero elements, we define the... The layer gradient sparsity index is:

[0076]

[0077] in, Indicates the first The total number of elements in the layer gradient. Quantified the first The strength of the overall impact of layer parameters on model performance The larger the value, the greater the impact of this layer's parameters on model updates, and the more critical they are in model updates. More parameters need to be retained to maintain model performance.

[0078] S23. Calculate the gradient importance index: for each client Normalize all gradient sparsity metrics to obtain gradient importance metrics for each layer of the local model after training:

[0079]

[0080] in, Indicates the total number of layers in the local model. It is a very small positive number used to prevent division by zero errors; obtained through calculation. It can measure the importance of each gradient layer. The larger the value, the greater the contribution of that layer to model training. Therefore, layers that are more important should be retained, while layers that are less important can be considered for removal.

[0081] S24. Dynamic Decision Gradient Compression Strategy: Based on the gradient importance index of each layer of the local model, an adaptive pruning rate and quantization level are assigned to each layer. Overall, it is required that... Layers with higher values ​​should retain more parameters, therefore a lower compression ratio is used. Layers with lower values ​​can use a higher compression ratio. Layer gradient pruning rate With quantification level The calculation method is as follows:

[0082]

[0083] in, This indicates rounding. This represents the upper limit of the pruning rate. This represents the lower limit of the pruning rate. This is the upper limit of the quantification level. This is the lower limit of the quantification level. for The maximum value, for The minimum value, and This is a constant used to control the gradient compression strength. For example, when... hour, Given the same value, compared to and When values ​​are the same, apply a larger pruning rate and a smaller quantification level.

[0084] S25. Adjust the gradient compression strategy as needed based on the client's resource constraints. In adjusting the gradient compression strategy, quantization level takes precedence over pruning rate. This considers each client... Resources include computing power Communication bandwidth and client battery level This requires ensuring that the client's energy consumption and gradient transmission latency during the entire local model training and gradient uploading process meet given constraints, specifically, they must be less than a set upper limit. When these constraints are not met, the quantization level can be reduced first to decrease communication overhead, and then the pruning rate can be increased to reduce computation.

[0085] During local model training, each client incurs communication, computational energy consumption, and latency during the process of obtaining and uploading model gradients. The gradient has weights for each layer with calculation formulas related to energy consumption and latency, as shown below:

[0086]

[0087]

[0088]

[0089]

[0090] in, To calculate energy consumption, For communication energy consumption, To calculate the delay, Due to communication delay, To calculate the energy consumption coefficient, This is the communication energy consumption coefficient. For the first The number of weight parameters for the layer; and Representing training and transmission respectively The computational and communication energy consumption generated by layer parameters; and Representing the first The corresponding computational and communication latency incurred by each layer. Consider the client. The following energy consumption and time delay constraints exist, as shown in the formula below:

[0091]

[0092]

[0093] in, Representing the client Preset maximum energy consumption limit; The maximum tolerable latency for each round of global model training is preset for federated learning. The pruning rate is obtained each time based on the adjustment strategy. and quantification level Afterwards, it is necessary to check whether the above constraints are met. If the parameters do not meet the constraints, adjust them according to priority. Only when the quantization level is the lowest should the pruning rate be increased.

[0094] 1. Reduce the quantization level (reduce communication overhead):

[0095]

[0096] 2. Increase pruning rate (reduce calculation workload):

[0097]

[0098] 3. Output the final pruning rate and quantification level.

[0099] S3. Based on the pruning rate obtained in S2, each client Perform a sparsification operation on its own gradient. (See reference for sparsification operation details.) Figure 3 As shown, the specific steps include:

[0100] S31. Calculate gradient increment: Any client The weights of the convolutional layer parameters in the local model are After completing the local model training, the weights of the convolutional layer parameters of the local model are: The corresponding model gradient increment is:

[0101]

[0102] in, Size is The sparsity operation employs a sparsity-based semi-structured pruning method, combined with a probability-driven kernel selection strategy to achieve sparsity of model parameters.

[0103] S32. Calculate the L2 norm and select important convolutional kernels: In order to locally select and retain important convolutional kernels within a layer, it is necessary to perform L2 norm calculations on each kernel. convolution kernel Calculate the L2 norm:

[0104]

[0105] in, convolution kernel The Middle line, number The weight value of the column. The larger the L2 norm, the greater the contribution (or influence) of the convolutional kernel to the model during feature extraction. Therefore, convolutional kernels with larger L2 norms should be retained; conversely, those with smaller L2 norms can be considered for pruning. Calculating the L2 norm of all convolutional kernels and stacking them yields the norm matrix. Size is .

[0106] S33. Determining the pruning threshold using probability-driven kernel selection: Probability-Driven Kernel Selection (PDKS) combines probability distribution characteristics with importance sampling. This is achieved by pre-setting the pruning rate. Scope, will As a quantile function Given a random input variable, the pruning threshold can be derived. This threshold can be expressed as:

[0107]

[0108] in, Here Represents the quantile position (e.g.) (representing the 30th percentile); It represents the infimum, ensuring the unique existence of the quantile; Represents random variables The cumulative distribution function in this method Importance filtering was achieved using quantiles.

[0109] S34. Generate a mask matrix for pruning: the pruning threshold is obtained. Then, based on the L2 norm values ​​of each convolutional kernel in the convolutional layer, the mask positions corresponding to convolutional kernels with values ​​not less than a threshold are set to 1, and those with values ​​less than a threshold are set to 0. Thus, a mask matrix can be generated using all the masks. Size is Generate a mask matrix. This can be expressed by a formula:

[0110]

[0111] S35. Prune the convolution kernel weights: The resulting mask matrix... Multiplying the corresponding elements of the weight matrix of the original convolutional layer, the mask matrix is ​​obtained. Elements with a value of 1 correspond to elements of the original convolutional layer weight matrix, which are preserved, while the mask matrix... Elements with a value of 0 are pruned from the original convolutional layer weight matrix.

[0112]

[0113] in, This represents the Hadamard product, which is the element-wise multiplication of matrices. Represented as the pruned weight matrix, with a size of .

[0114] S36. Further remove zero-value convolution kernels: Although the pruned weight matrix Compared to the original weight matrix, this is sparser, but some zero-value convolutional kernels remain unremoved in the previous steps. These kernels contribute nothing in the actual gradient iteration. To further reduce the amount of model data transmitted, only non-zero-value convolutional kernels can be retained, removing these zero-value parts to minimize the communication burden on model parameters during transmission. The final simplified weight matrix can be represented as follows: Size is .

[0115] S4. Based on the quantification level obtained in S2, each client The sparsed gradient obtained from S3 is then quantized and encoded. For the quantization operation, refer to... Figure 4 As shown, the specific steps include:

[0116] S41. Separate the sign bit and value bit: To facilitate subsequent quantization operations, they need to be saved separately. The sign information and value of each element are considered, because the quantized values ​​need to be restored to their original positive or negative sign information. The resulting sign matrix can be represented as:

[0117]

[0118] Through the By taking the absolute value of each element, the corresponding numerical matrix can be separated out:

[0119]

[0120] in, Represents the weight matrix The elements in for The maximum value, for The minimum value.

[0121] S42. Divide the quantization interval: Use the maximum value obtained earlier. and minimum value ,Will Divided into average There are 10 discrete quantization levels. Each quantization level corresponds to a quantization interval, and the size of each interval can be expressed as:

[0122]

[0123] Therefore, each sub-interval can be represented by a general expression: ,in, After the quantization intervals are divided, each weight value can be assigned to the corresponding quantization level based on its numerical value.

[0124] S43. Random Quantization: For a predefined quantization interval, the following quantization strategy is adopted:

[0125]

[0126]

[0127] in, For the weight value to fall within The probability within the interval. This probability is based on... It is calculated based on the distance from the boundary of the quantization interval.

[0128] S44. Generating the quantized index matrix: After random quantization, the weight matrix does not directly store the actual quantized values, but instead stores the quantization interval indices corresponding to the quantized values. For each weight... The corresponding quantization value is The quantization index matrix can be used express.

[0129] S45. Huffman Coding: Statistical analysis of the frequency of all characters is performed to assess the relative importance of each index; then, a Huffman tree is constructed based on the statistical results, and a corresponding binary code is assigned to each index according to its frequency; finally, the generated Huffman code replaces the quantized index matrix to obtain the encoded index matrix. .

[0130] S5, each client The compressed data is uploaded to the server, where the gradient data is decompressed. The specific steps include:

[0131] The compressed data is an encoded index matrix. And other auxiliary decompression data (minimum value of the numerical matrix) Symbol matrix Mask matrix (and Huffman code table), where the Huffman code table is obtained by uploading from the client.

[0132] S51, Huffman Decoding: Encoded index matrix for each client device. Huffman decoding is performed using a Huffman code table to obtain the index matrix. The index matrix With the quantized index matrix They are the same matrix;

[0133] S52. Quantization Decoding: Mapping the weights to the corresponding quantization intervals and recovering their positive and negative sign information using the sign matrix:

[0134]

[0135] The weight matrix for restoring positive and negative sign information With the sparse weight matrix They are the same matrix;

[0136] S53, Pruning and Decompression: Using a Mask Matrix This process restores the previously zeroed convolutional kernel positions to zero, retaining only the positions of non-zero convolutional kernels, ultimately yielding the quantized sparse model parameter increments. ,here .right The data uploaded by each client device is decompressed through the above process to obtain... .

[0137] The server receives compressed incremental data, which includes quantization error and encoding noise, and is processed through a mask matrix. Explicitly restoring to zero values ​​ensures parameter increments after decompression. It is completely consistent with the client's original pruning structure, avoiding the occurrence of tiny values ​​in positions that should be zero due to potential errors in the transmission or calculation process, such as floating-point rounding.

[0138] S6. The server aggregates the decompressed gradients element-wise and updates the global model, such as... Figure 5 As shown, the server aggregates the obtained model parameters element by element, and updates the global model parameters through aggregation. This "aggregation of parameters element by element" can be understood as averaging out the number of parameters in each layer that have been updated compared to the original global parameters. For example, assuming there are three client gradients with updates of "0", "e", and "g" in a certain layer, the original average aggregation would yield aggregated parameters "(0+e+g) / 3". After element-wise aggregation, the aggregated parameters are "(e+g) / 2". This can be understood as follows: because some convolutional kernels are set to zero during sparsification, the server should only consider those non-zero convolutional kernels during aggregation, meaning each parameter is aggregated based on the number of elements contributing to the update.

[0139] make Indicates client The uploaded weight increments after sparse quantization The first convolutional layer Each convolutional kernel weight increment, This is its corresponding mask matrix. The global model update gradient increment after aggregation can be expressed as... .

[0140]

[0141] Therefore, the global model after the current round of updates for:

[0142]

[0143] S7. Starting from the local training in S1 and ending in S6, repeat this process until the global model converges.

[0144] A flowchart for implementing sparse-driven dynamic compression federated learning for resource-constrained scenarios, such as... Figure 6 As shown below, the effectiveness of the method is demonstrated through specific experimental simulations. Consider a federated learning system with 10 clients. The training task is a classic CIFAR10 dataset classification task, with 50 global iterations, 5 local iterations, and a learning rate of 0.02. The CIFAR10 dataset consists of 60,000 32-bit images across 10 classes. The dataset consists of 32 color images, with 6000 images per category. 50,000 training images and 10,000 test images are selected, with a batch size of 128. The model chosen is the VGG11 convolutional neural network, defined with 9 layers (8 convolutional layers and 1 fully connected layer). Considering resource heterogeneity and client-side computation speed... Meeting the requirements of [1e6, 1e10] FLOPS, communication bandwidth The compression strategy satisfies [0.1, 1e3] Mbits / s, the maximum energy range of the client satisfies [10, 1000] Joules, the client computational energy consumption coefficient satisfies a uniform distribution of [0.5, 0.7] J / TFLOPs, and the client communication energy consumption coefficient satisfies [1e-9, 2e-7] J / bit. The initial pruning rate in the compression strategy is... It is 0.9. The initial quantization level is 0.5. It is 16. It is 4. It is 1.2. It is 1.2.

[0145] Simulation results are as follows Figure 7 As shown, information containing "dynamic" represents the results of the dynamic gradient compression method in this patent, while information containing "FedAvg" represents the results of the traditional federated learning method. The figure demonstrates that this method achieves gradient compression while sacrificing only 0.8% of model performance, significantly reducing communication overhead, saving approximately 97.79% of communication overhead compared to the FedAvg method.

[0146] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A dynamic gradient compression learning method for a federated learning system, characterized in that, The federated learning system includes one server and one client, the server aggregates local models shared by all clients, and assigns different communication bandwidths to different clients, each client communicates data with the server through a wireless channel, the method comprising the following steps: S1. Construct the learning model of the federated learning system. The server allocates different sampling communication bandwidths to each client. The client obtains a local model after training based on the built-in local dataset. S2. Considering the resource constraints and model gradient state of each client's local model, determine and calculate the pruning rate and quantization level of the dynamic gradient compression strategy in this round. Calculate the average of the absolute values ​​of the non-zero elements in the gradients of each layer of the local model. ; average Convert to sparsity index ,in For the first The number of non-zero elements in the layer gradient. For the first The total number of elements in the layer gradient; sparsity index Normalization yields importance indicators ; A dynamic decision gradient compression strategy is adopted for the importance indicators. The median values ​​of the pruning rate and quantization level are dynamically adjusted through mapping; each client is considered. If the resource and energy consumption and gradient propagation delay meet the preset constraints, adjust the gradient adjustment strategy, and finally obtain the pruning rate. and quantification level ; S3. Based on the pruning rate obtained in S2, each client performs a sparsification operation on its own gradient to obtain the sparsified gradient. S4. Based on the quantization level obtained in S2, each client performs quantization and encoding operations on the sparsified gradient obtained in S3 to obtain compressed gradient data. S5. Each client uploads the compressed gradient data to the server. The server decompresses the compressed gradient data to obtain the decompressed gradient data. S6. The server aggregates the decompressed gradient data element by element and updates the global model; S7. Repeat S1 to S6 until the global model converges.

2. The method according to claim 1, characterized in that, S1 includes: Build a federated learning model, with the server aggregating all clients. Shared local model parameters, in Independent random sampling within the bandwidth range and allocation of different sampling communication bandwidths For each client Following a continuous uniform distribution ; The client It has a built-in local dataset and communicates with the server via wireless channels; In each round of global model training, each client The local model is trained based on a pre-set local dataset, and the local model is obtained.

3. The method according to claim 2, characterized in that, In the gradient adjustment strategy, the quantization level has a higher priority than the pruning rate. When the constraint requirements are not met, the quantization level is reduced first to reduce communication overhead. Then, the pruning rate is increased to reduce the amount of computation. Only when the quantization level reaches the minimum is the pruning rate increased. The preset constraint requirement is as follows: in To calculate energy consumption, For communication energy consumption, To calculate the delay, Due to communication delay, For the client Preset maximum energy consumption limit The maximum tolerable latency for each round of global model training is preset for federated learning.

4. The method according to claim 3, characterized in that, S3 includes: S31, Client After completing the local model training, obtain the parameter weights of the convolutional layers of the local model. and local model gradient increment ; S32. Calculate the L2 norm of the convolution kernels of each layer in the local model. The L2 norm Expressed in matrix form, the norm matrix is ​​obtained. ; S33. Select and set the pruning rate using probability-driven kernel selection. The range of pruning thresholds is obtained ; S34. The L2 norm described in S32 The pruning threshold described in S33 Compare and generate a mask matrix. ; S35, the mask matrix With the weight matrix of the original convolutional layer Multiply the elements one by one and retain the mask matrix. The value above corresponds to the original convolutional layer weight matrix. The elements of the mask matrix are removed. The value above 0 corresponds to the original convolutional layer weight matrix. The elements are used to obtain the pruned weight matrix. ; S36. The pruned weight matrix retains non-zero convolution kernels and removes zero convolution kernels, finally obtaining a sparse weight matrix. .

5. The method according to claim 4, characterized in that, Based on the L2 norm of each convolutional kernel in the convolutional layer The values ​​are set such that the mask positions corresponding to convolution kernels with values ​​not less than a threshold are set to 1, and those with values ​​less than a threshold are set to 0. All masks are then merged to generate a mask matrix. mask matrix Expressed as a formula .

6. The method according to claim 5, characterized in that, S4 includes: S41. Save the sparsified weight matrices respectively. The sign matrix is ​​obtained by combining the sign information and value of each weight value. sum numerical matrix The symbol matrix and the numerical matrix Expressed as a formula: For the symbol matrix For elements less than 0, decrement by 1; for elements greater than or equal to 0, decrement by 1. For the numerical matrix... The maximum value of the numerical matrix is ​​obtained by taking the absolute value of all elements within it. Minimum value of the sum of numerical matrices ; S42. The maximum value of the numerical matrix as described in S41 Minimum value of the sum of numerical matrices For numerical matrices The weight matrix is ​​divided into several quantization levels, each corresponding to a quantization interval, and the sparsed weight matrix is... Each weight value in the algorithm is assigned to a corresponding quantization level based on its numerical value. S43. For the quantization interval described in S42, random quantization is performed using the following quantization strategy to obtain the quantized index matrix. : in, For the weight value to fall within The probability within the interval, said probability is based on It is calculated based on the distance from the boundary of the quantization interval; Indicates the first Each quantization boundary relative to the minimum value The offset; S44, the index matrix after quantization in S43 The frequency of occurrence of all characters is statistically analyzed to evaluate the relative importance of each index. The statistical results are used to construct a Huffman tree. Each index is assigned a corresponding binary code according to its frequency, resulting in Huffman coding and Huffman code table. S45. Replace the quantized index matrix with the Huffman coding. The encoded index matrix is ​​obtained. ; The compressed gradient data is an encoded index matrix. Other auxiliary decompression data, including the minimum value of the numerical matrix. Symbol matrix Mask matrix and Huffman code table.

7. The method according to claim 6, characterized in that, S5 includes: S51. Encoded index matrix corresponding to the client device Huffman decoding is performed using a Huffman code table to obtain the decoded index matrix. The decoded index matrix With the quantized index matrix They are the same matrix; S52, the decoded index matrix The weight values ​​are mapped to the corresponding quantization intervals through the symbol matrix. Obtain the weight matrix for recovering symbolic information The weight matrix for recovering symbolic information With the sparse weight matrix They are the same matrix; S53, the position of the zero-value convolution kernel is determined by the mask matrix. Restore the parameters to zero, retaining only the positions of non-zero convolution kernels, to obtain the increments of model parameters after quantization sparsity. ; The server decompresses the compressed data uploaded by all client devices using the above process to obtain decompressed data, which consists of the quantized and sparsed model parameter increments from all clients. ,in This represents the number of clients.

8. The method according to claim 7, characterized in that, The server receives compressed incremental data, which includes quantization error and encoding noise, and is processed through a mask matrix. Explicitly restoring to zero values ​​ensures parameter increments after decompression. It is completely consistent with the client's original pruning structure, avoiding the occurrence of tiny values ​​in positions that should be zero due to potential errors in the transmission or calculation process, such as floating-point rounding.

9. The method according to any one of claims 7-8, characterized in that, Step S6 includes: make For the client The uploaded weight increments after sparse quantization, where The first convolutional layer The increment of each convolutional kernel weight, after aggregation, is expressed as the global model update gradient increment. To obtain the global model after the current round of updates. Update the global model parameters until the global model converges. During aggregation, only non-zero convolution kernels are considered.

Images