An adaptive-based hierarchical compression federated learning method, device and medium

Abstract

The application belongs to the technical field of distributed machine learning, and discloses a layered compression federated learning method based on self-adaption, equipment and medium, the method comprises the following steps: the server side selects a client based on the dynamically calculated client participation rate and sparse rate and distributes global model parameters; after the client is locally trained, the gradient change is calculated, and in turn, the layered, dynamic sparsification and sparse threshold recording, ternary quantization and 1-bit compression sensing processing are carried out, and the obtained compression value, sparse threshold and training loss are uploaded to the server; after the server side receives the data, on the one hand, based on the training loss uploaded by all clients, the participation rate and the sparse rate of the next round are calculated and updated through the self-adaptive participation rate algorithm and the dynamic sparse rate algorithm, and on the other hand, the received compressed parameters are reconstructed and weighted aggregated to update the global model. The method can significantly reduce the communication overhead while maintaining almost the same model performance as the lossless transmission scheme.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning technology, specifically to a method for optimizing communication efficiency in federated learning, and particularly to an adaptive hierarchical compression federated learning method, device, and medium. Background Technology

[0002] In the rapid development of the information age, machine learning, especially deep learning, has achieved remarkable results in many fields such as computer vision, natural language processing, communication, and image recognition. The success of machine learning relies heavily on large amounts of training data. Traditional centralized machine learning algorithms require the collection of data from various devices or institutions for centralized training; for example, training a medical diagnostic model requires collecting relevant medical data from multiple medical institutions. However, frequent data breaches and misuse incidents have led to increasing public concern about the privacy threats posed by centralized data processing. At the same time, the increasing independence of data among organizations has highlighted the problem of data silos, resulting in the ineffective utilization of valuable data resources.

[0003] To address this issue, the concept of Federated Learning (FL) was introduced. Federated Learning is a distributed machine learning architecture that allows multiple participants (clients) to jointly train a global model without sharing their private data. In Federated Learning, the training of the global model typically involves multiple iterations. In each iteration, a central server randomly selects a subset of clients to participate in the training. The selected clients update their model parameters based on the current global model using their local data, and then upload these local updates to the server for aggregation, ultimately generating a new global model.

[0004] However, federated learning faces a key bottleneck in practice: communication overhead. Modern deep learning models typically contain millions or even hundreds of millions of parameters, reaching scales of several gigabytes. During hundreds of thousands of training iterations on large datasets, the total communication volume per client can easily exceed petabytes (PB). Furthermore, unstable network conditions on the client side can lead to significant communication latency. In environments with limited bandwidth and unstable transmission, the enormous communication volume severely impacts the efficiency of federated learning. Therefore, developing a communication-efficient federated learning method is crucial to solving these problems.

[0005] In existing technologies, the Deep Learning-based Adaptive Ternary Quantization (FTTQ) method compresses the local model parameters trained on the client side into ternary values ​​before sending them to the server. While this method reduces the number of bits representing each parameter through quantization, thereby reducing communication overhead, it still has shortcomings: First, the number of parameters transmitted remains unchanged; second, its global model update strategy is relatively simple, leading to significant model accuracy loss in complex scenarios such as heterogeneous data, resulting in unsatisfactory overall performance. Therefore, there is an urgent need for a communication optimization method that can achieve higher compression rates and higher training efficiency while maintaining model accuracy. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, the present invention aims to provide an adaptive hierarchical compression federated learning method, device and medium to solve the problems of high communication cost and low model accuracy in existing federated learning.

[0007] In a first aspect, the present invention provides an adaptive hierarchical compressed federated learning method, comprising the following steps:

[0008] Get the initialized global model parameters;

[0009] In each training round:

[0010] Step 1: On the server side: Based on the client participation rate and sparsity rate of the current training round, select one or more clients from multiple clients; send the global model parameters and the sparsity rate to the selected one or more clients;

[0011] Step 2: On the client side: Any selected client receives the global model parameters and the sparsity rate; trains the received global model using a local private dataset to obtain local model parameters and training loss; calculates the difference between the local model parameters and the global model parameters to obtain the gradient change; performs sparsity processing on the gradient change based on the sparsity rate to obtain a sparse gradient and a sparse threshold; quantizes the sparse gradient to obtain a parameter update trend; compresses the parameter update trend to obtain a compressed value; and sends the compressed value, the sparse threshold, and the training loss to the server.

[0012] Step 3: On the server side: Receive the compressed value, the sparsity threshold, and the training loss from one or more clients; calculate the client participation rate for the next training round based on the received training loss; calculate the dynamic sparsity rate for the next training round based on the received training loss; reconstruct each received compressed value using the corresponding sparsity threshold to restore the personalized model parameters for one or more clients; aggregate and update the global model parameters for the current training round based on the restored personalized model parameters to obtain the global model parameters for the next training round.

[0013] Repeat steps 1 to 3 until the preset training stop condition is met.

[0014] As an optional implementation of the first aspect of this application, step 2, which involves sparse processing of the gradient change based on the sparsity rate to obtain a sparse gradient and a sparse threshold; quantizing the sparse gradient to obtain a parameter update trend; and compressing the parameter update trend to obtain a compressed value, specifically includes: arranging all elements in the gradient change in ascending order according to their absolute values; calculating an index value k based on the sparsity rate, and determining the absolute value of the kth element after ascending order as the sparse threshold for this round of training; traversing all elements in the gradient change, retaining elements with absolute values ​​greater than the sparse threshold as the sparse gradient, and setting the remaining elements to zero; applying the sign function to the non-zero elements in the sparse gradient to obtain the parameter update trend consisting of positive and negative ones; sampling and compressing the parameter update trend using a Gaussian random matrix to obtain sampled values; and applying the sign function again to the sampled values ​​to obtain compressed values ​​conforming to a 1-bit compression form.

[0015] As an optional implementation of the first aspect of this application, in step 2, when performing sparsity processing on the gradient change, the processing is performed in layers according to the structure of the global model parameters, that is, the sparsity processing, quantization processing and compression are performed independently on each layer of parameters of the gradient change.

[0016] As an optional implementation of the first aspect of this application, step 3, the step of calculating the client participation rate for the next training round based on the received one or more training losses, includes: performing a federated average of the received one or more training losses to obtain the average loss for the current training round. ; Calculate the average loss for the current training round. Average loss compared to the previous training round The difference is used to obtain the magnitude of the loss change. and change values Determine the magnitude of the loss change. Is it less than the preset amplitude stability threshold? If the value is less than the set value, the stabilization counter is incremented by one; if it is not less than the set value, the stabilization counter is reset to zero; if the value of the stabilization counter c is greater than the preset convergence count value, the model is determined to have converged, and the current client participation rate is reduced according to the preset decay factor to obtain the client participation rate for the next training round. Determine the change value. Is it not greater than the preset stable value threshold? If it is not greater than, then the client participation rate for the next training round is dynamically calculated using a proportional-integral-derivative adjustment mechanism. .

[0017] As an optional implementation of the first aspect of this application, the step of dynamically calculating the client participation rate in the next training round through a proportional-integral-derivative adjustment mechanism specifically includes: calculating the current error. The current error The average loss for the current training round Average loss compared to the previous training round The difference; calculate the integral term. The integral term The summation of error values ​​from all historical rounds; calculation of the differential term. The differential term Current error Error compared to the previous round The difference, if If empty, the difference is 0; based on the preset proportional hyperparameter. Integral hyperparameters Differential hyperparameters Calculate the intermediate adjustment value The calculation method is as follows Determine the distribution type of the client's local private dataset. If it is of type IID, the client participation rate in the next training round will be based on the optimization parameters. and intermediate adjustment value The calculation is as follows: If it is a non-IID type, the client participation rate in the next training round will be based on the optimization parameters. and intermediate adjustment value Calculated.

[0018] As an optional implementation of the first aspect of this application, step 3, the step of calculating the dynamic sparsity rate of the next training round based on the received one or more training losses, includes: performing a federated average of the received one or more training losses to obtain the average loss of the current training round. ; Update history maximum loss If the current average loss Greater than the largest historical loss This would result in the greatest historical loss. Updated to current average loss Based on the aforementioned historical maximum loss Current average loss Preset transition sparsity Maximum sparsity and minimum sparsity The dynamic sparsity rate for the next training round is calculated. .

[0019] As an optional implementation of the first aspect of this application, the step of calculating the sparsity rate for the next training round further includes: calculating the average loss for the current training round. Average loss compared to the previous training round The difference is used to obtain the magnitude of the loss change. Determine the magnitude of the loss change. Is it less than the preset stability threshold? If the value is less than 1, the model is considered to be converging. The sparsity of the next training round is obtained using the first sparsity calculation rule, which is as follows: If the magnitude of the loss changes Not less than the stability threshold If the model has not yet converged, the sparsity of the next training round is obtained using the second sparsity calculation rule, which is as follows: ;in The average loss in round t is expressed as... With maximum loss The ratio, This represents the t-th iteration. Indicates the maximum number of rounds.

[0020] As an optional implementation of the first aspect of this application, step 3, which involves reconstructing each received compressed value using a corresponding sparse threshold to restore the personalized model parameters of one or more clients; and aggregating and updating the global model parameters of the current training round based on the restored personalized model parameters to obtain the global model parameters of the next training round, specifically includes: applying a binary iterative hard thresholding algorithm to iteratively reconstruct the compressed value from any client to obtain a restored value of the parameter update trend; extracting the sign bit of each element in the restored value as the reconstructed parameter update trend; restoring the personalized model parameters of the client based on the reconstructed parameter update trend, the sparse threshold uploaded by the client, and the global model parameters of the current training round; and weighting and aggregating the personalized model parameters of each restored client according to the size of the client's local private dataset to obtain the global model parameters of the next training round; wherein the weight of any client is the proportion of the number of samples in the client's local private dataset to the total number of samples from all clients participating in the training round.

[0021] In a second aspect, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0022] Thirdly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] (1) The adaptive participation rate (APR) algorithm designed in this invention can intelligently adjust the number of clients participating in training according to the dynamic changes in the model training loss. When the model tends to converge, the number of participating clients is reduced, and when the model needs to learn quickly, the number of participating clients is increased. In this way, while ensuring the training effect, unnecessary communication and computing overhead are effectively reduced, and the overall efficiency of federated learning is improved.

[0025] (2) The Dynamic Sparse Hierarchical Compressed Sensing (DHCS) method proposed in this invention combines dynamic sparsity and 1-bit compressed sensing techniques. It is not simply quantization, but first filters out the most important gradient information through dynamic sparsity rate, and then performs extreme compression on it, transmitting only symbol information. This dual compression mechanism greatly reduces the amount of data in a single communication. At the same time, the BIHT algorithm can be used on the server side to reconstruct the model update trend better, reducing information loss.

[0026] (3) The method of this invention shows superiority on non-independent and identically distributed (Non-IID) datasets. Traditional federated learning methods show a significant performance degradation in Non-IID scenarios, while the adaptive participation rate mechanism (especially the PID adjustment strategy designed for Non-IID) and dynamic sparsity rate mechanism of this invention can better adapt to the challenges brought about by data heterogeneity, and experiments have shown that higher model accuracy can be obtained. Attached Figure Description

[0027] Figure 1 A flowchart illustrating an adaptive hierarchical compressed federated learning method provided in an embodiment of the present invention;

[0028] Figure 2 This is a comparison chart of the experimental results of the method of this invention on the MNIST handwritten digit dataset;

[0029] Figure 3 This is a comparison chart of the experimental results of the method of this invention on the F-MNIST fashion dataset;

[0030] Figure 4 This is a comparison chart of the experimental results of the method of this invention on the CIFAR-10 image recognition dataset. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0033] Example 1

[0034] Please see Figure 1 The flowchart below illustrates an adaptive hierarchical compressed federated learning method provided in an embodiment of the present invention. Before executing this method, system initialization is performed first.

[0035] Establish the basic framework for federated learning, setting up a central server and multiple clients. The central server initializes a global model to be trained, and its parameters are denoted as... During the system initialization phase, the server sets the following parameters: total number of training rounds T, maximum number of participating clients. Initial participation rate (e.g., 0.5), initial sparsity (e.g., 80%), PID controller parameters for adaptive participation control (proportional hyperparameter kp, integral hyperparameter ki, derivative hyperparameter kd), and optimization parameters for updating the participation rate. and maximum participation rate Initialization.

[0036] After initialization, the method specifically includes the following steps:

[0037] Step 1: On the server side: Based on the client participation rate and sparsity rate of the current training round, select one or more clients from multiple clients; send the global model parameters and the sparsity rate to the selected one or more clients.

[0038] Specifically, the server-side is based on the participation rate. Select the clients participating in this round and distribute the global model parameters. sparsity For the selected client, t This indicates the current training round.

[0039] Step 2: On the client side: Any selected client receives the global model parameters and the sparsity rate; trains the received global model using a local private dataset to obtain local model parameters and training loss; calculates the difference between the local model parameters and the global model parameters to obtain the gradient change; performs sparsity processing on the gradient change based on the sparsity rate to obtain a sparse gradient and a sparse threshold; quantizes the sparse gradient to obtain a parameter update trend; compresses the parameter update trend to obtain a compressed value; and sends the compressed value, the sparse threshold, and the training loss to the server.

[0040] Specifically, this step involves the following operations:

[0041] S21: Selected clients, after receiving the distributed model, use their respective private datasets. Train them to obtain the corresponding parameters of their respective local models. and training loss ,in i Indicates the selected first i There are 1 client, where t represents the t-th training round;

[0042] S22: Select the local model parameters obtained from the client's training. With the global model parameters in this round The gradient change of the model is obtained by subtraction. .

[0043]

[0044] The obtained gradient change value The global model parameters are processed hierarchically according to their structure, and sparse ternary compression is performed as shown in Table 1. This represents the sparsity of the i-th client after round t. This represents the value to be compressed after sparse quantization in round t for the i-th client. First, the model gradient parameters... All elements are sorted in ascending order of absolute value (as shown in row 1 of Table 1; row 2 of Table 1 indicates sorting by absolute value). The numerical information of the update symbol value is calculated. ), will the first The absolute value of each element is used as the sparsity threshold. Where the value of k is determined by the given sparsity rate of this round; based on the sparsity rate sent by the server. A sparse threshold is obtained, and data values ​​less than the threshold are set to 0. Then, the sign function is applied to the non-zero elements in the sparse gradient to obtain the parameter update trend consisting of positive and negative ones. (As shown in row 4 of Table 1, according to) Set empty strings in the sequence that are less than the threshold to 0, and then retain the sign information of the remaining numbers to obtain... ), and record the sparsity threshold for this round. For example, based on a sparsity rate SR = 50%, a sparsity threshold is selected. =0.0146.

[0045] Table 1. Numerical information of the sparse ternary compression process

[0046]

[0047] Parameter update trend Sampling compression is performed using a Gaussian matrix to obtain sampled values. Furthermore, regarding The sign function is used to obtain 1-bit symbol information, resulting in the final compressed value. Finally, the compressed values ​​after layering and the threshold are combined, along with the training loss. It is uploaded to the server along with the gradient changes. It should be noted that when performing sparsity processing on gradient changes, it is done in layers according to the structure of the global model parameters. That is, sparsity processing, quantization processing, and compression are performed independently on the parameters of each layer of gradient changes.

[0048] Step 3: On the server side: Receive the compressed value, the sparsity threshold, and the training loss from one or more clients; calculate the client participation rate for the next training round based on the received training loss; calculate the dynamic sparsity rate for the next training round based on the received training loss; reconstruct each received compressed value in combination with the corresponding sparsity threshold to restore the personalized model parameters for one or more clients; aggregate and update the global model parameters for the current training round based on the restored personalized model parameters to obtain the global model parameters for the next training round.

[0049] Specifically, this step involves the following operations:

[0050] S31: The training loss received by the server from the client The average loss was calculated using the average aggregation method. .

[0051] Because we need to compare the loss with the previous stage, we first need to determine whether the current training epoch is greater than or equal to the second stage; if so, we calculate the current average loss. Compared to the average loss in the previous round range of change and change values By judging the magnitude of change Is it greater than the amplitude stability threshold? If the change is too large, the value of the stabilizing counter will be activated. Set to 0, otherwise stabilize the counter value. Add 1. (Among them) This represents the absolute value function.

[0052] pass The value can be used to determine whether the network has converged. A continuous increase indicates that the network is gradually converging, at which point the participation rate can be reduced. To fine-tune the network with fewer client involvement, This represents the network decay factor, used to slowly reduce participation rate without affecting accuracy.

[0053] Continue to assess the change in loss value Is it greater than the numerical stability threshold? If the conditions are met, the new participation rate will be... If the conditions are not met, then the participation rate needs to be dynamically adjusted.

[0054] The dynamic adjustment of the participation rate first determines the loss error between the two rounds. If the current error is greater than the previous round's error, the participation rate can be increased directly. Conversely, it is based on the current error of the training loss. Integral terms and differential terms The updated intermediate value is obtained by calculating using equation (2). Based on the characteristics of the dataset, different participation rate calculation methods are designed as follows;

[0055]

[0056] Where the error value Integral term ,like If empty, the value is 0; differential term ,like If the result is empty, the difference is 0. If the dataset distribution is of type IID, the participation rate is calculated according to equation (3); otherwise, it is calculated using equation (4). Finally, the new participation rate is returned. Equation (3) designs a calculation method suitable for IID data, uses cumulative error to judge the stability of the network, and simultaneously... p Limit the maximum participation rate to dynamically adjust the client participation ratio; (4) then through 3. To adjust the participation rate updates for other data distributions.

[0057] Dynamic sparsity Based on the current maximum loss Current losses Excessive sparsity The maximum and minimum sparsity rates are calculated. First, the maximum loss is initialized, and then the following formula is used to obtain... .

[0058]

[0059] in Represents the average loss in round t. With maximum loss The ratio, This represents the t-th iteration. Indicates the maximum number of rounds.

[0060] The convergence of the network is judged by setting a network stability threshold. If the value is less than this stable threshold, it indicates that the network has gradually converged. Using equation (6) with a high sparsity rate, the client transmits a small number of updates to fine-tune the global network. Conversely, using equation (7) with a low sparsity rate, the client transmits a large number of updates to allow the global network to obtain more update information.

[0061] S32: The server processes the received compressed values ​​and thresholds one by one. It performs layered processing on the compressed values ​​uploaded by each client according to the layered compression method. Because the update trend uses 1-bit compression and the sparsity rate is sent from the server to the client in each round, the received compressed values ​​are reconstructed using the Binary Iterative Hard Thresholding (BIHT) reconstruction method to obtain the partial parameter update trends uploaded by the client. Finally, the reconstructed values ​​from each layer are restored and combined to obtain the complete parameter update trends from the client. Using the obtained parameter update trends, sparsity thresholds, and the global model for this round... The personalized model parameters of the client can be restored according to equation (8). .

[0062]

[0063] in express t Round i The first client's j One element, This represents the j-th element in round t of the global parameter. Let N represent the j-th element of the i-th client after round t reconstruction, and N represent the number of parameters.

[0064] S33: The server-side performs weighted aggregation of the selected client-specific model parameters based on the size of the dataset to obtain new global model parameters. The weighted aggregation method used is as follows:

[0065]

[0066] in Indicates the first t Rounds i Dataset size per client Indicates the first t The total amount of data involved in each training round; This represents the selected set of clients.

[0067] Repeat steps 1 to 3 above until the stopping condition is met to obtain the final global model parameters.

[0068] Specifically, the method of this invention mainly reduces communication overhead during model training by using client selection and model parameter compression. In this embodiment, the invention considers that in federated learning, a fixed participation rate leads to a linear increase in the workload of update communication and server aggregation with each training round. Even if the network has gradually converged, the number of participating clients remains fixed, resulting in unnecessary communication overhead. Therefore, this invention proposes an Adaptive Client Participation Rate (APR) method to determine how many clients should participate in training to aggregate local updates in the next round, thereby making communication more efficient by limiting the number of participating clients. Specifically, in this method, the server determines whether the network has gradually converged based on the training loss of the local model of the selected clients in this round. If the network is close to convergence, the number of participating clients needs to be gradually reduced to a minimum. If the network has not converged and the decrease is significant, the number of clients in the next round can be appropriately increased. If the network has not converged and the fluctuation is small, an Adaptive Proportional PID algorithm can be used to dynamically calculate the number of clients participating in the federated learning system in the next round based on the average training loss of this round, the maximum participation rate, and the historical average training loss.

[0069] Meanwhile, this invention proposes a Dynamically Sparse Hierarchical Compression (DHCS) method. On the client side, the client utilizes private data and the updated gradient and training loss obtained from the current round's global model. Based on the current round's sparsity rate, it performs sparsity ternary compression to obtain a sparsified update trend and sparsity threshold. Then, using a 1-bit compression-aware method, the sparsified update trend is compressed and sent to the server along with the training loss and sparsity threshold. On the server side, the server reconstructs and aggregates the compressed parameters transmitted by the client in this round to obtain a new global model. Based on the average loss, it calculates a suitable sparsity rate using the Dynamically Sparse Rate Adjustment (DS) algorithm and distributes it to the selected client for the next round. By combining dynamic sparse ternary compression with a 1-bit compression method, the communication efficiency of federated learning can be further improved while maintaining high accuracy.

[0070] In the above embodiments, as one possible implementation, the model accuracy (acc) is used to characterize model performance. It should be noted that after completing this round of aggregation, the server will evaluate and update the performance (acc) of the aggregated global model for use in comparing model performance in the next training round.

[0071] In the above embodiments, as one possible implementation method, the server uses a weighted aggregation method to aggregate the parameters of all client model participants in the training.

[0072] To verify the effectiveness of the proposed efficient federated learning scheme (A-DHCS), the present invention also provides the following experimental data; at the same time, the experiment also compares the method of the present invention with federated learning schemes such as the classic baseline algorithm FedAvg, dynamic threshold 1-bit-cs, fixed threshold 1-bit-cs, T-FedAvg, CAFL, and fedcams, highlighting the advantages of the present invention.

[0073] (I) Experimental Setup

[0074] The experimental parameters are shown in Table 2. Three datasets were used in the experiment: the MNIST handwritten digit dataset, the F-MNIST fashion dataset, and the CIFAR-10 image recognition dataset.

[0075] (1) The MNIST handwritten digit dataset is a benchmark dataset for machine learning algorithms, containing 60,000 training images and 10,000 test images. Experiments used an MLP model with a 5-layer network structure to train the MNIST dataset.

[0076] (2) The F-MNIST dataset contains 60,000 training images and 10,000 test images for 10 different fashion categories. The experiment used an MLP model with a 5-layer network structure to train the F-MNIST dataset.

[0077] (3) The CIFAR-10 dataset contains 10 different categories, with 6,000 images in each category for image recognition. The experiment used a CNN model with a six-layer network structure to train the CIFAR-10 image recognition dataset.

[0078] Table 2. Experimental Parameter Settings

[0079]

[0080] (II) Dataset Setup

[0081] In the field of federated learning, the distribution of data among client devices has a significant impact on the learning outcomes. Typically, data is divided into independent and identically distributed (IID) data: each client receives randomly assigned data from the dataset, and each client receives an equal amount of data; and non-independent and identically distributed (Non-IID) data: each client is restricted to a few specific data categories, but does not necessarily have to be restricted to the same amount of data (in the most extreme case, a client may only have data labeled as a single category).

[0082] (III) Precision

[0083] To evaluate the accuracy of the A-DHCS scheme, the model was trained on the MNIST, F-MNIST, and CIFAR-10 datasets, and the results were compared with the baseline federated learning model (FedAvg), dynamic threshold 1-bit-cs, fixed threshold 1-bit-cs, T-FedAvg, CAFL, and fedcams federated learning schemes.

[0084] For the MNIST dataset, the training results are as follows: Figure 2 As shown, the left figure represents the IID setting, and the right figure represents the Non-IID setting. On the IID dataset, the scheme designed in this invention achieves higher accuracy than other compression algorithms at both compression ratios, with its accuracy approaching that of the uncompressed benchmark FedAvg method. Furthermore, when CR is set to 0.03, the A-DHCS method converges faster. On the Non-IID dataset, the seven methods show significant differences in performance, while the A-DHCS method designed in this invention achieves slightly higher final accuracy than the benchmark FedAvg method at both compression ratios.

[0085] For the F-MNIST dataset, the training results are as follows: Figure 3 As shown, the left figure represents the IID setting, and the right figure represents the Non-IID setting. On the IID dataset, the A-DHCS method converges the fastest and stabilizes after about 50 rounds, with a final accuracy of about 85%, which is comparable to the benchmark FedAvg method. On the Non-IID dataset, the performance of all six methods is not as good as that on the IID dataset, but the performance of the A-DHCS method is still comparable to that of the benchmark FedAvg method.

[0086] For the CIFAR-10 dataset, the training results are as follows: Figure 4 As shown, the left figure represents the IID setting, and the right figure represents the Non-IID setting. On the IID dataset, although the accuracy curve of the A-DHCS method is not as smooth as the benchmark FedAvg method in the early stages, its accuracy is higher than all methods around 30-80 rounds. Finally, it is still close to the benchmark FedAvg method and higher than other compression algorithms. In the Non-IID data environment, the experimental results are similar to those of the F-MNIST experiment. The performance of the A-DHCS method is close to the benchmark FedAvg method and better than other compression methods.

[0087] The above results show that the A-DHCS scheme proposed in this invention outperforms the baseline federated learning models FedAvg, dynamic threshold 1-bit-cs, fixed threshold 1-bit-cs, T-FedAvg, CAFL, and fedcams federated learning schemes, especially when the MNIST data is not independent and identically distributed.

[0088] (iv) Communication overhead

[0089] To evaluate the communication overhead of the A-DHCS scheme, the communication cost will be measured by the amount of model parameters transmitted. In the experiments, the MNIST dataset is used, the model network is an MLP, the model parameters are of type float64 (each parameter occupies 64 bits), and the number of model parameters is [number missing]. Among them, A-DHCS, dynamic thresholding, and fixed thresholding 1-bit-CS methods use 1-bit compressed sampling. Therefore, the parameter transmission amount per round for these methods is the model parameter amount multiplied by the compression ratio. Other methods calculate the parameter transmission amount per round according to the compression method in the paper. In the experiment, the compression ratio of A-DHCS, dynamic thresholding 1-bit compressed sensing method, fixed thresholding 1-bit compressed sensing method, and CAFL is 0.03. T-FedAvg and fedcams use the best quantization method in the literature. The final results are shown in Tables 3 and 4, and the data unit in the tables is MB.

[0090] Table 3. Relationship between client upload communication cost and accuracy when using IID data.

[0091]

[0092] Table 4. Relationship between client upload communication cost and accuracy when using Non-IID data.

[0093]

[0094] As can be clearly seen from the experimental data in Tables 3 and 4, the A-DHCS method proposed in this invention exhibits significant advantages in terms of client-side upload communication cost and model accuracy.

[0095] On the IID dataset, both dynamic and fixed thresholding methods achieved 80% accuracy with minimal communication, but failed to reach the highest accuracy of 96%. While the A-DHCS method of this invention transmits slightly more data at 80% accuracy than the fixed and dynamic thresholding methods, it subsequently achieves the same high accuracy as the benchmark FedAvg method, with a communication cost only 1 / 3230th of FedAvg.

[0096] On the Non-IID dataset, the dynamic thresholding 1-bit-CS method achieves 75% accuracy with minimal communication, but it fails to reach 90% accuracy. In contrast, the A-DHCS method achieves the same 93% accuracy as the benchmark FedAvg method with only 1 / 4207th the communication cost, and its final accuracy is 2% higher than the benchmark FedAvg method.

[0097] Experimental results show that the efficient federated learning scheme A-DHCS provided by this invention optimizes the communication bottleneck in federated learning, significantly improving communication efficiency without sacrificing model accuracy and convergence speed. Especially in scenarios where client data is not independently and identically distributed, the model accuracy of the A-DHCS scheme is superior to baseline federated learning and other parameter-compressed federated learning methods.

[0098] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of an adaptive hierarchical compression federated learning method and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0099] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of an adaptive hierarchical compressed federated learning method and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0100] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0101] It should be noted that in this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0103] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An adaptive hierarchical compressed federated learning method, characterized in that, Includes the following steps: Get the initialized global model parameters; In each training round: Step 1: On the server side: Select one or more clients from multiple clients based on the client participation rate and sparsity rate of the current training round; Send the global model parameters and the sparsity rate to one or more selected clients; Step 2: In the client: Any selected client receives the global model parameters and the sparsity rate; trains the received global model using the local private dataset to obtain the local model parameters and training loss; The gradient change is obtained by calculating the difference between the local model parameters and the global model parameters. Based on the sparsity rate, the gradient change is sparsified to obtain a sparse gradient and a sparse threshold. The sparse gradient is quantized to obtain the parameter update trend; The parameter update trend is compressed to obtain a compressed value; The compressed value, the sparse threshold, and the training loss are sent to the server. Specifically, the process includes: sorting all elements in the gradient change in ascending order of their absolute values; calculating an index value k based on the sparsity rate, and determining the absolute value of the k-th element after ascending order as the sparsity threshold for this round of training; traversing all elements in the gradient change, retaining elements with absolute values ​​greater than the sparsity threshold as the sparse gradient, and setting the remaining elements to zero; applying the sign function to the non-zero elements in the sparse gradient to obtain the parameter update trend consisting of positive and negative ones; sampling and compressing the parameter update trend using a Gaussian random matrix to obtain sampled values; and applying the sign function again to the sampled values ​​to obtain compressed values ​​conforming to 1-bit compression. Step 3: On the server side: Receive the compressed value, the sparsity threshold, and the training loss from one or more clients; calculate the client participation rate for the next training round based on the received training loss; calculate the dynamic sparsity rate for the next training round based on the received training loss; reconstruct each received compressed value using the corresponding sparsity threshold to restore the personalized model parameters for one or more clients; aggregate and update the global model parameters for the current training round based on the restored personalized model parameters to obtain the global model parameters for the next training round. Repeat steps 1 to 3 until the preset training stop condition is met.

2. The method according to claim 1, characterized in that, In step 2, when the gradient change is sparsified, it is processed in layers according to the structure of the global model parameters. That is, the sparsification, quantization and compression are performed independently on each layer of parameters of the gradient change.

3. The method according to claim 1, characterized in that, Step 3, which involves calculating the client participation rate for the next training round based on one or more received training losses, includes: The received training losses are federated and averaged to obtain the average loss for the current training round. ; Calculate the average loss for the current training round. Average loss compared to the previous training round The difference is used to obtain the magnitude of the loss change. and change values ; Determine the magnitude of the loss change Is it less than the preset amplitude stability threshold? If the value is less than 1, the stabilization counter is incremented by one; if the value is not less than 1, the stabilization counter is reset to zero. If the value c of the stabilization counter is greater than the preset convergence count value, the model is determined to have converged. The current client participation rate is then reduced according to a preset decay factor to obtain the client participation rate for the next training round. ; Determine the change value Is it not greater than the preset stable value threshold? If it is not greater than, then the client participation rate for the next training round is dynamically calculated using a proportional-integral-derivative adjustment mechanism. .

4. The method according to claim 3, characterized in that, The step of dynamically calculating the client participation rate for the next training round using a proportional-integral-derivative adjustment mechanism specifically includes: Calculate the current error The current error The average loss for the current training round Average loss compared to the previous training round The difference; Calculate the integral term The integral term This is the cumulative sum of error values ​​from all historical rounds. Calculate the differential term The differential term Current error Error compared to the previous round The difference, if If empty, the difference is 0; Based on preset proportional hyperparameters Integral hyperparameters Differential hyperparameters Calculate the intermediate adjustment value The calculation method is as follows ; Determine the distribution type of the client's local private dataset. If it is of type IID, the client participation rate in the next training round will be based on the optimization parameters. and intermediate adjustment value The calculation is as follows: If it is a non-IID type, the client participation rate in the next training round will be based on the optimization parameters. and intermediate adjustment value Calculated.

5. The method according to claim 1, characterized in that, Step 3, which involves calculating the dynamic sparsity rate for the next training round based on one or more received training losses, includes: The received training losses are federated and averaged to obtain the average loss for the current training round. ; Maximum loss in updating history If the current average loss Greater than the largest historical loss This would result in the greatest historical loss. Updated to current average loss ; Based on the historical maximum loss Current average loss Preset transition sparsity Maximum sparsity and minimum sparsity The dynamic sparsity rate for the next training round is calculated. .

6. The method according to claim 5, characterized in that, The step of calculating the sparsity rate for the next training round further includes: Calculate the average loss for the current training round. Average loss compared to the previous training round The difference is used to obtain the magnitude of the loss change. ; Determine the magnitude of the loss change Is it less than the preset stability threshold? If the value is less than 1, the model is considered to be converging. The sparsity of the next training round is obtained using the first sparsity calculation rule, which is as follows: ; If the magnitude of the loss changes Not less than the stability threshold If the model has not yet converged, the sparsity of the next training round is obtained using the second sparsity calculation rule, which is as follows: ; in Represents the average loss in round t. With maximum loss The ratio, This represents the t-th iteration. Indicates the maximum number of rounds.

7. The method according to claim 1, characterized in that, In step 3, each received compressed value is reconstructed using a corresponding sparse threshold to restore one or more personalized model parameters for the client; based on the restored one or more personalized model parameters, the global model parameters for the current training round are aggregated and updated to obtain the global model parameters for the next training round. This step specifically includes: For the compressed value from any client, an iterative reconstruction using a binary iterative hard thresholding algorithm is performed to obtain the recovered value of the parameter update trend; Extract the sign bit of each element in the recovered value as the parameter update trend for reconstruction; Based on the reconstructed parameter update trend, the sparse threshold uploaded by the client, and the global model parameters of the current training round, the personalized model parameters of the client are restored. For each client's personalized model parameters that have been restored, a weighted aggregation is performed based on the size of the client's local private dataset to obtain the global model parameters for the next training round. The weight of any client is the proportion of the number of samples in that client's local private dataset to the total number of samples from all clients participating in the training round.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of an adaptive hierarchical compressed federated learning method as described in any one of claims 1-7.

9. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, they implement the steps of the adaptive hierarchical compressed federated learning method as described in any one of claims 1-7.

Images