Federal learning communication optimization method based on adaptive rarefaction and quantization

Abstract

The invention belongs to the field of federated learning communication optimization, and provides a federated learning communication optimization method based on adaptive rarefaction and quantization, so as to reduce communication overhead and improve energy efficiency. The method comprises the following steps: a central server issues a global model, randomly selects part of clients for local training, calculates gradient update, dynamically screens important gradients for transmission by adopting a dynamic quantization strategy, and further compresses residual gradients by applying an adaptive rarefaction method; meanwhile, an error feedback mechanism is introduced, and quantization errors are accumulated to improve the convergence effect of the model. And the client uploads the compressed gradients to the server, the server aggregates and updates the gradients and then continues to distribute the gradients, loop iteration is carried out in this way, and finally an efficient convergence global model is obtained. According to the method, the communication cost in federated learning is remarkably reduced by combining adaptive rarefaction, dynamic quantization and error compensation, and convergence is accelerated on the premise of ensuring the model performance, so that the method is more suitable for a distributed learning environment with limited resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed machine learning, and in particular relates to a federated learning communication optimization method based on adaptive sparsification and quantization. Background Art

[0002] As a distributed machine learning paradigm, federated learning achieves multi-node collaborative modeling under the requirements of privacy protection and data compliance through localized data training and parameter aggregation mechanisms. Its core framework includes a five-stage iterative process: global model initialization, local model training, parameter encryption upload, central server aggregation, and global model distribution. This technology has been applied to fields such as healthcare, financial risk control, and smart Internet of Things, effectively solving the problem of data islands. However, in the actual deployment of edge computing scenarios, federated learning faces three major technical bottlenecks. The first is the efficiency bottleneck under the constraints of communication resources. Typical federated learning requires the transmission of millions of parameters per round. For example, the ResNet-50 model contains 25 million parameters. In a 4G / 5G hybrid network environment, the upload time of a single edge device for a single iteration can reach 2-5 minutes. When there are more than 100 participating nodes, the peak bandwidth demand of the central server will exceed 1Gbps, causing significant communication delays. Existing solutions such as TernGrad (ternary quantization) proposed by DeepMind can compress the communication volume to 32.6%, but the fixed 3-bit quantization strategy leads to 3.2%-5.7% accuracy loss on data sets such as CIFAR-10. The second is the imbalance of energy consumption of heterogeneous devices. The computing power of edge devices varies by three orders of magnitude, from 13.5 GFLOPS of Raspberry Pi 4B to 1370 GFLOPS of NVIDIA Jetson AGX Xavier. When a unified training strategy is adopted, the single-round training time of low-end devices can reach 7.3 times that of high-end devices, resulting in more than 61% of the system synchronization waiting time. At the same time, intensive floating-point operations make the device power consumption reach 3.8-5.2W. With a battery capacity of 4000mAh, the battery life of continuous training conditions is less than 4 hours. The third is the defect of dynamic environment adaptability. Existing gradient compression methods, such as Google's sparse masking technology, use static thresholds, which are usually set to retain the top 1% gradients and cannot respond to network bandwidth fluctuations. Actual measurements show that the coefficient of variation of edge environment bandwidth is as high as 0.78 and the computing load of the device changes. When the channel quality deteriorates, the fixed compression rate strategy causes the effective information loss rate to increase by 2.1 times and the number of iterations required for model convergence increases by 37%.

[0003] Based on the above problems, some gradient compression methods have been proposed in the prior art. For example, fixed-bit quantization (QSGD algorithm) uses uniform quantization intervals and does not consider the long-tail characteristics of parameter distributions, resulting in an 18.4% loss of feature map information entropy on the ImageNet dataset; static sparsification strategies (Top-K gradient selection) ignore the real-time computing power of devices, and the time-consuming ratio of gradient screening on low-computing-power nodes reaches 29%; non-adaptive compression schemes lack a dynamic adjustment mechanism, and when the node offline rate exceeds 15%, the sensitivity of model update delay increases by 4.7 times. There are also some improvement attempts, including the dynamic quantization bit-width adjustment method (DQA-FL), which dynamically adjusts the 4-8 bit quantization interval by monitoring the remaining battery power of the device, improving the energy efficiency ratio by 23%. However, this method does not couple with the sparsification strategy, and there is still 14% communication redundancy in the 128-node scale test. These methods cannot dynamically adjust the strategy according to the computing and communication resources of the device, resulting in insufficient energy efficiency optimization. Summary of the Invention

[0004] To solve the above problems, the present invention proposes a federated learning communication optimization method based on adaptive sparsification and quantization. It includes the following steps: The central server initializes the global model parameters and distributes them to the client cluster, and performs multiple rounds of communication iterations; in each iteration, a part of the clients are dynamically selected to perform local model training, and the local gradient update amount is calculated; the gradient update amount is successively subjected to adaptive sparsification screening, dynamic quantization compression, and error accumulation compensation operations to generate compressed gradient data packets; the clients upload the compressed gradients to the server side, and the server aggregates all the client gradients and then applies the momentum acceleration algorithm to update the global model parameters to complete a single communication iteration;

[0005] The training process includes the following:

[0006] S1. The central server initializes the global model parameters to be trained;

[0007] S2. In each global training round, the central server randomly samples a fixed number of clients from all available clients and distributes the current global model parameters to the sampled clients;

[0008] S3. Each sampled client performs local training and calculation quantization;

[0009] S4. Model update difference calculation. After the client completes the local SGD iteration, it calculates the update difference in this round, which is the difference between the local model parameters before and after the update;

[0010] S5. Communication quantization and error correction: Perform communication quantization on the updated difference. Use different precisions to quantize the difference for different clients. At the same time, correct the quantization error generated during the quantization process together with the difference to ensure that the updated result is not distorted in the expected sense. The untransmitted quantization error is automatically compensated through weight updates in iterative training;

[0011] S6. Update data sparsification: Apply a sparsification operator to the quantized updated difference. The sparsification parameter only retains the most critical part of the update to further reduce the data transmission volume. At the same time, update the local error record;

[0012] S7. The client uploads the updated result after quantization and sparsification to the central server; the clients that are not sampled to participate in this round of training keep their cumulative errors unchanged. The central server aggregates all the received updated results and uses the weighted average method to generate new global model parameters;

[0013] S8. Repeat the above steps. When the global iteration number is satisfied or the server verifies whether the aggregated result meets the delay constraint τ, end the training to obtain the final training model.

[0014] Further, the client sampling step is random sampling, and only a part of the clients are selected to participate in the training in each round of global training to reduce the communication overhead and make full use of the limited network bandwidth resources. The quantization probability formula is:

[0015]

[0016] Further, the dynamic quantization compression adopts piecewise non-uniform quantization coding, including: dividing the amplitude interval of the gradient component, adaptively adjusting the quantization bit width according to the dynamic range of the gradient value, and generating a compressed coding sequence including the sign bit, exponent bit, and mantissa bit. The quantization formula is:

[0017] including a compressed coding sequence with a sign bit, an exponent bit, and a mantissa bit. The quantization formula is:

[0018]

[0019] Further, the adaptive sparsification screening adopts a dynamic threshold control strategy, specifically including: calculating the decision threshold of gradient importance according to the current training round and gradient distribution characteristics, retaining the gradient components with absolute values greater than this threshold to form a sparse gradient set, and the remaining gradient components enter the quantization processing link. The condition formula satisfied by the sparsification strategy is:

[0020]

[0021] Further, the loss function formula is:

[0022]

[0023] Furthermore, the communication termination judgment formula is:

[0024] F(ω t+1 ) - F(ω * ) ≤ ε

[0025] Furthermore, the model difference calculation formula is:

[0026]

[0027] Furthermore, the gradient sparsification calculation formula is:

[0028]

[0029] Furthermore, the global model update rule calculation formula is:

[0030]

[0031] Advantages of the present invention:

[0032] (1) The present invention adopts a dual dynamic quantization strategy, using dynamic bit computation quantization to replace 32-bit floating-point operations in the local training phase, and quadratic quantization compression in the communication phase, improving communication efficiency. The integrity of the gradient direction is maintained through non-uniform random quantization.

[0033] (2) The present invention adopts adaptive sparsification control, proposes a dynamic threshold screening mechanism, automatically adjusts the parameter transmission ratio according to parameters, and supports dual modes of Top-K and random sparsification.

[0034] (3) The present invention adopts joint optimization of energy efficiency and accuracy, establishes a quantization error - sparsification error - convergence rate mathematical model, and the server side adopts an improved Adam aggregation algorithm, reducing the number of communication rounds and improving the convergence speed. Description of the Drawings

[0035] The following further elaborates the present invention in detail in conjunction with the drawings and specific embodiments.

[0036] Figure 1 is the detailed architecture diagram of the server - client of the present invention;

[0037] Figure 2 is the overall framework diagram of the present invention; Specific Embodiments

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0039] A federated learning communication optimization method based on adaptive sparsification and quantization, as Figure 1 shown in the detailed server-client architecture diagram of the method of the present invention. The central server initializes the global model parameters and distributes them to the client cluster, and performs multiple rounds of communication iterations; in each round of iteration, some clients are dynamically selected to perform local model training and calculate the local gradient update amount; the gradient update amount is successively subjected to adaptive sparsification screening, dynamic quantization compression, and error accumulation compensation operations to generate compressed gradient data packets; the clients upload the compressed gradients to the server side, and the server aggregates all the client gradients and then applies the momentum acceleration algorithm to update the global model parameters, completing a single communication iteration, and then distributes the new global model to each client participating in the next round of training until the training end condition is met, and finally obtains the trained target model.

[0040] Figure 2 shown in the overall framework diagram of the present invention:

[0041] S1. The central server initializes the global model parameters to be trained;

[0042] Among them, the input of the model:

[0043] The training model is a CNN, which is a deep learning model with the characteristics of local connection and weight sharing. It automatically extracts the local features of the input data through structures such as convolutional layers and pooling layers, and is widely used in fields such as image recognition, target detection, and natural language processing. The set parameters are 2×10 6 pieces.

[0044] Among them, the input of the data set:

[0045] The data sets used are the image classification data sets CIFAR-10 and MNIST widely used in the fields of machine learning and computer vision. Among them, the CIFAR-10 data set contains 60,000 color images of 32*32 pixels, which are divided into 10 different categories, with 6,000 images in each category. The data set is divided into 50,000 training images and 10,000 test images. The MNIST data set contains 70,000 grayscale images of handwritten digits (0-9) of 28*28 pixels. The data set is divided into 60,000 training images and 10,000 test images.

[0046] S2. In each global training round, the central server randomly samples a fixed number of clients from all available clients and distributes the current global model parameters to the sampled clients;

[0047] S3. Each sampled client performs local training and computational quantization;

[0048] Specifically, client k performs τ rounds of SGD iterative training locally to update the local model parameters:

[0049]

[0050] where, is the model parameter in the (t + 1)-th round, η is the learning rate, L k is the local loss function, represents the local training dataset of client k. The update quantity is dynamically precision-quantized, and the formula is:

[0051]

[0052] where, ⊙ is the Hadamard product, is the b k bit quantization operator, and different clients adaptively select b according to the channel conditions k . The update quantity calculation formula is:

[0053]

[0054] where, τ is the total number of training rounds.

[0055] S4. Model update difference calculation. After the client completes the local SGD iteration, it calculates the update difference in this round. This difference is the difference between the local model parameters before and after the update, and the formula is:

[0056]

[0057] where, is the communication quantization error, is the local computation error.

[0058] S5. Communication quantization and error correction. The update difference is processed by communication quantization. Different precisions are used to quantize the difference for different clients. At the same time, the quantization error generated during the quantization process and the difference are corrected together to ensure that the update result is not distorted in the expected sense. The untransmitted quantization error is automatically compensated by the weight update in the iterative training, and the formula is:

[0059]

[0060] where, Denote random rounding to the neighboring quantization centers with probability proportion, δ k is the quantization step size, and the system assigns an independent quantization step size δ to each client k , which is dynamically adjusted according to the statistical characteristics of the local update of the client (the variance of the parameter update amplitude), to achieve an optimized configuration where clients with high variance use low precision and clients with low variance use high precision. However, errors will accumulate during this process, and the quantization error is compensated through two channels during the iteration process;

[0061] Specifically, explicit compensation: the accumulated error participates in the next round of quantization as a correction term; implicit compensation: during local training, the parameter update process automatically absorbs the error components that are not transmitted.

[0062] Among them, the cumulative quantization error formula is:

[0063]

[0064] Specifically, the update after quantization satisfies:

[0065]

[0066] Among them, denotes the expectation.

[0067] S6. Update data sparsification, apply the sparsification operator to the updated differences after quantization, where the sparsification parameter only retains the most critical part of the update to further reduce the data transmission volume, and at the same time update the local error record; The formula for the Top-k sparsification operator is:

[0068]

[0069] Among them, k represents retaining the first k elements with the largest absolute values, and Mask represents the mask matrix. By retaining the first k elements with the largest absolute values in the parameter vector, it ensures the high-fidelity transmission of key gradient information; the mask matrix realizes element selection through binary marking (0 / 1), where the non-zero positions correspond to the indices of the parameters to be retained. The sparsification rate directly determines the communication compression ratio and needs to be dynamically adjusted according to the convergence characteristics of the model. Too high will weaken the compression effect, and too low may cause the loss of important gradient information. The sparsification rate formula is:

[0070]

[0071] Among them, d is the parameter dimension.

[0072] S7. The client uploads the updated results after quantization and sparsification to the central server; the clients that are not sampled and participate in this round of training keep their accumulated errors unchanged. The central server aggregates all the received updated results and generates new global model parameters by using weighted average; the global update formula is:

[0073]

[0074] where S t is a set of clients randomly selected according to a fixed ratio, n k is the data volume of client k, and the clients not selected keep

[0075] S8. Repeat the above steps. When the global iteration number is satisfied or the server verifies whether the aggregated result meets the latency constraint τ, end the training and obtain the final trained model.

[0076] Specifically, end the training when any of the following conditions is met:

[0077] Condition 1: The global iteration number

[0078] T ≥ T max

[0079] Condition 2: The validation set loss converges

[0080]

[0081] Condition 3: The communication latency reaches the threshold

[0082]

[0083] where R k is the upload rate of the client, and τ is the latency constraint.

[0084] The above embodiments further elaborate on the purpose, technical solution, and advantages of the present invention. It should be understood that the above embodiments are only the preferred embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made to the present invention within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A federated learning communication optimization method based on adaptive sparsification and quantization, characterized in that, It includes the following steps: The central server initializes the global model parameters and distributes them to the client cluster, and performs multiple rounds of communication iteration; In each round of iteration, some clients are dynamically selected to perform local model training, and the local gradient update amount is calculated; the adaptive sparse screening, dynamic quantization compression and error accumulation compensation operations are sequentially performed on the gradient update amount to generate a compressed gradient data packet; the client uploads the compressed gradient to the server side, and the server aggregates all the client gradients and then applies the momentum acceleration algorithm to update the global model parameters to complete a single communication iteration; Training process It includes the following processes: S1. The central server initializes the global model parameters to be trained; S2. In each global training round, the central server randomly samples a fixed number of clients from all available clients and distributes the current global model parameters to the sampled clients; S3. Each sampled client performs local training and calculation quantization; S4. Model update difference calculation. After the client completes the local SGD iteration, it calculates the update difference in this round, and this difference is the difference between the local model parameters before and after the update; S5. Communication quantization and error correction. The update difference is processed by communication quantization, and different precisions are used to quantize the difference for different clients. At the same time, the quantization error generated during the quantization process and the difference are corrected together to ensure that the update result is not distorted in the expected sense. The untransmitted quantization error is automatically compensated by the weight update in the iterative training; S6. Update data sparsification. The sparsification operator is applied to the quantized update difference, and only the most critical part of the update is retained in the sparsification parameter to further reduce the data transmission volume, and the local error record is updated at the same time; S7. The client uploads the updated result after quantization and sparsification to the central server; the clients not sampled to participate in this round of training keep their accumulated errors unchanged. The central server aggregates all the received updated results and generates new global model parameters by weighted average; S8. Repeat the above steps. When the global iteration number is satisfied or the server verifies whether the aggregation result meets the delay constraint τ, the training is terminated and the final training model is obtained.

2. The federated learning communication optimization method based on adaptive sparsification and quantization according to claim 1 thereof, characterized in that The client sampling step is random sampling, and only some clients are selected to participate in the training in each round of global training to reduce the communication overhead and make full use of the limited network bandwidth resources. The quantization probability formula is: Among them, P k represents the sampling probability of client k, n k represents the data volume of client k, and N represents the total data volume.

3. A federated learning communication optimization method based on adaptive sparsification and quantization according to claim 1 thereof, characterized in that The dynamic quantization compression adopts piecewise non-uniform quantization coding, including: dividing the amplitude interval of the gradient component, adaptively adjusting the quantization bit width according to the dynamic range of the gradient value, and generating a compressed coding sequence including a sign bit, an exponent bit and a mantissa bit. The quantization formula is: Among them, w represents any element in the model, represents the quantization result of the w element, z i and z i+1 represent the upper and lower bounds of the quantization of the w element.

4. An optimization method for federated learning communication based on adaptive sparsification and quantization according to claim 1 thereof, characterized in that, The adaptive sparsification screening adopts a dynamic threshold control strategy, which specifically includes: calculating a determination threshold for gradient importance according to the current training round and gradient distribution characteristics, retaining gradient components with absolute values greater than the threshold to form a sparse gradient set, and the remaining gradient components entering the quantization processing link. For any satisfies the following property, and the formula is: Among them, S k (.) represents the sparsification operator, represents the expectation of the sparsifier S k ; ω represents the d-dimensional parameter model, γ k represents the proportion of the updated information retained by client k during the sparsification process, and its value ranges from 0 to 1. The larger γ k is, the more information is retained (i.e., the lower the sparsification degree), and vice versa, the more information is discarded (i.e., the higher the sparsification degree), thus controlling the amount of transmitted data and the corresponding error level.

5. A federated learning communication optimization method based on adaptive sparsification and quantization according to claim 1 thereof, characterized in that The loss function formula is: where N represents the total number of clients, represents the data set of N clients, D k represents the local data set of the k-th client, |D k | represents the size of this data set, f k represents the loss function of the k-th client locally, {x k ,y k} represents the input-output pair of the model.

6. An optimization method for federated learning communication based on adaptive sparsification and quantization according to claim 1 thereof, characterized in that Communication termination condition judgment module. When the accuracy improvement of the global model in consecutive T communication rounds is less than the preset threshold ε, or when the maximum communication round T is reached, the training process is terminated and the final model is output. The judgment formula is: F(ω t+1 ) - F(ω * ) ≤ ε Among them, ω * represents a model trained in a certain round, and ε represents a preset threshold value.

7. An optimization method for federated learning communication based on adaptive sparsification and quantization according to claim 4 thereof, characterized in that In the method, the client updates the quantization error after each round of training to ensure that the cumulative error brought by the previous round of quantization is compensated during the subsequent local training and uploading process. The model difference calculation formula is: Among them, represents the model of the k-th client in the (t + 1)-th round, which will be used as the input parameter for subsequent calculation of quantization communication.

8. A federated learning communication optimization method based on adaptive sparsification and quantization according to claim 1 thereof, characterized in that The gradient sparsification calculation formula is: where m k represents the quantization precision number of client k, represents the quantization operation, and S represents the sparsification operation.

9. A federated learning communication optimization method based on adaptive sparsification and quantization according to claim 1 thereof, characterized in that, The calculation formula for the global model update rule is as follows: Among them, ω t+1 represents the model in the (t + 1)-th global round, and K represents the total number of participating clients.

Images