Dynamic Aggregation Federated Learning System Based on Pruning and Bidirectional Quantization

Abstract

This invention discloses a dynamic aggregation federated learning system based on pruning and bidirectional quantization, comprising: a group of clients and a server; each client uses the global quantized weights issued by the server as the weight vector of its local model, trains the local model, prunes the trained weight vector, calculates the contribution value and quantization gradient of the corresponding client, and uploads them to the server; the server uses the contribution value as the aggregation weight, updates the global weight vector of the global model, quantizes the global weight vector to obtain the global quantized weights, and distributes them to each client, which then performs the next round of training until the global model converges or reaches the required accuracy. The clients construct lightweight local models through pruning and quantization methods, significantly reducing the number of parameters transmitted; the server employs a dynamic weight allocation strategy, optimizing parameter aggregation based on the data scale and relevance of the clients, reducing the weight of bias parameters, and accelerating the convergence rate of the global model.

Description

Technical Field

[0001] This invention belongs to the field of federated learning technology, and more specifically, this invention relates to a dynamic aggregation federated learning system based on pruning and bidirectional quantization. Background Technology

[0002] Federated learning (FL), as a novel distributed machine learning paradigm, allows multiple clients to collaboratively train a globally optimal model based on their local data, coordinated by a central server. In each iteration cycle, clients independently train their local models and periodically upload their local parameters as updates to the server for aggregation. Simultaneously, the server optimizes and updates the global model, then distributes the updated global model back to the clients for the next training round. Because this process does not require sharing users' original data, it effectively overcomes the data silo problem. Therefore, FL is widely used in fields with high security requirements, such as computer vision, recommender systems, traffic flow prediction, and smart cities. However, with a large number of clients or a large number of model parameters, FL is limited by communication resources and network connectivity, resulting in a heavy bandwidth burden. Furthermore, significant differences in the distribution and quality of client data cause some local parameters to deviate from the global convergence direction during training, severely impacting the model's convergence rate and performance.

[0003] Traditional federated learning methods reduce communication rounds by increasing the number of client training rounds or limiting the number of clients participating in training, thereby reducing communication overhead and bandwidth requirements. However, such methods require all local parameters to be received before aggregation can be performed, resulting in a relatively long communication delay.

[0004] Server aggregation methods rely solely on the weighted average of data uploaded by clients, ignoring the diversity of data distribution. This leads to the accumulation of biased parameters during aggregation, causing global model weight skew and model training instability. To address data heterogeneity and promote model convergence, existing research primarily employs dynamic selection strategies, choosing only unbiased parameters aligned with the global convergence direction for server aggregation. However, these selected parameters fail to fully reflect the characteristics of the data distribution, resulting in incomplete and inaccurate model training. Summary of the Invention

[0005] This invention provides a dynamic aggregation federated learning system based on pruning and bidirectional quantization, which aims to improve at least one of the above-mentioned problems.

[0006] This invention is implemented as follows: a dynamic aggregation federated learning system based on pruning and bidirectional quantization, the system comprising:

[0007] A dynamic aggregation federated learning system based on pruning and bidirectional quantization, characterized in that the system comprises:

[0008] A group of clients and one server;

[0009] The client receives the global quantization weights sent by the server. quantize the global weights As a local model m c The weight vector w, based on local samples, affects the local model m. c Training is performed, the trained weight vector w is pruned, and then the contribution value cv of the corresponding client and the quantization gradient are calculated. And upload it to the server;

[0010] The server receives the contribution value (cv) and quantization gradient uploaded by the client. Use the contribution value cv as the aggregation weight to update the global model M. G The global weight vector W is then quantized to obtain the global quantized weights. The data is distributed to each client, which then performs the next round of training until the global model M is reached. G Convergence or achieving the required accuracy.

[0011] Furthermore, the client comes from the set of participants C. t After quantizing the updated global weight vector W, the server randomly selects T clients in each round according to a set ratio to form participants, and puts them into the participant set C. t .

[0012] Furthermore, client c interacts with local model m. c The importance of the convolutional kernels in each convolutional layer is ranked, and the current local model m is adjusted according to the preset model pruning rate pr. c Pruning is performed by resetting the weights of low-importance convolution kernels in each layer to zero.

[0013] Furthermore, the importance of each layer's convolutional kernels is ranked based on their importance score, where the importance score is the sum of the kernel's L1 norm and gradient.

[0014] Furthermore, based on the preset model pruning rate pr, the pruning rate pr-l of each convolutional layer l is set. The specific method for determining the pruning rate pr-l of each convolutional layer l is as follows:

[0015] The local model m is obtained from the test set. c The initial accuracy αp0 is obtained by removing convolutional kernels with low importance according to a set ratio for the l-th convolutional layer and then updating the local model m. c The accuracy is αp l and the accuracy difference Δαp l =αp l -αp0, estimates the effect of pruning a convolutional kernel in the l-th convolutional layer on the local model m.c Average loss due to accuracy The number of kernels pruned in each layer is determined to ensure that the pruning effect on the local model accuracy is approximately the same for each layer. The pruning effect on the local model accuracy of the l-th convolutional layer is equal to the sum of the kernel pruning data and the average loss of the l-th convolutional layer. The product of.

[0016] Furthermore, the contribution value cv of client c c The specific calculation formula is as follows:

[0017] cv c =ds c *dc c

[0018] Among them, dc c For the local model m c The remaining weight vector w after pruning c With the corresponding global quantization weights from the previous round The correlation between the data, ds c Let c be the sample size of client c, that is, the sample size of client c as a percentage of the participant set C. t The ratio of the total number of samples from all clients.

[0019] Furthermore, data correlation DC c The weight vector w after pruning in the current round in client c c And the previous round of global quantization weights The number of weight parameters with the same direction and the weight vector w c The ratio of the total number of weighted parameters Right now:

[0020]

[0021] Where n is the weight vector w c The total number of weight parameters in the dataset.

[0022] Furthermore, the specific formula for updating the global weight vector is as follows:

[0023]

[0024] Among them, W t+1 W t The global model M is respectively G In the weight vector of the next round and the current round, This represents the contribution value and quantization gradient uploaded by client c in the current round.

[0025] Furthermore, the global model M G The loss function F(W) is as follows:

[0026]

[0027] Where N represents the total number of samples from all clients, N c w represents the sample size of client c. c F c (w c ) represent the weight parameters of client c and the local model m, respectively. c The objective function.

[0028] This invention proposes a dynamic aggregation federated learning system based on pruning and bidirectional quantization. The client constructs a lightweight local model through pruning and quantization methods, significantly reducing the number of parameters transmitted in subsequent transfers. The server adopts a dynamic weight allocation strategy to optimize parameter aggregation based on the client's data scale and relevance, reducing the weight of bias parameters and accelerating the convergence rate of the global model. Finally, the new round of global model is quantized to further improve the overall training efficiency, enabling the model to achieve efficient training in heterogeneous data environments. Attached Figure Description

[0029] Figure 1 A schematic diagram of the structure of a dynamic aggregation federated learning system based on pruning and bidirectional quantization provided in an embodiment of the present invention;

[0030] Figure 2 The MNIST dataset provided in this embodiment of the invention has two partitioning types, where (a) is IID data and (b) is Non-IID data;

[0031] Figure 3 The different model accuracies on the IID data provided in the embodiments of the present invention are shown, wherein (a) is the LeNet-5 model on the MNIST dataset, (b) is the ResNet-20 model on the CIFAR-10 dataset, and (c) is the VGG-16 model on the CIFAR-10 dataset.

[0032] Figure 4 The different model accuracies on the Non-IID data provided in the embodiments of the present invention are shown, wherein (a) is the LeNet-5 model on the MNIST dataset, (b) is the ResNet-20 model on the CIFAR-10 dataset, and (c) is the VGG-16 model on the CIFAR-10 dataset.

[0033] Figure 5 The compression effects of different algorithms provided in the embodiments of the present invention are shown, where (a) represents different model sizes and (b) represents different FLOPs. Detailed Implementation

[0034] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0035] Figure 1 This is a schematic diagram of the structure of a dynamic aggregation federated learning system based on pruning and bidirectional quantization provided in an embodiment of the present invention. For ease of explanation, only the parts related to the embodiments of the present invention are shown. The system includes:

[0036] A set of clients and a server, with the same model integrated on both the clients and the server. The model on the client is trained based only on local parameters and is called a local model. The model on the server updates its weights using parameters uploaded by each client and is called a global model.

[0037] Client C receives the global quantization weights from the server. quantize the global weights As a local model m c The weight vector w, based on local samples, affects the local model m. c After training, the trained weight vector w is pruned, and the contribution value cv and quantized gradient are calculated. And upload it to the server;

[0038] Client c obtains the gradient value g for the current round. c And the gradient value g is obtained by using stochastic gradient quantization. c Perform quantization to obtain the quantization gradient.

[0039] The server receives the contribution value (cv) and quantization gradient uploaded by each client. Use the contribution value cv as the aggregation weight to update the global model M. G After the global model is updated, the global weight vector W is quantized to obtain the global quantized weights. The data is distributed to each client, which then performs the next round of training until the global model M is reached. G Convergence or achieving the required accuracy.

[0040] In round 0, server S initializes the global model. The global parameter W0 is set and distributed to all clients. In each subsequent training iteration, server S randomly selects T clients (10%) to form the participant set C. t The server distributes the global quantization weights of the current global model to the participant set C. t The client in the middle, the set of participants C tClient c in the middle divides the local sample set d according to the batch sample size B. c Divide into several batches, and use the global quantization weights of each batch of samples. The local model is trained.

[0041] In this embodiment of the invention, client c modifies the local model m. c The weight vector of the l-th convolutional layer To achieve model sparsity, pruning is performed. First, the importance of the convolutional kernels in the corresponding layers is ranked, and then the current local model m is pruned according to the preset model pruning rate pr. c Pruning is performed by resetting the weights of low-importance convolutional kernels to zero, so that these kernels will not participate in the local model m. c The update is as follows. The following uses the local model m... c The pruning process is illustrated using the pruning of the l-th convolutional layer as an example. If the local model m c If the l-th convolutional layer has n convolutional kernels, then the local model m c The weights of the convolutional kernel in the l-th convolutional layer are The L1 norm of the convolution kernel is in, Simultaneously calculate the gradient of each convolutional kernel. The gradient of the convolution kernel is obtained by differentiating the loss function with respect to the weights of the corresponding convolution kernel. Among them, F c Let c be the loss function of the client. The sum of the L1 norm of the convolution kernel and the gradient value is used as the importance score. Based on the preset model pruning rate pr, the weights of the m convolution kernels with low importance scores are reset to zero.

[0042] In this embodiment of the invention, in order to balance the accuracy loss during multi-layer pruning, before performing the pruning operation, the pruning rate pr-l of each convolutional layer l is set based on a preset model pruning rate pr, ensuring that the impact of the pruned convolutional kernels of each convolutional layer on the local model accuracy is approximately equal. In the subsequent pruning process, the number of prunes is determined based on the pruning rate pr-l of each layer. The specific method for determining the pruning rate pr-l of each convolutional layer l is as follows:

[0043] The initial local model m is obtained from the test set. c To achieve the accuracy αp0, for the l-th convolutional layer, after removing a set proportion (60%) of the less important convolutional kernels, update the local model m. c The accuracy is αp l and the accuracy difference Δαp l ,Δαp l Divide by the number of convolutional kernels at the set ratio (60%) of this layer, and estimate the effect of pruning one convolutional kernel in the l-th convolutional layer on the local model m. c Average loss due to accuracy The number of convolutional kernels pruned in each layer is determined to ensure that the local model accuracy loss due to pruning is basically the same for each layer. The accuracy loss of the local model due to pruning in the l-th convolutional layer is equal to the sum of the number of convolutional kernels pruned in the l-th layer and the average loss. The product of.

[0044] In this embodiment of the invention, the set of participants C t The client sample size set is DS = {ds} 1 ,ds 2 ,...,ds T}, sample size ds c The sample size of client c accounts for a proportion of the participant set C. t The ratio of the total number of samples from all clients is used to measure the local model m using formula (1). c The weight vector w after pruning c Compared with the previous round of global quantization weights The correlation between the data, and the previous round of global quantization weights. The global quantization weights issued by the server are used as the local model m. c The initial weight vector w for the current round c If the weight vector w after the current round of pruning c And the previous round of global quantization weights If the directions are consistent, then Otherwise, it is 0. The set of participants, C. t The client set in the data is DC = {dc} 1 ,dc 2 ,...,dc T}, data correlation of client c dc c The weight vector w after pruning in the current round in client c c And the previous round of global quantization weights The number of weight parameters with the same direction and the weight vector w c (Previous round of global quantization weights) The ratio of the total number of weighted parameters A higher ratio indicates fewer deviation parameters in client c.

[0045]

[0046] Where n is the weight vector w c Or, the total number of weight parameters of the global quantization weight W in the previous round, where the contribution value cv of client c is... c The specific calculation formula is as follows:

[0047] cv c =ds c *dc c (2).

[0048] Set of participants C t Each client in the process will record its contribution value (cv) for the current round. c and quantization gradient Uploaded to the server.

[0049] Server detection participant set C t Are the data sizes of all clients the same? If so, then adjust the contribution value cv. c As the aggregation weight, the client's contribution value cv is used here. c data correlation DC c Regarding client-side data sizes, when different data sizes exist, the server considers both data size and data relevance, multiplying their product as the contribution value cv. c To perform weight aggregation and ensure a reasonable allocation of aggregation weights, it is necessary to perform weight aggregation on the set of participants C. t The contribution values ​​of all clients are normalized so that the set of participants C is equal. t The sum of the weights of all clients in the process equals 1.

[0050] By analyzing the scale and relevance of data uploaded by clients, aggregation weights are dynamically allocated, aggregation bias parameters are precisely optimized to prevent global weight shifts, ensuring smooth and rapid convergence of the global model when aggregating data from multiple sources. The server is based on the set of participants C. t The contribution value (cv) of all clients c Update global model M G Weight vector W t →W t+1 To obtain the global weight vector W for the next round. t+1 The updated formula is as follows:

[0051]

[0052] Among them, W t For the global model M G In the weight vector of the current round, The contribution value and quantization gradient uploaded by client c in the current round are then used to quantize the global weight vector W for the next round using a ternary quantization method. t+1 Quantization is performed to obtain global quantization weights. At the same time, T participating clients are randomly selected and added to the participant set C. t Quantize the global weights Distributed to the set of participants C t All clients in the system.

[0053] Although the pruning operation can significantly reduce the size of the local model, the communication burden in the transmission phase is still excessively heavy, and clients and servers need to further quantize model parameters to reduce communication overhead. The present invention uses a bidirectional quantization method to implement model compression at both global and local levels simultaneously, and adopts a differentiated quantization strategy for weights and gradients to reduce information loss and optimize model performance. Clients quantize local gradients and then transmit them to the server, and the server quantizes the updated global weights and then distributes them to randomly selected clients.

[0054] In the embodiment of the present invention, the global model M G has the following specific loss function F(W):

[0055]

[0056] Wherein, N represents the total number of samples of all clients, N c is the sample size of client c, w c and F c (w c ) respectively represent the weight parameter of client c and the local model m c 's objective function. When the loss value calculated by the loss function and the model training accuracy curve converge, or the recognition accuracy of the global model M G meets the required accuracy requirement, the training is stopped.

[0057] Assume that the neural network with only one hidden layer has a total of N parameters, the batch size is B, the client only performs one mini-batch training per round, and the time complexity of uploading local parameters to the server is negligible. The time complexity of FedAvg is O(NB). The time complexity of the dynamic aggregated federated learning algorithm (referred to as FL-STMV) based on pruning and bidirectional quantization proposed by the present invention mainly includes the following three key components: multi-indicator pruning is based on model parameters, and the time complexity of performing one parameter evaluation and sorting is O(N+NlogN); the time complexity of quantizing the sparse model is O(N′), where N′ is the number of remaining parameters after pruning, and O(N′) < O(N); the time complexity of calculating the client contribution value is O(N′). Taken together, the total time complexity of FL-STMV on each client can be expressed as: O(N(1+logN)+2N′).

[0058] On the server side, the average time complexity of FedAvg is O(lCN), where l represents the client participation rate, C is the total number of clients, and lC≥1. The time required for FL-STMV to dynamically weighted aggregate according to the contribution values of clients is also O(lCN). Finally, the time complexity of quantizing the global weights is O(N), so the total time complexity of FL-STMV on the server side is O(N(1+lC)).

[0059] The experiment involved 100 clients and 1 server. In each round, 10% of the clients were randomly selected to train the model on two widely used public datasets: MNIST and CIFAR-10. MNIST is a 28×28 grayscale handwritten digit image dataset with 10 classes, consisting of 60,000 training images and 10,000 test images. CIFAR-10 contains 10 classes of 32×32 color image labeled samples, with 6,000 images per class, of which 50,000 were used for training and 10,000 for testing.

[0060] LeNet-5 was chosen as the training model for MNIST. This model has two convolutional layers and three fully connected layers, totaling approximately 60,000 parameters, making it suitable for simple image recognition tasks. For the more complex CIFAR-10, ResNet-20 and VGG-16 network models were selected. ResNet-20 is a deep residual network containing six residual blocks, each consisting of three convolutional layers and one skip connection, totaling approximately 270,000 parameters. VGG-16 includes 13 convolutional layers and three fully connected layers, with a total of approximately 138 million parameters.

[0061] The training samples for MNIST and CIFAR-10 were divided into two types: Independent and Identically Distributed (IID) and Non-Identically Distributed (Non-IID), while the test samples remained independent. This was used for subsequent accurate evaluation of model performance. Taking MNIST as an example, the specific distribution is as follows: Figure 2 As shown in the figure, the y-axis represents the number of samples and the z-axis represents the sample categories (10 categories in total). The partitioning process is described in detail below.

[0062] (1) IID data: In order to simulate the real data distribution and ensure the randomness and independence of the samples, we randomly shuffled the order of the samples in the dataset and appropriately allocated the samples according to the category to meet the needs of each client to hold the same category and amount of data.

[0063] (2) Non-IID data: The client randomly selects different numbers and categories of samples according to the distribution ratio k of each category in the dataset, thereby simulating the heterogeneity of data distribution in reality. k represents the number of sample categories held by each client. The experiment sets k = 5.

[0064] Furthermore, this invention selects three mainstream algorithms most relevant to the proposed FL-STMV as benchmarks:

[0065] (1) FedAvg is a federated learning FL algorithm without any improvements; (2) FTTQ algorithm only uses quantization to reduce the amount of data transmitted uplink and downlink; (3) GWEP algorithm combines pruning and quantization compression methods in model training.

[0066] The experiments were conducted on an Ubuntu 18.04 system equipped with an Intel Ultra9-185H CPU, an RTX 4060 GPU, and 16GB of RAM. All models were built and trained using Python within the Tensorflow framework. Table 1 shows the models selected for the experiments and their parameter settings.

[0067] Table 1 Model and parameter settings

[0068]

[0069] This section compares the accuracy of three models (local and global models) – LeNet-5, ResNet-20, and VGG-16 – on MNIST and CIAFR-10 using the four algorithms described above. 100 iterations of training were performed on both IID and Non-IID data distributions. Each client independently trained for 5 iterations locally. The cross-entropy loss function was used to evaluate model performance. The pruning threshold (pr) and quantization bit depth were set to 0.6 and 8, respectively. Experiments showed that this effectively simplified the model while maintaining accuracy. A detailed analysis of the performance of each algorithm under different datasets and data distributions is provided.

[0070] (1) Precision of IID data

[0071] On MNIST and CIFAR-10, each client stores a certain number of training samples. The four algorithms are iteratively trained on the IID data. The accuracy curves of the LeNet-5, ResNet-20, and VGG-16 models on the test set are shown below. Figure 3 As shown, detailed test accuracy is shown in Table 2.

[0072] Table 2 Detailed Model Accuracy

[0073]

[0074] from Figure 3As shown in Table 2, FL-STMV performed best when training the LetNet-5 model on MNIST, achieving a test accuracy of up to 94.74%, which is 1.83% and 3.19% higher than GWEP and FTTQ, respectively. Furthermore, FL-STMV reached a stable state in approximately 60 epochs, while GWEP and FTTQ required 90 epochs to achieve similar convergence. This is mainly attributed to the pruning and quantization processes, which comprehensively consider multiple factors and finely control the model's accuracy loss. Similarly, when training the ResNet-20 model on CIAFR-10, FL-STMV achieved higher test accuracy than FedAvg and the other two compression algorithms, and had the fastest convergence speed. Although the accuracy of the VGG-16 model was slightly lower than FedAvg, FL-STMV reached the required accuracy faster, significantly reducing communication costs. In summary, FL-STMV not only significantly improves model training efficiency but also maintains high accuracy.

[0075] (2) Precision of Non-IID data

[0076] When k is less than the total number of classes in the training dataset, the client's local data satisfies Non-IID. In this case, the union of all client data equals the entire dataset, but the number of classes contained in each client is not equal to the total number of classes in the dataset, and there is some overlap in the data between different clients. Figure 4 The model accuracy curves of the four algorithms on the Non-IID dataset are shown. Detailed test accuracy is shown in Table 3.

[0077] Table 3 Detailed Model Accuracy

[0078]

[0079] Figure 4It can be seen that when k=5, FL-STMV improves the accuracy of MNIST by 1.05% and 2.4% compared to GWEP and FTTQ, respectively, while the accuracy loss between FL-STMV and FedAvg is negligible. This is mainly attributed to FL-STMV's optimization of the aggregation process; the dynamic aggregation strategy effectively accelerates the training process without sacrificing model accuracy. When training the other two models on CIAFR-10, FL-STMV performs best, achieving test accuracies of 91.17% and 88.18% for ResNet-20 and VGG-16, respectively, both superior to GWEP and FTTQ. In terms of convergence speed, FL-STMV also outperforms GWEP and FTTQ, further demonstrating its effectiveness and efficiency on the Non-IID dataset. The model compression effect of different algorithms is comprehensively evaluated using two key metrics: model size and FLOPs. Model size directly reflects the storage space occupied by each model and the number of parameters it contains during training, while FLOPs effectively measures the computational cost required to train the model. Intermediate variables generated during training are ignored. Because the number of layers and structural complexity of the LeNet-5 model differ significantly from the other two models, the sizes of LeNet-5 and ResNet-20 models are measured in MB, while VGG-16 is measured in GB. Similarly, the computational cost of LeNet-5 is expressed in FLOPs, while ResNet-20 and VGG-16 models use GFLOPs. Figure 5 It visually demonstrates the memory size and FLOPs required for a single client to train three models under different algorithms.

[0080] like Figure 5 As shown in (a), the dense model obtained by FedAvg has a memory footprint as high as 2.06GB, resulting in a high storage burden. FTTQ and GWEP have limited compression effects, only compressing the model size to about 1 / 5 of its original size. FL_STMV significantly outperforms the above two methods, achieving a compression effect of approximately 17.1 times. This indicates that each model has too many redundant parameters. Figure 5 As shown in (b), training the VGG-16 model with FedAvg requires as much as 15.5 GFLOPs of computation. For resource-constrained clients, this high computational requirement leads to a slow training process. The FL_STMV algorithm reduces the computational complexity by at least 15.2 times, significantly reducing the client's resource consumption. Furthermore, the compression process should follow clear metrics; blindly reducing the model size can lead to a significant loss of accuracy.

[0081] The experiment set the target accuracy for the LeNet-5 model on MNIST to 85%, while the target accuracy for the ResNet-20 and VGG-16 models on CIFAR-10 was set to 80%. The number of communication rounds and the average training time per round required to train the models to achieve the preset accuracy using the four algorithms were observed. The specific results are shown in Table 4.

[0082] Table 4. Number of communication rounds and time required for different algorithms to achieve preset accuracy.

[0083]

[0084] As shown in Table 4, when training the LeNet-5 model, FL-STMV and FedAvg reached the preset accuracy in similar times. However, FL-STMV required significantly fewer communication rounds than the other three algorithms, mainly due to the dynamic aggregation strategy accelerating model convergence. On the more complex CIFAR-10, although the number of communication rounds and time required for all four algorithms to reach the target accuracy increased significantly, FL-STMV still maintained high training efficiency. Particularly when training the VGG-16 model, its communication rounds were reduced by 22 and 24 rounds compared to GWEP and FTTQ, respectively. These results demonstrate that FL-STMV effectively reduces communication costs while maintaining high efficiency.

[0085] To address communication constraints and model weight bias caused by data heterogeneity during transmission, this invention proposes a resource-friendly and efficient communication algorithm, FL-STMV. By pruning redundant model parameters, quantizing transmitted data, and dynamically allocating aggregate weights to the client, FL-STMV improves the model's generalization ability under different data distributions while effectively utilizing client resources and efficiently processing data transmission. Experimental data on two public datasets demonstrate that FL-STMV outperforms comparative algorithms in training efficiency and performance, without significant loss of model accuracy.

[0086] The present invention has been described by way of example. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A dynamic aggregation federated learning system based on pruning and bidirectional quantization, characterized in that, The system includes: A group of clients and a server are used. In each round, a set proportion of clients are randomly selected to train the model on the MNIST and CIFAR-10 datasets. MNIST is a 28×28 grayscale handwritten digit image dataset, and CIFAR-10 is a 32×32 color image labeled sample dataset. The client receives the global quantization weights sent by the server. Quantize the global weights As a local model weight vector Local model based on local samples Training is performed on the trained weight vector. Pruning is performed, followed by calculating the contribution value of the corresponding client. and quantization gradient And upload it to the server; The server receives the contribution value uploaded by the client. and quantization gradient , contribution value As aggregate weights, update the global model. global weight vector For the global weight vector Perform quantization to obtain global quantization weights. The data is then distributed to each client, which performs the next round of training until the global model is fully implemented. Convergence or achieving the required accuracy; Client For local models The importance of the convolutional kernels in each convolutional layer is ranked, and the pruning rate is set according to the preset model. For the current local model Pruning is performed by resetting the weights of low-importance convolutional kernels in each layer to zero; The importance of each convolutional kernel is ranked based on its importance score, where the importance score is the sum of the L1 norm and gradient of the convolutional kernel. Based on the preset model pruning rate Set the pruning rate for each convolutional layer l pruning rate of each convolutional layer l The specific method for determining it is as follows: Local models are obtained from the test set. initial accuracy For the l-th convolutional layer, after removing convolutional kernels with a set proportion of low importance, the local model is updated. The accuracy is and accuracy difference It is estimated that removing one convolutional kernel from the l-th convolutional layer affects the local model. Average loss due to accuracy Determine the number of convolutional kernels to be pruned in each layer, ensuring that the pruning effect on the local model accuracy is roughly the same for each layer. The pruning effect on the local model accuracy of the l-th convolutional layer is equal to the sum of the pruning data of the convolutional kernels in the l-th convolutional layer and the average loss. The product; Contribution value of client c The specific calculation formula is as follows: ； in, For local models Residual weight vector after pruning With the corresponding global quantization weights from the previous round The correlation between the data Let c be the sample size of client c, that is, the sample size of client c as a percentage of the total number of participants. The ratio of the total number of samples from all clients in the sample; Data Relevance The weight vector after pruning in the current round in client c. And the previous round of global quantization weights Number of weight parameters with consistent direction and weight vector The ratio of the total number of weighted parameters ,Right now: ； in, Weight vector The total number of weight parameters in the data; The specific formula for updating the global weight vector is as follows: ； in, , Global model In the weight vector of the next round and the current round, This represents the contribution value and quantization gradient uploaded by client c in the current round.

2. The dynamic aggregation federated learning system based on pruning and bidirectional quantization as described in claim 1, characterized in that, The client comes from the set of participants. On the updated global weight vector After quantification, the server randomly selects T clients each round according to a set ratio to form the participants and adds them to the participant set. .

3. The dynamic aggregation federated learning system based on pruning and bidirectional quantization as described in claim 1, characterized in that, Global Model loss function Specifically as follows: ； in, This represents the total number of samples from all clients. This represents the sample size of client c. , These represent the weight parameters of client c and the objective function of the local model, respectively.

Images