Federal learning dynamic cutting method and device based on gradient correlation

Abstract

The invention provides a federal learning dynamic cutting method and device based on gradient correlation, and the method comprises the steps: receiving a global model issued by a server; training the global model according to the target local data to obtain a local model; determining an update of each level of the local model and an update of each level of the global model; determining a correlation value for each level between the local model and the global model; determining hierarchies of which the correlation values are smaller than a preset correlation threshold value in the local model as cutting hierarchies of the local model; determining hierarchies of which the correlation values are greater than a preset correlation threshold value in the local model as reserved hierarchies of the local model; cutting the invalid update corresponding to the cutting hierarchy of the local model, and returning the valid update corresponding to the reserved hierarchy of the local model to the server; according to the invention, federal learning efficiency and model performance can be improved.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to a federated learning dynamic pruning method and device based on gradient correlation. Background Art

[0002] Federated Learning (FL) is indeed a cutting-edge and important machine learning paradigm, especially in the current era of data privacy protection. It allows multiple clients (such as different organizations or personal devices) to jointly train a global model without directly sharing their local data. This approach not only helps improve the generalization ability of the model, but also effectively protects the data privacy of each participant.

[0003] However, due to the differences in data distribution among clients, this difference may lead to a large deviation between the local model (i.e., the model trained by each client based on its local data) and the global model, thus affecting the effectiveness and efficiency of federated learning.

[0004] It can be seen that the federated learning method in the related technology has a technical problem of poor federated learning efficiency due to the local data deviation of each client. Summary of the invention

[0005] The present invention provides a method and device for dynamic clipping of federated learning based on gradient correlation, so as to solve the defect of poor efficiency of federated learning caused by local data deviation of each client in the prior art, and improve the efficiency and model performance of federated learning.

[0006] The present invention provides a federated learning dynamic pruning method based on gradient correlation, comprising the following steps.

[0007] Receive a global model sent by a server, wherein the global model is constructed based on local models of multiple participants in a distributed learning environment; train the global model according to target local data to obtain a local model; determine the update of each level of the local model and the update of each level of the global model, wherein the global model is trained based on the local data of the multiple participants respectively; determine the correlation value of each level between the local model and the global model based on the update of each level of the local model and the update of each level of the global model; determine the level in the local model whose correlation value is less than a preset correlation threshold as the trimmed level of the local model; determine the level in the local model whose correlation value is greater than the preset correlation threshold as the retained level of the local model; trim the invalid updates corresponding to the trimmed level of the local model, and return the valid updates corresponding to the retained level of the local model to the server, so that the valid updates are synchronized to the global model.

[0008] According to a method for dynamic clipping of federated learning based on gradient correlation provided by the present invention, after determining the level in the local model whose correlation value is greater than the preset correlation threshold as the retained level of the local model, the method further includes: clipping the clipped levels of the local model, and retaining the retained levels of the local model to obtain a clipped local model; determining the L2 norm mean of the gradient of the retained levels of the local model as the clipping threshold; determining the L2 norm mean of the gradient of the clipped local model as the target norm mean; when the target norm mean is greater than the clipping threshold, performing gradient clipping on the clipped local model to obtain an updated gradient.

[0009] According to a federated learning dynamic clipping method based on gradient correlation provided by the present invention, the gradient clipping is expressed by the following formula: ; in, represents the updated gradient, represents the clipping threshold, represents the gradient of the pruned local model, represents the target norm mean.

[0010] According to a federated learning dynamic clipping method based on gradient correlation provided by the present invention, after determining the L2 norm mean of the gradient of the clipped local model as the target norm mean, the method also includes: when the target norm mean is less than the clipping threshold, adding noise to the clipped local model to obtain an updated local model.

[0011] According to a method for dynamic pruning of federated learning based on gradient correlation provided by the present invention, after determining the level in the local model whose correlation value is greater than the preset correlation threshold as the retained level of the local model, the method further includes: allocating a privacy budget for valid updates corresponding to the retained level of the local model to obtain a target privacy budget; and performing differential privacy perturbation on the target privacy budget.

[0012] According to a method for dynamic pruning of federated learning based on gradient correlation provided by the present invention, the correlation value of each level between the local model and the global model is determined based on the update of each level of the local model and the update of each level of the global model, including: calling a correlation update function to determine the correlation value of the update of each level of the local model in the current training round and the update of each level of the global model in the previous training round, and an update mark of each level of the local model; wherein the update mark is used to mark the correlation between the current level of the local model and the corresponding level of the global model.

[0013] The present invention also provides a federated learning dynamic pruning device based on gradient correlation, comprising the following modules: a receiving module for receiving a global model sent by a server, wherein the global model is constructed based on local models of multiple participants in a distributed learning environment; a training module for training the global model according to target local data to obtain a local model; a determination module for determining the update of each level of the local model and the update of each level of the global model, wherein the global model is trained based on the local data of the multiple participants respectively; a correlation module for determining the update of each level of the local model and the update of each level of the global model based on the update of each level of the local model and the local data of the participants respectively; The update of each level of the global model determines the correlation value of each level between the local model and the global model; a trimming module is used to determine the level in the local model whose correlation value is less than a preset correlation threshold as the trimmed level of the local model; a retention module is used to determine the level in the local model whose correlation value is greater than the preset correlation threshold as the retained level of the local model; an updating module is used to trim the invalid updates corresponding to the trimmed levels of the local model, and return the valid updates corresponding to the retained levels of the local model to the server, so that the valid updates are synchronized to the global model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the dynamic pruning method for federated learning based on gradient correlation as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for dynamic pruning of federated learning based on gradient correlation.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned federated learning dynamic pruning methods based on gradient correlation.

[0017] The present invention provides a method and device for dynamic clipping of federated learning based on gradient correlation. The global model is the result of joint training by multiple participants. After receiving the global model, local training is performed on this basis without starting from scratch, thereby accelerating the learning process of the model; local training enables the model to learn the data distribution characteristics unique to the client and enhances the personalization ability of the model; by comparing the updates of the local model and the global model at each level, the correlation value is determined, which helps to quantify the degree of association between the local model update and the global model update; clipping updates at low correlation levels (lower than a preset correlation threshold) can reduce the transmission of noise and redundant information, reduce communication costs, and also help maintain the stability and consistency of the global model; retaining updates at high correlation levels (higher than a preset correlation threshold) can ensure that the global model can absorb valuable updates that have a positive impact on performance improvement, thereby accelerating the optimization process of the global model; by clipping invalid updates and transmitting only valid updates, communication efficiency can be significantly improved and unnecessary resource consumption can be reduced; thereby solving the technical problem of poor federated learning efficiency caused by local data deviations of each client in the federated learning method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the dynamic pruning method of federated learning based on gradient correlation provided by the present invention.

[0020] Figure 2 It is a schematic diagram of the model cutting process provided by the present invention.

[0021] Figure 3 It is an overall flow chart of the dynamic pruning method of federated learning based on gradient correlation provided by the present invention.

[0022] Figure 4It is a schematic diagram comparing the communication overhead of different threshold settings under MNIST provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the accuracy comparison of different threshold models under MNIST provided by the present invention.

[0024] Figure 6 This is a comparison chart of communication overhead of multiple algorithm models under MNIST provided by the present invention.

[0025] Figure 7 This is a comparison chart of the accuracy of multiple algorithm models under MNIST provided by the present invention.

[0026] Figure 8 This is a comparison chart of communication overheads under different threshold settings under CIFAR-10 provided by the present invention.

[0027] Fig. 9 This is a comparison chart of the accuracy of different threshold models under CIFAR-10 provided by the present invention.

[0028] Fig.10 This is a comparison chart of communication overhead of multiple algorithm models under CIFAR-10 provided by the present invention.

[0029] Fig.11 This is a comparison chart of the accuracy of multiple algorithm models under CIFAR-10 provided by the present invention.

[0030] Fig.12 It is a structural schematic diagram of the gradient correlation-based federated learning dynamic clipping device provided by the present invention.

[0031] Fig.13 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Federated learning is an emerging distributed machine learning method that protects data privacy by enabling multiple clients to collaborate on training a global model without sharing the original data. However, since the data distribution of each client may be significantly different, the difference between the local model and the global model is large, which poses a challenge to the convergence speed and accuracy of the model. Therefore, it is of great significance to study how to effectively quantify and select local model parameters that contribute more to the global model. This paper proposes a dynamic pruning method for federated learning based on gradient correlation to improve the efficiency and model performance of federated learning.

[0034] Federated learning is a concept proposed for the optimization of keyboard input methods for individual users. At the same time, they proposed the Federated Averaging Algorithm (FedAvg), the core idea of ​​which is to optimize the local stochastic gradient descent process of the data owner separately and perform aggregation operations on the central server. Since then, the concept of federated learning has entered the public's field of vision. This algorithm also provides a basic process for horizontal federated learning. It is also the most classic and widely used algorithm in federated learning, laying a solid foundation for the subsequent development of federated learning.

[0035] Compared with centralized deep learning, the FedAvg algorithm converges slowly and has poor stability in heterogeneous environments. Several federated learning algorithms have been proposed to better deal with the above problems. In the related art, a federated optimization algorithm for heterogeneous networks, FedProx, is proposed. By adding a proximal term to the model update, the gradient update is corrected to make it closer to the current model, effectively limiting the impact of some local updates that differ too much from the global model. However, the value of the proximal term is very sensitive to the performance of FedProx, and its adjustment depends on the prior knowledge of the statistical distribution. In the related art, client and server control variables are introduced, and the difference between the control variables is calculated during the communication process to correct the model parameters with excessive local update offsets, so that it performs better in heterogeneous environments, but due to the additional control variables, its per-round communication cost is doubled. In the related art, FedADMM is proposed, which is a federated learning algorithm derived from the primal-dual optimization method. One of its unique properties is the use of dual variables to guide the local training process, which allows automatic adaptation to data distribution without parameter tuning.

[0036] Federated learning generally has communication problems in large-scale global learning scenarios. To solve this problem, QSGD is proposed in related technologies to reduce the communication burden through discrete simulation and encoding; the CMFL solution is further proposed to reduce the number of clients by excluding irrelevant updates, thereby reducing communication overhead. Other technical solutions, such as COFEL and FedCPF, have proposed more sophisticated communication optimization mechanisms on this basis, which achieve more efficient communication through fine-grained selection and local fractional privacy protection. At the same time, there is also an adaptive learning algorithm based on evolutionary strategies that can personalize the construction of client sub-models and significantly reduce communication costs.

[0037] In federated learning, the global model is obtained by aggregating a large number of local models of users. Since the local model is trained with the local data of a specific user, there are usually differences between the local model and the global model. The model pruning process proposed in this topic is as follows: In each learning iteration, the client first receives feedback information about the global update from the central server. Next, the client continues local training and generates local updates. The client then compares it with the global update to check the degree to which the two gradients are aligned with each other. Through a correlation measurement method, local models with smaller differences from the global model are uploaded to the server to participate in aggregation, while models with larger differences from the global model, that is, models with smaller contributions, are not uploaded. The correlation measurement method only considers the similarity of the gradient signs, while the model proposed in the embodiment of the present invention also takes the similarity of the gradient size into account.

[0038] In addition, during the federated learning process, the client's gradient L2 norm will decrease as the number of iterations increases. If a unified gradient clipping threshold is used, it will have a negative impact on the accuracy of the model. Therefore, a dynamic adaptive gradient clipping solution is proposed. In each iteration, the L2 norm of all unclipped gradients is calculated and their average is calculated.

[0039] The present invention proposes a dynamic pruning method for federated learning based on gradient correlation. Each client does not need to upload the entire model, but selects part of the model layer to upload. The amount of model parameters uploaded is reduced, and the privacy budget is only allocated to the parameters that need to interact, thereby reducing the waste of the privacy budget. In federated learning, the global model is obtained by aggregating a large number of local models of users. Since the local model is trained with the local data of a specific user, there are usually differences between the local model and the global model. Through the method of quantifying the correlation based on the layer perspective, local model parameters that contribute more to the global model can be selected, thereby improving the training efficiency and performance of the model. At the same time, through the method of dynamically adaptively adjusting the pruning threshold, the pruning threshold of the gradient can be effectively controlled, the overall noise amount of the model, the consumption of the privacy budget and the communication cost can be reduced, and finally a more efficient and accurate global model can be obtained.

[0040] Optionally, the dynamic pruning method of federated learning based on gradient correlation in an embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device. For example, the dynamic pruning method of federated learning based on gradient correlation in this embodiment is executed by a client (terminal) of a federated learning participant.

[0041] Figure 1 is a flow chart of the dynamic pruning method of federated learning based on gradient correlation provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0042] Step 101: receiving a global model sent by a server, wherein the global model is constructed based on local models of multiple participants in a distributed learning environment.

[0043] In federated learning, the global model is constructed by aggregating local models from many clients. Each local model is trained based on a specific local dataset.

[0044] For example, in a distributed learning environment, each client (or participant) has a specific local dataset. The client uses its local dataset to train an initial model (e.g., a copy of the global model or a completely independent model) to obtain a local model.

[0045] After training is complete, the client uploads the parameters (or updates) of the local model to the server. These updates include changes in weights, gradients, or other forms of model parameters. The server receives local model updates from all (or some) clients; uses aggregation algorithms (such as weighted average, federated average, etc.) to merge these updates into a global model update; the server applies the global model update to the current global model to obtain a new global model.

[0046] The server sends the new global model to all (or some) clients. The clients can continue training based on the new global model to form new local models and repeat the above process.

[0047] Step 102: Train the global model according to the target local data to obtain a local model.

[0048] In an embodiment of the present invention, each client obtains the latest global model from the server. The global model is obtained by aggregating the local model updates of multiple clients. The client uses the obtained global model as a starting point or basis to create a copy of the local model; this copy will be used for training on the local data set. The client trains the local model using its local data set, and the training process includes steps such as forward propagation, loss calculation, back propagation, and weight update.

[0049] After a certain number of training iterations, the local model will gradually adapt to the local dataset and may show better performance than the initial global model (at least on the local dataset). At this point, a local model has been trained based on the target local data.

[0050] Step 103, determining the update of each level of the local model and the update of each level of the global model, wherein the global model is trained based on the local data of multiple participants.

[0051] The local model is a model trained on the client based on the local data set. During the training process, the weights of each layer of the local model (such as convolutional layer, fully connected layer, etc.) are updated according to the characteristics of the local data.

[0052] For example, local models usually use the back-propagation algorithm to calculate gradients and update weights based on the gradients. The weight update may use optimization algorithms such as stochastic gradient descent (SGD) and Adam.

[0053] The global model is a model obtained by aggregating local model updates from multiple clients on the server. The update of the global model is usually performed on the server, and the performance of the global model is improved by aggregating local model updates from different clients.

[0054] For example, how the global model is updated depends on the specific aggregation algorithm, such as FedAvg; during the aggregation process, the server calculates the average or weighted average of all local model updates and applies it to the global model.

[0055] It should be noted that the local model usually starts from a copy or initialized version of the global model, so before training begins, each layer of the local model is the same as the corresponding layer of the global model.

[0056] During the training process, due to the differences between the local dataset and the global dataset, each level of the local model will gradually adapt to the characteristics of the local data and differ from the corresponding level of the global model; these differences reflect the uniqueness and diversity of the local data.

[0057] Step 104 , based on the update of each level of the local model and the update of each level of the global model, determine the correlation value of each level between the local model and the global model.

[0058] For each layer of the local model and the global model, extract its weight update vector; the weight update vector can be expressed as the change in the weight of the layer during the training process.

[0059] Use appropriate correlation measurement methods (such as Pearson correlation coefficient, cosine similarity, mutual information, etc.) to calculate the correlation value between the weight update vectors of the local model and the global model at each level. The correlation value is used to reflect the similarity or consistency of the two models at the corresponding level.

[0060] For example, the Pearson correlation coefficient between the weight update vectors of the local model and the global model at each level is calculated to evaluate the linear correlation between them; or the cosine similarity between the weight update vectors of the local model and the global model at each level is calculated to evaluate the directional consistency between them.

[0061] According to a dynamic pruning method for federated learning based on gradient correlation provided by the present invention, based on the update of each level of the local model and the update of each level of the global model, the correlation value of each level between the local model and the global model is determined, including: Call the correlation update function to determine the correlation value of the update of each level of the local model in the current training round and the update of each level of the global model in the previous training round, and the update mark of each level of the local model; The update mark is used to mark the correlation between the current level of the local model and the corresponding level of the global model.

[0062] In federated learning, the global model is constructed by aggregating local models from many clients. Each local model is trained based on a specific local dataset, so there are often differences between the local model and the global model. Specifically, when the model is updated, the sign and size of the parameter indicate the direction of the model improvement in that parameter dimension. In this case, the server hopes that users will upload parameters that contribute more to the global model, that is, those that are roughly the same in sign and size as the global model parameters, so as to improve the accuracy and convergence speed of the model.

[0063] A common practice is to use a correlation-based method to compare the difference between the local model and the global model, and upload the model with higher correlation to the server for aggregation. For the model with lower correlation, it is not allowed to participate in the aggregation of the global model because it may have a negative impact on the global model.

[0064] In the current scenario, the client selectively uploads the model strategy in the way that the client either uploads the entire model or does not upload it at all. This method does not take into account the unique characteristics that different layers in the neural network may exhibit and the differences in training performance and correlation. Therefore, the present invention proposes a new method to quantify the correlation between layers from a hierarchical perspective, so that the decision to upload the model can be more detailed and accurate.

[0065] Refer to the following Algorithm 1, which is used to describe the layer-based perspective quantization correlation.

[0066] The measure of correlation can be measured by the similarity of the client and global model gradients. From the perspective of optimization, the core goal of federated learning is to fit an approximate solution to an unknown function, which usually involves minimizing one or more loss functions to make the fitted function as close as possible to the true unknown function. In this process, the gradient used to fit the unknown function indicates the direction of optimization. Therefore, if the training gradient obtained by the combined data is closer to the theoretically optimal gradient direction, the model is considered to be closer to the ideal state, indicating that the data is more valuable.

[0067] Specifically, by calculating the cosine similarity between each layer of gradient updates obtained by the client training in the federated model and each layer of global model gradient updates trained by all participating parties’ data, the contribution of this layer to the global model can be evaluated.

[0068] The global optimization process first uses multiple client local models of the current round 𝑡 and the last round of global model update As input, the goal is to synchronize updates between each client’s local model and the global model. In the iterative loop of global optimization, each training round 𝑡 is passed, and each client’s local model will be updated in parallel. During this process, the local model will calculate the relevance of its updates layer by layer.

[0069] During the update process, call the RelevanceUpdate function to determine the current layer update The correlation with the previous round of global model update. This function returns two results: correlation value and update flag For layers marked as relevant, their updates are collected into the set 𝑆 and used to update the global model. Specifically, the RelevanceUpdate subprocess comprehensively calculates the cosine similarity and distance between the current layer update and the previous round of global update. When the threshold 𝑄 is exceeded, the current layer is considered to contribute more to the global model, True is returned and the update is retained; otherwise, False is returned and the update is discarded to avoid invalid or negative impact on the global model. Finally, all the filtered valid updates are synchronized to the global model.

[0070] Through the embodiments of the present invention, only updates that have made significant contributions to the global model will be integrated, thereby reducing unnecessary data transmission and calculation and improving the efficiency of federated learning.

[0071] Step 105, determining the level in the local model whose correlation value is less than a preset correlation threshold as the clipping level of the local model.

[0072] Based on the consideration of model performance, complexity and computing resource requirements, a relevance threshold is pre-set. The preset relevance threshold is used to determine which layers have low relevance values ​​and may become candidates for pruning.

[0073] In some embodiments, ensure that the local model and the global model are structurally aligned so that weight updates of corresponding levels can be compared; extract weight update vectors from each level of the local model and the global model; and calculate the correlation value of each level using a suitable correlation metric method (such as Pearson correlation coefficient, cosine similarity, etc.).

[0074] Compare the relevance value of each layer with the preset relevance threshold to determine the layers with relevance values ​​below the threshold; mark the layers with relevance values ​​below the preset relevance threshold as pruned layers. Pruned layers may be removed or replaced in subsequent model optimization processes.

[0075] Through the embodiments of the present invention, the clipping level in the local model can be effectively determined, and the complexity of the model and the computing resource requirements can be reduced while maintaining the model performance.

[0076] Step 106, determining the levels in the local model whose correlation values ​​are greater than a preset correlation threshold as the reserved levels of the local model.

[0077] In the embodiment of the present invention, the relevance value of each level is compared with a preset relevance threshold, and the levels with relevance values ​​higher than the threshold are identified; the levels with relevance values ​​higher than the threshold are marked as reserved levels. The reserved levels should be retained in the subsequent model optimization or pruning process.

[0078] Through the embodiments of the present invention, the retention level in the local model can be effectively determined, and key parts can be ensured to be retained when optimizing the model.

[0079] Step 107 , the invalid updates corresponding to the clipping level of the local model are clipped, and the valid updates corresponding to the retaining level of the local model are returned to the server, so that the valid updates are synchronized to the global model.

[0080] Based on the weight update of the local model and the global model at each level, the correlation value is calculated; the preset correlation threshold is used to determine the pruned level (correlation value below the threshold) and the retained level (correlation value above the threshold).

[0081] At the pruning level, updates that do not contribute significantly to model performance or may cause performance degradation are identified and are considered invalid updates. Invalid updates may include weight updates that are in the wrong direction, too small, or inconsistent with the global model update direction.

[0082] Remove invalid updates from pruned layers, i.e. set weight updates of these layers to zero or replace them with other means (e.g. averaging global model weights). Pruning should be done in a way that does not disrupt the overall structure or functionality of the model.

[0083] In the reserved layers, extract those updates that have significantly contributed to the model performance improvement, and these updates are considered valid updates. Valid updates should contain complete information about weight updates, including the direction and magnitude of the updates. Package the valid updates of the reserved layers and send them to the server for global model updates.

[0084] The server receives valid updates from each local model and verifies them to ensure the legitimacy and validity of the updates; aggregates all legal and valid updates into the global model to update the weights of the global model; the aggregation operation may adopt weighted average, federated average or other aggregation algorithms, depending on the framework and strategy of distributed learning or federated learning.

[0085] Through the above steps of the embodiment of the present invention, the global model is the result of joint training by multiple participants. After receiving the global model, local training is performed on this basis without starting from scratch, thereby accelerating the learning process of the model; local training enables the model to learn the data distribution characteristics unique to the client and enhances the personalization ability of the model; by comparing the updates of the local model and the global model at each level, the correlation value is determined, which helps to quantify the degree of association between the local model update and the global model update; cutting the updates of the low correlation level (lower than the preset correlation threshold) can reduce the transmission of noise and redundant information, reduce communication costs, and also help maintain the stability and consistency of the global model; retaining the updates of the high correlation level (higher than the preset correlation threshold) can ensure that the global model can absorb valuable updates that have a positive impact on performance improvement, thereby accelerating the optimization process of the global model; by cutting invalid updates and only transmitting valid updates, the communication efficiency can be significantly improved and unnecessary resource consumption can be reduced; thereby solving the technical problem of poor efficiency of federated learning due to local data deviation of each client in the federated learning method in the related technology.

[0086] According to a method for dynamic pruning of federated learning based on gradient correlation provided by the present invention, after determining the level in the local model whose correlation value is greater than a preset correlation threshold as the retained level of the local model, the method further includes: The cropping level of the local model is cropped, and the retained level of the local model is retained to obtain a cropped local model; Determine the L2 norm mean of the gradient of the retained layer of the local model as the clipping threshold; Determine the L2 norm mean of the gradient of the pruned local model as the target norm mean; When the target norm mean is greater than the clipping threshold, the clipped local model is gradient clipped to obtain the updated gradient.

[0087] refer to Figure 2 , Figure 2 It is a schematic diagram of the model clipping process provided by the present invention, which includes: calculating the correlation between the local model and the global model by layer; retaining the network layers with high correlation and clipping the network layers with low correlation; calculating the L2 norm mean of all unclipped gradients after layer screening; and calculating the clipped local gradient.

[0088] Model pruning is a technique for reducing the size and computation of neural network models. It usually involves removing (or "pruning") weights or neurons that have little impact on the network output. Layer pruning refers to selectively keeping or removing entire layers between different levels of the model.

[0089] Gradient clipping is a technique used to prevent gradient explosion during neural network training. It does this by limiting the value of the gradient to a specific range. The L2 norm (also known as the Euclidean norm) is a common metric used in gradient clipping to calculate the "magnitude" of the gradient.

[0090] In the federated learning framework under differential privacy, the threshold setting of gradient clipping has a direct impact on the level of privacy protection and the convergence performance of the model.

[0091] In the stochastic gradient descent process that implements the differential privacy strategy, the gradient clipping technique is used to limit the impact of each data sample on the model update. This method aims to evaluate the sensitivity of a single sample in the sum of gradients by setting the maximum threshold of the L2 norm of the gradient, while reducing the excessive impact of individual data points on the entire learning process. By adding noise to the training parameters, gradient clipping helps maintain the balance of the model while controlling outliers.

[0092] Therefore, selecting an appropriate clipping threshold is crucial to achieving efficient differential privacy federated learning. If the clipping threshold C is set too low, the processed gradient average may deviate from the direction of the true gradient, resulting in inaccurate model updates; and if the clipping threshold C is too high, more noise will need to be added to the gradient parameters because the amount of Gaussian noise that needs to be added is proportional to the clipping threshold.

[0093] In addition, during the federated learning process, the client's gradient L2 norm will decrease as the number of iterations increases.

[0094] Therefore, if a uniform gradient clipping threshold is used, it will have a negative impact on the accuracy of the model. Therefore, the present invention proposes a dynamic adaptive gradient clipping scheme, where the L2 norm of all unclipped gradients is calculated in each iteration and their average is calculated.

[0095] In some embodiments, the local model is pruned in layers, that is, some layers are selectively removed while other layers are retained to obtain a pruned local model, and the size and computational complexity of the pruned local model are reduced compared to the original model.

[0096] Next, calculate the L2 norm mean of the gradient of the retained layer and use this value as the clipping threshold. The clipping threshold is used to determine whether the gradient of the clipped model is too large; calculate the L2 norm mean of the gradient of the clipped local model as the target norm mean. The target norm mean reflects the gradient size of the clipped model during the training process.

[0097] If the mean value of the target norm is greater than the clipping threshold, it means that the gradient of the clipped model is too large, which may cause unstable training. In this case, the gradient of the clipped local model is clipped to limit the value of the gradient to a reasonable range and obtain the updated gradient.

[0098] Through the embodiments of the present invention, it is possible to ensure that the model remains stable during the training process while reducing the risk of gradient explosion.

[0099] According to a federated learning dynamic clipping method based on gradient correlation provided by the present invention, the gradient clipping is expressed by the following formula: ; in, represents the updated gradient, represents the clipping threshold, represents the gradient of the pruned local model, represents the target norm mean.

[0100] In the embodiment of the present invention, the dynamic adaptive adjustment of the clipping threshold may refer to the following algorithm 2:

[0101] Here, epoch is used to describe the number of times the entire training data set is traversed during the training process. An epoch means that each training sample is used once in the model to update the model weights.

[0102] Through the embodiments of the present invention, gradient clipping can be performed on the pruned local model, the gradient value can be limited to a reasonable range, and an updated gradient can be obtained.

[0103] According to a federated learning dynamic pruning method based on gradient correlation provided by the present invention, after determining the L2 norm mean of the gradient of the pruned local model as the target norm mean, the method further includes: When the target norm mean is less than the clipping threshold, noise is added to the clipped local model to obtain an updated local model.

[0104] During the training of the local model, when the target norm mean (i.e. the gradient L2 norm mean of the pruned local model) is less than the pruned threshold, in order to maintain the stability and exploratory nature of the model, one strategy is to add an appropriate amount of noise to the pruned local model. This step aims to enhance the generalization ability of the model by introducing tiny random perturbations, prevent the model from falling into a local optimal solution, and help the model learn more features of the data.

[0105] After each training cycle (or iteration), the mean L2 norm of the gradient of the pruned local model (i.e., the target norm mean) is calculated; this mean is compared with the preset pruned threshold; if the target norm mean is less than the pruned threshold, the noise addition step is performed.

[0106] Select an appropriate noise type, such as Gaussian noise, uniform noise, or Laplace noise. Determine the amplitude or standard deviation of the noise, which usually needs to be set based on the complexity of the model, the characteristics of the data, and other hyperparameters during training. Add the selected type of noise to the weights or activation values ​​of the pruned local model in a random manner.

[0107] After the noise is added, the updated local model is obtained, and the next training cycle or iteration is continued, using the updated model for forward propagation, loss calculation, and backpropagation.

[0108] Through the embodiments of the present invention, by introducing noise, the risk of model overfitting can be alleviated to a certain extent, and the robustness and generalization ability of the model can be improved.

[0109] According to a method for dynamic pruning of federated learning based on gradient correlation provided by the present invention, after determining the level in the local model whose correlation value is greater than a preset correlation threshold as the retained level of the local model, the method further includes: A privacy budget is allocated to the valid updates corresponding to the retained levels of the local model to obtain the target privacy budget; and a differential privacy perturbation is performed on the target privacy budget.

[0110] Differential Privacy (DP) is a privacy-preserving technique that ensures the privacy of individual data by adding noise to the data while allowing meaningful statistical analysis of the data.

[0111] The privacy budget (usually denoted as ε or (ε,δ)) is a parameter that measures the risk of privacy leakage. Smaller ε values ​​indicate stronger privacy protection, but may sacrifice some model accuracy.

[0112] For the retained levels of the local model, a privacy budget needs to be allocated based on the importance of each level, the sensitivity of the data, and the overall privacy requirements of the model. For example, the gradients, weight updates, or activation values ​​of each level of the model are assessed for privacy risk, and the corresponding privacy budget shares are allocated accordingly. Based on the above allocation, the target privacy budget for each retained level can be calculated. This budget will be used to guide the subsequent differential privacy perturbation process.

[0113] Differential privacy is usually achieved by adding noise to data or model updates. Common noise distributions include Laplace distribution and Gaussian distribution. The choice of noise distribution depends on the specific differential privacy mechanism and privacy budget.

[0114] Based on the target privacy budget and the selected noise distribution, the amount of noise that needs to be added can be calculated. This amount is usually inversely proportional to the privacy budget, that is, a smaller privacy budget requires more noise to be added.

[0115] The calculated noise is added to the gradients, weight updates, or activation values ​​of the retained layers. This process ensures that even if a single data point changes, the output of the model does not change significantly, thereby preserving the privacy of the data.

[0116] Through the embodiments of the present invention, differential privacy technology ensures the privacy of individual data by adding noise to data or model updates; the privacy budget (such as the ε value) is used as a parameter to measure the risk of privacy leakage, so that the degree of privacy protection can be quantified. By adjusting the size of the privacy budget, the risk of privacy leakage during data publishing or querying can be controlled.

[0117] refer to Figure 3 , Figure 3 It is an overall flow chart of the dynamic pruning method of federated learning based on gradient correlation provided by the present invention, which includes a client and a central server.

[0118] The client is specifically used to perform the following steps: receive the global model sent by the central server; perform local training; calculate the correlation between the local model and the global model by layer; determine whether it is greater than a preset threshold; retain the layer in the local model (that is, the layer greater than the preset threshold); calculate the L2 norm mean of all unclipped gradients after layer screening; calculate the clipped local gradient; dynamically allocate the privacy budget according to the gradient size, iteration rounds, and correlation; perform differential privacy perturbations on the gradients and upload the client model.

[0119] The central server is specifically used to perform the following steps: aggregating and updating the global model, and issuing the global model.

[0120] In order to verify the performance of the gradient correlation dynamic gradient clipping algorithm and this model, the present invention designs the following experiments for verification.

[0121] This experiment uses Python as the programming language, version 3.8.20, the GPU model is NVIDIA GeForce RTX2080 Ti, the computing node includes 12 virtual CPUs, the CPU model is Intel(R) Xeon(R) Platinum 8255C CPU, the main frequency is 2.50GHz, the operating system is Ubuntu22.04, and the deep learning framework used is Pytorch, version 2.3.0. Refer to Table 1 for details, which is a description of the hardware and software environment of the experiment provided by the present invention.

[0122] Table 1 Description of the software and hardware environment of the experiment

[0123] The present invention uses two standard and commonly used image datasets: MNIST and CIFAR-10 for relevant experiments, which facilitates performance comparison with existing advanced models and methods.

[0124] MNIST is a dataset of handwritten digits, including a training set of 60,000 images and a test set of 10,000 images. There are 10 categories of images, corresponding to Arabic numerals 0 to 9. In the original MNIST dataset, each image is represented by a 28*28 matrix, and the value of each pixel ranges from 0 to 1. It is generally converted into a 784-dimensional vector as input.

[0125] CIFAR-10 is a small dataset compiled by Hiton's students for identifying universal objects. It contains a total of 10 categories of RGB color images: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks.

[0126] The image size of the CIFAR-10 dataset is 32*32; there are 50,000 training images and 10,000 test images in the dataset.

[0127] CIFAR-10 does not have image files by default, but is saved in pickle format. The result of reading is a dictionary. It contains "data" and "labels" key values, where "data" is a 10000×3072 numpy array, and each row is a 32×32 image. "labels" is 10,000 labels between 0 and 9. For details, please refer to Table 2 and Table 3.

[0128] Table 2 Network structure of the MNIST dataset

[0129] Table 3 Network structure of CIFAR dataset

[0130] In order to compare and evaluate the performance of the model framework proposed in this invention, the experiment will be compared with the most widely used FedAvg and the more advanced COFEL algorithm. FedAvg is the basic algorithm of federated learning. It generates a global model by averaging the models of participating clients in each round, ensuring that the data does not leave the device and only shares model updates, thereby protecting data privacy. COFEL (Communication-Efficient and Optimized Federated Learning with LocalDifferential Privacy) transmits only the most relevant parts of each client model through layer-based parameter selection, and perturbs these parameters in combination with the local differential privacy mechanism to enhance privacy protection and reduce data transmission and training time.

[0131] For comparison, we also apply LDP to the traditional federated learning system and CMFL. The data perturbation setting is the same as in COFEL. For privacy considerations, when we compare the performance of FedAvg and CMFL, we use .

[0132] The experimental results are compared with other solutions through the accuracy rate (Acc), communication volume and aggregation speed of the test set. The accuracy calculation formula is as follows: Among them, TP (True Positive) represents the number of true positive examples, that is, the number of samples correctly predicted as positive examples by the model; TN (True Negative) represents the number of true negative examples, that is, the number of samples correctly predicted as negative examples by the model; FP (False Positive) represents the number of false positive examples, that is, the number of samples where the model incorrectly predicts negative examples as positive examples; FN (False Negative) represents the number of false negative examples, that is, the number of samples where the model incorrectly predicts positive examples as negative examples. In the formula, the numerator is the number of samples correctly classified by the model, and the denominator is the number of all samples. The higher the accuracy, the more consistent the classification results of the model are with the actual situation.

[0133] During the entire federated learning process, the communication volume includes: (1) Parameter quantity uploaded by the client: After a local training, each client uploads its local model parameters to the server.

[0134] (2) Global model parameter quantity sent by the server: After each round of communication, the server sends the updated global model to all clients.

[0135] In federated learning, multiple clients participate in each communication round, sending their local model parameters to the server for aggregation, and the server then sends the aggregated global model to all clients. Therefore, this code estimates the communication volume by calculating the size of parameters uploaded and sent during one communication process.

[0136] in, is the number of elements of each parameter. is the element size of each parameter (in bytes, usually 4 bytes or 8 bytes, depending on the data type).

[0137] The communication overhead and accuracy of the FedAvg framework without gradient layer pruning are compared with those of the pruning model under different thresholds on the MNIST dataset.

[0138] Refer to Table 4, which is the MNIST data set model architecture provided by the present invention.

[0139] Table 4 MNIST dataset model architecture

[0140] refer to Figure 4 , Figure 4 It is a schematic diagram comparing the communication overhead of different threshold settings under MNIST provided by the present invention.

[0141] refer to Figure 5 , Figure 5 This is a schematic diagram of the accuracy comparison of different threshold models under MNIST provided by the present invention.

[0142] The model proposed in the present invention is tested on the MNIST data set, and the threshold is modified. After layer pruning, the communication volume is significantly reduced compared with the original model.

[0143] By adjusting the threshold and observing the model effect, we found that when the model is pruned to a certain extent, it will affect the stability of the model and cause a certain degree of oscillation. Therefore, when selecting a more appropriate threshold, we can ensure that the communication volume is reduced while not affecting the convergence effect of the model.

[0144] The communication overhead and accuracy of the classic framework FedAvg and COFEL algorithms are compared on the MNIST dataset, and the privacy budget saving rate of each framework is calculated.

[0145] Refer to Table 5, which is the parameter setting of the MNIST multi-algorithm comparison experiment provided by the present invention.

[0146] Table 5 MNIST multi-algorithm comparison experiment parameter settings

[0147] refer to Figure 6 , Figure 6 This is a comparison chart of communication overhead of multiple algorithm models under MNIST provided by the present invention.

[0148] refer to Figure 7 , Figure 7 This is a comparison chart of the accuracy of multiple algorithm models under MNIST provided by the present invention.

[0149] By comparing the classic framework FedAvg and COFEL framework, experiments have shown that the gradient clipping in this project has greatly improved the communication efficiency without affecting the convergence of the model.

[0150] The communication overhead and accuracy of the FedAvg framework without gradient layer pruning are compared with those of the pruning model under different thresholds on the CIFAR-10 dataset.

[0151] Refer to Table 6, which is the parameter setting of the CIFAR-10 multi-threshold comparison experiment provided by the present invention.

[0152] Table 6 CIFAR-10 multi-threshold comparison experiment parameter settings

[0153] refer to Figure 8 , Figure 8 This is a comparison chart of communication overheads under different threshold settings under CIFAR-10 provided by the present invention.

[0154] refer to Fig. 9 , Fig. 9 This is a comparison chart of the accuracy of different threshold models under CIFAR-10 provided by the present invention.

[0155] Through threshold adjustment, it is found that when the threshold is adjusted appropriately, the accuracy of the model increases, and the number of rounds of convergence is not much different from that before cropping. When the threshold is set too high and there is more cropping, the accuracy of the model will fluctuate to a certain extent, but the accuracy after final convergence is not much different from that before cropping.

[0156] The communication overhead and accuracy of the classic framework FedAvg and COFEL algorithms are compared on the CIFAR-10 dataset, and the privacy budget saving rate of each framework is calculated.

[0157] Refer to Table 7, which is the CIFAR-10 multi-algorithm comparison experiment parameter setting provided by the present invention.

[0158] Table 7 CIFAR-10 multi-algorithm comparison experiment parameter settings

[0159] refer to Fig.10 , Fig.10 This is a comparison chart of communication overhead of multiple algorithm models under CIFAR-10 provided by the present invention.

[0160] refer to Fig.11 , Fig.11 This is a comparison chart of the accuracy of multiple algorithm models under CIFAR-10 provided by the present invention.

[0161] The algorithm proposed in the present invention can effectively reduce the number of parameters for model training. Since the reduced number of parameters increases in direct proportion to the saved privacy budget, when the number of users and the number of training times are increased, the algorithm proposed in the present invention can exert its advantages to a greater extent in large-scale deep learning scenarios, which is conducive to the realization of lightweight federated learning.

[0162] This paper analyzes that in federated learning, data distribution may be significantly different, resulting in large differences between local models and global models, which affects the convergence speed and accuracy of the model. Based on this problem, this paper proposes a layer-based perspective quantification correlation method and a strategy for dynamically adaptively adjusting the clipping threshold to improve the efficiency and model performance of federated learning. Experimental results show that compared with existing algorithms, the algorithm proposed in this paper achieves better model accuracy and communication efficiency, and its overall performance is better than other algorithms.

[0163] This paper proposes a dynamic gradient clipping method for model clipping of federated learning algorithms under the framework of differential privacy. This method quantifies the contribution of local gradients to the global model, selectively uploads gradients at the level, and dynamically adjusts the clipping threshold strategy to reduce the impact of noise on the model and improve the accuracy of the model, thereby optimizing communication efficiency and reducing privacy budget consumption. Experimental results show that the proposed method outperforms existing baseline methods on standard datasets, not only significantly reducing communication costs, but also effectively improving model performance, providing a robust and efficient solution for federated learning under differential privacy protection.

[0164] The following is a description of the federated learning dynamic clipping device based on gradient correlation provided by the present invention. The federated learning dynamic clipping device based on gradient correlation described below and the federated learning dynamic clipping method based on gradient correlation described above can refer to each other.

[0165] refer to Fig.12 , Fig.12 It is a structural schematic diagram of the gradient correlation-based federated learning dynamic clipping device provided by the present invention.

[0166] A receiving module 1201 is used to receive a global model sent by a server, wherein the global model is constructed based on local models of multiple participants in a distributed learning environment; A training module 1202 is used to train the global model according to the target local data to obtain a local model; A determination module 1203, for determining an update of each level of a local model and an update of each level of a global model, wherein the global model is trained based on local data of multiple participants; A correlation module 1204 for determining a correlation value for each level between the local model and the global model based on the update of each level of the local model and the update of each level of the global model; A clipping module 1205 is used to determine a level in the local model whose correlation value is less than a preset correlation threshold as a clipping level of the local model; A retaining module 1206, configured to determine a level in the local model whose correlation value is greater than a preset correlation threshold value as a retaining level of the local model; The updating module 1207 is used to prune the invalid updates corresponding to the pruning level of the local model, and return the valid updates corresponding to the retaining level of the local model to the server, so that the valid updates are synchronized to the global model.

[0167] Specifically, the above-mentioned federated learning dynamic clipping device based on gradient correlation provided by the present invention can implement all the method steps implemented by the above-mentioned federated learning dynamic clipping method embodiment based on gradient correlation, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0168] Fig.13 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Fig.13As shown, the electronic device may include: a processor (processor) 1310 , a communication interface (Communications Interface) 1320 , a memory (memory) 1330 and a communication bus 1340 , wherein the processor 1310 , the communication interface 1320 , and the memory 1330 communicate with each other through the communication bus 1340 . The processor 1310 can call the logic instructions in the memory 1330 to execute the dynamic pruning method of federated learning based on gradient correlation, which includes: receiving the global model sent by the server, wherein the global model is constructed based on the local models of multiple participants in a distributed learning environment; training the global model according to the target local data to obtain a local model; determining the update of each level of the local model and the update of each level of the global model, wherein the global model is trained based on the local data of multiple participants respectively; based on the update of each level of the local model and the update of each level of the global model, determining the correlation value of each level between the local model and the global model; determining the level in the local model whose correlation value is less than a preset correlation threshold as the pruning level of the local model; determining the level in the local model whose correlation value is greater than the preset correlation threshold as the retained level of the local model; pruning the invalid updates corresponding to the pruning level of the local model, and returning the valid updates corresponding to the retained level of the local model to the server, so that the valid updates are synchronized to the global model.

[0169] In addition, the logic instructions in the above-mentioned memory 1330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0170] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the dynamic pruning method of federated learning based on gradient correlation provided by the above methods, and the method includes: receiving a global model sent by a server, wherein the global model is constructed based on local models of multiple participants in a distributed learning environment; training the global model according to the target local data to obtain a local model; determining the update of each level of the local model and the update of each level of the global model. , wherein the global model is trained based on the local data of multiple participants; based on the update of each level of the local model and the update of each level of the global model, the correlation value of each level between the local model and the global model is determined; the level with a correlation value less than a preset correlation threshold in the local model is determined as the pruned level of the local model; the level with a correlation value greater than a preset correlation threshold in the local model is determined as the retained level of the local model; the invalid updates corresponding to the pruned level of the local model are pruned, and the valid updates corresponding to the retained level of the local model are returned to the server, so that the valid updates are synchronized to the global model.

[0171] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the dynamic pruning method of federated learning based on gradient correlation provided by the above methods, the method comprising: receiving a global model sent by a server, wherein the global model is constructed based on local models of multiple participants in a distributed learning environment; training the global model according to target local data to obtain a local model; determining the update of each level of the local model and the update of each level of the global model, wherein the global model is trained based on the local data of multiple participants respectively; determining the correlation value of each level between the local model and the global model based on the update of each level of the local model and the update of each level of the global model; determining the level in the local model whose correlation value is less than a preset correlation threshold as the pruning level of the local model; determining the level in the local model whose correlation value is greater than the preset correlation threshold as the retained level of the local model; pruning invalid updates corresponding to the pruning level of the local model, and returning valid updates corresponding to the retained level of the local model to the server, so that the valid updates are synchronized to the global model.

[0172] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic pruning method for federated learning based on gradient correlation, characterized in that: include: Receiving a global model sent by a server, wherein the global model is constructed based on local models of multiple participants in a distributed learning environment; Training the global model according to the target local data to obtain a local model; Determining an update of each level of the local model and an update of each level of the global model, wherein the global model is trained based on local data of each of the plurality of participants; Determining a correlation value for each level between the local model and the global model based on an update of each level of the local model and an update of each level of the global model; Determine a level in the local model whose correlation value is less than a preset correlation threshold as a clipping level of the local model; Determine a level in the local model whose correlation value is greater than the preset correlation threshold as a reserved level of the local model; Invalid updates corresponding to the clipping level of the local model are clipped, and valid updates corresponding to the retaining level of the local model are returned to the server, so that the valid updates are synchronized to the global model.

2. The method for dynamic pruning of federated learning based on gradient correlation according to claim 1, characterized in that: After determining the level in the local model whose correlation value is greater than the preset correlation threshold as the reserved level of the local model, the method further includes: The cropping level of the local model is cropped, and the reserved level of the local model is reserved to obtain a cropped local model; Determine the L2 norm mean of the gradients of the retained layers of the local model as a clipping threshold; Determine the L2 norm mean of the gradient of the pruned local model as the target norm mean; When the target norm mean is greater than the clipping threshold, gradient clipping is performed on the clipped local model to obtain an updated gradient.

3. The method for dynamic pruning of federated learning based on gradient correlation according to claim 2, characterized in that: The gradient clipping is expressed by the following formula: ； in, represents the updated gradient, represents the clipping threshold, represents the gradient of the pruned local model, represents the target norm mean.

4. The method for dynamic pruning of federated learning based on gradient correlation according to claim 2, characterized in that: After determining the L2 norm mean of the gradient of the pruned local model as the target norm mean, the method further includes: When the target norm mean is less than the clipping threshold, noise is added to the clipped local model to obtain an updated local model.

5. The method for dynamic pruning of federated learning based on gradient correlation according to claim 1, characterized in that: After determining the level in the local model whose correlation value is greater than the preset correlation threshold as the reserved level of the local model, the method further includes: A privacy budget is allocated for valid updates corresponding to the retained level of the local model to obtain a target privacy budget; and a differential privacy perturbation is performed on the target privacy budget.

6. The method for dynamic pruning of federated learning based on gradient correlation according to claim 1, characterized in that: The determining of the correlation value of each level between the local model and the global model based on the update of each level of the local model and the update of each level of the global model comprises: Calling a correlation update function to determine a correlation value of an update of each level of the local model in a current training round and an update mark of each level of the global model in a previous training round, and an update mark of each level of the local model; The update mark is used to mark the correlation between the current level of the local model and the corresponding level of the global model.

7. A federated learning dynamic pruning device based on gradient correlation, characterized in that: include: A receiving module, configured to receive a global model sent by a server, wherein the global model is constructed based on local models of multiple participants in a distributed learning environment; A training module, used for training the global model according to target local data to obtain a local model; A determination module, configured to determine an update of each level of the local model and an update of each level of the global model, wherein the global model is trained based on local data of the plurality of participants respectively; a correlation module for determining a correlation value of each level between the local model and the global model based on the update of each level of the local model and the update of each level of the global model; A clipping module, used to determine the level in the local model whose correlation value is less than a preset correlation threshold as the clipping level of the local model; A retention module, used to determine the level in the local model whose correlation value is greater than the preset correlation threshold as the retention level of the local model; An update module is used to prune invalid updates corresponding to the pruning level of the local model, and return valid updates corresponding to the retention level of the local model to the server, so that the valid updates are synchronized to the global model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the dynamic pruning method of federated learning based on gradient correlation as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the dynamic pruning method of federated learning based on gradient correlation as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the dynamic pruning method of federated learning based on gradient correlation as described in any one of claims 1 to 6 is implemented.

Images