Communication efficient federal learning method based on gradient cosine similarity and differential privacy
By adopting the federated learning method of gradient cosine similarity and differential privacy in non-IID data environments, efficient communication and privacy protection of federated models are achieved, solving the problem of inefficient communication in the existing technology, and significantly improving the model convergence speed and Privacy and security.
Patent Information
- Application Number
- CN202411138937.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-19
AI Technical Summary
In non-IID data environments, federated learning has low communication efficiency and risks of privacy leakage. Existing methods such as FedAvg, FedProx, and MOON perform poorly in this scenario.
Adopting an efficient federated learning method based on gradient cosine similarity and differential privacy, designing an adaptive weighted aggregation strategy by defining local cumulative gradients on the user side and defining pseudo-global gradients on the server side, using differential privacy technology to protect user privacy and improve federated The convergence speed and communication efficiency of the model.
It significantly reduces the communication rounds required to achieve target test accuracy, improves the communication efficiency of federated learning in non-IID data environments, and effectively overcomes privacy leakage issues.
Smart Images

Figure CN119089980A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to distributed machine learning technology and privacy computing technology, and in particular to a communication-efficient federated learning method based on gradient cosine similarity and differential privacy. Background Art
[0002] In recent years, mobile devices and the Internet of Things have been increasingly widely used, and massive amounts of valuable data are constantly being generated in users' local devices. With the rapid development of machine learning (ML) technology, the value of big data can be mined through increasingly complex and efficient model training. In the traditional machine learning paradigm, model training is centralized and requires the use of a large amount of data and computing resources. The centralized machine learning paradigm requires collecting local raw data from the user's device, then uploading it to a central server for data processing, and then using it for model training. This approach brings many disadvantages, including the huge communication burden caused by massive data uploads, the limited possibility of real-time training models, and so on. More importantly, centralized machine learning relies too much on the use of sensitive raw data, and there is a serious risk of privacy leakage.
[0003] Federated Learning (FL) is an emerging distributed machine learning framework and is also regarded as a privacy computing technology. It aims to solve the privacy and security issues inherent in the centralized machine learning paradigm, thereby effectively breaking down data barriers. A typical federated learning system consists of a central server and several distributed independent participating users (individuals or organizations). Both the server and the user devices have certain computing power, data storage capacity and communication capabilities. The general federated training process is as follows: the server broadcasts the initialized global model to several users; the user trains the local model using a private data set on the local device, and then transmits the updated local model to the server; after receiving a certain number of local models, the server aggregates these models to update the global model; the server broadcasts the updated global model to the user and repeats the above training and communication process until the pre-set global model performance index or the maximum number of communication rounds is reached.
[0004] However, in real-world application scenarios, users of federated learning collect and store data independently of each other, resulting in their local data sets often being statistically non-independent and identically distributed (non-IID) characteristics, i.e., data distribution heterogeneity. In a non-IID data environment, the local model update directions of federated learning participants (which can be reflected by the gradient or the difference before and after the model update) differ significantly, leading to the following problem: the communication efficiency of federated training is low. Therefore, non-IID data has become a key challenge facing federated learning technology.
[0005] McMahan et al. proposed the FedAvg method in the paper "Communication-efficient learning of deep networks from decentralized data". The core idea is that users train local models locally and then upload local model parameters to the central server. Then, the server updates the global model parameters by weighted averaging these local models according to the size of the user's local data set. In addition, in order to optimize the above-mentioned federated learning training process, FedAvg reduces the system communication overhead and accelerates the convergence of the global model by increasing the amount of iterative calculations in the user's local area. However, FedAvg performs poorly in non-IID data environments.
[0006] Li et al. proposed the FedProx method in the paper "Federated optimization in heterogeneous networks", which improved on FedAvg. FedProx introduces a regularization term into the local loss function to limit the divergence of the local model compared to the latest federated model (i.e., the previous round of federated model), making federated training more stable in non-IID data environments. However, FedProx still cannot significantly improve the communication efficiency of federated learning in non-IID data environments.
[0007] Li et al. proposed the MOON method in the paper "Model-contrastive federated learning". MOON improves the generalization performance of federated models by performing contrastive learning between global and local models. Compared with traditional federated learning methods, MOON significantly improves the communication efficiency of federated learning. However, MOON still performs suboptimally in non-IID data environments.
[0008] Cosine similarity is a mathematical method used to measure the similarity between two non-zero vectors. The basic idea is to measure the directional similarity of two vectors by calculating the cosine of the angle between them. Given two vectors, the cosine similarity between them can be obtained by dividing their inner product by the product of their norms (length / modulus, indicating the size of the vector). Cosine similarity has the following characteristics: ① The value range is [-1,1]. ② The larger the cosine similarity value of two vectors, the closer the directions of the two vectors are. In particular, when the value is 1, it means that the directions of the two vectors are exactly the same; when the value is 0, it means that the two vectors are orthogonal; when the value is -1, it means that the directions of the two vectors are completely opposite. ③ The size of the vector is not considered. The size of the two vectors has no effect on the calculation of cosine similarity.
[0009] Differential Privacy (DP) is a privacy computing technology that protects the privacy of individual data by introducing controllable noise during data publishing or query processing. The core idea of DP is to ensure the accuracy of data analysis results while making it difficult for attackers to infer the original data from the data analysis results, thereby maximizing the protection of user privacy. Differential privacy technology seeks a balance between privacy protection and data utilization, and is widely used in data mining, machine learning and other fields. Summary of the invention
[0010] In view of the problem of low communication efficiency of the above-mentioned federated learning in a non-IID data environment, the present invention proposes a communication-efficient federated learning method (Federated Learning with Noise-added Gradient Cosine Similarity, FedNGCS) based on gradient cosine similarity and differential privacy, thereby improving the communication efficiency of FL training on non-IID data and overcoming the derived privacy leakage problem. Specifically, a local cumulative gradient is defined on each user side and continuously updated during the local iteration process to reflect the user's local data distribution; a pseudo-global gradient is defined on the server side and continuously updated after federated aggregation to indicate the latest global model update direction; based on the cosine similarity between the above two real-time updated gradients, an adaptive weighted aggregation strategy is designed to accelerate the convergence of the federated model and improve the communication efficiency of federated training. The federated training process of the method of the present invention requires users to upload local cumulative gradients. Compared with the original FL method, this step will derive additional privacy leakage risks. Therefore, the present invention introduces differential privacy technology as a privacy protection means.
[0011] The technical solution of the present invention is as follows:
[0012] A communication-efficient federated learning method based on gradient cosine similarity and differential privacy includes the following steps:
[0013] Step 1: Start federated learning and the server initializes the global model;
[0014] Step 2: The server determines whether the previous moment is the FL start moment or a global model aggregation moment. If so, the server randomly selects a fixed number of users to participate in a new round of training and broadcasts the latest global model to these users. If not, the server continues to use the selected user set of the previous moment to participate in training.
[0015] Step 3: Each selected user uses the gradient descent algorithm on the local data set to calculate the stochastic gradient, updates the local model, and obtains the iterated local model;
[0016] Step 4: The server determines whether it is in the first communication round at the current moment. If so, it executes step 5; if not, each selected user updates the local cumulative gradient based on the stochastic gradient;
[0017] Step 5: The server determines whether the current time t is the global model aggregation time. If not, it returns to step 2. If so, the server calculates the standard deviation of the global sensitivity and DP noise in turn, and sends the standard deviation of the DP noise to each selected user. The selected user clips the local cumulative gradient and adds DP noise to obtain the perturbed local cumulative gradient that meets the privacy protection requirements.
[0018] Step 6: The server determines whether the current time t is the first global model aggregation time. If so, the server weights the users based on the data volume proportion of each selected user; if not, the server calculates the cosine similarity between the perturbed local cumulative gradient of each selected user and the latest pseudo-global gradient, and weights the users based on this; the server weights and aggregates the local models of all selected users to update the global model; the server updates the pseudo-global gradient;
[0019] Step 7: Repeat steps 2 to 6 until the preset model performance index or maximum communication rounds are reached.
[0020] Beneficial effects:
[0021] The present invention improves the communication efficiency of federated learning in a non-IID data environment, specifically reducing the communication rounds required for the federated model to achieve target test accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a flowchart of the communication efficient federated learning method based on gradient cosine similarity of the present invention.
[0023] Figure 2This is one of the comparison diagrams of the accuracy of the method of the present invention and three traditional federated learning methods as the number of communication rounds changes in the embodiments.
[0024] Figure 3 This is the second comparison diagram of the accuracy of the method of the present invention and three traditional federated learning methods as the number of communication rounds changes in the embodiments.
[0025] Figure 4 This is the third comparison diagram of the accuracy of the method of the present invention and three traditional federated learning methods as the number of communication rounds changes in the embodiments. DETAILED DESCRIPTION
[0026] The technical solution provided by the present application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of the present application will become more apparent with the following description.
[0027] This paper proposes a communication-efficient federated learning method based on gradient cosine similarity and differential privacy, aiming to improve the communication efficiency of FL in non-IID data environments. Figure 1 As shown, the method flow is as follows:
[0028] Step 1: Start federated learning. The server initializes the global model w (0) .
[0029] Step 2: The server determines whether the previous time t-1 is the FL opening time or a global model aggregation time.
[0030] If so, perform the following operations: ① The server randomly selects K users to form a user set represents the user selected for local training at time t; ② the latest global model w (t-1) Broadcast to each user U m , as their respective latest local models Where, t = 1, ..., T; T = Rτ, R represents the total number of communication rounds, and τ represents the time required for each round of communication; M is the total number of users participating in federated learning; m=1,......,M.
[0031] If not, perform the following operations: Continue to use the selected user set at the previous moment, that is,
[0032] Step 3: Each selected user U m In local dataset The stochastic gradient is calculated using the gradient descent algorithm For local models Update to obtain the iterated local model in, For user U m Private datasets, represents an F-dimensional feature vector, Represents a tag.
[0033] Step 4: The server determines whether the current time t is in the first communication round.
[0034] If so, do nothing.
[0035] If not, perform the following operations: Each selected user U m Based on stochastic gradient Update the local accumulated gradient The formula is as follows:
[0036]
[0037] in, is the latest local cumulative gradient after update, is the local cumulative gradient of the previous moment. In particular, user U m Before the first local iteration in this round of communication begins, As a clearing operation, the reason is: User U m It is possible to participate in two consecutive rounds of federal training.
[0038] Step 5: The server determines whether the current time t is the global model aggregation time.
[0039] If not, go back to step 2.
[0040] If yes, do the following:
[0041] ①The server calculates the global sensitivity Δs G , the formula is as follows:
[0042]
[0043] Then, according to the privacy protection requirements of the Gaussian (∈, δ)-DP mechanism, the standard deviation σ of the DP noise is calculated as follows:
[0044]
[0045] Among them, ∈>0 is the privacy budget, which is used to constrain the distribution differences of adjacent data sets, and δ>0 is the failure probability, which is used to indicate the probability of constraint failure.
[0046] Next, the server sends the standard deviation σ of the DP noise to each selected user.
[0047] ②Each selected user Um The local accumulated gradient To crop:
[0048]
[0049] Where C is the clipping threshold.
[0050] Next, the user accumulates the gradient to the clipped local Add DP noise to obtain the perturbed local cumulative gradient that meets the privacy protection requirements of (∈,δ)-DP The formula is as follows:
[0051]
[0052] in, represents Gaussian noise, d represents Dimension.
[0053] Finally, the user will perturb the local accumulated gradient Upload to the server.
[0054] Step 6: The server determines whether the current time t is the first global model aggregation time.
[0055] If so, the server performs the following operations:
[0056] ① Based on each selected user U m The proportion of data volume is used to empower users. m Weight The calculation formula is as follows:
[0057]
[0058] ② Collect all selected users U m The latest local model These models are then weighted aggregated to update the global model w (t) , the update formula is as follows:
[0059]
[0060] ③ Update the pseudo-global gradient, the formula is as follows:
[0061]
[0062] in, is the latest pseudo global gradient obtained after updating.
[0063] If not, the server performs the following actions:
[0064] ① Calculate each selected user U mThe perturbed local cumulative gradient and the latest pseudo-global gradient The cosine similarity between The formula is as follows:
[0065]
[0066] ② For each selected user U m To empower user U m Weight The calculation formula is as follows:
[0067]
[0068] ③ Collect all selected users U m The latest local model These models are then weighted aggregated to update the global model w (t) , the update formula is as follows:
[0069]
[0070] ④Update the pseudo-global gradient, the formula is as follows:
[0071]
[0072] Step 7: Repeat steps 2 to 6 until the preset model performance index or maximum communication rounds are reached.
[0073] In the present invention, an adaptive weighted aggregation strategy is designed, which is reflected in that before each round of model weighted aggregation, the method of the present invention can adapt to The randomness of the user composition and Based on the differences in the non-IID levels of different users in the algorithm, a reasonable weight is assigned to each selected user. The higher the local non-IID level of the user, the more the local cumulative gradient deviates from the pseudo-global gradient direction, and the greater the cosine similarity between the two. This strategy assigns weights to users after performing softmax normalization on the cosine similarity between the perturbed local cumulative gradient and the pseudo-global gradient of each selected user. By restricting users with higher non-IID levels, the convergence of the federated model is accelerated and the communication efficiency of federated training is improved.
[0074] Experimental verification:
[0075] In order to intuitively demonstrate the effectiveness and superiority of this method, the following experimental results are provided.
[0076] The FedNGCS method proposed in this paper is implemented using PyTorch and compared with three classic FL methods: FedAvg (proposed in the paper "Communication-efficient learning of deep networks from decentralized data"), FedProx (proposed in the paper "Federated optimization inheterogeneous networks"), and MOON (proposed in the paper "Model-contrastive federated learning"). The superiority of the FedNGCS proposed in this paper under non-IID data conditions is demonstrated through three sets of image classification experiments.
[0077] In the embodiment, three widely used public data sets are selected: MNIST, FashionMNIST and CIFAR-10. Two neural network models are built for different data sets: for the single-channel data sets MNIST and FashionMNIST, a CNN (Convolutional Neural Network) model is trained for each, referred to as CNN-M, including 2 convolutional layers, 2 maximum pooling layers, 2 fully connected layers and 1 softmax layer; for the three-channel data set CIFAR-10, a CNN model is trained, referred to as CNN-C, including 3 convolutional layers, 2 maximum pooling layers, 3 fully connected layers and 1 softmax layer. The non-IID data environment setting method of this embodiment is as follows: set 10 users with ids 1 to 10 to participate in federated training, the data volume of each user is a random value between 400 and 800, and the data volume between each other is not equal; users with ids 1, 2, and 3 randomly obtain sample data of 1 fixed category from the training set; users with ids 4, 5, 6, and 7 randomly obtain sample data of 2 fixed categories from the training set; users with ids 8, 9, and 10 randomly obtain sample data of all categories from the training set. In order to more intuitively compare the communication efficiency of each method, focus on observing the communication rounds required for each method to achieve the target accuracy. In the "MNIST+CNN-M" experimental group, the target accuracy is set to 95%; in the "FashionMNIST+CNN-M" experimental group, the target accuracy is set to 80%; in the "CIFAR-10+CNN-C" experimental group, the target accuracy is set to 40%.
[0078] Other main experimental hyperparameters are set as follows: maximum communication rounds R = 200; local iterations per round τ = 3; the number of users participating in each round of training K = 5. Referring to the paper "Federated learning with differential privacy: Algorithms and performance analysis", let: privacy budget = 10; failure probability δ = 0.001; the value of the pruning threshold C is the median of the two norms of all parameters to be pruned; the batch size used for each user training is equal to the amount of local data. In particular, the initial learning rate in the CNN-M experiment is set to 0.01 and the decay rate is set to 0.995; the initial learning rate in the CNN-C experiment is set to 0.1 and the decay rate is set to 0.992.
[0079] The experimental results based on three public datasets are as follows Figure 2 , Figure 3 and Figure 4 Table 1 lists the number of communication rounds required for each method to achieve the target accuracy in the three groups of experiments, corresponding to Figure 2 , Figure 3 and Figure 4 Among them, the experimental data with the best communication efficiency performance are marked in bold; for experiments that still do not reach the target accuracy after the maximum number of communication rounds, their experimental data are marked with the "N / A" symbol, and the highest test accuracy achieved during the training process is recorded in the brackets behind.
[0080] Experimental results show that compared with FedAvg, FedProx and MOON, the FedNGCS proposed in this invention can effectively accelerate the convergence speed of the model, thereby improving the communication efficiency.
[0081] Table 1: Number of communication rounds required for the proposed method and three traditional federated learning methods to achieve target accuracy.
[0082]
[0083] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.
Claims
1. A communication-efficient federated learning method based on gradient cosine similarity and differential privacy, It is characterized in that Includes steps: Step 1: Start federated learning and the server initializes the global model; Step 2: The server determines whether the previous moment is the FL start moment or a global model aggregation moment. If so, the server randomly selects a fixed number of users to participate in a new round of training and broadcasts the latest global model to these users. If not, the server continues to use the selected user set of the previous moment to participate in training. Step 3: Each selected user uses the gradient descent algorithm on the local data set to calculate the stochastic gradient, updates the local model, and obtains the iterated local model; Step 4: The server determines whether it is in the first communication round at the current moment. If so, it executes step 5; if not, each selected user updates the local cumulative gradient based on the stochastic gradient; Step 5: The server determines whether the current time t is the global model aggregation time. If not, it returns to step 2; If yes, the server calculates the standard deviation of global sensitivity and DP noise in turn, and sends the standard deviation of DP noise to each selected user. The selected user clips the local cumulative gradient and adds DP noise to obtain the perturbed local cumulative gradient that meets the privacy protection requirements. Step 6: The server determines whether the current time t is the first global model aggregation time. If so, the server weights the users based on the data volume proportion of each selected user; if not, the server calculates the cosine similarity between the perturbed local cumulative gradient of each selected user and the latest pseudo-global gradient, and weights the users based on this; the server weights and aggregates the local models of all selected users to update the global model; the server updates the pseudo-global gradient; Step 7: Repeat steps 2 to 6 until the preset model performance index or maximum communication rounds are reached.
2. A communication-efficient federated learning method based on gradient cosine similarity and differential privacy as claimed in claim 1, It is characterized in that Step 2: The server determines whether the previous time t-1 is the FL opening time or a global model aggregation time; If so, perform the following operations: ① The server randomly selects K users to form a user set represents the user selected for local training at time t; ② the latest global model w (t-1) Broadcast to each user U m , as their respective latest local models Where, t = 1, ..., T; T = Rτ, R represents the total number of communication rounds, and τ represents the time required for each round of communication; K≤M, M is the total number of users participating in federated learning; m=1,......,M; If not, continue to use the selected user set at the previous moment, that is, 3. A communication-efficient federated learning method based on gradient cosine similarity and differential privacy as claimed in claim 1, It is characterized in that Step 3: Each selected user U m In local dataset The stochastic gradient is calculated using the gradient descent algorithm For local models Update to obtain the iterated local model in, For user U m Private datasets, represents an F-dimensional feature vector, Represents a tag.
4. The communication-efficient federated learning method based on gradient cosine similarity and differential privacy as claimed in claim 1, It is characterized in that In step 4, when the server determines that the current time t is not in the first communication round, the following operations are specifically performed: Each selected user U m Based on stochastic gradient Update the local accumulated gradient The formula is as follows: in, is the latest local cumulative gradient after update, is the local cumulative gradient of the previous moment; user U m Before the first local iteration in this round of communication begins, As a clear operation.
5. The communication-efficient federated learning method based on gradient cosine similarity and differential privacy as claimed in claim 1, It is characterized in that In step 5, when the server determines that the current time t is the global model aggregation time, the following operations are specifically performed; ①The server calculates the global sensitivity Δs G , the formula is as follows: Then, according to the privacy protection requirements of the Gaussian (∈, δ)-DP mechanism, the standard deviation σ of the DP noise is calculated as follows: Among them, ∈>0 is the privacy budget, which is used to constrain the distribution difference of adjacent data sets, and δ>0 is the failure probability, which is used to indicate the probability of constraint failure; Next, the server sends the standard deviation σ of the DP noise to each selected user; ②Each selected user U m The local accumulated gradient To crop: Among them, C is the clipping threshold; Next, the user accumulates the gradient towards the clipped local Add DP noise to obtain the perturbed local cumulative gradient that meets the privacy protection requirements of (∈,δ)-DP The formula is as follows: in, represents Gaussian noise, d represents Dimensions; Finally, the user will perturb the local accumulated gradient Upload to the server.
6. A communication-efficient federated learning method based on gradient cosine similarity and differential privacy as claimed in claim 1, It is characterized in that Step 6: If the current time t is the first global model aggregation time, the server performs the following operations: ① Based on each selected user U m The proportion of data volume is used to empower users; user U m Weight The calculation formula is as follows: ② Collect all selected users U m The latest local model These models are then weighted aggregated to update the global model w (t) , the update formula is as follows: ③ Update the pseudo-global gradient, the formula is as follows: in, is the latest pseudo global gradient obtained after updating; If the current time t is not the first global model aggregation time, the server performs the following operations: ① Calculate each selected user U m The perturbed local cumulative gradient and the latest pseudo-global gradient The cosine similarity between The formula is as follows: ② For each selected user U m To empower user U m Weight The calculation formula is as follows: ③ Collect all selected users U m The latest local model These models are then weighted aggregated to update the global model w (t) , the update formula is as follows: ④Update the pseudo-global gradient, the formula is as follows:
Citation Information
Patent Citations
Cluster resource scheduling method and device, cluster system and readable storage medium
CN116684420A
Communication efficient federal learning method based on differential privacy and dynamic weighted aggregation strategy
CN118333138A
Asynchronous gradient averaging distributed stochastic gradient descent
WO2019042571A1
Cited By
Satellite hopping beam method based on federated learning
CN119834871A
Communication optimization federated learning method based on self-guiding evolutionary strategy
CN119940478A
A communication-optimized federated learning method based on self-guided evolutionary strategy
CN119940478B
