An efficient communication method for federated learning based on singular value decomposition

Abstract

The present invention provides an efficient communication method for federated learning based on singular value decomposition. The method comprises: a cloud server uses a similarity metric on model gradients synchronously updated by clients, and utilizes a clustering algorithm to group the clients to obtain several groups of gradient matrices; extracts shared basis vectors from each group's gradient matrix through singular value decomposition, and retains valid basis vector matrices according to a contribution rate strategy; the cloud server reconstructs the gradients within the group based on the basis vector matrix and an aggregation coefficient to obtain a reconstructed gradient; and employs a heterogeneous weight-aware allocation strategy to globally aggregate the reconstructed gradients of each group to update the global gradient and proceed to the next iteration. By projecting the gradients onto the basis vectors, efficient gradient compression is achieved, thereby effectively reducing communication overhead and improving the efficiency of federated learning; and a heterogeneous-aware group weight allocation mechanism is designed to optimize the performance and generalization capability of the global model.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to an efficient communication method for federated learning based on singular value decomposition. Background Art

[0002] Federated learning is a distributed machine learning paradigm that protects user privacy. Its core advantage lies in the ability to collaboratively train models without sharing original data, thereby ensuring data security while protecting user privacy. However, this approach also comes with a series of challenges. A major challenge is dealing with the non-independent and identically distributed (IID) problem of local data among the participants. This leads to the subsequent problem that the model's generalization ability may be limited, as the gradient updates generated by different participants may not be consistent, thus affecting model accuracy. In addition, this also leads to a slowdown in the convergence of the global model. To ensure model accuracy and generalization ability, more training rounds are often required, significantly increasing the communication overhead required for model training.

[0003] When dealing with data heterogeneity, researchers have proposed a variety of solutions at different levels. However, these solutions often only focus on the accuracy and robustness of the model, while ignoring the communication overhead caused by model training. For example, regularization techniques can enhance the adaptability of the model to different data distributions and reduce the risk of overfitting, but they usually transmit more parameters during the communication process, thereby increasing the communication burden. With the rapid development of large models, the scale and complexity of the models continue to increase. A high-performance model usually requires multiple rounds of communication. At the same time, large-scale neural network models often contain millions of parameters, which leads to huge communication overhead. Therefore, in scenarios where data is not independent and identically distributed, another key challenge facing federated learning is how to effectively deal with the communication overhead and model performance impact brought about by data heterogeneity.

[0004] Existing methods for addressing the communication overhead of federated learning fall into two main categories: one focuses on reducing the number of communication rounds during training, and the other on reducing the amount of data transmitted in each round. Among the methods for reducing communication rounds, Fedavg allows the client to perform multiple local updates in each round. This involves computing gradients multiple times on local data, and then sending the updated parameters to the server for aggregation. Other methods introduce momentum to accelerate model convergence and reduce communication rounds. However, these methods can lead to model convergence instability when dealing with highly non-IID data distributions. Methods for reducing the amount of data transmitted per round can result in the loss of important model information. In some cases, the pruned or quantized gradient components may contain information crucial for model updates, thus affecting the model's convergence speed and ultimate performance. Therefore, the difficulties and challenges faced by existing methods for addressing federated communication overhead are: how to address the speed and instability of model convergence, and how to prevent the model from losing critical information, which can affect performance. Summary of the Invention

[0005] In view of the above problems, the present invention provides an efficient communication method for federated learning based on singular value decomposition, which can achieve efficient gradient compression, thereby effectively reducing communication overhead and improving the efficiency of federated learning.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] An efficient communication method for federated learning based on singular value decomposition, the efficient communication method for federated learning comprising the following:

[0008] The cloud server uses similarity measurement on the model gradients updated synchronously by the clients and uses a clustering algorithm to group the clients to obtain several groups of gradient matrices;

[0009] Extract shared basis vectors from the gradient matrix of each group through singular value decomposition, and retain the valid basis vector matrix according to the contribution rate strategy;

[0010] The cloud server reconstructs the intra-group gradient based on the basis vector matrix and the aggregation coefficient to obtain the reconstructed gradient;

[0011] A heterogeneous weight-aware allocation strategy is adopted to globally aggregate the reconstructed gradients of each group to update the global gradient and proceed to the next iteration.

[0012] Furthermore, the gradient matrix is ​​decomposed using SVD to extract the basis vector matrix:

[0013] The server receives the gradient matrix G of the kth group containing m clients;

[0014] Perform SVD decomposition on the gradient matrix G, ,in, is a left singular vector matrix whose column vector The basis vectors that constitute the gradient space, d is the dimension of the data transmitted by each client, that is, the dimension of the column vector in the basis vector; is a diagonal matrix with diagonal elements is a singular value; is the right singular vector matrix;

[0015] Adopting the contribution rate strategy: the cumulative energy contribution rate determines the number of retained basis vectors r:

[0016] ,

[0017] Get a set of basis vector matrices ; j represents a singular value The lower identification value of r is the value that satisfies the cumulative energy contribution rate greater than the set threshold The singular values ​​of The minimum lower identification value j.

[0018] Furthermore, the intra-group gradient reconstruction is specifically as follows:

[0019] The cloud server receives the coefficient vector in the group and calculates the aggregation coefficient ,in, is the aggregation coefficient of the kth group, m is the number of clients in the kth group;

[0020] The server implements the intra-group reconstruction gradient based on the basis vector matrix and the aggregation coefficient: ,in, It is represented as the kth group t+1th round reconstruction gradient; is the shared basis vector matrix of the kth group of clients.

[0021] Furthermore, the coefficient vector is obtained by encoding the gradient using an orthogonal projection method by the client in the group. The specific operation is:

[0022] The client receives the basis vector matrix of the group to which it belongs from the cloud server;

[0023] Approximate the gradient of any client in the group through linear combination , and update the gradient vector of the new round of local training to ;

[0024] in, represents the shared basis vector matrix of the kth group of clients, is the coefficient vector of the i-th group of clients; It is represented as the coefficient value of the rth client in the i-th group;

[0025] Gradient encoding is achieved through orthogonal projection: , obtain the coefficient vector and transmit it to the cloud server.

[0026] Furthermore, the heterogeneous weight-aware allocation strategy is specifically as follows:

[0027] Calculate client local data distribution and ideal independent and identically distributed Cosine similarity of:

[0028] ;

[0029] The distribution difference value is defined as: , then the distribution difference of group k is It can be expressed as:

[0030] ;

[0031] Comprehensive sample size and distribution difference value, the weight of group k for:

[0032] ;

[0033] in, and are hyperparameters, representing the sample quantity weight coefficient and the distribution difference penalty coefficient respectively; is the sample size of group k; is the distribution difference value of the kth group.

[0034] Weighted aggregation obtains the global gradient: , where K represents the number of groups into which the client is divided using the clustering algorithm, which is expressed as the clustering coefficient; Expressed as the weight of the kth group; It is expressed as the reconstructed gradient of the kth group at the t+1th round.

[0035] Furthermore, the global gradient is updated using a batch gradient descent strategy and sent to each client for the next iteration: ,in, is the learning rate; Denotes the global model parameters of round t; is represented as the global gradient.

[0036] The beneficial effects of this application are as follows:

[0037] To address the data heterogeneity problem, this paper uses a gradient similarity grouping strategy based on MiniBatchKMeans to maximize the similarity of the data of devices in each group, and reduces the impact of heterogeneous data on the accuracy of the global model by allocating weights between groups.

[0038] This paper uses the similarity of gradients during training on different devices and the correlation between different rounds to extract group basis vectors in each cluster group based on the singular value decomposition technique. The large-scale gradients aggregated in each group are projected into the group basis vectors to achieve efficient gradient compression, thereby effectively reducing communication overhead and improving the efficiency of federated learning.

[0039] During the cloud server aggregation process, the present invention adopts a group weight allocation mechanism based on heterogeneous perception, which ensures that groups with larger data volumes and smaller category differences contribute more to the global model, thereby optimizing the performance and generalization ability of the global model.

[0040] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 A flowchart of an efficient communication method for federated learning based on singular value decomposition provided by the present invention;

[0043] Figure 2 This figure shows a schematic diagram of an efficient communication method for federated learning based on singular value decomposition provided by the present invention. The following definitions are provided in the diagram: Global Model, Group Gradient, Basis Vector, Client Clusters, and Client Model.

[0044] Figure 3 A schematic diagram showing the impact of heterogeneous perception weight allocation strategies under different hyperparameter selections on a federated learning efficient communication method based on singular value decomposition provided by the present invention;

[0045] Figure 4This is a performance diagram comparing the accuracy of different methods provided in the embodiments of the present invention on different datasets. The datasets in the diagram are as follows: MNIST, a dataset containing handwritten digits; AG News, a dataset of over one million news articles collected by ComeToMyHead; CIFAR-10, an image collection widely used in machine learning and computer vision algorithms.

[0046] Figure 5 This figure shows a performance comparison of the communication overhead of different methods provided in an embodiment of the present invention under different datasets. The datasets in the figure are as follows: MNIST, a dataset containing handwritten digits; AG News, a dataset of over one million news articles collected by ComeToMyHead; CIFAR-10, an image collection widely used in machine learning and computer vision algorithms. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of this application more clearly understood, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] MiniBatchKMeans: Mini-batch K-means, a variant of the K-Means clustering algorithm for large datasets;

[0049] MNIST: A dataset containing tens of thousands of handwritten digits.

[0050] CIFAR-10: a collection of images widely used in machine learning and computer vision algorithms;

[0051] AG News: A dataset of over one million news articles collected by ComeToMyHead, including 496,835 articles from over 2,000 news sources in the AG News corpus. This dataset uses only the title and description fields, with 30,000 training examples and 1,900 test examples for each category.

[0052] FedAvg (Federated Averaging): Federated Averaging algorithm, a distributed machine learning method;

[0053] Fedprox (Federalized Proximal Algorithm): an optimization algorithm designed in the federated learning environment;

[0054] Scaffold (Stochastic Controlled Averaging for Federated Learning): An optimization algorithm for the non-independent and identically distributed (IID) problem of client data in federated learning.

[0055] The following is a brief introduction to the design concept of the embodiments of this application.

[0056] The method of the present invention is implemented in a federated learning system of cloud servers, clients, including but not limited to mobile devices, edge servers or Internet of Things terminals and distributed computing nodes.

[0057] like Figure 1 As shown in FIG, a schematic diagram of an efficient communication method for federated learning based on singular value decomposition is shown. This embodiment can be applied to scenarios of heterogeneous federated learning of data, such as Figure 2 The figure shows a schematic diagram of the application scenario of this embodiment. The method specifically includes the following steps:

[0058] Step S1: The cloud server uses a similarity metric on the model gradients updated synchronously by the clients and uses a clustering algorithm to group the clients to obtain several groups of gradient matrices.

[0059] Specifically, the cloud server initializes the global model parameters and distributes the initial model to the clients participating in the training. After receiving the initial model, the client performs training to update the gradient and uploads the new gradient to the cloud server. At this point, the cloud server synchronously updates the model gradient.

[0060] The cloud server uses a clustering algorithm to group clients, specifically including: using principal component analysis to reduce the dimension of client gradients, retaining features with a cumulative variance contribution rate of 95%; based on the gradient after dimensionality reduction, using the MiniBatchKMeans algorithm for dynamic clustering to divide the clients into K groups, and optimizing the number of clusters K according to the silhouette coefficient; for clients in empty clusters or extremely small clusters, they are redistributed to adjacent clusters based on cosine similarity.

[0061] Among them, when using clustering algorithms to group clients, dynamic clustering uses gradient vectors rather than original data as similarity measurement to avoid data privacy leakage; further combined with local sensitive hashing (LSH) to encrypt the gradient, so that the server can only infer client similarity but cannot restore the original gradient.

[0062] In each update round, each client transmits the gradient vector of its local training process to the server. The gradient vector is defined as the derivative of the client's local loss function with respect to the global model parameters:

[0063] ;

[0064] in For the client The local loss function on is the global model parameter, For the client The gradient vector reflects the direction in which the client adjusts the model parameters during training and is an effective implicit representation of the data distribution characteristics.

[0065] In order to reduce the clustering complexity of high-dimensional gradients, the original gradients are firstly reduced in dimension by principal component analysis (PCA), retaining more than 95% of the cumulative variance contribution rate and compressing the feature dimension to ,in , significantly reducing the computational load while ensuring that the information loss rate is less than 5%. Subsequently, based on the gradient features after dimensionality reduction, the MiniBatchKMeans algorithm is used to dynamically cluster the clients and divide them into K groups. Compared with traditional K-Means, this algorithm can reduce the computational complexity from 100% to 100% through random subsampling. down to ( is the batch size, is the number of iterations), while maintaining a clustering accuracy loss of no more than 2%, ensuring efficient processing of large-scale gradient data. To ensure grouping quality, the silhouette coefficient is introduced to evaluate clustering quality, ensuring the selection of the optimal clustering coefficient K. Furthermore, a repair mechanism is designed for empty and extremely small clusters that may arise during the clustering process. Clients within these clusters are reassigned to adjacent clusters based on cosine similarity, preventing overly dispersed grouping from impacting subsequent gradient compression efficiency.

[0066] Step S2: extracting shared basis vectors from the gradient matrix of each group through singular value decomposition, and retaining the valid basis vector matrix according to the contribution rate strategy;

[0067] The cloud server stacks each group of gradients into a matrix, performs singular value decomposition layer by layer, extracts the shared basis vectors of each group, determines the number of retained basis vectors by cumulative energy contribution rate, and then distributes the basis vectors to the clients in the corresponding group;

[0068] For the gradient matrix, SVD decomposition is used to extract the basis vector matrix:

[0069] Assume that group k contains Client, in In the round of training, the server accepts the gradient vector set of the group as , where the gradient vector of each client in group k is is obtained by expanding the original gradient according to the network order layer, where d represents the dimension of each client transmission data. By stacking these gradient vectors, we get a Matrix : .

[0070] Perform SVD decomposition on the gradient matrix G (need to be processed independently layer by layer to maintain the gradient structure), ,in, is a left singular vector matrix whose column vector Constitute the basis vectors of the gradient space; and is a diagonal matrix with diagonal elements is a singular value; is the right singular vector matrix.

[0071] Since the gradients within a group are highly correlated, only a small number of basis vectors in U are needed, and any column vector in the matrix G can be represented by a linear combination of these basis vectors. According to the properties of the singular value decomposition, The value of reflects the corresponding basis vector The importance of , so you can choose Basis vectors As the principal basis vectors, .

[0072] The cumulative energy contribution rate is used to determine the number of retained basis vectors r:

[0073] ;

[0074] Get a set of basis vector matrices , j represents the singular value The lower identification value of r is the value that satisfies the cumulative energy contribution rate greater than the set threshold The singular values ​​of The minimum lower identification value j; specifically, the threshold Set it to 0.95 to ensure that 95% of the gradient information is retained.

[0075] Step 3: The cloud server reconstructs the intra-group gradient based on the basis vector matrix and the aggregation coefficient to obtain the reconstructed gradient.

[0076] After receiving the basis vector, the client calculates the optimal coefficient of the gradient in the basis vector space through orthogonal projection and uploads only the low-dimensional coefficient vector to the server;

[0077] The coefficient vector is the gradient encoded by the client in the group using the orthogonal projection method, and the specific operation is:

[0078] The client receives the basis vector matrix of the group to which it belongs from the cloud server;

[0079] Approximate the gradient of any client in the group through linear combination ,

[0080] in, represents the shared basis vector matrix of the kth group of clients, is the coefficient vector of the i-th group of clients, and the gradient vector of the new round of local training is updated as ;

[0081] Gradient encoding is achieved through orthogonal projection: , obtain the coefficient vector and transmit it to the cloud server;

[0082] Among them coefficients Represents the gradient in basis vectors The client only needs to transmit the low-dimensional coefficient vector (Dimension ), the communication data volume increases from Compress to .

[0083] Step 4: Adopt the heterogeneous weight-aware allocation strategy to globally aggregate the reconstructed gradients of each group to update the global gradient and proceed to the next iteration.

[0084] Once the coefficients are determined, the client returns them to the server. The server then aggregates the coefficients of each group and calculates the aggregated coefficients ,in, is the aggregation coefficient of the kth group, m is the number of clients in the kth group;

[0085] After the server completes the calculation of the aggregation coefficients of each group, the global gradient update is achieved through a two-step core process: intra-group gradient reconstruction and heterogeneous perception weight allocation.

[0086] The server implements the intra-group reconstruction gradient based on the basis vector matrix and the aggregation coefficient: ,in, It is represented as the reconstructed gradient of the kth group in the t+1th round; Represented as the shared basis vector matrix of the k-th group of clients.

[0087] Heterogeneous perception weight allocation balances the impact of the number and distribution differences of samples within a group on the global model. The weight calculation formula can be specifically expressed as: , moderate The value can effectively suppress model drift caused by heterogeneous data by penalizing client updates whose distribution deviates from IID.

[0088] The heterogeneous weight-aware allocation strategy is specifically as follows:

[0089] Calculate client local data distribution and ideal independent and identically distributed Cosine similarity of:

[0090] ;

[0091] The distribution difference value is defined as: , then the distribution difference of group k is It can be expressed as:

[0092] ;

[0093] Comprehensive sample size and distribution difference value, the weight of group k for:

[0094] ;

[0095] in, and are hyperparameters, representing the sample quantity weight coefficient and the distribution difference penalty coefficient respectively; is the sample size of group k; is the distribution difference value of the kth group.

[0096] Weighted aggregation obtains the global gradient:

[0097] ;

[0098] in, Expressed as the weight of the kth group; It is represented as the reconstructed gradient of the kth group in the t+1th round; K represents the number of groups the client is divided into.

[0099] The cloud server adopts a heterogeneous weight-aware allocation strategy to globally aggregate the reconstruction gradients of each group and finally obtain the global gradient; and uses batch gradient descent to update the global model parameters.

[0100] The cloud server determines whether to update the basis vectors and sends an update signal to the client. Every R rounds, the cloud server regroups the clients and updates the basis vectors, where R represents the basis vector update period. The client then receives the global model and the update signal again, resynchronizing its model gradients with the cloud server. Otherwise, the client gradient vectors are updated again. This process continues until the global model converges.

[0101] The global gradient is updated using a batch gradient descent strategy and sent to each client for the next iteration: ,in, is the learning rate, Denotes the global model parameters of round t; is represented as the global gradient.

[0102] Specifically, during the update process, significant latency may be introduced due to the need for clustering and singular value decomposition, as all clients must send their complete gradient information to the server and wait for the server to calculate and distribute new basis vectors.

[0103] To address this issue, an asynchronous update mechanism is employed. With this mechanism, the client can use existing basis vectors for the next round of training, even if new basis vectors have not yet been calculated. Due to the correlation of gradients, these basis vectors still have the ability to represent the gradients of the next round. This approach not only reduces the time delay caused by waiting for basis vector updates, but also significantly reduces the overall time cost and improves the efficiency of the training process.

[0104] Calculate the communication cost. The formula for calculating the communication cost is ,in, Indicates the number of clients, Indicates the number of communication rounds required for the global model to reach a certain target accuracy or convergence, Indicates the size (in bits) of data transmitted from the client to the server each time.

[0105] In the implementation described above, all simulation experiments were performed on a machine equipped with an NVIDIA RTX 4090D GPU (24GB VRAM), an Intel Xeon(R) Platinum CPU, and 120GB of memory. The experiments were conducted on Ubuntu 22.04, using PyTorch version 2.1.2 and FLGO version 0.3.25.

[0106] The effectiveness of the present invention is verified from four aspects: accuracy comparison of different methods under different data sets, communication overhead comparison of different methods under different data sets, selection of different basis vector update strategies of the present method, and selection of hyperparameters of the present method under heterogeneous-aware weight allocation strategy. In order to evaluate the performance and advantages of our proposed method in heterogeneous data environments, we first conducted accuracy comparison experiments using SVDFed (a popular collaborative gradient compression method) under heterogeneous conditions. Subsequently, we conducted communication overhead comparison experiments using several federated learning algorithms that have shown good performance in data heterogeneity scenarios, including Scaffold and FedProx, as well as two baseline models FedAvg and SVDFed. In our experiments, the hyperparameters of FedProx are Set to 0.01.

[0107] In order to be closer to the actual situation, this paper uses Dirichlet distribution to simulate the skewness of the client data label distribution (non-independent and identically distributed). Among them, the parameters of Dirichlet distribution are Determines the heterogeneity of data distribution: when When is large, the generated data distribution is close to uniform distribution, and the data tends to be independent and identically distributed; when When is smaller, the generated probability distribution is more concentrated in the minority category, the dataset is more skewed, and is closer to non-independent and identically distributed. To simulate higher heterogeneity, this paper Under these parameter settings, we randomly sampled the MNIST, AG News, and CIFAR-10 datasets and distributed them to clients. Local training used SGD as the optimizer with a learning rate of 0.05. Training involved 100 clients with a batch size of 50. To prevent overfitting, 20% of the training data was retained for validation.

[0108] Specifically, to verify the rationality of the basis vector update period R, we set R to {0, 3, 5, 8, 10} (where R = 0 indicates no basis vector updates during training). The experimental results are shown in Table 1. When R = 0, the test accuracy is only 73.2%, the communication overhead is as high as 85.6 GB, and the accuracy fluctuates significantly. This result highlights the necessity of a dynamic update mechanism, as static basis vectors can lead to severe model bias in heterogeneous scenarios. When R = 3, frequent updates accelerate basis vector adjustment, but the short regrouping interval leads to frequent basis vector reconstruction. When R = 5, the communication overhead is further reduced to 60.3 GB, the test accuracy peaks at 82.3%, and the accuracy fluctuation is reduced to ±0.9%. This demonstrates that an appropriate update frequency achieves an optimal balance between communication overhead and model performance. When R > 5, the insufficient update frequency reduces the representation power of the basis vectors. The basis vectors cannot adapt to the dynamic changes of the client gradients in a timely manner, resulting in slower convergence, increased communication overhead, and increased accuracy fluctuation. Therefore, all our experiments described below are tested with an update cycle of 5.

[0109] Table 1 The impact of the method of the present invention under different basis vector update strategy selections:

[0110]

[0111] Specifically, the influence of the number of samples and distribution differences within the balance group on the global model is analyzed by grid search. (sample size weight factor) and (distribution difference penalty coefficient) for joint optimization. and The system performance of various parameter combinations is systematically evaluated in the parameter space with step sizes of 0.1 and 0.05. Figure 3 The model test accuracy heat map under different parameter combinations is shown in the figure, where the horizontal and vertical coordinates correspond to and The value of , the color depth represents the accuracy. The experimental results show that the model performance is significantly sensitive to the parameter selection. At the same time, we observed that when When the ratio is around 2:1 (i.e. the sample size weight is about twice the distribution difference penalty strength), the test accuracy is stable above 80% (as shown in the highlighted area in the figure). When the ratio is high, the model tends to prioritize learning the features of clients with sufficient data, but the model will produce serious overfitting in highly heterogeneous scenarios. Therefore, we need a moderate The value of suppresses the model drift caused by heterogeneous data by penalizing the client updates whose distribution deviates from the independent and identical distribution. All our experiments below are conducted on as well as tested under the circumstances.

[0112] Specifically, we compared the convergence accuracy of our method with that of the SVDFed method on the MNIST and CIFAR-10 datasets and the AG News dataset. The experimental results are as follows: Figure 4 As shown in the figure, it is clear that under different datasets, the SVDFed algorithm experiences significant model accuracy fluctuations under the influence of non-IID data, exhibiting significant instability. This fluctuation not only affects the overall performance of the algorithm but also demonstrates SVDFed's limited adaptability to varying data distributions. Specifically, in the case of non-IID data, the significantly reduced correlation between client gradients makes it difficult for the global model to effectively capture key feature information during the aggregation process, thus affecting overall performance. In contrast, our proposed method demonstrates significant advantages under the same conditions. First, in terms of accuracy, our method consistently outperforms SVDFed, achieving higher test accuracy across multiple datasets. This demonstrates that our method is more effective in optimizing the model's learning capabilities when dealing with non-IID data. Second, our method exhibits greater stability during training. As can be seen from the curves, the oscillation amplitude during training is significantly lower than that of SVDFed. This stability not only accelerates convergence but also further enhances the reliability and robustness of the model in practical applications.

[0113] Specifically, we set the target accuracy of the three datasets MNIST, CIFAR-10, and AG News to 0.9, 0.7, and 0.6, respectively, and compared the communication overhead of our method with the other four methods to achieve the target accuracy. The results are as follows: Figure 5Experimental results demonstrate that our method significantly reduces communication costs through dynamic grouping driven by gradient clustering and a low-rank basis vector compression mechanism. Taking the CIFAR-10 dataset as an example, at Epoch 1 and a model accuracy of 60%, our method consumes only 78.04GB of communication overhead. This represents a reduction of 77.15%, 77.43%, 92.51%, and 48.04% compared to FedAvg, FedProx, Scaffold, and SVDFed, respectively. Furthermore, increasing the number of local training epochs from 1 to 2 accelerates global model convergence, and the communication overhead of all methods decreases. Our method achieves an average reduction of 53.10% in communication overhead across all datasets, with FedAvg, FedProx, Scaffold, and SVDFed achieving reductions of 36.31%, 35.49%, 39.89%, and 57.97%, respectively. Our reduction is only slightly inferior to SVDFed, significantly outperforming other methods. Specifically, we decouple the fine-grained feature learning of the local model (multiple rounds of local training) from the coarse-grained direction correction of the global model (transferring sparse coefficients) through grouped gradient reconstruction and heterogeneous perception aggregation. This not only reduces sensitivity to non-IID (non-independent and identically distributed) data, but also achieves a coordinated optimization of communication efficiency and model accuracy.

[0114] Compared with the prior art, the technical effects of this application are:

[0115] To address the data heterogeneity issue, a gradient similarity grouping strategy based on MiniBatchKMeans is proposed to maximize the similarity of the data of devices in each group. The impact of heterogeneous data on the accuracy of the global model is reduced by weight distribution between groups.

[0116] By leveraging the similarity of gradients during training on different devices and the correlation between different rounds, we extract the group basis vectors in each cluster group based on the singular value decomposition technique. We then project the large-scale gradients aggregated by each group onto the group basis vectors, achieving efficient gradient compression. This effectively reduces communication overhead and improves the efficiency of federated learning.

[0117] During the cloud server aggregation process, a heterogeneity-aware group weight allocation mechanism was designed to ensure that groups with large data volumes and small category differences contribute more to the global model, thereby optimizing the performance and generalization ability of the global model.

[0118] An asynchronous update mechanism is used. Under this mechanism, the client can use existing basis vectors for the next round of training even if new basis vectors have not yet been calculated. Due to the correlation of gradients, these basis vectors still have the ability to represent the gradients of the next round. This approach not only reduces the time delay caused by waiting for basis vector updates, but also significantly reduces the overall time cost and improves the efficiency of the training process.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An efficient communication method for federated learning based on singular value decomposition, characterized in that: The efficient communication method for federated learning includes the following: The cloud server uses similarity measurement on the model gradients updated synchronously by the clients and uses a clustering algorithm to group the clients to obtain several groups of gradient matrices; Extract shared basis vectors from the gradient matrix of each group through singular value decomposition, and retain the valid basis vector matrix according to the contribution rate strategy; The cloud server reconstructs the intra-group gradient based on the basis vector matrix and the aggregation coefficient to obtain the reconstructed gradient; A heterogeneous weight-aware allocation strategy is adopted to globally aggregate the reconstructed gradients of each group to update the global gradient and proceed to the next iteration; The gradient matrix is ​​decomposed by SVD to extract the basis vector matrix: The server receives the gradient matrix G of the kth group containing m clients; Perform SVD decomposition on the gradient matrix G, ,in, is a left singular vector matrix whose column vector Constitute the basis vectors of the gradient space; and is a diagonal matrix with diagonal elements is a singular value; is the right singular vector matrix; Adopting the contribution rate strategy: the cumulative energy contribution rate determines the number of retained basis vectors r: ， Get a set of basis vector matrices .

2. The efficient communication method for federated learning based on singular value decomposition according to claim 1, characterized in that: The intra-group gradient reconstruction is specifically as follows: The cloud server receives the coefficient vector in the group and calculates the aggregation coefficient ,in, is the aggregation coefficient of the kth group, m is the number of clients in the kth group; The server implements the intra-group reconstruction gradient based on the basis vector matrix and the aggregation coefficient: ,in, Denoted as the reconstructed gradient of the kth group.

3. The efficient communication method for federated learning based on singular value decomposition according to claim 2, characterized in that: The coefficient vector is the gradient encoded by the client in the group using the orthogonal projection method. The specific operation is: The client receives the basis vector matrix of the group to which it belongs from the cloud server; Approximate the gradient of any client in the group through linear combination , and update the gradient vector of the new round of local training to ; in, represents the shared basis vector matrix of the kth group of clients, is the coefficient vector of the i-th group of clients; Gradient encoding is achieved through orthogonal projection: , obtain the coefficient vector and transmit it to the cloud server.

4. The efficient communication method for federated learning based on singular value decomposition according to claim 1, characterized in that: The heterogeneous weight-aware allocation strategy is specifically as follows: Calculate client local data distribution and ideal independent and identically distributed Cosine similarity of: ， The distribution difference value is defined as: , then the distribution difference of group k can be expressed as: ， Comprehensive sample size and distribution difference value, the weight of group k for ， in, and are hyperparameters, representing the sample quantity weight coefficient and the distribution difference penalty coefficient respectively; Weighted aggregation obtains the global gradient: 。 5. The efficient communication method for federated learning based on singular value decomposition according to claim 1, characterized in that: The global gradient is updated using a batch gradient descent strategy and sent to each client for the next iteration: ,in, is the learning rate.

Images