A gradient aggregation method for heterogeneous distributed robust learning combining SVD and K-means

Abstract

The present invention belongs to the technical field of distributed machine learning, and more specifically, relates to a heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means. The method comprises: building a distributed learning system; calculating the gradient using a stochastic gradient descent algorithm and uploading it to a parameter server; calculating the SVD score of the gradient; calculating the K-means score of the gradient; combining the SVD score and the K-means score to calculate the comprehensive score of each node, and selecting the n-f gradients with the highest comprehensive scores for average aggregation to obtain a final global gradient, and using the aggregated global gradient for parameter update. The present invention can effectively identify and eliminate Byzantine nodes in a heterogeneous distributed machine learning system, thereby improving the performance of the heterogeneous distributed machine learning system.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed machine learning, and more specifically, relates to a heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means. Background Art

[0002] As data continues to grow in size and complexity, traditional single-machine learning methods are no longer able to meet increasingly complex computing demands. Distributed machine learning, by distributing data and computing tasks across multiple nodes, not only improves computing efficiency but also accelerates model training, offering high scalability and fault tolerance. This has led to widespread application of distributed machine learning in areas such as internet services, financial analysis, and scientific research.

[0003] However, data in real-world applications often exhibits heterogeneous distributions, meaning that the data distribution held by each node varies significantly. This heterogeneity exacerbates the difficulty of identifying Byzantine nodes, which can impact the accuracy of global models by uploading malicious or erroneous information. Due to the heterogeneous data distribution, traditional aggregation methods often struggle to effectively distinguish between legitimate and malicious nodes in the face of Byzantine attacks, leading to information contamination during model training and degrading overall system performance. Therefore, designing effective methods to resist Byzantine attacks and achieve Byzantine robustness is an important research direction in heterogeneous distributed machine learning.

[0004] To address the Byzantine robustness problem under heterogeneous data distributions, existing research has proposed various methods, but they all have limitations. The NBS method only considers gradient norm information and ignores other important features, which may result in ineffective identification of malicious nodes in complex data environments. The Krum method aggregates by selecting updates that are similar to other gradients. However, when the proportion of malicious nodes is high, it has difficulty distinguishing between legitimate nodes, and its performance degrades significantly when the data is highly heterogeneous. The bucketing method divides nodes into multiple buckets and aggregates them independently. While this can reduce the impact of malicious nodes within a single bucket, if the data distribution within a bucket is too specific, the model may not generalize well to the entire dataset. While incorporating a regularization term into the objective function can enhance the model's resilience to outliers, the selection and adjustment of the regularization term increases the complexity of model training and may lead to unstable performance under different data distributions. Existing scoring-based methods such as ByGARS and GAS each have shortcomings in improving Byzantine robustness under heterogeneous data distributions. The ByGARS method uses reputation scores to aggregate gradients. Although it can effectively deal with any number of Byzantine nodes, it relies on additional auxiliary data sets to calculate reputation scores, which may lead to performance degradation in cases of scarce or unbalanced data. In addition, the GAS method aggregates the gradient by dividing it into multiple sub-vectors and calculates the sum of the identification scores of each sub-vector to eliminate malicious updates. This may cause updates of normal nodes to be misjudged, especially when the data distribution is extremely uneven. Overall, the above methods have difficulty in effectively identifying and eliminating Byzantine nodes when processing complex heterogeneous data, resulting in reduced robustness and performance of the model in heterogeneous distributed learning environments, and difficulty in dealing with Byzantine attacks in heterogeneous environments. Therefore, in heterogeneous distributed machine learning, designing a method that can effectively identify and eliminate Byzantine nodes remains a key issue that needs to be solved urgently.

[0005] Chinese patent document CN111967015A discloses a defense proxy method for improving the Byzantine robustness of distributed learning systems. This invention utilizes an adaptive credibility assessment module based on a neural network structure to dynamically assess the credibility of each submitted gradient, update the global classifier parameters maintained on the current master node, generate a reward signal, and adjust the parameters of the adaptive credibility assessment module based on the reward signal within the framework of reinforcement learning. During the training process, the feasibility assessment value of each working node is dynamically adjusted to mitigate the impact of tampered gradients submitted by malicious working nodes on the system training process, thereby improving the Byzantine robustness of the distributed learning system. However, this patent relies on an additional auxiliary dataset to calculate the reputation score, which may lead to performance degradation in cases of data scarcity or imbalance. In particular, in environments with heterogeneous data distribution, it is difficult to ensure the accuracy of the assessment. Chinese patent document CN111898763A discloses a robust Byzantine fault-tolerant distributed gradient descent algorithm. The method includes the following steps: Step 1: Initialize the structure and hyperparameters of the model to be trained; Step 2: Each worker node in the parameter server framework calculates local gradients using the gradient descent method and sends them to the parameter server; Step 3: At the beginning of training, the parameter server first uses the multi-Krum aggregation algorithm to aggregate local gradients. After the global model has been trained to a certain degree, the parameter server then uses the Acc-based aggregation algorithm to aggregate local gradients; Step 4: The parameter server updates the global model based on the aggregated gradients and sends the global model to the worker nodes for the next iteration. Training is terminated when the number of training iterations reaches a set threshold T. However, this patent primarily relies on the multi-Krum and Acc-based algorithms for gradient aggregation. These two methods primarily filter gradients based on the distance or norm between gradients and lack a comprehensive assessment of gradient characteristics (such as gradient direction and singular values), making it difficult to effectively identify and eliminate carefully designed Byzantine attacks.

[0006] In view of this, the present invention designs a heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means. Summary of the Invention

[0007] The present invention aims to overcome at least one of the above-mentioned shortcomings of the prior art and provide a heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means, which can effectively identify and eliminate Byzantine nodes in heterogeneous distributed machine learning systems, thereby improving the performance of heterogeneous distributed machine learning systems.

[0008] The detailed technical solutions of the present invention are as follows:

[0009] A heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means, the method comprising:

[0010] S1. Build a distributed learning system with a parameter server and n nodes, where the nodes are divided into honest nodes and Byzantine nodes.

[0011] S2. The honest node extracts some data samples from the local classification data set and calculates the gradient using the stochastic gradient descent algorithm, and uploads the calculated gradient information to the parameter server;

[0012] S3. The parameter server performs singular value decomposition on the gradient matrices submitted by all nodes, extracts singular values ​​and corresponding eigenvectors, and calculates the SVD score of the gradient to evaluate the effectiveness and importance of each gradient, providing a basis for subsequent aggregation steps.

[0013] S4. The parameter server uses the K-means clustering algorithm to cluster the gradients of all nodes, obtain k cluster centers, and record the distance between each node and its cluster center. Based on the distance, the K-means score of the gradient is calculated for subsequent comprehensive scoring and gradient aggregation.

[0014] S5. Combine the SVD score and K-means score to calculate the comprehensive score of each node, and select the nf gradients with the highest comprehensive scores for average aggregation to obtain the final global gradient. The aggregated global gradient is used to update the global model parameters.

[0015] Preferably, according to the present invention, the specific steps of step S1 are as follows:

[0016] Construct a distributed learning system with a parameter server and n nodes; the n nodes include G honest nodes and nG Byzantine nodes. The Byzantine nodes send false information to the parameter server in order to disrupt the normal operation of the system.

[0017] Next, we will optimize the system. The optimization goal is to find the parameter that minimizes the expected value of the average loss function of all honest nodes. :

[0018] (1)

[0019] In formula (1), It is the ultimate goal of system optimization and the final parameters of the model after training. It represents the model parameters that perform best on the data of honest nodes. represents the local data set of node i, and the data distribution among nodes is heterogeneous; Represents node i with respect to its local dataset The loss function of Indicates that it is based on the local dataset Calculated loss function The expected value of Represents the model parameters, which are used to calculate the predicted value of the model and are continuously updated through the optimization algorithm to minimize the loss function.

[0020] Preferably, according to the present invention, the specific steps of step S2 are as follows:

[0021] S21, honest node i first receives the current global model parameters from the parameter server ;

[0022] S22, honest node i from its local classification dataset Extract some data samples , and calculate the global model parameters Under local stochastic gradient :

[0023] (2)

[0024] In formula (2), t represents the tth iteration, Represents the current global model parameters;

[0025] S23, honest node i calculates the local stochastic gradient Upload it to the parameter server, and the Byzantine node will send malicious gradients to the parameter server.

[0026] Preferably, according to the present invention, the specific steps of step S3 are as follows:

[0027] S31. After receiving the gradients uploaded by all nodes, the parameter server constructs all the gradients into a matrix M:

[0028] (3)

[0029] In formula (3), represents the stochastic gradient sent by node n at iteration t;

[0030] S32. Perform singular value decomposition on the gradient matrix M to obtain:

[0031] (4)

[0032] In formula (4), U is the left singular vector matrix, Σ is a diagonal matrix containing singular values, is the right singular vector matrix;

[0033] S33. Extract the singular value corresponding to node i from the diagonal matrix Σ , the singular value reflects the importance of node i. Subsequently, the SVD score of each node is calculated based on the extracted singular value:

[0034] (5)

[0035] In formula (5), represents the SVD score of node i, represents the singular value corresponding to node i, represents the singular value corresponding to node j, n represents node n, Represents the sum of the singular values ​​corresponding to all nodes. Through the above steps, the contribution of each node can be effectively evaluated, providing a basis for the subsequent calculation of the comprehensive score and aggregation process.

[0036] Preferably, according to the present invention, the specific steps of step S4 are as follows:

[0037] S41. The parameter server uses the K-means clustering algorithm to cluster the gradients of all nodes and randomly selects k cluster centers. ,in ;

[0038] S42. Assign the gradient of each node to the cluster with the nearest cluster center, and record the distance between each node and its corresponding cluster center:

[0039] (6)

[0040] In formula (6), Represents the cluster center of node i and its cluster The distance between represents the gradient sent by node i at the tth iteration, represents the cluster center of the cluster to which node i belongs;

[0041] S43, based on distance , calculate the K-means score for each node:

[0042] (7)

[0043] In formula (7), Represents the K-means score of node i. The closer the node is to the cluster center, The higher the score, the more effective the K-means score of each node is calculated through the above steps, providing a basis for subsequent comprehensive scoring and aggregation.

[0044] Preferably, according to the present invention, the specific steps of step S5 are as follows:

[0045] S51. Combine the SVD score and the K-means score to get the comprehensive score of each node:

[0046] (8)

[0047] In formula (8), represents the comprehensive score of node i, represents the SVD score of node i, represents the K-means score of node i, represents the SVD score of node j, represents the K-means score of node j;

[0048] S52: Select the gradients sent by nf nodes with the highest comprehensive scores, average them and calculate the global gradient :

[0049] (9)

[0050] S53. Using global gradient Update global model parameters:

[0051] (10)

[0052] In formula (10), represents the global model parameters at the t+1th iteration, is the global model parameter at the tth iteration, represents the learning rate;

[0053] S54. Use this method to continuously iterate the global model parameters until the objective function is satisfied or the set maximum number of iterations is reached.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] This paper combines singular value decomposition (SVD) and a comprehensive scoring mechanism based on K-means clustering to effectively evaluate the contribution of each node, prioritizing nodes with higher comprehensive scores. Compared with existing technologies, this paper comprehensively considers node effectiveness to accurately identify and filter Byzantine nodes, significantly improving the robustness of distributed learning systems against Byzantine attacks in heterogeneous data distributions. Ultimately, it achieves accurate aggregation of information from honest nodes in complex data environments, thereby improving the performance and stability of the global model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flowchart of a heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means described in the present invention.

[0057] Figure 2 This is a comparison of the test accuracy of the SK Score method of the present invention and other clustering methods in the no-attack scenario in Example 1 of the present invention.

[0058] Figure 3 This is a comparison of the test accuracy of the SK Score method of the present invention and other clustering methods under the Bit Flipping attack scenario in Example 1 of the present invention.

[0059] Figure 4 This is a comparison of the test accuracy of the SK Score method of the present invention and other clustering methods under the IPM attack scenario in Example 1 of the present invention. DETAILED DESCRIPTION

[0060] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0061] Example 1

[0062] Ginseng Figure 1 This embodiment provides a heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means, the method comprising the following steps:

[0063] S1. Build a heterogeneous distributed learning system. The specific steps are as follows:

[0064] A distributed learning system with a parameter server and n nodes is constructed. The n nodes include G honest nodes and nG Byzantine nodes. The Byzantine nodes send false information to the parameter server in order to disrupt the normal operation of the system.

[0065] Next, we will optimize the system. The optimization goal is to find the parameter that minimizes the expected value of the average loss function of all honest nodes. :

[0066] (1)

[0067] In formula (1), It is the ultimate goal of system optimization and the final parameters of the model after training. It represents the model parameters that perform best on the data of honest nodes. represents the local data set of node i, and the data distribution among nodes is heterogeneous; Represents node i with respect to its local classification dataset The loss function of Represents a local classification dataset Calculated loss function The expected value of Represents model parameters, used to calculate the predicted value of the model, and continuously updated through the optimization algorithm to minimize the loss function. Preferably, the local classification dataset is an image dataset.

[0068] S2. The honest node extracts some data samples from its local classification data set and calculates the gradient using the stochastic gradient descent algorithm, and uploads the calculated gradient information to the parameter server;

[0069] S21. At each iteration, honest node i first receives the current global model parameters from the parameter server. , preferably, the global model is an MLP model;

[0070] S22, honest node i from its local classification dataset Extract some data samples , and calculate the global model parameters Under local stochastic gradient :

[0071] (2)

[0072] In formula (2), t represents the tth iteration, Represents the current global model parameters;

[0073] S23, honest node i calculates the local stochastic gradient Upload it to the parameter server, and the Byzantine node will send malicious gradients to the parameter server.

[0074] S3. The parameter server performs singular value decomposition on the gradient matrices submitted by all nodes, extracts singular values ​​and corresponding eigenvectors, and calculates the SVD score of the gradient to evaluate the effectiveness and importance of each gradient, providing a basis for subsequent aggregation steps. The specific steps are as follows:

[0075] S31. After receiving the gradients uploaded by all nodes, the parameter server constructs all the gradients into a matrix M:

[0076] (3)

[0077] In formula (3), represents the stochastic gradient sent by node n at iteration t;

[0078] S32. Perform singular value decomposition on the gradient matrix M to obtain:

[0079] (4)

[0080] In formula (4), U is the left singular vector matrix, Σ is a diagonal matrix containing singular values, is the right singular vector matrix;

[0081] S33. Extract the singular value corresponding to node i from the diagonal matrix Σ , the singular value reflects the importance of node i. Subsequently, the SVD score of each node is calculated based on the extracted singular value:

[0082] (5)

[0083] In formula (5), represents the SVD score of node i, represents the singular value corresponding to node i, represents the singular value corresponding to node j, n represents node n, Represents the sum of the singular values ​​corresponding to all nodes. Through the above steps, the contribution of each node can be effectively evaluated, providing a basis for the subsequent calculation of the comprehensive score and aggregation process.

[0084] S4. The parameter server uses the K-means clustering algorithm to cluster the gradients of all nodes, obtain k cluster centers, and record the distance between each node and its cluster center. Based on the distance, the K-means score of the gradient is calculated for subsequent comprehensive scoring and gradient aggregation. The specific steps are as follows:

[0085] S41. The parameter server uses the K-means clustering algorithm to cluster the gradients of all nodes and randomly selects k cluster centers. ,in ;

[0086] S42. Assign the gradient of each node to the cluster with the nearest cluster center, and record the distance between each node and its corresponding cluster center:

[0087] (6)

[0088] In formula (6), Represents the cluster center of node i and its cluster The distance between represents the gradient sent by node i at the tth iteration, represents the cluster center of the cluster to which node i belongs;

[0089] S43, based on distance , calculate the K-means score for each node:

[0090] (7)

[0091] In formula (7), Represents the K-means score of node i. The closer the node is to the cluster center, The higher the score, the more effective the K-means score of each node is calculated through the above steps, providing a basis for subsequent comprehensive scoring and aggregation.

[0092] S5. Combine the SVD score and K-means score to calculate the comprehensive score of each node, and select nf gradients with the highest comprehensive scores for average aggregation to obtain the final global gradient. The aggregated global gradient is used for parameter update. The specific steps are as follows:

[0093] S51. Combine the SVD score and the K-means score to get the comprehensive score of each node:

[0094] (8)

[0095] In formula (8), represents the comprehensive score of node i, represents the SVD score of node i, represents the K-means score of node i, represents the SVD score of node j, represents the K-means score of node j;

[0096] S52: Select the gradients sent by nf nodes with the highest comprehensive scores, average them and calculate the global gradient :

[0097] (9)

[0098] S53. Using global gradient Update global model parameters:

[0099] (10)

[0100] in, represents the global model parameters at the t+1th iteration, is the global model parameter at the tth iteration, represents the learning rate;

[0101] S54. Use this method to continuously iterate the global model parameters until the objective function is satisfied or the set maximum number of iterations is reached.

[0102] Experimental example,

[0103] The experiment uses the FEMNIST image dataset, and the loss function considers cross entropy loss to evaluate the training effect of the model. The cross entropy loss function is defined as follows:

[0104] (11)

[0105] Where N represents the total number of data samples, represents the true label, Indicates that the model predicts the input data sample as probability.

[0106] This method (SK Score) is compared with the Median, RFA, Bulyan, and Multi-Krum aggregation methods, considering three experimental scenarios: no attack, Bit Flipping attack, and IPM attack. In the Bit Flipping attack scenario, after the Byzantine node calculates the stochastic gradient, it negates the gradient and then sends the gradient -∇f(x) to the parameter server. In the IPM attack scenario, the Byzantine node first calculates the average of the gradients of all honest nodes, then multiplies this average gradient by a negative coefficient -ϵ, and finally sends this gradient to the parameter server. Figure 2 It can be seen that in the non-attack scenario, the method described in the present invention achieves the highest test accuracy, which is better than other methods. Figure 3 It can be seen that in the Bit Flipping attack scenario, the method described in the present invention achieved the highest test accuracy, which is better than other methods. Figure 4 It can be seen that in the IPM attack scenario, the method described in the present invention achieves the highest test accuracy, which is better than other methods.

[0107] Example 2

[0108] This embodiment applies the method described in this invention to a personalized recommendation system in Internet services and a fraud detection system in financial analysis, specifically as follows:

[0109] In internet services, personalized recommendation systems, such as those on e-commerce platforms, video sites, and social media platforms, often rely on distributed machine learning to train recommendation models. These systems must process massive amounts of data from users around the world, and user behavior data is often heterogeneously distributed. For example, user preferences vary significantly across regions, leading to uneven distribution of gradient matrices. Furthermore, malicious nodes, such as compromised servers or malicious users, may exist in the system. These nodes may upload false data or malicious gradients, disrupting the training of recommendation models.

[0110] The proposed method first extracts global features of user behavior data through singular value decomposition and evaluates the contribution of each node. Cluster analysis then distinguishes between normal and abnormal nodes, identifying malicious nodes. Finally, combining SVD scores with K-means scores accurately identifies and removes malicious nodes, ensuring the robustness of the recommendation model. By effectively aggregating gradients from normal nodes, the accuracy of the recommendation model is significantly improved. Even in the presence of malicious nodes, the system maintains stable operation and prevents interference with recommendation results.

[0111] In the field of financial analytics, fraud detection systems often rely on distributed machine learning to train fraud detection models. These systems must process heterogeneous data (such as transaction records and user behavior) from multiple financial institutions. This data distribution can vary significantly, leading to uneven distribution of gradient matrices. Furthermore, malicious nodes (such as attacked financial institutions or malicious users) may exist in the system. These nodes may upload false data or malicious gradients, interfering with the training of fraud detection models and resulting in inaccurate detection results.

[0112] The method first extracts global features of transaction data through singular value decomposition and evaluates the contribution of each node. Cluster analysis then distinguishes normal and abnormal nodes, identifying malicious nodes. Finally, combining SVD scores with K-means scores accurately identifies and removes malicious nodes, ensuring the robustness of the fraud detection model. By effectively aggregating gradients from normal nodes, the accuracy of the fraud detection model is significantly improved. Even in the presence of malicious nodes, the system maintains stable operation and prevents interference with detection results.

[0113] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means, characterized by: The method comprises: S1. Build a distributed learning system with a parameter server and n nodes, where the nodes are divided into honest nodes and Byzantine nodes. S2. The honest node extracts some data samples from the local classification data set and calculates the gradient using the stochastic gradient descent algorithm, and uploads the calculated gradient information to the parameter server; S3. The parameter server performs singular value decomposition on the gradient matrices submitted by all nodes, extracts the singular values ​​and corresponding eigenvectors, and calculates the SVD score of the gradient. S4. The parameter server uses the K-means clustering algorithm to cluster the gradients of all nodes, obtain k cluster centers, and record the distance between each node and its cluster center. Based on the distance, the K-means score of the gradient is calculated; S5. Combine the SVD score and K-means score to calculate the comprehensive score of each node, and select nf gradients with the highest comprehensive scores for average aggregation to obtain the final global gradient. The aggregated global gradient is used to update the global model parameters. The specific steps are as follows: S51. Combine the SVD score and the K-means score to get the comprehensive score of each node: （1） In formula (1), represents the comprehensive score of node i, represents the SVD score of node i, represents the K-means score of node i, represents the SVD score of node j, represents the K-means score of node j; S52: Select the gradients sent by nf nodes with the highest comprehensive scores, average them and calculate the global gradient : （2） S53. Using global gradient Update global model parameters: （3） In formula (3), represents the global model parameters at the t+1th iteration, is the global model parameter at the tth iteration, represents the learning rate; S54. Use this method to continuously iterate the global model parameters until the objective function is satisfied or the set maximum number of iterations is reached.

2. The heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means according to claim 1, characterized in that: The specific steps of step S1 are as follows: Construct a distributed learning system with a parameter server and n nodes; the n nodes include G honest nodes and nG Byzantine nodes; Next, we will optimize the system. The optimization goal is to find the parameter that minimizes the expected value of the average loss function of all honest nodes. : （4） In formula (4), It is the ultimate goal of system optimization and the final parameters of the model after training. It represents the model parameters that perform best on the data of honest nodes. represents the local data set of node i, and the data distribution among nodes is heterogeneous; Represents node i with respect to its local dataset The loss function of Indicates that it is based on the local dataset Calculated loss function The expected value of Represents the model parameters, which are used to calculate the predicted value of the model and are continuously updated through the optimization algorithm to minimize the loss function.

3. The heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means according to claim 1, characterized in that: The specific steps of step S2 are as follows: S21, honest node i first receives the current global model parameters from the parameter server ; S22, honest node i from its local classification dataset Extract some data samples , and calculate the global model parameters Under local stochastic gradient : （5） In formula (5), t represents the tth iteration, Represents the current global model parameters; S23, honest node i calculates the local stochastic gradient Upload it to the parameter server, and the Byzantine node will send malicious gradients to the parameter server.

4. The heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means according to claim 1, characterized in that: The specific steps of step S3 are as follows: S31. After receiving the gradients uploaded by all nodes, the parameter server constructs all the gradients into a matrix M: （6） In formula (6), represents the stochastic gradient sent by node n at iteration t; S32. Perform singular value decomposition on the gradient matrix M to obtain: （7） In formula (7), U is the left singular vector matrix, Σ is a diagonal matrix containing singular values, is the right singular vector matrix; S33. Extract the singular value corresponding to node i from the diagonal matrix Σ , the singular value reflects the importance of node i. Subsequently, the SVD score of each node is calculated based on the extracted singular value: （8） In formula (8), represents the SVD score of node i, represents the singular value corresponding to node i, represents the singular value corresponding to node j, n represents node n, Represents the sum of the singular values ​​corresponding to all nodes.

5. The heterogeneous distributed robust learning gradient aggregation method combining SVD and K-means according to claim 1, characterized in that: The specific steps of step S4 are as follows: S41. The parameter server uses the K-means clustering algorithm to cluster the gradients of all nodes and randomly selects k cluster centers. ,in ; S42. Assign the gradient of each node to the cluster with the nearest cluster center, and record the distance between each node and its corresponding cluster center: （9） In formula (9), Represents the cluster center of node i and its cluster The distance between represents the gradient sent by node i at the tth iteration, represents the cluster center of the cluster to which node i belongs; S43, based on distance , calculate the K-means score for each node: （10） In formula (10), Represents the K-means score of node i. The closer the node is to the cluster center, The higher the score.

Images