A distributed policy gradient fault-tolerant mechanism method based on graph structure similarity analysis

Abstract

The application relates to a distributed policy gradient fault-tolerant mechanism method based on graph structure similarity analysis, and specifically comprises the following steps: step 1, initializing a worker node and a server, initializing a model structure to be trained and hyperparameters, step 2, graph structure feature calculation: taking the gradient vector uploaded by a node as a point, calculating a gradient distance and combining a node weight to obtain a directed graph, step 3, calculating the similarity between all nodes according to the graph structure features, step 4, each node giving a vote to the node with the maximum similarity, step 5, updating the weight of each node to the product of the weight of the last round and a decay rate plus the sum of the votes of the current round, and step 6, the weighted average of the gradients of the nodes with the maximum votes will be used as the aggregated gradient of the current round. The algorithm has strong practicability and stronger robustness, can be applied to all machine learning models based on the policy gradient descent method, and can resist various Byzantine attacks.

Description

Technical Field

[0001] This invention relates to a distributed strategy gradient fault-tolerance mechanism based on graph structure similarity analysis, belonging to the field of distributed machine learning technology. Background Technology

[0002] In distributed machine learning, effectively distributing massive amounts of data and computational tasks across multiple machines can improve computation speed, protect user data privacy, and increase scalability. Taking a commonly used centralized framework in distributed machine learning as an example, this framework has a central parameter server node and several worker nodes. Each worker node uses its own data and current model parameters to obtain a local model and sends it to the central server. The central server then aggregates all received local models and updates the global model.

[0003] However, the nature of distributed learning makes it vulnerable to errors and adversarial attacks. Devices may experience computational failures or be attacked during computation or communication, potentially sending malicious messages to the data center and misleading the learning process. This invention focuses primarily on malicious attacks known as Byzantine attacks. Taking a centralized distributed machine learning framework as an example, both worker nodes and the central node are susceptible to Byzantine attacks. If a worker node is attacked by Byzantine attacks, it sends incorrect local models to the central node, degrading the performance of the trained model; if the central node is attacked by Byzantine attacks, it receives incorrect global gradients and obtains an incorrect machine learning model.

[0004] To date, some progress has been made in the study of Byzantine attacks in distributed machine learning. However, the following problems remain: 1. Existing distributed policy gradient descent algorithms with Byzantine fault tolerance have poor robustness and cannot effectively defend against some Byzantine attacks, such as inner product operation attacks, sign inversion attacks, or combinations of multiple attacks. 2. Existing work assumes that the central node is honest and reliable, and the algorithms used to defend against Byzantine attacks on training nodes have poor robustness. Furthermore, when the central node is attacked by Byzantine attacks, existing consensus algorithms cannot identify malicious nodes. In practical applications, attackers often only need to attack a few nodes, monitor the gradient distribution of the attacked nodes, estimate the correct gradient distribution, and backpropagate specific gradients to disable most fault-tolerant algorithms. The few distributed learning systems that can resist these attacks rely on local computation of the original data on the server to verify the correctness of the node gradients. This not only increases server computational overhead but is also difficult to apply when data is private. Therefore, a new solution is urgently needed to address these technical problems. Summary of the Invention

[0005] This invention addresses the problems existing in current technologies by providing a distributed strategy gradient fault-tolerance mechanism based on graph structure similarity analysis. This solution is designed for distributed machine learning and provides an algorithm to address the problem of nodes being susceptible to Byzantine attacks. Furthermore, the algorithm provided by this invention has the advantage of not requiring known trusted nodes; that is, it does not need to assume that the central node is honest and reliable, as in previous works.

[0006] To achieve the above objectives, the technical solution of this invention is as follows: a distributed strategy gradient fault-tolerant mechanism method based on graph structure similarity analysis. This method constructs a directed graph using the weights of gradient distance and node credibility, constructs the graph structure features of nodes using an ordered set of directed edge lengths, and selects gradients for aggregation by calculating similarity based on these graph structure features. The algorithm includes the following steps:

[0007] Step 1, System Initialization: System initialization includes worker node initialization and server initialization.

[0008] The worker node initialization determines the model structure and hyperparameters to be trained.

[0009] Server initialization determines the number of worker nodes. attenuation rate The number of confidence nodes is determined as follows: Initial weights .

[0010]

[0011] in, This is the Gaussian rounding symbol.

[0012]

[0013]

[0014] in, express The first node in the nth node 1 node Set the initial weight for this node.

[0015] Step 2, Graph Structure Feature Calculation: Using the gradient vector uploaded by each node as a point, calculate the gradient distance and combine it with the node weights to construct directed edges between points, obtaining a directed graph. The graph structure feature of a node is an ordered set of the lengths of the directed edges formed by the points represented by its gradient. The specific implementation is as follows:

[0016] Distributed learning frameworks include The node. At the... In the first iteration, the... Each node calculates its gradient using the policy gradient method. And upload it to the server.

[0017] After receiving the gradient, the server calculates the graph structure features in the following way:

[0018] Step 1: Calculate the gradient distance.

[0019] The specific process for calculating the distance between the gradients of all nodes is as follows:

[0020]

[0021]

[0022] in, Represents a node With nodes Gradient distance between them For nodes In the Gradients uploaded in each iteration For nodes In the Gradients uploaded in each iteration It is the Euclidean norm.

[0023] Step 2: Calculate the confidence distance.

[0024] The confidence distance between all nodes is calculated based on their weights, as follows:

[0025]

[0026]

[0027] in, For nodes With nodes Confidence distance between For nodes With nodes Gradient distance between them Let be the weight of node j in the previous iteration.

[0028] Step 3: Construct the graph structure features of the nodes.

[0029] For any node , will be with the node Confidence distance smallest The other nodes form a set within the set Each node and node gradient distance between ( An ordered set is formed from smallest to largest. ,in . Set Called a node The graph structure features.

[0030] Step 3, Calculate the similarity of the graph structure between nodes: Calculate the similarity between all nodes based on the graph structure features. The specific process is as follows:

[0031]

[0032] in, For nodes With nodes The similarity between them. This represents the number of confidence nodes.

[0033] Step 4, Voting: Each node votes for the node with the highest similarity to itself. Each node casts one vote, and each node casts a total of [number] votes. Tickets. Nodes The number of votes received was .

[0034] Step 5, Update Weights: The weight of each node is the sum of the weight from the previous round multiplied by the decay rate and the number of votes received in the current round. The specific process is as follows:

[0035]

[0036] in, For the first wheel node The weight, The attenuation rate, For nodes The number of votes received.

[0037] Step 6, aggregate gradients: the one with the most votes The gradient-weighted average of the nth node will be used as the gradient weighted average of the nth node. The gradient of the round aggregation. The specific process is as follows:

[0038] The one with the most votes The set consists of nodes.

[0039]

[0040] in, For the first The gradient after round aggregation. For nodes In the Gradients uploaded in round-robin fashion. For nodes No. The weight of the wheel.

[0041] This reduces the impact of Byzantine nodes being selected in the initial rounds and allows for the rapid elimination of Byzantine nodes.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] 1) The distributed policy gradient fault tolerance mechanism based on graph structure similarity analysis provided by this technical solution has strong practicality and can be applied to all machine learning models based on policy gradient descent.

[0044] 2) The algorithm proposed in this invention has stronger robustness than existing algorithms and can resist various Byzantine attacks;

[0045] 3) The algorithm provided by this invention has the advantage of not requiring known trusted nodes.

[0046] 4) The algorithm provided by this invention does not require data samples to supervise fault-tolerant aggregation.

[0047] 5) Calculate similarity through the graph structure of gradient distribution. A series of fault-tolerant aggregation algorithms, such as Median, Krum, Multi-Krum, and Bullyan, calculate similarity based on the distance between individual gradients—isolated point-to-point distances, where closer distances are considered more reliable—without taking into account the overall distribution. This approach, however, constructs a graph structure with surrounding gradients and calculates the correlation between nodes based on the similarity between these graph structures, thus incorporating the gradient distribution. Even if two gradients are far apart, they can still be correlated through the similarity of the graph structure formed by their surrounding gradients. This essentially grants the ability to perform convolution and self-attention within spatial point clouds.

[0048] 6) High applicability. As a fault-tolerant mechanism, it can be easily integrated into distributed learning systems without requiring additional data or privacy data. Its computational complexity is no higher than existing algorithms, it does not require the assumption of trusted nodes, and it can automatically separate abnormal nodes. For example, it can be applied to distributed learning in hospital disease treatment. In this scenario, patient data is privacy data and cannot be directly obtained or trusted nodes can be preset. It can only be aggregated through intermediate data after distributed learning. Attached Figure Description

[0049] Figure 1 This is a schematic diagram showing the performance comparison of the model obtained by using different aggregation methods when none of the seven nodes were subjected to Byzantine attacks during the local gradient calculation in this embodiment of the invention.

[0050] Figure 2In this example of the invention, when performing local gradient calculation, 3 out of 7 nodes are subjected to sign inversion attack (dispersion). The model performance comparison diagram obtained by using different aggregation methods is shown.

[0051] Figure 3 In this example of the invention, when performing local gradient calculation, 3 out of 7 nodes are subjected to a sign inversion attack (concentrated). The model performance comparison diagram obtained by using different aggregation methods is shown.

[0052] Figure 4 In this example of the invention, when performing local gradient calculation, 3 out of 7 nodes are attacked by inner product operations (concentrated). The diagram shows the performance comparison of the model obtained by using different aggregation methods.

[0053] Figure 5 In this example of the invention, when performing local gradient calculation, 3 out of 7 nodes are attacked (dispersed) by inner product operation. The diagram shows the performance comparison of the model obtained by using different aggregation methods. Detailed Implementation

[0054] To enhance understanding of the present invention, the following detailed description of the solution is provided in conjunction with the accompanying drawings.

[0055] Example 1: See Figures 1-5 Generally, the longer the duration, the better the performance. A distributed strategy gradient fault-tolerance mechanism based on graph structure similarity analysis is proposed. The algorithm includes the following steps:

[0056] Step 1, System Initialization: System initialization includes worker node initialization and server initialization.

[0057] The worker node initialization determines the model structure and hyperparameters to be trained.

[0058] Server initialization determines the number of worker nodes. attenuation rate The number of confidence nodes is determined as follows: Initial weights .

[0059]

[0060] in, This is the Gaussian rounding symbol.

[0061]

[0062]

[0063] in, express The first node in the nth node 1 node Set the initial weight for this node.

[0064] Step 2, Graph Structure Feature Calculation: Using the gradient vector uploaded by each node as a point, calculate the gradient distance and combine it with the node weights to construct directed edges between points, obtaining a directed graph. The graph structure feature of a node is an ordered set of the lengths of the directed edges formed by the points represented by its gradient. The specific implementation is as follows:

[0065] Distributed learning frameworks include The node. At the... In the first iteration, the... Each node calculates its gradient using the policy gradient method. And upload it to the server.

[0066] After receiving the gradient, the server calculates the graph structure features in the following way:

[0067] Step 1: Calculate the gradient distance.

[0068] The specific process for calculating the distance between the gradients of all nodes is as follows:

[0069]

[0070]

[0071] in, Represents a node With nodes Gradient distance between them For nodes In the Gradients uploaded in each iteration For nodes In the Gradients uploaded in each iteration It is the Euclidean norm.

[0072] Step 2: Calculate the confidence distance.

[0073] The confidence distance between all nodes is calculated based on their weights, as follows:

[0074]

[0075]

[0076] in, For nodes With nodes Confidence distance between For nodes With nodes Gradient distance between them Let be the weight of node j in the previous iteration.

[0077] Step 3: Construct the graph structure features of the nodes.

[0078] For any node , will be with the node Confidence distance smallest The other nodes form a set within the set Each node and node gradient distance between ( An ordered set is formed from smallest to largest. ,in . Set Called a node The graph structure features.

[0079] Step 3, Calculate the similarity of the graph structure between nodes: Calculate the similarity between all nodes based on the graph structure features. The specific process is as follows:

[0080]

[0081] in, For nodes With nodes The similarity between them. This represents the number of confidence nodes.

[0082] Step 4, Voting: Each node votes for the node with the highest similarity to itself. Each node casts one vote, and each node casts a total of [number] votes. Tickets. Nodes The number of votes received was .

[0083] Step 5, Update Weights: The weight of each node is the sum of the weight from the previous round multiplied by the decay rate and the number of votes received in the current round. The specific process is as follows:

[0084]

[0085] in, For the first wheel node The weight, The attenuation rate, For nodes The number of votes received.

[0086] Step 6, aggregate gradients: the one with the most votes The gradient of the nth node will be used as the gradient of the nth node. The gradient of the round aggregation. The specific process is as follows:

[0087] The one with the most votes The set consists of nodes.

[0088]

[0089] in, For the first The gradient after round aggregation. For nodes In the Gradients uploaded in round-robin fashion. For nodes No. The weight of the wheel.

[0090] This reduces the impact of Byzantine nodes being selected in the initial rounds and allows for the rapid elimination of Byzantine nodes.

[0091] Our aggregation scheme was tested against the following attack methods and compared with fault-intolerant schemes and fault-tolerant mechanisms such as Krum and Median:

[0092] The experiment consisted of 7 nodes, learning in the cartpole balancing-v1 scenario. The longer the balance maintenance time, the better the performance.

[0093] 1. No attack; none of the 7 nodes were subjected to Byzantine attacks. See [link / reference] Figure 1 ,

[0094] 2. Symbol inversion attack (dispersion), Byzantine node In the Backpropagation gradients during training rounds , For nodes In the The correct local gradient of the wheel, It is a negative constant, here Of the 7 nodes, 3 are Byzantine nodes; see [link / reference]. Figure 2 ,

[0095] 3. Symbol inversion attack (centralized): All Byzantine nodes backpropagate the same gradient. Byzantine nodes In the Backpropagation gradients during training rounds , For node 0 in the th The correct local gradient of the wheel, It is a negative constant, here Of the 7 nodes, 3 are Byzantine nodes; see [link / reference]. Figure 3 ,

[0096] 4. Inner product operation attack (centralized): All Byzantine nodes return the same gradient. Byzantine nodes In the Backpropagation gradients during training rounds , Normal node In the The correct local gradient of the wheel, It is a negative constant, here Of the 7 nodes, 3 are Byzantine nodes, see [link / reference]. Figure 4 ,

[0097] 5. Inner product operation attack (distributed): All Byzantine nodes return the same gradient. Byzantine nodes In the Backpropagation gradients during training rounds , Normal node In the The correct local gradient of the wheel, It is a negative constant, here Of the 7 nodes, 3 are Byzantine nodes, see [link / reference]. Figure 5 ,

[0098] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A distributed strategy gradient fault-tolerant mechanism method based on graph structure similarity analysis, characterized in that, The method includes the following steps: Step 1, System Initialization: System initialization includes worker node initialization and server initialization. Worker node initialization determines the model structure and hyperparameters to be trained, while server initialization determines the number of worker nodes. attenuation rate Number of confidence nodes Initial weights , Step 2, Graph Structure Feature Calculation: Using the gradient vector uploaded by the node as a point, calculate the gradient distance and combine it with the node weights to construct directed edges between the points, obtaining a directed graph. The graph structure feature of a node is an ordered set of the lengths of the directed edges formed by the points represented by its gradient. The specific implementation method is as follows: Distributed learning frameworks include The node at the node In each iteration, all nodes upload gradients, where the th... Each node calculates its gradient using the policy gradient method. And upload it to the server. After receiving the gradient, the server calculates the graph structure features in the following way: Step 1: Calculate the gradient distance. For the gradient of all nodes, calculate the distance between them. , Step 2: Calculate the confidence distance. For all nodes, calculate the confidence distance between them based on their weights. , Step 3: Construct the graph structure features of the nodes, for any node , will be with the node Confidence distance smallest The other nodes form a set within the set Each node and node gradient distance between ( An ordered set is formed from smallest to largest. ,in , will set Called a node The graph structure features, Step 3, Calculate the similarity of the graph structure between nodes: Calculate the similarity between all nodes based on the graph structure features. Step 4, Voting: Each node votes for the node with the highest similarity to itself. Each node casts one vote, and each node casts a total of [number] votes. ticket, Step 5, Update Weights: The weight of each node is the sum of the weight from the previous round multiplied by the decay rate and the number of votes received in the current round. Step 6, aggregate gradients: the one with the most votes The gradient-weighted average of the nth node will be used as the gradient weighted average of the nth node. The gradient of the aggregation wheel.

2. The distributed strategy gradient fault-tolerant mechanism method based on graph structure similarity analysis according to claim 1, characterized in that, The calculation method for the confidence distance described in step 2, part two: The confidence distance between all nodes is calculated based on their weights, as follows: in, For nodes With nodes Confidence distance between For nodes With nodes Gradient distance between them Let be the weight of node j in the previous iteration.

3. The distributed strategy gradient fault-tolerant mechanism method based on graph structure similarity analysis according to claim 1, characterized in that, The similarity calculation method for the node-to-graph structure described in step 3 is expressed as follows: in, For nodes With nodes Similarity between them This represents the number of confidence nodes.

4. The distributed strategy gradient fault-tolerant mechanism method based on graph structure similarity analysis according to claim 1, characterized in that, The process of aggregation using graph structure feature similarity includes: Voting mechanism: Each node votes for the node with the highest similarity to itself. Each node casts one vote, and each node casts a total of [number] votes. Tickets, nodes The number of votes received was , Update weights: The weight of each node is the sum of the product of the previous round's weight and the decay rate, plus the number of votes received in this round, as shown in the following expression: in, For the first wheel node The weight, The attenuation rate, For nodes The number of votes received, Aggregate gradient: This will aggregate the gradients of the most voted individuals. The set consists of nodes. The aggregate expression is as follows: in, For the first The gradient after round aggregation, For nodes In the Gradients uploaded in round-robin fashion For nodes No. The weight of the wheel.

Images