A gradient privacy leakage detection method for graph federated learning

Abstract

The application designs a gradient privacy leakage detection method for graph federated learning, and belongs to the field of federated learning data privacy protection. The detector with prior knowledge is allowed to restore the original graph data of the target data owner, including the topological structure and node attributes, through the shared gradient and model parameter information in the graph federated learning, effectively decomposes the topological structure and node attribute restoration task according to the gradient calculation result, realizes higher accuracy of the graph data restoration, and further detects the graph data privacy leakage risk of the target data owner.

Description

Technical Field

[0001] This invention belongs to the field of data privacy protection in federated learning, and relates to a method for detecting privacy leaks in federated learning gradients, specifically a method for detecting privacy leaks in federated learning gradients for graph data. Background Technology

[0002] Graph data structures can efficiently model complex relationships between entities and are widely used in fields such as biology, transportation, and medicine. However, in real-world scenarios, massive amounts of graph data are typically distributed among multiple data owners. Due to privacy concerns, it is difficult to combine multi-party graph data to improve the performance of graph machine learning models and their applications. To address this issue, graph federated learning technology has emerged. It extends the distributed machine learning paradigm of federated learning to the field of graph data processing, enabling the utilization of multi-party data by passing model gradients without directly accessing the original graph data.

[0003] Existing work shows that gradients transmitted in federated learning still pose a privacy risk; that is, the original training data can be reconstructed from the gradients shared in federated learning, leading to privacy leaks. This phenomenon is called deep gradient leakage. To detect the degree of gradient privacy leakage, it is necessary to design methods to reconstruct the original data from the gradients, and then assess the leakage by comparing the reconstructed data with the original data. Existing reconstruction methods are mainly divided into two categories: The first category is based on objective optimization, which uses prior knowledge to iteratively optimize the reconstructed data using gradient descent to minimize the distance between the generated gradient and the true gradient generated by the original data, thereby achieving the reconstruction of the original data. This type of method can achieve good reconstruction of image and text data when using homogeneous prior knowledge. The second category is based on gradient analysis, which constructs an equation relating the gradient to the original input data and reconstructs the original data by solving the closed-form solution of the equation. This type of method has more accurate reconstruction results, stronger interpretability, and higher reconstruction efficiency, but it is currently limited to linear layers (MLP) and convolutional neural networks (CNN).

[0004] Therefore, existing reconstruction methods are mainly designed for image and text data, and cannot effectively reconstruct graph data, making it difficult to effectively assess the gradient privacy leakage risk in graph federated learning. Unlike image and text data, graph data, in addition to node attributes, also possesses a unique topological structure. In common graph machine learning models, such as graph convolutional neural networks (GCNs), the topological structure and node attribute information are deeply coupled after graph convolution operations, making it difficult to effectively reconstruct both from the gradients simultaneously. This highlights that the main challenge lies in effectively decomposing the topology and attribute reconstruction tasks to achieve efficient reconstruction of graph data.

[0005] In summary, there is still a lack of methods for detecting privacy leaks in federated learning gradients for graph data. Because the topology and node attributes of graph data are deeply coupled in gradients, it is difficult to simultaneously reconstruct the original topology and node attributes, thus failing to effectively reveal the hidden leakage risks in graph federated learning gradients. Summary of the Invention

[0006] By analyzing and comparing existing technologies, this invention designs a gradient privacy leakage detection method for graph federation learning. It allows detectors with prior knowledge to reconstruct the original graph data (topology and node attributes) of the target data owner through shared gradients and model parameter information in graph federation learning. Based on the gradient calculation results, the task of reconstructing the topology and node attributes is effectively decomposed to achieve more accurate graph data reconstruction, thereby detecting the graph data privacy leakage risk of the target data owner.

[0007] The gradient privacy leakage detection method for graph federated learning is divided into the following steps:

[0008] Step 1: For the graph model in graph federation learning, calculate the gradients of its linear layers and graph convolutional networks respectively, and construct the relationship between the gradients and the original graph data.

[0009] The original graph data is represented as The relation, in which Represents the node attribute matrix. The adjacency matrix of the graph. This refers to data from the target graph dataset. The graph data, i.e., the target to be restored.

[0010] The graphical model consists of a Graph Convolutional Network (GCN) and a Linear Layer (MLP). The gradient calculation results of the linear layer are shown below:

[0011]

[0012] in , These are the gradients of the linear layer weights and the offset term, respectively. For the graph pooling operation matrix, for Node embeddings output by layer graph convolutional networks. The result of the loss function, This is the output of a linear layer. The actual meaning is graph embedding (denoted as ). This result establishes the relationship between the gradient of a linear layer and the graph embedding of its input.

[0013] In graph convolutional networks, for any i Layered graph convolutional network (total number of layers is) The gradient calculation results are as follows:

[0014] ,

[0015] in Representing the Gradients and parameters of layered graph convolutional networks Indicates the first The node embeddings output by the layer graph convolutional network (i.e., the inputs to the next layer). It is the normalized result of the original adjacency matrix. Representing the The layer outputs the derivative of the activation function ReLU. This is the introduced coefficient matrix. This gradient calculation result constructs a recursive relationship between the gradient of each layer of the graph convolutional network and its input node embeddings and adjacency matrices.

[0016] Step 2: Reconstruct the graph embedding based on the linear layer gradient, and input the graph embedding into the pre-trained topology decoder to output the reconstructed adjacency matrix. This enables the restoration of the topology.

[0017] The restored image is embedded as / ;

[0018] Topology decoder The model structure is a multilayer perceptron, which learns the prior knowledge held by the detector (i.e., additional graph dataset) through pre-training. This allows for the reconstruction of the original adjacency matrix from the graph embedding.

[0019] The loss function used in the pre-training of the topology decoder is:

[0020]

[0021] The first term is the summation of the adjacency matrix reconstruction loss for the additional graph data under the autoencoder training framework. It's an encoder. The second term is a regularization term for heterogeneous prior knowledge. For coefficients, This refers to the maximum mean difference between the graph embeddings of the additional graph dataset and the target graph dataset, with the aim of achieving alignment.

[0022]

[0023] in This is the kernel function.

[0024] Step 3: Based on the restored adjacency matrix, recursively work backward from the last layer of the graph convolutional network to the first layer to obtain the original output of the graph neural network.

[0025] Step 301, for the first A layer (i.e., the last layer) of a graph convolutional network, solving for its input. .

[0026] First, for the restored adjacency matrix Normalization is performed to obtain , yes The degree matrix.

[0027] Secondly, calculation ,in, These are all model parameters or gradients shared in graph federated learning. It is approximated by an all-one matrix.

[0028] Finally, solve the matrix equations. You can get .

[0029] at this time ;

[0030] Step 302, for the first Layered graph neural networks recursively solve for their inputs. .

[0031] First, calculate based on the recursive relation. ,in For shared parameters, Already solving the first... Calculate the layer time. It can be derived from the already calculated The calculation is obtained. Next, the matrix equation is solved. You can get .

[0032] Step 303: Determine the current layer number of the graph convolutional network. Is it equal to 1? If yes, it means the original input of the graph neural network has been solved. ,Will As a node attribute to be restored Otherwise, it means that we need to continue recursively solving for the input of the previous layer, so let Repeat step 302.

[0033] Step four: Calculate the similarity between the restored graph data and the original graph data, and use this as the result of privacy leakage detection.

[0034] For node attributes, MSE is used as the similarity measure:

[0035] MSE=

[0036] The larger the calculated MSE value, the greater the difference between the restored node attributes and the original values, the lower the restoration accuracy, and the smaller the degree of privacy leakage.

[0037] For the topological structure, the value at each position in the adjacency matrix is ​​treated as a binary classification, and AUC is used as the similarity measure:

[0038] AUC=

[0039] In the above formula , , representing the sets of positive and negative samples in a binary classification, respectively. and It is the corresponding position index in the adjacency matrix; , These are the first in the set. , One sample; This is an indicator function. The smaller the calculated AUC value, the greater the difference between the restored topology and the original value, the lower the restoration accuracy, and the smaller the degree of privacy leakage.

[0040] The advantages of this invention are:

[0041] (1) This invention constructs the relationship expression between the gradient of the federated graph model and the topology and node attributes of the original graph data through gradient calculation, and designs a restoration method based on the expression rules to restore the topology first and then restore the node attributes, effectively decomposing the task of restoring the complete graph data from the gradient, thereby improving the graph data restoration effect and more fully detecting the risk of privacy leakage.

[0042] (2) In the process of restoring the topology of the graph with the help of prior knowledge, the present invention designs a regularization term that aligns the heterogeneity between prior knowledge and target graph data, so that heterogeneous prior knowledge can be fully utilized in the restoration process, thereby improving the restoration accuracy and optimizing the detection effect.

[0043] (3) The present invention designs a recursive closed-form solution method to restore the graph node attributes. By accurately describing the relationship between the gradient of different layers and the graph data and its intermediate calculation results through relational formulas, the node attributes are obtained step by step, which improves the restoration accuracy and more accurately detects the privacy leakage of graph data. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the gradient privacy leakage detection method for graph federation learning proposed in this invention.

[0045] Figure 2 This is a comparison chart of experimental results on the image data restoration effect of a specific embodiment of the present invention. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to the accompanying drawings.

[0047] This invention relates to a gradient privacy leakage detection method for graph federated learning. It aims to enable a detection party with prior knowledge to reconstruct the graph data of the target data owner using shared gradients and model parameter information in graph federated learning, and then assess the privacy leakage risk based on the similarity between the reconstructed graph data and the original graph data. The workflow of this invention is as follows: Figure 1 As shown, the method includes four stages: gradient calculation, graph topology reconstruction, graph node attribute reconstruction, and privacy metric calculation, ultimately achieving the detection and assessment of gradient privacy leakage risks in graph federated learning scenarios. Specifically, it consists of the following steps:

[0048] Step 1: Calculate the gradient of the graph model in the graph federated learning, and then construct the model gradient. Compared with the original graph data The relation, in which Represents the node attribute matrix. The adjacency matrix of the graph. This refers to data from the target graph dataset. The graph data is the target to be restored. Since the graph model consists of a graph convolutional network (GCN) and a linear layer (MLP), the gradients of these two parts will be calculated separately, and the relationship between the gradients and the original graph data will be constructed.

[0049] In the linear layer, the gradient calculation results are shown below:

[0050]

[0051] in , These are the gradients of the linear layer weights and the offset term, respectively. For the graph pooling operation matrix, for Node embeddings output by layer graph convolutional networks. The result of the loss function, This is the output of a linear layer. It's worth noting that... The actual meaning is graph embedding (denoted as ). This result establishes a relationship between the linear layer gradient and the graph embedding of its input, demonstrating that the graph embedding can be accurately reconstructed from the linear layer gradient, and that subsequent reconstructions can fully utilize the graph embedding information.

[0052] In graph convolutional networks, for any i Layered network (total number of layers is) The gradient calculation results are as follows:

[0053] ,

[0054] in Representing the Gradients and parameters of layered graph convolutional networks Indicates the first The node embeddings output by the layer graph convolutional network (i.e., the inputs to the next layer). It is the normalized result of the original adjacency matrix. Representing the The layer outputs the derivative of the activation function ReLU. This is the introduced coefficient matrix. This gradient calculation result constructs a recursive relationship between the gradient of each layer of the graph convolutional network and its input node embeddings and adjacency matrices.

[0055] The recursive pattern in the above relational expression shows that once the adjacency matrix is ​​restored, Given that, node embedding The solution can be recursively applied layer by layer, starting from the last layer, and the final result is the original node attributes. Therefore, in subsequent steps, this invention will first restore the adjacency matrix and then restore the node attributes. Compared to the intuitive solution of directly restoring both topology and attributes through iterative optimization, this invention effectively decomposes the tasks of graph topology restoration and node attribute restoration based on the patterns obtained from gradient calculation, thereby improving the graph data restoration effect.

[0056] Step 2: Based on the relationship established in Step 1 for the linear layer, reconstruct the embedding graph by dividing the weight gradient of the linear layer by the gradient of the offset term. ,Right now / .

[0057] Step 3: Embed the image restored in Step 2 Input to the pre-trained topology decoder In the output, the reconstructed adjacency matrix is ​​obtained. This enables the restoration of the topology.

[0058] The decoder's model structure is a multilayer perceptron, which learns the prior knowledge held by the detector (i.e., additional graph datasets) through pre-training. This allows for the reconstruction of the original adjacency matrix from the graph embedding.

[0059] Specifically, the pre-training process employs an autoencoder training framework, inputting the topological structure and node attributes of the additional graph data into an encoder with the same structure as the federated graph model. The graph is encoded to obtain the corresponding graph embedding, which is then input into the decoder to reconstruct the adjacency matrix. The loss function used in pre-training is...

[0060]

[0061] The first term is the summation of the adjacency matrix reconstruction loss for the additional graph data. The second term is a regularization term for heterogeneous prior knowledge. The regularization term (where MMD is a coefficient) essentially aligns the graphs by minimizing the maximum mean discrepancy (MMD) between the graph embeddings of the additional graph dataset and the target graph dataset. This ensures that the decoder trained on the additional graph dataset still performs well in reconstructing the adjacency matrix on the target graph dataset. The specific formula for calculating this regularization term is as follows:

[0062]

[0063] The graph embeddings of the target graph dataset used above can be obtained through step 2. This is the kernel function.

[0064] Compared to existing methods that also rely on prior knowledge for reconstruction, this invention is the first to focus on the heterogeneity between prior knowledge and the target graph data. In the graph data reconstruction process addressed in this invention, the heterogeneity of prior knowledge will affect the reconstruction performance of the decoder. To fully utilize heterogeneous prior knowledge, this invention incorporates the aforementioned regularization term into the pre-trained loss function, enabling the decoder to maintain good reconstruction performance on the target graph dataset.

[0065] Step 4: For the first The layer (i.e., the last layer) graph convolutional network, its input is solved according to the formula obtained in step 2. .

[0066] First, the adjacency matrix restored in step 3... Normalization is performed to obtain , yes The degree matrix. Next, calculate... ,in, These are all model parameters or gradients shared in graph federated learning. Approximated by an all-one matrix. Finally, solve the matrix equation. You can get .make Then proceed with the next step of the assessment.

[0067] Step 5: Determine the current layer number of the graph convolutional network. Is it equal to 1? If so... An equal value of 1 indicates that the original input of the graph neural network has been solved. ,Will As a node attribute to be restored If not, proceed to step 7; otherwise, it means that we need to continue recursively solving the input of the previous layer, so let Skip to step 6.

[0068] Step 6: For the first Layered graph neural networks, whose inputs are solved according to the formula obtained in step 2. First, calculate according to the recursive relation. ,in For shared parameters, Already solving the first... Calculate the layer time. It can be derived from the already calculated The calculation is obtained. Next, the matrix equation is solved. You can get Finally, skip to step 5 to determine whether the solution has been completed.

[0069] Compared to existing methods that iteratively optimize and restore data, this invention uses an expression to accurately characterize the relationship between gradients and the original graph data, thereby improving the graph data restoration effect and more fully revealing the gradient privacy leakage risk of graph data.

[0070] Step 7: Calculate the similarity between the restored graph data and the original graph data, and use this as the result of privacy leakage detection.

[0071] For node attributes, MSE is used as the similarity measure, i.e., MSE = The larger the calculated MSE value, the greater the difference between the restored node attributes and the original values, the lower the restoration accuracy, and the smaller the degree of privacy leakage.

[0072] For the topological structure, the value of each position in the adjacency matrix is ​​considered as a binary classification, and AUC is used as the similarity measure, i.e., AUC = In the formula , , representing the sets of positive and negative samples in a binary classification, respectively. and It is the corresponding position index in the adjacency matrix; , These are the first in the set. , One sample; This is an indicator function. The smaller the calculated AUC value, the greater the difference between the restored topology and the original value, the lower the restoration accuracy, and the smaller the degree of privacy leakage.

[0073] This invention was validated on five chemical molecule / protein (figure) datasets. The reduction effect of one specific embodiment is shown below. Figure 2 As shown, compared to existing methods, this invention achieves higher accuracy in restoring the topology and node attributes of graph data, and the visualization results are closer to the original data, thus enabling more thorough detection of gradient privacy leakage risks in graph federated learning.

Claims

1. A gradient privacy leakage detection method for graph federated learning, characterized in that, The steps are as follows: Step 1: For the graph model in graph federation learning, calculate the gradients of its linear layers and graph convolutional networks respectively, and construct the relationship between the gradients and the original graph data. The original graph data is represented as The relation, in which Represents the node attribute matrix. The adjacency matrix of the graph. This refers to data from the target graph dataset. The graph data, i.e., the target to be restored; The graph model consists of a graph convolutional network and linear layers. The gradient calculation results of the linear layers are shown below: ； in , These are the gradients of the linear layer weights and the offset term, respectively. For the graph pooling operation matrix, for Node embeddings from the output of a layered graph convolutional network. The result of the loss function, Output for linear layers; The actual meaning is graph embedding, denoted as ; The total number of layers in the convolutional network is: For any number of The gradient calculation results for the layered graph convolutional network are as follows: , ； in Representing the Gradients and parameters of layered graph convolutional networks Indicates the first Node embeddings from the output of a layered graph convolutional network. It is the normalized result of the original adjacency matrix. Representing the The layer outputs the derivative of the activation function ReLU. It is the introduced coefficient matrix; Step 2: Reconstruct the graph embedding based on the linear layer gradient, and input the graph embedding into the pre-trained topology decoder to output the reconstructed adjacency matrix. To achieve the restoration of the topology; Step 3: Based on the restored adjacency matrix, recursively work backward from the last layer of the graph convolutional network to the first layer to obtain the original output of the graph neural network. Step 301, for the first Layered graph convolutional networks, solving for their input ; First, for the restored adjacency matrix Normalization is performed to obtain , yes The degree matrix; Secondly, calculation ,in, These are all model parameters or gradients shared in graph federated learning. Approximated by an all-one matrix; Finally, solve the matrix equations. You can get ; at this time ; Step 302, for the first Layered graph neural networks recursively solve for their inputs. ; First, calculate based on the recursive relation. ,in For shared parameters, Already solving the first... Calculate the layer time. It can be derived from the already calculated The calculations were performed; secondly, the matrix equations were solved. You can get ; Step 303: Determine the current layer number of the graph convolutional network. Is it equal to 1? If yes, it means the original input of the graph neural network has been solved. ,Will As a node attribute to be restored Otherwise, it means that we need to continue recursively solving for the input of the previous layer, so let Repeat step 302; Step 4: Calculate the similarity between the restored graph data and the original graph data, and use this as the privacy leak detection result; For node attributes, MSE is used as the similarity measure; for topology, the value of each position in the adjacency matrix is ​​treated as a binary classification, and AUC is used as the similarity measure.

2. The gradient privacy leakage detection method for graph federated learning according to claim 1, characterized in that, The restored image is embedded as / .

3. The gradient privacy leakage detection method for graph federated learning according to claim 1, characterized in that, Topology decoder The model structure is a multilayer perceptron, which learns the prior knowledge held by the detector through pre-training, i.e., additional graph datasets. This allows for the reconstruction of the original adjacency matrix from the graph embedding.

4. The gradient privacy leakage detection method for graph federated learning according to claim 1, characterized in that, The loss function used in the pre-training of the topology decoder is: ； The first term is the summation of the adjacency matrix reconstruction loss for the additional graph data under the autoencoder training framework. It is the encoder; the second term is the regularization term for heterogeneous prior knowledge. For coefficients, This refers to the maximum mean difference between the graph embeddings of the additional graph dataset and the target graph dataset, with the aim of achieving alignment. ； in This is the kernel function.

5. The gradient privacy leakage detection method for graph federated learning according to claim 1, characterized in that, For node attributes, MSE is used as the similarity measure: MSE= ; The larger the calculated MSE value, the greater the difference between the restored node attributes and the original values.

6. The gradient privacy leakage detection method for graph federated learning according to claim 1, characterized in that, For the topological structure, the value at each position in the adjacency matrix is ​​treated as a binary classification, and AUC is used as the similarity measure: AUC= ; In the above formula , , representing the sets of positive and negative samples in a binary classification, respectively. and It is the corresponding position index in the adjacency matrix; , These are the first in the set. , One sample; For indicator functions; The smaller the calculated AUC value, the greater the difference between the restored topology and the original value.

Images