Differential privacy satisfying longitudinal federated graph learning method and device

Abstract

The application provides a longitudinal federated graph learning method and device satisfying differential privacy, which comprises the following steps: sending a learning request to a service party and other parties participating in longitudinal federated graph learning except a local party, obtaining graph structure coding information based on target conjecture graph structure information and an updated community set to generate disturbed graph structure coding information satisfying differential privacy; obtaining target shared graph structure information satisfying differential privacy based on the disturbed graph structure coding information and initial shared graph structure information to share with other parties; receiving target shared graph structure information from other parties, determining global graph structure information based on the target shared graph structure information and target conjecture graph structure information to reconstruct a global graph structure; and performing distributed training on a preset global graph model based on the global graph structure and local partial graph data by sending intermediate results to the service party to obtain a trained global graph model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of longitudinal federated graph learning, and particularly relates to a longitudinal federated graph learning method and device meeting differential privacy. BACKGROUND

[0002] Graph neural networks can combine node attribute information and high-order neighborhood information to generate node embeddings with rich representation capabilities, achieving the best performance in graph learning-based tasks such as node classification and edge classification, and thus attracting widespread attention from academia and industry. In the process of computation, graph neural networks use neighborhood information of nodes as guidance to aggregate attribute and feature information of adjacent nodes and generate node embeddings containing both attribute information and structural information. Therefore, the model accuracy of graph neural networks depends on the complete attribute information and structural information provided by the graph data holder.

[0003] However, in real-world scenarios, due to data regulations and commercial competition, global graph data may be scattered in multiple institutions in the form of subgraphs or local graphs. Each institution can collect graph data information from users who trust the institution and construct local subgraphs, but institutions that do not trust each other cannot directly share these subgraphs for training graph neural networks. Longitudinal federated graph learning is a typical method for graph learning problems in the context of collaborative multi-party graph data. Longitudinal federated graph learning assumes that each participant is node-aligned, that is, each participant has all the nodes in the global graph, but the edges and node attributes in the global graph are divided into subgraphs held by different participants. In the longitudinal federated graph learning method, the computation process of the graph neural network needs to aggregate the features of each node based on the complete global graph structure to calculate the node embedding, but any participant only holds an incomplete subgraph of the global graph structure and cannot directly share the subgraph data held by each participant with other participants, so each participant must collaborate to aggregate features while meeting privacy protection.

[0004] Existing longitudinal federated graph learning methods that meet differential privacy generally use secure multi-party computation technology to encrypt subgraph structures and perform feature aggregation on the complete neighborhood of nodes in the ciphertext state to ensure that no additional information is revealed during the feature aggregation process. However, secure computation in ciphertext has a huge computational overhead, resulting in low computational efficiency of existing methods. SUMMARY

[0005] In view of this, the embodiments of the present application provide a longitudinal federated graph learning method and device meeting differential privacy to eliminate or improve one or more defects in the prior art.

[0006] One aspect of the present application provides a longitudinal federated graph learning method meeting differential privacy, which comprises the following steps:

[0007] sending a longitudinal federated graph learning request to the service party and each of the other parties participating in the longitudinal federated graph learning except the local party, obtaining graph structure encoding information based on target speculative graph structure information and an updated community set, to generate perturbed graph structure encoding information satisfying differential privacy, wherein the target speculative graph structure information is obtained by using a local graph model to iteratively optimize speculative graph structure information initialized based on local graph data, and the updated community set is obtained by updating an initial community set obtained by performing community division on the local graph data based on the target speculative graph structure information;

[0008] obtaining target shared graph structure information satisfying differential privacy based on the perturbed graph structure encoding information and initial shared graph structure information, to share with the other parties, wherein the perturbed initial shared graph structure information is initial shared graph structure information satisfying differential privacy generated based on initial shared graph structure information obtained by merging the target speculative graph structure information and the local graph data;

[0009] receiving each target shared graph structure information from the other parties, determining global graph structure information based on the each target shared graph structure information and the target speculative graph structure information to reconstruct the global graph structure;

[0010] performing distributed training on a preset global graph model based on the global graph structure and the local graph data by sending intermediate results to the service party to obtain a trained global graph model, wherein the global graph model comprises a local feature extraction module, a global feature aggregation module located at the service party and a local classification module connected in sequence, and the intermediate results comprise local features extracted in each iteration and updated gradients, and in the first iteration, the updated gradients are initial gradients.

[0011] In some embodiments of the present application, the local graph data comprises a node set, an adjacency matrix representing an edge set, a node attribute set and a node class label set representing a real node class.

[0012] In some embodiments of the present application, the obtaining target shared graph structure information satisfying differential privacy based on the perturbed graph structure encoding information and the initial shared graph structure information comprises:

[0013] estimating an edge existence probability matrix based on the perturbed graph structure encoding information as a prior probability;

[0014] performing Bayesian estimation on the target speculative graph structure information based on the prior probability and the perturbed initial shared graph structure information as evidence to obtain an edge posterior existence probability matrix satisfying differential privacy as the target shared graph structure information.

[0015] In some embodiments of the present application, the edge existence probability matrix comprises an updated intra-community edge existence probability matrix obtained based on the disturbed neighbor number sequence in the disturbed graph structure encoding information and an updated inter-community edge existence probability matrix obtained based on the disturbed connection number matrix in the disturbed graph structure encoding information.

[0016] In some embodiments of the present application, the graph structure encoding information comprises a neighbor number sequence and a connection number matrix, the neighbor number sequence comprising a number of neighbor nodes of each node in the local partial graph data in each respective update community to which the node belongs in the update community set determined based on the target speculative graph structure information, and the connection number matrix comprising a number of connections between each two update communities in the update community set determined based on the target speculative graph structure information.

[0017] In some embodiments of the present application, the determination of the global graph structure information based on the respective target shared graph structure information and the target speculative graph structure information comprises:

[0018] For an edge between respective nodes in the respective target shared graph structure information of each participant involved in the vertical federated graph learning, if the sum of the edge posterior existence probabilities of the corresponding edge in the respective target shared graph structure information shared by other participants is greater than or equal to 1, or the value of the corresponding edge in the target speculative graph structure information is 1, the existence probability of the edge is determined to be 1, otherwise the existence probability of the edge is the sum of the edge posterior existence probabilities of the corresponding edge in the respective target shared graph structure information shared by other participants.

[0019] The global edge existence probability matrix is obtained based on the existence probability of each edge and is used as the global graph structure information.

[0020] The global graph structure information is hard-sampled according to a sampling threshold to reconstruct the global graph structure.

[0021] In some embodiments of the present application, the distributed training of the preset global graph model based on the global graph structure and the local partial graph data by sending intermediate results to the service party to obtain a trained global graph model comprises:

[0022] The neighborhood of each node in the global graph structure is obtained based on the global graph structure by the local feature extraction module, and feature aggregation is performed based on the neighborhood of each node and the node attribute set in the local partial graph data to extract local features.

[0023] The extracted local features are sent to the service party, so that the service party can receive the respective local features from each participant involved in the vertical federated graph learning, perform feature aggregation based on the respective local features by the global feature aggregation module to obtain global features, and send the global features to each participant.

[0024] receiving global features from the service party, performing classification based on the global features by the local classification module to output a predicted node category, and sending an updated gradient in each iteration to the service party to enable the service party to receive each updated gradient from each participant and aggregate the updated gradients to obtain an aggregated gradient until a loss between the predicted node category and a set of node category labels in the local partial graph data is minimized to obtain a trained global graph model, wherein the updated gradient is obtained by updating based on the aggregated gradient obtained in the last iteration through back propagation.

[0025] In some embodiments of the present application, the intermediate result is a perturbed intermediate result generated by adding noise perturbation satisfying differential privacy to the intermediate result.

[0026] In some embodiments of the present application, the number of node categories in the initial community in the initial community set and the number of node categories in the node set in the local partial graph data are the same; the number of neighbor nodes of each node in each updated community in the updated community set is the maximum, and the number of neighbor nodes is calculated by target speculation graph structure information; and the updated community set is a perturbed updated community set generated by adding noise perturbation satisfying differential privacy to each updated community in the updated community set.

[0027] Another aspect of the present application provides a differential privacy-compliant longitudinal federated graph learning device, which comprises a computer device including a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, so that the device implements the steps of the foregoing method.

[0028] Another aspect of the present application provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the steps of the foregoing method.

[0029] Another aspect of the present application provides a computer program product comprising computer instructions, which are executed by a processor to implement the steps of the foregoing method.

[0030] The differential privacy-compliant longitudinal federated graph learning method and device of the present application can be used to enable each participant in the longitudinal federated graph learning to cooperatively train a graph neural network model while satisfying differential privacy, and to share graph structure information critical to the utility of the graph neural network model for feature aggregation during the longitudinal federated graph learning process, thereby achieving necessary feature aggregation calculation of the graph while avoiding the introduction of computationally intensive secure multi-party computation technology, and achieving high computational efficiency while satisfying differential privacy constraints and maintaining high model accuracy, thereby greatly reducing computational overhead.

[0031] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following detailed description and drawings in which

[0032] Those skilled in the art will appreciate that the objects and advantages of the application can be implemented in a variety of ways, and that the application can be implemented using other structures besides those specifically described herein. The present application is thus not to be construed as limited to the specific embodiments set forth herein, but rather only by the claims that follow. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate the present application and, together with the description given below, serve to explain the principles of the application.

[0034] Figure 1 Flowchart of a longitudinal federated graph learning method satisfying differential privacy in an embodiment of the present application;

[0035] Figure 2 Process diagram for obtaining target conjecture graph structure information in step S110 in an embodiment of the present application;

[0036] Figure 3 Process diagram for community division and obtaining graph structure encoding information in step S110 in an embodiment of the present application;

[0037] Figure 4 Comparison diagram of model precision effects of the present method and prior art in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the embodiments and drawings. Herein, the illustrative embodiments of the present application and the descriptions thereof are used to explain the present application, but are not intended to limit the present application.

[0039] It should be noted that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0040] It should be emphasized that the terms "comprises / comprising" when used in this specification are taken to specify the presence of stated features, elements, steps or components, but do not preclude the presence or addition of one or more other features, elements, steps, components, or groups thereof.

[0041] It is also noted herein that, if it is not otherwise indicated, the term "connection" can refer not only to a direct connection, but also to an indirect connection via an intermediate object.

[0042] Embodiments of the present application will be described below with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0043] In order to solve the problems of large computational overhead and low training efficiency of the existing longitudinal federated graph learning method based on secure multi-party computing technology to meet differential privacy, the embodiments of the present application propose a longitudinal federated graph learning method and device that meets differential privacy, which can be used for each participant in longitudinal federated graph learning to cooperatively train a graph neural network model that meets differential privacy. Not only does it meet the differential privacy constraint, but it also greatly improves the computational efficiency while maintaining high model accuracy without introducing computationally intensive secure multi-party computing technology.

[0044] Figure 1 The flowchart of the longitudinal federated graph learning method that meets differential privacy in an embodiment of the present application is shown in FIG. Figure 1 As shown in the figure, the method is performed by any of the participants in longitudinal federated graph learning, and includes the following steps: step S110, sending a longitudinal federated graph learning request to the service party and each of the other participants in longitudinal federated graph learning except the local one, obtaining graph structure encoding information based on target speculative graph structure information and an updated community set to generate perturbed graph structure encoding information that meets differential privacy, wherein the target speculative graph structure information is obtained by using the local local graph model to iteratively optimize the speculative graph structure information initialized based on the local local graph data, and the updated community set is obtained by updating the initial community set obtained by community partitioning the local local graph data based on the target speculative graph structure information;

[0045] Step S120, obtaining target shared graph structure information that meets differential privacy based on the perturbed graph structure encoding information and the initial shared graph structure information, and sharing it with the other participants, wherein the perturbed initial shared graph structure information is initial shared graph structure information that meets differential privacy generated based on the initial shared graph structure information obtained by merging the target speculative graph structure information and the local local graph data;

[0046] Step S130, receiving each target shared graph structure information from the other participants, determining global graph structure information based on the target shared graph structure information and the target speculative graph structure information to reconstruct the global graph structure;

[0047] In step S140, the preset global graph model is distributed trained based on the global graph structure and the local partial graph data by sending the intermediate result to the service party to obtain a trained global graph model, wherein the global graph model comprises a local feature extraction module, a global feature aggregation module located at the service party and a local classification module which are sequentially communicatively connected, and the intermediate result comprises the local feature extracted in each iteration and the updated gradient, and in the first iteration, the updated gradient is the initial gradient.

[0048] The graph learning involved in the method mainly refers to the training of a graph neural network model, and the method is the most advanced graph learning method and has achieved the best performance in many graph learning-based tasks such as graph data mining, recommendation system and biological information analysis. The basic principle of graph learning is to aggregate the node attributes (for example, the node attributes can be the gender and age of a user) or node embeddings of adjacent nodes according to the graph structure, and to map the aggregated embeddings through a parameterized nonlinear transformation, so as to learn the feature representation of all nodes in the graph. Taking the node classification task as an example, which is a baseline task of the graph neural network, given a simple undirected graph, each layer of the graph neural network aggregates the node embeddings of the previous layer according to the adjacent matrix representing the graph structure, which describes the adjacent node set of a node, through a differentiable and permutation-invariant aggregation function (for example, a summation or average function), and performs a differentiable nonlinear transformation to obtain the node embedding of the node at the layer. In the input layer of the network, the node embedding is initialized as the node attribute itself. In the last layer of the network, the model outputs a feature vector for performing a downstream task, that is, the node embedding of the graph. In the training process of the graph neural network model for the node classification task, the node embedding is compared with the node category label through a softmax layer, and the model parameters are updated through a back propagation algorithm by using a predefined loss function (for example, a cross-entropy cost function), so as to complete the training of the graph neural network model.

[0049] In step S110, a vertical federated graph learning request is first sent by any participant to the service party and other participants, so that the service party and other participants and the requesting participant jointly participate in the vertical federated graph learning. Wherein, the participant can be a client, and the service party can be a server, and the server is a semi-honest server.

[0050] The local partial graph model can be obtained by pre-training based on the local partial graph data held. The local partial graph data held by the u-th participant can be G u = (V u ,A u ,X u ,Y u ), wherein V u represents a node set, A u is an edge set Eu the adjacency matrix of G, i.e., if edge (i, j) ∈ E u then On the contrary i represents a node, j represents a neighbor node of node i; X u represents a node attribute set, Y u represents a node category label set, the node category label is a real node category, and the node category may be, for example, an active user or an inactive user, a user gender, or the like. The participant obtains a local graph neural network model (referred to as a local graph model for short) f u based on local graph data G gnnl , and uses the pre-trained local graph model to learn and optimize the inferred graph structure information in a reverse learning manner, so that the optimized inferred graph structure information has similar utility with the adjacency matrix in feature aggregation on the local graph, that is, the useful information on the graph is retained while privacy leakage is avoided, thereby ensuring the utility of the shared information in longitudinal federated graph learning. In order to effectively learn the inferred graph structure information, the participant can use a two-layer GCN (Graph Convolutional Network) model as the preset local graph model, which can be represented as:

[0051]

[0052]

[0053] wherein Z u represents a node embedding set, ReLU represents a nonlinear activation function, represents a first predicted node category set, σ represents a Sigmod function, and respectively represent learnable parameters of a first layer and a second layer of the graph convolutional network, represents a cost function, and specifically, a cross-entropy cost function CE can be used to evaluate the difference between the first predicted node category set and the node category label set. The local graph neural network is pre-trained by minimizing the cost function, and the trained parameters and are obtained, that is, the pre-trained local graph model is obtained. The pre-trained local graph model is the above-mentioned pre-trained local graph model.

[0054] Then, the inferred graph structure information is optimized through the local graph model, the cost function in the training process of the local graph model, and the gradient descent method, so as to extract the key graph structure that most significantly affects the utility of the graph neural network. For example, Figure 2As shown, each element in the speculative graph structure matrix representing the speculative graph structure information is first initialized to 0, i.e., obtaining initialized speculative graph structure information, and then the local graph model is adopted to obtain the target speculative graph structure matrix based on the initialized speculative graph structure matrix Output the second predicted node class set which can be represented as:

[0055]

[0056] After obtaining the second predicted node class set from the speculative graph structure matrix, the target speculative graph structure matrix is finally obtained by iteratively optimizing the following backward learning cost function which can be represented as:

[0057]

[0058] wherein p represents a weight parameter of the cost function , represents a sparse regular term weight parameter, represents the Frobenius norm (i.e., F-norm, used to measure the overall "size" or "magnitude" of a matrix) of the speculative graph structure matrix . The cost function of the backward learning forces the second predicted node class set from the speculative graph structure matrix to be consistent with the first predicted node class set from the adjacency matrix A u of the real local graph and the real node class label set Y u , so that the speculative graph structure matrix can have good data utility on the GNN learning task and evaluate which edges in the real local graph structure have a greater importance to the accurate prediction of the node class. Moreover, the sparsity of the speculative graph structure matrix is controlled by penalizing the Frobenius norm thereof. In order to control the range of the elements in the speculative graph structure matrix , the elements a in the speculative graph structure matrix are projected to the value range (0, 1) after each gradient update by the participant, i.e., which can be represented as:

[0059]

[0060] wherein M represents a projection function, t represents the number of training steps or iterations, ζ represents the learning rate of the backward learning, and g represents the backward learning gradient.

[0061] After the backward learning is completed, the participant obtains the local graph model f gnn1 with the target speculative graph structure matrix ​​​and the adjacency matrix A of the local partial graph u and the target speculation graph structure matrix After merging, an initial shared graph structure matrix is obtained, which is used as initial shared graph structure information and can be expressed as:

[0062]

[0063] The initial shared graph structure information contains local partial graph privacy information from the participants, and if directly shared with other participants, it may cause privacy information leakage risk, so it needs to be shared under the constraint of differential privacy. To ensure that the shared graph structure information strictly satisfies edge differential privacy, the participants first perform binary processing on the initial shared graph structure information , which can be expressed as:

[0064]

[0065] This means that if the element value representing the importance of the edge in is higher than the sampling threshold, the edge is selected as part of the initial shared graph structure. The sampling threshold can be 0.5. Then, the participants add random noise disturbance to the initial shared graph structure information using a random response mechanism function with a privacy budget of ∈ r , and the calculation method can be:

[0066]

[0067] Based on this, the initial shared graph structure information after disturbance satisfies ∈ r -edge differential privacy.

[0068] To extract the graph structure features in the local partial graph data, i.e., to obtain the graph structure encoding information, the participants first perform initial community division according to the target speculation graph structure information, and then further adjust the initial community division result to obtain the final community division result, such as Figure 3 . Specifically, the multi-layer perception (MLP) can be pre-trained according to the node attribute set and node category label set (X u , Y u ) that are irrelevant to the graph structure information, and the community division network is obtained. The node set V of the local partial graph is divided into multiple initial communities using the community division network, and the number of initial communities is the same as the number of node categories in the node set, i.e., the initial community set denotes the node set divided into the corresponding initial community, and c denotes the number of initial communities or the number of node categories. Here, other community division manners can also be used for initial community division, and are not limited to the above manners. Since the community division network itself can have misclassification, leading to nodes being divided into wrong communities, it is necessary to adjust the community to which each node belongs, so as to more accurately calculate the graph structure encoding information. For each node in each initial community in the initial community set, the target speculation graph structure matrix The number of neighbor nodes of the node in each initial community is calculated, the initial community with the largest number of neighbor nodes containing the node is taken as a candidate community, and the node is added to the candidate community, to obtain an updated community (the communities to which the node belongs before and after the node is added are both updated), thereby obtaining an updated community set obtained after the community of each node is iteratively updated

[0069] Since the community adjustment process depends on the target speculation graph structure matrix To meet the edge differential privacy constraint, an exponential mechanism with a privacy budget of ∈ e is used to add noise disturbance to the result after each iterative update in the community adjustment process, wherein the number of neighbor nodes of the node in different communities is taken as the exponential mechanism. Therefore, in this process, the final updated community set obtained can also be a disturbed updated community set satisfying differential privacy, wherein denotes the cth disturbed updated community. Specifically, since the number of neighbor nodes of the nodes contained in the community will change to at most 1 after adding or deleting an edge in the target speculation graph structure matrix , the sensitivity ΔEM of the exponential mechanism is 1. Since adding or deleting an edge will affect the two end nodes of the edge in the community adjustment process, the privacy budget used in each iterative update is set to

[0070] The participant obtains the final community division result and the target speculation graph structure information are used together to calculate the coarse-grained graph structure encoding information, and noise disturbance is added to the graph structure encoding information, so that the disturbed graph structure encoding information satisfies differential privacy. Based on the final community division result, the edges in the target speculation graph structure matrix can be divided into community internal edges and community inter-edge. The final community division result can be an updated community set without adding noise disturbance, or an updated community set to which noise disturbance satisfying differential privacy is added. In this embodiment, as Figure 3As shown, the graph structure encoding information includes a neighbor number sequence and a connection number matrix, where the neighbor number sequence includes the number of neighbor nodes of each node in the local partial graph data in the respective update community to which each node belongs in the update community set determined based on the target speculative graph structure information, and the connection number matrix includes the number of connections between each two update community in the update community set determined based on the target speculative graph structure information. For each node i in the local partial graph, the number of neighbor nodes of node i in the update community to which node i belongs in the update community set may be expressed as:

[0071]

[0072] Then, the neighbor number sequence composed of the number of neighbor nodes of all nodes is n represents the number of nodes. For any pair of update communities and in the update community set , the number of connections between the two communities can be expressed as:

[0073]

[0074] where 1≤k<m≤c. Then, the connection number matrix composed of the number of connections of all update community pairs is ε u ∈R c×c .

[0075] Then, the participants add noise perturbation to the neighbor number sequence and the connection number matrix using the Laplace mechanism function with a privacy budget of ∈1, so that the perturbed neighbor number sequence and the connection number matrix satisfy ∈1-c edge difference privacy. Since the change of an edge in will cause the number of neighbor nodes of two nodes to change, the sensitivity of the neighbor number sequence to the Laplace mechanism is 2, and the perturbed neighbor node number can be expressed as:

[0076]

[0077] On the contrary, the change of an edge in will at most cause the number of connections between the corresponding two update communities to change to 1, so the sensitivity of the connection number matrix to the Laplace mechanism is 1, and the perturbed connection number can be expressed as:

[0078]

[0079] Since the neighbor number sequence and the connection number matrix ε u each count the target speculative graph structure matrix ​intersections, thus the two kinds of graph structure encoding information can use the same privacy budget ∈1. And, considering that the number of neighbor nodes and the number of connections cannot be negative, the disturbed neighbor number sequence and the disturbed connection number matrix are clipped to non-negative values.

[0080] In some embodiments, in step S120, the target shared graph structure information satisfying differential privacy is obtained based on the disturbed graph structure encoding information and the initial shared graph structure information, including the following steps:

[0081] The edge existence probability matrix is estimated based on the disturbed graph structure encoding information and taken as the prior probability;

[0082] The disturbed initial shared graph structure information is taken as the evidence, and the Bayesian estimation is performed on the target inferred graph structure information based on the prior probability and the evidence, to obtain the edge posterior existence probability matrix satisfying differential privacy and taken as the target shared graph structure information.

[0083] In step S120, the participant calculates the edge existence probability matrix as the prior probability of Bayesian estimation according to the disturbed coarse-grained graph structure encoding information, and performs Bayesian estimation with the disturbed fine-grained initial shared graph structure information as the evidence, so as to calculate the edge posterior existence probability matrix of the target inferred graph structure matrix as the post-processing of noise reduction, and share the edge posterior existence probability matrix as the target shared graph structure information to each participant participating in the federated graph learning.

[0084] Specifically, the edge existence probability matrix can be estimated by the Chung-Lu model according to the disturbed neighbor number sequence and the connection number matrix. The estimated edge existence probability matrix includes two parts, which are the updated intra-community edge existence probability matrix and the updated inter-community edge existence probability matrix. First, the probability p of the existence of an edge between each node i and each node j in the same update community is calculated ij , and the calculation method can be represented as:

[0085]

[0086] wherein, and respectively represent the disturbed neighbor node number of node i and node j in the update community to which they belong, and the sum of the disturbed neighbor node numbers of each node s in the update community. Thus, the intra-community edge existence probability matrix includes the intra-community edge existence probability of each update community in the update community set. Then, the edge existence probability matrix is estimated based on the disturbed connection number matrix Calculate the probability of edges existing between nodes in different update communities. For update communities... Each node i within and the update community The probability that there is an edge between each node j in the array. The calculation method can be expressed as:

[0087]

[0088] in, and Let represent the number of nodes in the k-th and m-th update communities, respectively. Therefore, the probability matrix of edge existence between update communities includes the probability of edge existence between each update community in the update community set.

[0089] For the perturbated initial shared graph structure matrix Each pair of elements Due to the privacy budget of the random response mechanism ∈ r The probability of flipping each edge is known, therefore... It is known. The participants can determine the flip probability p. r Calculate the observed initial shared graph structure matrix after perturbation, assuming an edge exists between nodes i and j and assuming no edge exists between nodes i and j. Each pair of elements The likelihood probability of whether a connection exists can be calculated as follows:

[0090]

[0091] Among them, l ij Indicated by When there is an edge between nodes i and j in the array, the perturbation result is observed. Likelihood probability; l′ ij Indicated by When there is no edge between nodes i and j in the array, the perturbation results are observed. The likelihood probability. Since the edge exists with probability p. ij , and likelihood probability l ij 、l′ ij It is the target inference graph structure matrix Based on independent observations, participants can estimate the posterior probability of edge existence using Bayesian rules. The calculation formula can be:

[0092]

[0093] Where 1≤i≠j≤|V|, and |V| represents the number of nodes in the node set V; Indicates that evidence was observed Afterwards, the posterior probability of the existence of an edge in the target shared graph structure matrix for the node pair (i, j); Pr denotes probability.

[0094] Since the Bayesian estimation is only based on the perturbed initial shared graph structure matrix The perturbed neighbor number sequence And the perturbed connection number matrix Three kinds of information satisfying the edge differential privacy constraint are calculated, so the Bayesian estimation can be regarded as a post-processing process of the differential privacy algorithm, and the edge posterior existence probability Still satisfies the differential privacy constraint. The edge posterior existence probability matrix Composed of each edge posterior existence probability As the final target shared graph structure information to be shared.

[0095] In some embodiments, step S130, determining the global graph structure information based on the target shared graph structure information and the target speculative graph structure information to reconstruct the global graph structure, includes the following steps:

[0096] For each edge between each node in the target shared graph structure information of each participant participating in the longitudinal federated graph learning, if the sum of the edge posterior existence probabilities of the corresponding edge in the target shared graph structure information shared by other participants is greater than or equal to 1, or the value of the corresponding edge in the target speculative graph structure information is 1, then the existence probability of the edge is determined to be 1, otherwise the existence probability of the edge is the sum of the edge posterior existence probabilities of the corresponding edge in the target shared graph structure information shared by other participants.

[0097] Based on the existence probability of each edge, a global edge existence probability matrix is obtained as the global graph structure information;

[0098] Hard sampling is performed on the global graph structure information according to the sampling threshold to reconstruct the global graph structure.

[0099] In step S130, the participant u receives the edge posterior existence probability matrix shared by other participants participating in the longitudinal federated graph learning except the local one and obtains Afterwards, U denotes the number of participants participating in the learning, and the edge posterior existence probability matrices from other participants are combined to calculate the global edge existence probability matrix That is, the global graph structure information is obtained, so that the global graph structure is reconstructed according to the global graph structure information. In this embodiment, since each participant holds a non-overlapping subgraph structure or local graph structure, The existence probability of each edge in the target shared graph structure information is the accumulation of the edge posterior existence probabilities of the corresponding edge shared by other participants, and the calculation method can be:

[0100]

[0101] wherein, when the cumulative sum of the edge posterior existence probability between nodes i and j shared by other participants is higher than 1, or there is an edge between nodes i and j in the target speculation graph structure matrix of participant u locally itself, is limited to 1, that is, the edge must exist. In other embodiments, each participant can also hold an overlapping subgraph structure, which does not constitute a limitation to the present application.

[0102] Then, the global edge existence probability matrix representing the global graph structure information is hard-sampled according to a sampling threshold to reconstruct the global graph structure which can be represented as:

[0103]

[0104] wherein, the sampling threshold can be 0.5. After obtaining the reconstructed global graph structure , the participants can perform distributed training and learning of the global graph model (i.e., the global graph neural network model) according to .

[0105] In some embodiments, the step S140 of performing distributed training on the preset global graph model based on the global graph structure and the local subgraph data by sending intermediate results to the service party to obtain the trained global graph model comprises the following steps: obtaining the neighborhood of each node in the global graph structure based on the global graph structure by the local feature extraction module, and performing feature aggregation based on the neighborhood of each node and the node attribute set in the local subgraph data to extract local features.

[0106] sending the extracted local features to the service party so that the service party can receive each local feature from each participant participating in the vertical federated graph learning, performing feature aggregation based on each local feature by the global feature aggregation module to obtain global features, and sending the global features to each participant;

[0107] receiving the global features from the service party, performing classification based on the global features by the local classification module to output predicted node categories, and sending the updated gradient in each iteration to the service party so that the service party receives each updated gradient from each participant and aggregates each updated gradient to obtain an aggregated gradient until the loss between the predicted node categories and the node category label set in the local subgraph data is minimized to obtain the trained global graph model, wherein the updated gradient is obtained by updating the aggregated gradient obtained in the last iteration by backpropagation.

[0108] ​In the model distributed iterative training process of step S140, the global graph model needs to perform global feature extraction first, and then perform classification according to the extracted global features, which is a semi-supervised learning process. In the feature extraction process, for each node in the global graph structure, the global graph neural network model needs to aggregate the complete high-order neighborhood structure information of the node and the node attributes or features of all neighbor nodes in the global graph structure to generate node embedding with rich feature representation capability, so as to obtain the node embedding of all nodes in the global graph structure. To achieve the above purpose, the method designs two processes of local feature extraction (aggregation) and global feature aggregation of the service party. Specifically, in the local feature aggregation process, for each node in the global graph structure, the participant first aggregates the complete high-order neighborhood structure information of the node and the part of the node attributes (i.e. the set of node attributes in the local local graph data) held by the participant (local) in the global graph structure to generate the local embedding of the node, so as to obtain the local embedding set composed of the local embedding of all nodes, and realize local feature extraction. Then, the participant sends the local embedding set to the server. After the server receives each local embedding set from each participant participating in learning, the global feature aggregation is performed to obtain the global embedding set, so as to realize the global feature aggregation at the service party. That is, the global graph model includes the local feature extraction module located at the participant, the global feature aggregation module located at the service party and the local classification module located at the participant which are connected in sequence.

[0109] The local feature aggregation process is similar to the feature aggregation process of the commonly used GNN (Graph Neural Network, graph neural network). The participant aggregates the neighborhood of each node in the reconstructed global graph structure to obtain the neighborhood of each node in the reconstructed global graph structure to obtain the neighborhood of each node in the reconstructed global graph structure

[0110]

[0111] wherein, represents the neighborhood embedding of node i at the hth layer, Agg represents the aggregation function, which is used to sample all neighbor nodes j of node i from the neighborhood N(i) of node i and aggregate the node embedding of these neighbor nodes at the (h-1)th layer to generate neighborhood embedding; represents the node embedding of node i at the hth layer, when h=0, the node embedding of node i is the node attribute held by the participant, that is S denotes a nonlinear activation function, W h denotes the learnable model parameters of the h-th layer, and Concat denotes a concatenation function. There are three types of aggregation functions in GraphSAGE, which are average function, LSTM and pooling function. In order to solve the computational efficiency bottleneck in vertical federated graph learning, the average function is used as the aggregation function in the embodiment.

[0112] According to the above formula, the neighborhood embedding of node i in the h-th layer of GraphSAGE is and the node embedding of node i in the h-1-th layer is concatenated, and the node embedding of node i in the h-th layer is obtained through the linear mapping layer in the local feature extraction module The node embedding aggregates the neighborhood structure information in the h-hop neighborhood of node i and the attribute information of the neighbor nodes held by the node. Therefore, the node embedding of node i in the h-th layer is that is, the local embedding of the node. Finally, the local embedding set (i.e., the local feature) of each node in the global graph structure is constructed.

[0113] In the distributed iterative training process, the participant needs to send the local embedding set obtained by the local feature extraction module in each iteration to the server for global feature aggregation. Further, in order to avoid the risk of privacy information leakage in the forward propagation process of model training, before sending the generated local embedding set to the server each time, the participant adds noise disturbance to the local embedding set through a Gaussian mechanism function with a total privacy budget of g to make the disturbed local embedding set satisfy differential privacy protection.

[0114] After that, the participant uploads the local embedding set (the local embedding set disturbed to satisfy differential privacy) to the server, so that the server can perform global feature aggregation based on each local embedding set uploaded by each participant participating in the vertical federated graph learning, so that the global feature contains the complete neighborhood structure information of each node and the attribute information of all neighbor nodes. Since each local embedding set uploaded by each participant is calculated using different node attribute sets held by each participant, the importance of each local embedding set to the global feature is different. In the embodiment, the global feature aggregation module uses a regression aggregation function to learn the importance difference between different local embedding sets. Specifically, the global embedding of the i-th node can be represented as:

[0115]

[0116] where W​g denotes learnable regression weight parameters, S denotes a nonlinear activation function, and denotes Hadamard product. Global embeddings of all nodes in the global graph structure form a global embedding set Z g (i.e., global features).

[0117] Next, the server sends the global embedding set Z g to each participant, in this embodiment, the participants use a softmax function (a normalized exponential function) as a local classification module, and train the classification module based on the global embedding set and the node class label set (part of the node labels in the global graph structure) in the local graph data, which can be represented as:

[0118]

[0119] wherein, denotes the predicted node class of node i, denotes the class label of node i in the local node class label set Y u . In the training process, the participants use the predicted node class and the corresponding label to calculate the cross-entropy cost function and calculate the model gradient by the backpropagation algorithm to update the global graph model parameters θ (which includes the parameters used in the local feature extraction module, the global feature aggregation module, and the local classification module) to minimize the difference and loss between and . Specifically, the participants upload the initial gradient calculated for the first time to the server, so that the server can receive each initial gradient from each participant, and aggregate the initial gradients to obtain an aggregated gradient, so as to update the parameters and gradients in the global feature aggregation module stored by the server according to the aggregated gradient, and continue to propagate the gradient back to each participant to update the parameters and initial gradients in the local feature extraction module stored by each participant, so that the participants obtain the updated gradient. Then, based on the updated gradient, the above similar process is performed for multiple times to iteratively train the global graph model, and finally a trained global graph model is obtained. In this process, the participants need to send the updated gradient in each iteration to the server for aggregation.

[0120] Further, in the backpropagation process of model training, the updated gradient uploaded by the participant to the server each time has the risk of leaking private information, so differential privacy protection is needed. Before sending the updated gradient to the server each time, the participant adds a Gaussian mechanism with a total privacy budget of ∈ g to the updated gradient The noise perturbation is added, so that the updated gradient after the perturbation satisfies differential privacy, so that all the model intermediate results leaving the local computing environment of the participants satisfy the differential privacy constraint.

[0121] In the method, the privacy budget consumption is determined by a standard matrix accounting method.

[0122] The embodiments of the present application also provide a longitudinal federated graph prediction method satisfying differential privacy, which is also executed by any of the participants participating in the longitudinal federated graph learning, and includes the following steps:

[0123] Local partial graph data is obtained, and a global graph structure is obtained according to a process similar to steps S110-S130 in the longitudinal federated graph learning method satisfying differential privacy described in the foregoing embodiments, wherein the local partial graph data includes a node set, an adjacency matrix representing an edge set, and a node attribute set;

[0124] The trained global graph model obtained by the longitudinal federated graph learning method satisfying differential privacy described in the foregoing embodiments is used to perform node category prediction based on the global graph structure and the local partial graph data. Specifically, the local feature is extracted by the trained local feature extraction module based on the global graph structure and the node attribute set in the local partial graph data; the extracted local feature is sent to the service party, so that the service party can receive each local feature from each participant participating in the longitudinal federated graph learning, the global feature is obtained by the trained global feature aggregation module based on each local feature, and the global feature is sent to each participant; the global feature from the service party is received, and the trained local classification module is used to classify based on the global feature to output the predicted node category.

[0125] In addition to the above node classification task, the method can also perform edge classification and other tasks. Specifically, the node attribute set and the node category label set in the local partial graph data can be replaced by an edge attribute set and an edge category label set, and the like.

[0126] Finally, the model training efficiency and prediction accuracy of the method are verified, and a comparison analysis is made with the prior art. The data sets used include three open source data sets, Cora, Citeseer and PubMed, as shown in Table 1:

[0127] Table 1: Data set information

[0128] Dataset Number of nodes Number of edges Dimension of node features Task Cora 2708 10556 1433 7-class node classification Citeseer 3327 9228 3703 6-class node classification PubMed 19717 88651 500 3-class node classification

[0129] The prior art includes five existing methods, Central, Local, VFGNN, DPVGL, and FDML, and their trained models. Among them, Central refers to a fully centralized (server) trained GraphSAGE model. Local refers to a fully locally trained GraphSAGE model by the participants. VFGNN refers to first generating a public-private key pair by a semi-honest server, then the participants use the public-private key pair to perform multi-party secure feature aggregation under ciphertext based on secret sharing technology, upload the decrypted feature aggregation result to the server for further feature aggregation, and generate the final node embedding for node classification tasks, and protect the privacy and security of the intermediate results transmitted between the participants and the server through the Gaussian mechanism. DPVGL refers to the subgraph structure held by the participants being directly disturbed and shared to each participant by adding noise through a random response mechanism, then the participants merge all the disturbed subgraph structures to generate a global graph structure, and perform feature aggregation according to the global graph structure, and finally upload the feature aggregation result to the server to generate node embedding for node classification tasks. Similarly, DPVGL protects the privacy and security of the intermediate results transmitted between the participants and the server through the Gaussian mechanism. FDML refers to the participants using local subgraphs and local graph neural networks to generate local node embeddings, and each local node embedding of each participant is uploaded to a semi-honest server for weighted summation and node classification tasks. FDML also protects the privacy and security of the intermediate results transmitted between the participants and the server through the Gaussian mechanism.

[0130] Figure 4 The present method and the above five existing technologies are compared in terms of model prediction accuracy under different privacy budgets ∈. As shown in Figure 4 The model prediction accuracy of the present method (Ours) changes with the privacy budget, and under moderate privacy protection strength, the present method can achieve similar model prediction accuracy to the VFGNN method. With the increase of privacy budget, the model prediction accuracy of the present method gradually approaches the VFGNN method based on secure multi-party computation technology and the Central method of complete centralization. In most cases, the model prediction accuracy of the present method is significantly higher than the Local method of complete local training and the FDML method that does not consider the feature aggregation calculation problem in graph neural networks, which shows that the present method can maintain high model prediction accuracy in the vertical federated graph learning process.

[0131] Table 2 shows the training efficiency comparison results of the present method and the existing methods under the privacy budget ∈ = 8, and gives the average value of the training time under 5 different random seeds in seconds.

[0132] Table 2: Training time comparison of the present method and the existing technologies (unit: seconds)

[0133]

[0134] It can be seen that, in the process of graph neural network calculation, the VFGNN which introduces secure multi-party computation technology for feature aggregation has the highest calculation overhead, and the training time is greater than that of other methods and has a large difference of orders of magnitude compared with other methods. In comparison, the method can significantly improve the calculation efficiency while maintaining high model accuracy, and the training time and the fully centralized Central method are in the same order of magnitude and are relatively close. Therefore, the method can achieve a balance of high training efficiency, low calculation overhead and high prediction accuracy.

[0135] Corresponding to the above method, the application also provides a longitudinal federated graph learning device satisfying differential privacy, which comprises a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, and the device implements the steps of the foregoing method when the computer instructions are executed by the processor.

[0136] The embodiment of the application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the foregoing method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0137] The embodiment of the application also provides a computer program product comprising computer instructions, which are executed by a processor to implement the steps of the foregoing method.

[0138] Those of ordinary skill in the art should understand that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether the implementation is in hardware or software depends on the specific application and design constraints imposed on the overall system. Those of skill can use various approaches to implement the described functionality depending on the specific application and design constraints. Such implementation should not be interpreted as causing a departure from the scope of the application. When implemented in hardware, the hardware can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and the like. When implemented in software, the elements of the application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link.

[0139] It is to be expressly understood that the invention is not limited to the specific configurations and process described above and illustrated in the accompanying drawings. For the sake of clarity, detailed descriptions of known methods are omitted. In the above-described embodiments, several specific steps are described and illustrated as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and various changes, modifications and additions can be made thereto by one of ordinary skill in the art without departing from the spirit of the present invention, and the order of the steps can be changed.

[0140] In the present invention, features described and / or illustrated with respect to one embodiment can be used in the same or a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0141] The above description is only preferred embodiments of the present invention, and is not intended to limit the present invention. The embodiments of the present invention can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the present invention.

Claims

1. A vertical federated graph learning method that satisfies differential privacy, characterized in that, The method includes: A vertical federated graph learning request is sent to the service provider and all other participants in the vertical federated graph learning process, excluding the local participant. Graph structure encoding information is obtained based on the target inferred graph structure information and the updated community set to generate perturbed graph structure encoding information that satisfies differential privacy. The target inferred graph structure information is obtained by iteratively optimizing the initialized inferred graph structure information based on the local local graph data using a local local graph model. The updated community set is obtained by updating the initial community set obtained by dividing the local local graph data into communities based on the target inferred graph structure information. Based on the perturbed graph structure encoding information and the initial shared graph structure information, a target shared graph structure information that satisfies differential privacy is obtained and shared with other participants. The perturbed initial shared graph structure information is an initial shared graph structure information that satisfies differential privacy, generated based on the initial shared graph structure information obtained by merging the target inferred graph structure information and the local local graph data. Receive target shared graph structure information from other participants, and determine global graph structure information based on the target shared graph structure information and target inferred graph structure information to reconstruct the global graph structure; Based on the global graph structure and local graph data, the global graph model is trained in a distributed manner by sending intermediate results to the service provider to obtain the trained global graph model. The global graph model includes a local feature extraction module, a global feature aggregation module located on the service provider, and a local classification module that are connected in sequence. The intermediate results include the local features extracted in each iteration and the updated gradient. In the first iteration, the updated gradient is the initial gradient.

2. The method according to claim 1, characterized in that, The local graph data includes a set of nodes, an adjacency matrix representing the set of edges, a set of node attributes, and a set of node category labels representing the actual node categories.

3. The method according to claim 1, characterized in that, The method for obtaining target shared graph structure information that satisfies differential privacy based on perturbed graph structure encoding information and initial shared graph structure information includes: The edge existence probability matrix is ​​estimated based on the perturbed graph structure encoding information and used as the prior probability. Using the perturbated initial shared graph structure information as evidence, Bayesian estimation of the target inferred graph structure information is performed based on the prior probability and the evidence, resulting in an edge posterior existence probability matrix that satisfies differential privacy, which is then used as the target shared graph structure information.

4. The method according to claim 3, characterized in that, The edge existence probability matrix includes the updated intra-community edge existence probability matrix obtained based on the perturbed neighbor number sequence in the perturbed graph structure encoding information and the updated inter-community edge existence probability matrix obtained based on the perturbed connection number matrix in the perturbed graph structure encoding information.

5. The method according to claim 1, 3, or 4, characterized in that, The graph structure encoding information includes a neighbor count sequence and a connection count matrix. The neighbor count sequence includes the number of neighbor nodes of each node in the local local graph data determined based on the target speculative graph structure information in each of the update communities to which it belongs in the update community set. The connection count matrix includes the number of connections between every two update communities in the update community set determined based on the target speculative graph structure information.

6. The method according to claim 3, characterized in that, The step of determining the global graph structure information based on the shared graph structure information of each target and the inferred graph structure information of the targets to reconstruct the global graph structure includes: For each node in the target shared graph structure information of each participant in the vertical federated graph learning, if the sum of the edge posterior existence probabilities of the corresponding edge in the target shared graph structure information shared by other participants is greater than or equal to 1, or the value of the corresponding edge in the target inferred graph structure information is 1, then the existence probability of the edge is determined to be 1; otherwise, the existence probability of the edge is the sum of the edge posterior existence probabilities of the corresponding edge in the target shared graph structure information shared by other participants. The global edge existence probability matrix is ​​obtained based on the existence probability of each edge and used as global graph structure information; The global graph structure information is hard-sampled based on the sampling threshold to reconstruct the global graph structure.

7. The method according to claim 1, characterized in that, The process involves distributed training of a pre-defined global graph model based on the global graph structure and local graph data, by sending intermediate results to the service provider, to obtain the trained global graph model, including: The local feature extraction module obtains the neighborhood of each node in the global graph structure based on the global graph structure, and performs feature aggregation based on the neighborhood of each node and the node attribute set in the local graph data to extract local features. The extracted local features are sent to the service provider so that the service provider can receive the local features from each participant in the vertical federated graph learning. The global feature aggregation module performs feature aggregation based on each local feature to obtain global features, and then sends the global features to each participant. The system receives global features from the service provider, performs classification based on the global features through the local classification module to output the predicted node category, and sends the updated gradients from each iteration to the service provider so that the service provider receives the updated gradients from each participant and aggregates the updated gradients to obtain the aggregated gradient. This process continues until the loss between the predicted node category and the node category label set in the local graph data is minimized to obtain the trained global graph model. The updated gradient is obtained by backpropagation based on the aggregated gradient obtained in the previous iteration. The number of node categories in the initial community set and the node set in the local local graph data are the same; the number of neighbor nodes in each updated community in the updated community set is the largest, and the number of neighbor nodes is calculated through the target speculative graph structure information; the updated community set is a perturbed updated community set generated by adding noise perturbation that satisfies differential privacy to each updated community in the updated community set; the intermediate result is a perturbed intermediate result generated by adding noise perturbation that satisfies differential privacy to the intermediate result.

8. A vertical federated graph learning device that satisfies differential privacy, comprising a processor, a memory, and computer instructions stored in the memory, characterized in that, The processor is configured to execute the computer instructions, and when the computer instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 7.

Images