Graph diffusion model learning method and device satisfying differential privacy

Abstract

The application provides a graph diffusion model learning method and device satisfying differential privacy, the learning method comprising: training a denoising network in a graph diffusion model based on a plurality of subgraphs in a held subgraph set and a plurality of noisy graphs corresponding to each subgraph, so that the denoising network performs node feature extraction based on each noisy graph to obtain a plurality of node feature sets; obtaining an edge feature set based on the node feature set, and using an attention mechanism to weight and fuse a plurality of edge features connected to two end nodes of each edge in the corresponding edge feature set based on node attribute similarity information of the noisy graph, to generate each fused edge feature to obtain an updated edge feature set; outputting a prediction result of the corresponding subgraph based on the updated edge feature set, minimizing the loss or privacy budget consumption between the prediction result of the corresponding subgraph and the subgraph, and reaching a preset threshold to obtain a trained denoising network satisfying differential privacy. The application can realize graph learning satisfying differential privacy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of graph learning, and in particular to a graph diffusion model learning method meeting differential privacy, a graph denoising method and equipment. BACKGROUND

[0002] Deep graph generation model is a typical graph learning method, which aims to learn the distribution of a given graph by means of a deep neural network and generate new graph data samples. Graph diffusion model is the most advanced deep graph generation method at present. Compared with previous deep graph generation models such as graph variational autoencoder or graph generative adversarial network, graph diffusion model has significant advantages in generating graph utility and training stability. A trusted institution that holds graph data can train and publish a graph diffusion model using graph data from users, for model users to generate graphs. However, the graph data held by the trusted institution contains personal privacy information of users, and untrusted model users may infer these personal information of users from model parameters or model generation results, thereby causing the risk of privacy information leakage.

[0003] Therefore, it is necessary to train the graph diffusion model in a privacy protection manner to protect the user privacy information in the graph data used for model training from being leaked in the process of model publishing and using. SUMMARY

[0004] In view of this, the embodiments of the present application provide a graph diffusion model learning method meeting differential privacy, a graph denoising method and equipment to eliminate or improve one or more defects existing in the prior art.

[0005] A first aspect of the present application provides a graph diffusion model learning method meeting differential privacy, comprising the following steps:

[0006] training a denoising network in the graph diffusion model based on a plurality of subgraphs in a held subgraph set and a plurality of noisy graphs corresponding to each subgraph, so that the denoising network performs node feature extraction based on each noisy graph to obtain a plurality of node feature sets, wherein the noisy graph is obtained by using a noisy model in the graph diffusion model to add noise to the corresponding subgraph;

[0007] obtaining an edge feature set based on the node feature set, and using an attention mechanism to weight and fuse a plurality of edge features connected by two end nodes of each edge in the corresponding edge feature set based on the node attribute similarity information of the noisy graph, to generate each fused edge feature, so as to obtain an updated edge feature set;

[0008] output a prediction result of the corresponding subgraph based on the updated edge feature set, minimize the loss or privacy budget consumption between the prediction result of the corresponding subgraph and the subgraph until the loss or privacy budget consumption reaches a preset threshold, so that the trained denoising network can meet differential privacy, and output a denoised graph based on a noisy graph generated based on non-private data carried by the trained denoising network, wherein the privacy budget consumption is obtained by adding noise to the perturbed gradient generated based on the gradient for updating the parameters of the denoising network in each iteration.

[0009] In some embodiments of the present application, in the step of minimizing the loss between the prediction result of the corresponding subgraph and the subgraph, the structure of the node degree vector of the subgraph is used to constrain the minimization of the loss between the prediction result of the corresponding subgraph and the subgraph.

[0010] In some embodiments of the present application, the method further comprises:

[0011] Based on each subgraph in the set of subgraphs, a node degree vector of each subgraph is calculated to obtain a node degree vector of the set of subgraphs, and a noisy perturbation is added to the node degree vector to generate a perturbed node degree vector that meets differential privacy, and the perturbed node degree vector and the trained denoising network parameters are published.

[0012] The second aspect of the present application provides a graph denoising method, which comprises the following steps:

[0013] The graph diffusion model is trained by the graph diffusion model learning method that meets differential privacy as described in the foregoing first aspect, and a noisy graph generated based on non-private data carried by the trained denoising network is used to output a denoised graph through multiple iterations, wherein in each iteration,

[0014] Node feature extraction is performed based on the noisy graph of the initial state to obtain a node feature set, and in the first iteration, the generated noisy graph is used as the noisy graph of the initial state;

[0015] An edge feature set is obtained based on the node feature set, and the attention mechanism is used to weight and fuse multiple edge features connected between two nodes of each edge in the corresponding edge feature set based on the node attribute similarity information of the noisy graph of the initial state, to generate each fused edge feature, so as to obtain an updated edge feature set;

[0016] A graph prediction result corresponding to the noisy graph of the initial state is output based on the updated edge feature set, and a noisy graph of the next state is output based on the graph prediction result through posterior probability prediction based on the Bayesian rule, and the noisy graph of the next state is used as the noisy graph of the initial state in the next iteration.

[0017] In some embodiments of the present application, the method further comprises:

[0018] The model trainer publishes a disturbed node degree vector, and the disturbed node degree vector is used to perform denoising post-processing on the denoised graph to generate a final graph, wherein the expected node degree of each node in the final graph is the same as the value of the corresponding element in the disturbed node degree vector.

[0019] In some embodiments of the present application, the denoising post-processing on the denoised graph based on the disturbed node degree vector to generate a final graph comprises:

[0020] The edge category prediction result of each edge in the denoised graph is converted into an edge existence probability to obtain an edge existence probability matrix; the edge existence probability matrix is normalized and scaled based on the disturbed node degree vector to obtain a processed edge existence probability matrix to generate a final graph.

[0021] In some embodiments of the present application, the posterior probability based on the Bayes rule is:

[0022]

[0023] wherein, P (X|Y) represents the posterior probability, represents the Hadamard product, Q' represents the transpose matrix of the transition probability matrix Q, e t (Q t ) represents the inverse transition probability from the noisy graph e t at the initial state to the noisy graph e t-1 at the next state, P (X|Y) represents the posterior probability, represents the Hadamard product, Q' represents the transpose matrix of the transition probability matrix Q, e P (X|Y) represents the posterior probability, represents the Hadamard product, Q' represents the transpose matrix of the transition probability matrix Q, e t-1 at the initial state to the noisy graph e t-1 at the next state.

[0024] In some embodiments of the present application, the non-private data comprises the number of edge categories in the subgraph set held by the model trainer.

[0025] The third aspect of the present application provides an electronic device, comprising: a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, so that the device implements the steps of the method of the first aspect or the steps of the method of the second aspect.

[0026] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method of the first aspect or the steps of the method of the second aspect.

[0027] A fifth aspect of the present application provides a computer program product comprising computer instructions which, when executed by a processor, implement the steps of the method of the first aspect described above, or implement the steps of the method of the second aspect described above.

[0028] The graph diffusion model learning method, the graph denoising method and the device of the present application meet differential privacy, can protect the user privacy information such as attributes, features and the like of nodes and edges contained in training graph data used for model training from being leaked in the process of model publishing and using, realize training of a graph diffusion model in a manner meeting differential privacy, and generate a graph with high data utility and high quality.

[0029] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following detailed description and drawings in which

[0030] Those skilled in the art will appreciate that the objects and advantages of the application can be implemented in a manner different than described above and that the above described and other objects and advantages can be obtained through the teachings contained in the detailed description below. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description, serve to explain the principles of the application.

[0032] Figure 1 Flowchart of the graph diffusion model learning method meeting differential privacy in an embodiment of the present application;

[0033] Figure 2 Flowchart of the graph denoising method in an embodiment of the present application;

[0034] Figure 3 Flowchart of the graph denoising method in another embodiment of the present application;

[0035] Figure 4 Specific flowchart of the graph diffusion model learning and denoising process in an embodiment of the present application;

[0036] Figure 5 Graph effect comparison diagram of the present method and prior art under different total privacy budgets for the dataset Ego-small in an embodiment of the present application;

[0037] Figure 6A comparison diagram of the generated graph effect of the method and the prior art under different total privacy budgets for the data set Cora-small in an embodiment of the present application is shown in FIG. 1.

[0038] Figure 7 A comparison diagram of the generated graph effect of the method and the prior art under different total privacy budgets for the data set IMDB-MULTI in an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be given to the present application in combination with embodiments and drawings. Herein, the illustrative embodiments of the present application and the description thereof are used to explain the present application, but not as a limitation to the present application.

[0040] It should be further noted that, in order to avoid the present application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0041] It should be emphasized that the term “comprise / comprising” is used herein to indicate the presence of a feature, element, step or component, but not to exclude the presence or addition of one or more other features, elements, steps or components.

[0042] It should be further noted that, unless otherwise specified, the term “connection” used herein can not only mean direct connection, but also indirect connection in the presence of an intermediate.

[0043] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0044] In order to protect the user privacy information such as attributes, features, etc. of nodes and edges contained in the training graph data used for model training from being leaked in the process of model publishing and use, realize training of the graph diffusion model in a manner satisfying differential privacy, and generate a graph with high data utility, an embodiment of the present application proposes a graph diffusion model learning method satisfying differential privacy, a graph denoising method and equipment, which can greatly alleviate the negative impact of the noise added for realizing differential privacy on model utility, while avoiding the user privacy information from being leaked, and can also perform a deep graph generation task based on the graph diffusion model satisfying differential privacy, to generate a high-quality graph which can effectively perform various tasks such as data enhancement, graph recognition and classification.

[0045] Figure 1 A flowchart of the graph diffusion model learning method satisfying differential privacy in an embodiment of the present application is shown in FIG. 3. Figure 4Fig. 1 is a schematic diagram of a specific process of a graph diffusion model learning and denoising process in an embodiment of the present application. As shown in Fig. 1, the method can be performed by a graph data holder or a model trainer, and includes the following steps: Figure 1 and Figure 4 As shown in Fig. 1, the method can be performed by a graph data holder or a model trainer, and includes the following steps:

[0046] In step S110, a denoising network in the graph diffusion model is trained based on a plurality of subgraphs in the held subgraph set and a plurality of noisy graphs corresponding to the respective subgraphs, so that the denoising network performs node feature extraction based on each noisy graph to obtain a respective set of node features, wherein the noisy graph is obtained by applying a noisy model in the graph diffusion model to the corresponding subgraph.

[0047] The subgraph set held by the graph data holder or the model trainer can be a graph set composed of a plurality of social network graphs in a network social application scenario, or a graph set composed of a plurality of biological molecule graphs in a biological application scenario, etc. The graph diffusion model includes a noisy model for implementing forward (forward) noise addition and a denoising network for implementing reverse (backward) denoising, wherein the denoising network is the core of the graph diffusion model. The noisy model needs to be able to convert the original graph data distribution into a limit distribution that is easy to sample noise state. Specifically, for each subgraph of the subgraph set, since the edge represents the structure of the graph, the edge category distribution of the subgraph can be converted with a certain probability by using the Markov transition probability of the edge category. The calculation method of the conversion probability can be represented as:

[0048] [Q t ] ij =q(e t =j|e t-1 =i)

[0049] wherein [Q t ] ij represents the element of the i-th row and the j-th column of the conversion probability matrix Q t , i.e., represents the probability that the category of the edge e at time t is i when the category of the edge e at time t-1 is i; the conversion probability matrix is formed by the conversion probability [Q t ] ij of each edge in the subgraph. Then, the edge category probability distribution can be sampled to obtain the noise state G t at different times, and the edge category probability distribution q(G t |G t-1 ) can be represented as:

[0050] q(G t |G t-1 )=E t-1 Q t

[0051] wherein q(G t |Gt-1 represents the noise state G at a given time t-1 t-1 represents the noise state G at time t t represents the probability of the noise state G t-1 represents the noise state G at time t-1 t-1 represents the edge set in the noise state G t has the Markov property, so that the cumulative transition probability matrix Therefore, the noise state G t at each time can be generated by combining the noise-free state G and the cumulative transition probability matrix, which can be calculated as:

[0052]

[0053] where the noise-free state G is the original subgraph data, and E represents the edge set in the original subgraph data. In order to protect the original graph data, when the maximum time of the forward noise diffusion process, i.e., the maximum value of t, T→∞, the limit of the cumulative product of the transition probability matrix needs to be independent of the original subgraph G, so as to ensure that the limit distribution itself does not leak the user privacy information contained in the subgraph data. Therefore, the noise model can use a discrete uniform transition noise model, i.e., Q t = α t I+β t 11′ / K, where α and β represent weight parameters, I represents a unit matrix; K represents the number of edge categories in the subgraph set, and the absence of an edge is also regarded as an edge category; 11′ represents a diagonal matrix with non-zero elements being 1. The limit of the cumulative product converges to a uniform distribution over the edge categories when T→∞, which is independent of the graph structure of the original subgraph data. In this case, the cumulative transition probability matrix can be calculated as:

[0054]

[0055] where, the cumulative weight parameter s can be any decimal number; the cumulative transition probability matrix can map the original subgraph G to the noise state G t that is discretely added with noise over edge categories, i.e., G t can be regarded as a graph containing noisy edges, so the noise state G t can also be called a noisy graph. Of course, other ways can also be used to generate a noisy graph with added noise on the graph structure.

[0056] For each subgraph G, a plurality of noisy graphs {Gt1 , G t2 ,..., G tm}, the plurality of noisy graphs and the corresponding subgraphs form a training sample, and a plurality of training samples constitute a training dataset for training a denoising network in a graph diffusion model, wherein each subgraph serves as a label.

[0057] The denoising network includes an encoder and a decoder, both of which can be a kind of graph neural network. Specifically, the encoder first takes each noisy graph G t and the corresponding subgraph (each training sample) as input, and for each noisy graph, extracts node embeddings (node features) of all nodes in the noisy graph through a graph neural network to form a node embedding set (node feature set) Z V . Then, the decoder calculates the corresponding edge embedding set according to the node embedding set, and the edge embedding set can perform edge classification tasks through a Softmax function to predict the next noise state. In order to parameterize the denoising network φ θ to calculate the edge embedding set, a two-layer graph convolutional network (GCN) can be used as the encoder, as follows:

[0058]

[0059] wherein, denotes the node embedding set, denotes the normalized adjacency matrix of the noisy graph in the input training sample, denotes the degree matrix of the adjacency matrix A of the noisy graph, N denotes the number of nodes in the noisy graph, u and v denote the u-th node and the v-th node in the noisy graph respectively, and u≠v, denotes the node ID one-hot encoding for identifying the generated graph nodes, ReLU denotes a nonlinear activation function, and W0 and W1 denote the learnable parameters of the GCN.

[0060] In step S120, based on the node feature set, an edge feature set is obtained, and based on the node attribute similarity information of the noisy graph, the multiple edge features connected by the two end nodes of each edge in the corresponding edge feature set are weighted and fused to generate each fused edge feature, to obtain an updated edge feature set.

[0061] In this step, for each noisy graph, a decoder based on a multi-layer perceptron can be used to calculate an edge embedding set (edge feature set, composed of edge embeddings (edge features) of each edge of the noisy graph) Z E , as follows:

[0062]

[0063] Where f(·) denotes a multilayer perceptron, N represents the number of nodes u and v in the noisy graph, and Z is the set of edge embeddings. E Consider each edge e in the noisy graph uv The nodes at the two endpoints u and v of Z are embedded. u and Z v To predict the category of the corresponding edge.

[0064] To improve the utility and performance of the denoising network based on homogeneity, for each noisy graph, the cosine similarity of node attributes between the two nodes in each node pair is calculated to construct a similarity matrix S representing the node attribute similarity information. The calculation formula can be expressed as:

[0065]

[0066] Wherein, the element S in the u-th row and v-th column of matrix S uv The attribute X of node u u With the attribute X of node v v The cosine similarity between them is then calculated. Next, for each edge of each noisy graph, an attention mechanism is used to further weight and fuse the relevant edge embeddings (edge ​​embeddings of all edges connected to the two endpoints) of the edge under the guidance of the similarity matrix. This achieves weighted fusion of each edge and obtains an updated set of edge embeddings. This set is then combined with the endpoint neighborhood structure and the edge embeddings in the endpoint neighborhood to more accurately predict the existence of the edge and its subdivided edge categories. This can mitigate the negative impact of noise added to achieve differential privacy on model performance.

[0067] Step S130: Based on the updated edge feature set, output the prediction result of the corresponding subgraph, minimize the loss or privacy budget consumption between the prediction result of the corresponding subgraph and the subgraph to reach a preset threshold, so that the trained denoising network can satisfy differential privacy, and output the denoised graph based on the noisy graph generated by the non-privacy data carried by the trained denoising network. The privacy budget consumption is obtained by adding the noise scale to the gradient after perturbation based on the gradient generated to update the parameters of the denoising network in each iteration, which satisfies differential privacy.

[0068] Specifically, the following formula can be used to predict the data distribution of the corresponding original subgraph based on the updated edge feature set, that is, to predict the edge category of each edge in the original graph:

[0069]

[0070] in, This represents the prediction results for the subgraph, including the edge category prediction results for each edge; Softmax represents the normalized exponential function, and 'a' represents the residual connection weights. denotes the updated edge embedding set, W att denote the learnable parameters of the attention mechanism.

[0071] In some embodiments, in the step of minimizing the loss between the prediction result of the respective subgraph and the subgraph, the loss between the prediction result of the respective subgraph and the subgraph is minimized with a structural regularization constraint on the node degree vector of the subgraph. With the structural regularization constraint during the training process, the data utility and quality of the graph structure generated by the subsequent inverse denoising are improved.

[0072] The optimization objective of the training process of the denoising network is to minimize the cost function of the predicted subgraph distribution and the respective edge e uv in the real original subgraph G, that is, to minimize the difference between the prediction result of the subgraph and the subgraph in the training data set, wherein the cost function can adopt a cross-entropy cost function. And the structure of the generated (predicted) graph is corrected based on the structural regularization constraint on the node degree vector to minimize the loss between the prediction result of the subgraph and the subgraph, and the calculation formula can be represented as:

[0073]

[0074] wherein, denotes the loss between the prediction result of the subgraph and the subgraph G; λ denotes a weight hyperparameter for controlling the weight of the structural regularization constraint term in the overall cost function; CE denotes the cross-entropy cost function, e uv denotes the edge class label between nodes u and v in the subgraph, denotes the edge class prediction probability distribution between nodes u and v output by the denoising network; d u denotes the degree of node u in the subgraph, and the node degree vector of the subgraph is formed by the degrees of each node u in the subgraph. In this embodiment, the prediction result of the subgraph that is, the edge class prediction result set, is composed of each edge class prediction result ; the subgraph G that is, the edge class label set, is to minimize the difference between the edge class prediction result set and the edge class label set by the above formula.

[0075] At each step of the denoising network training process, i.e. in each iteration, a noise perturbation is added to the gradient for parameter updating of each layer of the network by a DP-SGD (differential privacy stochastic gradient descent) method to generate a perturbed gradient satisfying differential privacy, so that the gradient used at each step of the training process satisfies the differential privacy constraint. And in the training process, a non-overlapping subgraph B = qM can be randomly sampled from the training dataset M according to the sampling rate q, and in each step, the node embedding set, the edge embedding set and the updated edge embedding corresponding to the respective noise state are calculated according to the different noise states Gt of the corresponding subgraph, and then the predicted graph data distribution is calculated according to the updated edge embedding and the original subgraph G (and the node degree vector of the subgraph G ), according to the cost function The perturbed gradient is calculated by the back propagation algorithm to update the denoising network parameters θ. According to the differential privacy serial composition theorem, the network parameters as the cumulative result of the gradient satisfy the differential privacy constraint. Specifically, the DP-SGD method first clips the gradient to limit the sensitivity of the gradient to the change of the training sample, and then adds a noise perturbation based on the Gaussian mechanism to the gradient according to the sensitivity to limit the degree of change of the network parameters when the training graph data changes an edge, and the calculation method can be represented as:

[0076]

[0077] where C represents the gradient clipping threshold, which is used to limit the influence of each edge in the training graph data on the network parameters θ; v u represents the u-th node in the u-th row of the corresponding edge matrix E, and σ represents the noise scale, represents the sensitivity, and ||·||2 represents the L2 norm, represents the Gaussian distribution, and I represents the identity matrix. The privacy budget consumption in the training process is determined by the moment accounting method in DP-SGD, which is a standard method for calculating privacy budget consumption when applying the Gaussian mechanism in the model training process. This standard method first calculates the logarithmic moment of privacy consumption as a random variable according to the added noise scale and the sampling rate, and then calculates the privacy budget consumption of each iteration round according to the logarithmic moment. If the consumed privacy budget reaches the preset threshold, the training process ends.

[0078] In some embodiments, the method further comprises the following steps:

[0079] The node degree vector of each subgraph in the subgraph set is calculated to obtain the node degree vector of the subgraph set, and a noise perturbation is added to the node degree vector to generate a perturbed node degree vector satisfying differential privacy, and the perturbed node degree vector and the trained denoising network parameters are published.

[0080] Since the node degree vector of the subgraph set calculated by the graph data holder based on the held subgraph set represents the connection of the node and its neighbor nodes, it contains user privacy information and needs to be published in a way that meets privacy protection constraints. By adding noise disturbance to the node degree vector through the Laplace mechanism, the graph data holder can publish the node degree vector containing noise disturbance, that is, the disturbed node degree vector that meets differential privacy. The influence of the change of the edge in the original graph data on the node degree is disturbed by the noise disturbance, so that the model user cannot infer the change of the edge in the original graph data through the change of the node degree vector. The disturbed node degree vector of the subgraph set can be calculated as follows:

[0081]

[0082] where d r and represent the node degree vector and the disturbed node degree vector of the rth subgraph G in the subgraph set respectively, Lap(0, 2 / ∈ d ) represents the Laplace noise, ∈ d represents the privacy budget, and R represents the number of subgraphs in the subgraph set. Considering that the Laplace noise may disturb the node degree to be negative, which is impossible in a real scenario, the graph data holder truncates the lowest node degree in the disturbed node degree vector of each subgraph in the subgraph set to 1, which can be represented as:

[0083]

[0084] where represents the final disturbed node degree vector of the rth subgraph. Since the average node degree of the graph data is usually greater than 1, and the expected error of the Laplace noise Lap(0, 2 / ∈ d ) is 2 / ∈ d , when ∈ d is relatively large (for example, ∈ d > 2), the expected error of the disturbed node degree that meets the differential privacy protection accounts for a small proportion of the value of the original node degree before disturbance, so the disturbed node degree vector is less affected by the negative effect of the differential privacy noise, and can be used to further correct the graph structure generated by reverse de-noising, thereby improving the quality of the generated graph.

[0085] Figure 2 is a flowchart of the graph de-noising method in an embodiment of the present application. As shown in Figure 2 and Figure 4 , the method can be performed by a model user and includes the following steps:

[0086] Step S210, using the trained denoising network in the graph diffusion model trained by the graph diffusion model learning method satisfying differential privacy as described in the foregoing embodiments, generating a noisy graph based on the non-private data carried by the trained denoising network, and through multiple iterations until a denoised graph is predicted, wherein in each iteration,

[0087] Step S211, performing node feature extraction based on the noisy graph in the initial state to obtain a node feature set, and in the first iteration, the generated noisy graph is used as the noisy graph in the initial state;

[0088] Step S212, obtaining an edge feature set based on the node feature set, and using an attention mechanism to weight and fuse multiple edge features connected by two nodes of each edge in the corresponding edge feature set based on the node attribute similarity information of the noisy graph in the initial state, to generate each fused edge feature, so as to obtain an updated edge feature set;

[0089] Step S213, outputting a graph prediction result corresponding to the noisy graph in the initial state based on the updated edge feature set, and outputting a noisy graph in the next state based on the posterior probability prediction based on the Bayesian rule based on the graph prediction result, and using the noisy graph in the next state as the noisy graph in the initial state in the next iteration.

[0090] The above graph denoising method is the inference application process of the trained denoising network in the graph diffusion model after the graph diffusion model is learned to satisfy differential privacy. After the denoising network is trained and the trained network parameters are published, the model user can use the trained denoising network to perform a reverse denoising process, that is, to generate new graph data by denoising. In the reverse denoising process, in order to generate new graph data samples, the trained denoising network φ θ is used to predict the initial noise state G T of the maximum time T t , that is, the noisy graph in the initial state, to predict the noise state G t-1 of the t-1 time, that is, the noisy graph in the next state, that is, in each iteration, the noise state G t is input, and the trained denoising network φ θ predicts the category of each edge t-1 in the noise state G , which can be expressed as:

[0091]

[0092] The formula represents that the prediction probability distribution p t-1 (G θ |G t-1 ) of the noise state G t can be decomposed into Gt-1 each edge Cumulative product of the predicted probability distribution.

[0093] for edges between nodes u and v The above equation can be further expanded as follows:

[0094]

[0095] where, denotes the probability of edge e t in the noisy state G θ predicted by the trained denoising network φ uv for each edge e in the edge set E uv predicted by the trained denoising network φ The existence probability of edge e may be chosen as follows:

[0096]

[0097] The meaning of this equation is that if the probability of edge e uv in the noisy state G at time t is greater than 0 for some class, then the probability of edge e in the noisy state G at time t-1 is This probability is taken to be equal to the probability that edge e uv in the noisy state G at time t can have been reached from edge e t-1 in the noisy state G t at time t-1 by any of the possible classes (according to the Markov property) in a reversible manner.

[0098]

[0099] where denotes the Hadamard product, Q' denotes the transpose of the transition probability matrix Q; e t (Q t )'denotes the inverse transition probability from the noisy state G t at time t to the noisy state G t-1 at time t-1, i.e., the inverse transition probability from the noisy graph at the initial state to the noisy graph at the next state; denotes the transition probability from the original edge e t-1The cumulative transition probability. Since an edge belongs to only one of the categories, this posterior probability is normalized to a sum of probabilities for all categories being 1, and the posterior probability value is calculated for sampling. Therefore, it is possible to obtain the cumulative transition probability from the posterior probability q(e t-1 |e,e t Sampling noise state G t-1 This makes the trained denoising network φ θ Based on the noise state G t-1 Continue predicting the noise state G t-2 This process continues until a new graph data sample G is generated that follows the noise-free state of the original graph data distribution.

[0100] Specifically, the non-privacy data includes the number of edge categories in the subgraph set held by the model trainer, such as the subgraph set containing both male and female categories, which is distinct from user privacy information such as whether a user is male or female. The model user can first sample initial noise G from a uniform distribution P(E=k)=1 / K. T (Noise state at maximum time T), i.e., generating the noisy image G. T , where K represents the number of edge classes in the subgraph set, and the uniform distribution indicates that the probability of edge E belonging to class k is 1 / K. Subsequently, the model user uses the trained denoising network φ θ With noise state G T As input, the noise state G T As the initial noisy graph, we first calculate the node feature set, edge feature set, and updated edge feature set obtained by weighted fusion at time T. Based on the updated edge feature set, we predict the distribution of the original graph data at time T, thus obtaining the graph prediction result corresponding to the noisy graph in the initial state. Then you can As the posterior probability based on Bayesian rules The input is used to calculate the edges e of the previous time step and the next state. T-1 The probability of existence, i.e., obtaining the noise state G. T-1 (The noisy graph of the next state), where the predicted edges From graph prediction results By iteratively repeating the above process, the noise state G at time T is obtained. T Predict the noise state G at time T-1 T-1 The noisy image G of the next state T-1 The noisy map, serving as the initial state for the next iteration, is formed by the noise state G at time T-1. T-1 Predict the noise state G at time T-2 T-2 This process continues until the graph prediction probability distribution p, which follows the original graph data distribution at time 0, is obtained. θ (G|G1 ) or p θ (G), that is, output the denoised graph G in a noise-free state.

[0101] Figure 3 is a flowchart of a graph denoising method in another embodiment of the present application. As shown in Figure 3 and Figure 4 , the method further includes the following steps:

[0102] In step S220, the disturbed node degree vector published by the model trainer is obtained, and the denoised graph is post-processed based on the disturbed node degree vector to generate a final graph, wherein the expected node degree of each node in the final graph is the same as the value of the corresponding element in the disturbed node degree vector.

[0103] In some embodiments, in step S220, the post-processing of the denoised graph based on the disturbed node degree vector to generate a final graph includes the following steps:

[0104] The edge category prediction result of each edge in the denoised graph is converted into an edge existence probability to obtain an edge existence probability matrix; the edge existence probability matrix is normalized and scaled based on the disturbed node degree vector to obtain a processed edge existence probability matrix to generate a final graph.

[0105] After obtaining the graph prediction probability distribution p θ (G) that follows the distribution of the original graph data, a common method for post-processing correction of the graph is to set a selection threshold (which can be 0.5, etc.) and select the edge category with a prediction probability higher than the threshold to output the final generated graph. However, the above method cannot guarantee that the number of neighbors of each node in the final generated graph meets the requirements, thereby resulting in a low quality of the generated graph. Therefore, the model user obtains the disturbed node degree vector published by the model trainer and combines the graph prediction probability distribution p θ (G) to perform edge sampling, so that the sampling expectation value of the number of neighbors of each node is equal to the corresponding element in the node degree vector. Specifically, first, the edge category probability of each edge in the graph prediction probability distribution p θ (G) is converted into an edge existence probability ε uv to obtain an edge existence probability matrix ε, which can be represented as:

[0106]

[0107] wherein p(e uv =k) represents the probability that the prediction category of edge e θ in the graph prediction probability distribution p uv (G) is k, and the prediction category does not include the edge missing category, and ε uvdenotes the probability of the existence of an edge between nodes u and v. Subsequently, the u-th row of the edge existence probability matrix corresponding to node u is normalized and scaled in size, and the calculation formula can be expressed as:

[0108]

[0109] wherein, denotes the perturbed node degree of node u. After sampling all edges, the expected value of the number of neighbor nodes of each node is that is, the expected node degree of each node in the finally generated graph and the value of the corresponding element in the perturbed node degree vector are the same. According to the edge existence probability matrix, each edge is sampled to obtain the finally generated graph Through the above denoising post-processing and correction operations, the quality of the generated graph can be greatly improved.

[0110] In summary, the graph diffusion model learning method and the graph denoising method provided by the embodiments of the present application can satisfy the differential privacy training, learning and publishing of the graph diffusion model, while making the model have high performance, and can realize the generation of high-quality depth map satisfying differential privacy.

[0111] Finally, the quality and utility of the graph generated by the foregoing graph denoising method are verified, and a comparative analysis is made with the prior art. The data set information used is shown in Table 1:

[0112] Table 1: Data set information

[0113] Dataset Sample size Feature dimension Average node number Average edge number Ego-small 200 3703 6.4 8.7 Cora-small 51 1433 38.7 61.6 IMDB-MULTI 91 - 13 65.9

[0114] The existing technologies involved in the comparison include the following four existing methods: 1) GGAN is a deep graph generation model that learns the original graph data distribution using a generative adversarial network (GAN). The basic idea is to train a graph generator that starts from a Gaussian distribution to try to restore the original graph data distribution, while alternately training a binary classifier that judges whether the generated graph is real data or false data. This is a non-private comparison method. 2) GDiff is a discrete graph diffusion model that models the mapping of the original graph data distribution to a discrete uniform distribution on the edge category through a forward noise diffusion process, and generates data samples for training the denoising network through the forward noise diffusion process. This is a non-private comparison method. 3) DPGGAN is a deep graph generation model that satisfies edge differential privacy. A graph generator with an encoder-decoder structure is trained by a graph variational autoencoder (VAE) to generate false graph data, where the encoder maps the original graph data distribution to a parameterized Gaussian distribution, and the decoder tries to restore the original graph data distribution from the Gaussian distribution, and protects the privacy information in the training graph data by DP-SGD. In order to reduce the influence of noise in the model parameters on the effectiveness of the generated graph data, DPGGAN introduces a binary classifier according to the adversarial learning idea to judge whether the generated graph is real data or false data, and can simultaneously judge whether the global graph structure of the generated graph is similar to the original graph Figure 1

[0115] Figures 5-7 The generated graph effect comparison of the present method and the above four existing technologies under different total privacy budgets is shown. As shown in FIG. 6, the present method has the best performance in terms of the global structure of the generated graph, and the generated graph is closest to the original graph. The GGAN method is the worst in terms of the global structure of the generated graph, and the generated graph is farthest from the original graph. The GDiff method is better than the GGAN method in terms of the global structure of the generated graph, but still far from the original graph. The DPGGAN method is better than the GDiff method in terms of the global structure of the generated graph, but still far from the original graph. The GDiff-SC method is better than the DPGGAN method in terms of the global structure of the generated graph, but still far from the original graph. Figures 5-7 ​As shown, five indexes, including largest connected component (LCC), triangle count (TC), characteristic path length (CPL), Gini index (GINI) and relative edge distribution entropy (REDE), are used to evaluate the utility of the generated graphs in terms of graph structure statistics, and the average value of the absolute difference of each pair of indexes between the generated graph and the corresponding original graph in the dataset is calculated for comparison of the utility of the generated graphs. In addition, the average cosine similarity of the degree distribution, the average cosine similarity of the motif subgraph count, and the average graph classification accuracy under five-fold cross-validation using the GIN model between each pair of generated graph and corresponding original graph are used to evaluate the global utility of the generated graph structure and the quality of the generated graph. It can be seen that in most cases, the performance of the present method (Ours) in terms of data utility of the generated graph structure is superior to the most advanced existing deep graph generation model that satisfies the edge difference privacy constraint. When the total privacy budget gradually decreases, the utility of the existing deep graph generation method that satisfies the differential privacy constraint decreases, because as the privacy protection strength increases, more differential privacy noise is introduced into the model parameters, thus reducing the utility of the generated graph. As the total amount of differential privacy disturbance noise increases, the difference between the graph structure statistics of the generated graph and the original graph increases, and the global utility of the generated graph structure also decreases. However, compared with the existing method, the present method realizes the optimal trade-off between privacy protection and the utility of the generated graph, that is, it can realize privacy protection while improving the quality of the generated graph, so that it has high utility in data enhancement, graph recognition and classification and other tasks.

[0116] Corresponding to the above method, the present application also provides an electronic device, which comprises a computer device including a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, so that the device implements the steps of the foregoing method when the computer instructions are executed by the processor.

[0117] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the foregoing method. The computer readable storage medium can be a tangible storage medium, such as random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable memory disk, CD-ROM, or any other form of storage medium known in the technical field.

[0118] This invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the aforementioned method.

[0119] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0120] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0121] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0122] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A graph diffusion model learning method that satisfies differential privacy, characterized in that, The method includes: The denoising network in the graph diffusion model is trained based on multiple subgraphs in the held subgraph set and multiple noisy graphs corresponding to each subgraph, so that the denoising network extracts node features based on each noisy graph to obtain each node feature set. The noisy graph is obtained by adding noise to the corresponding subgraph using the noisy model in the graph diffusion model. Based on the node feature set, an edge feature set is obtained. Then, an attention mechanism is used to perform weighted fusion of multiple edge features connected to the two end nodes of each edge in the corresponding edge feature set based on the node attribute similarity information of the noisy graph, so as to generate each fused edge feature and obtain an updated edge feature set. Based on the updated edge feature set, the prediction result of the corresponding subgraph is output. The loss or privacy budget consumption between the prediction result of the corresponding subgraph and the subgraph is minimized to reach a preset threshold, so that the trained denoising network can satisfy differential privacy. The denoised graph is output based on the noisy graph generated by the non-privacy data carried by the trained denoising network. The privacy budget consumption is obtained by adding the noise scale to the gradient after the gradient is generated to update the parameters of the denoising network in each iteration, based on the gradient perturbation that satisfies differential privacy.

2. The method according to claim 1, characterized in that, In the step of minimizing the prediction result of the corresponding subgraph and the loss between the subgraph, the structural regularization constraint of the node degree vector of the subgraph is used to minimize the prediction result of the corresponding subgraph and the loss between the subgraph.

3. The method according to claim 1, characterized in that, The method further includes: The node degree vector of each subgraph in the subgraph set is calculated to obtain the node degree vector of the subgraph set. Noise perturbation is added to the node degree vector to generate a perturbed node degree vector that satisfies differential privacy. The perturbed node degree vector and the trained denoised network parameters are then published.

4. An image denoising method, characterized in that, The method includes: The denoising network in the graph diffusion model trained using the differential privacy-satisfied graph diffusion model learning method as described in any one of claims 1 to 3 is used. Based on the noisy graph generated from the non-privacy data carried by the trained denoising network, the process iterates multiple times until a denoised graph is predicted as the output. In each iteration, Node features are extracted from the noisy graph of the initial state to obtain a set of node features. In the first iteration, the generated noisy graph is used as the noisy graph of the initial state. Based on the node feature set, an edge feature set is obtained. Then, an attention mechanism is used to perform weighted fusion of multiple edge features connected to the two end nodes of each edge in the corresponding edge feature set based on the node attribute similarity information of the noisy graph in the initial state, so as to generate each fused edge feature and obtain an updated edge feature set. Based on the updated edge feature set, the graph prediction result corresponding to the noisy graph of the initial state is output, and based on the graph prediction result, the noisy graph of the next state is predicted and output through the posterior probability prediction based on Bayesian rules. The noisy graph of the next state is used as the noisy graph of the initial state in the next iteration.

5. The method according to claim 4, characterized in that, The method further includes: Obtain the perturbed node degree vector published by the model trainer, perform denoising post-processing on the denoised graph based on the perturbed node degree vector, and generate the final graph. The expected node degree of each node in the final graph is the same as the value of each corresponding element in the perturbed node degree vector.

6. The method according to claim 5, characterized in that, The step of performing denoising post-processing on the denoised graph based on the perturbed node degree vector to generate the final graph includes: The edge category prediction results of each edge in the denoised graph are converted into the edge existence probability to obtain the edge existence probability matrix. The edge existence probability matrix is ​​then normalized and scaled based on the perturbed node degree vector to obtain the processed edge existence probability matrix to generate the final graph.

7. The method according to claim 4, characterized in that, The posterior probability based on Bayesian rules is: in, Let ⊙ denote the posterior probability, ⊙ denote the Hadamard product, Q′ denote the transpose of the transition probability matrix Q, and e t (Q t )′ represents the noisy graph e from the initial state. t Noisy map e to the next state t-1 The reverse transition probability, This represents the predicted edges derived from the graph prediction results of each iteration output. Noisy map e to the next state t-1 The cumulative transition probability; The non-privacy data includes the number of edge categories in the subgraph set held by the model trainer.

8. An electronic device comprising a processor, a memory, and computer instructions stored in the memory, characterized in that, The processor is configured to execute the computer instructions, and when the computer instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 7.

Images