A training method for a Bayesian graph neural network model with noise resistance

Abstract

The present invention discloses a training method for a Bayesian graph neural network model with noise resistance, which relates to the fields of Bayesian neural networks, graph neural networks and adversarial defense in the direction of deep learning, including: step 1, updating the representation vector of each node through a Bayesian update module; step 2, aggregating the updated representation vectors of each node through a Bayesian aggregation module to obtain a final output vector; step 3, calculating the expected upper bound of the spectral norm of the weight matrix; step 4, calculating the cross-entropy loss function of the output vector; step 5, summing the above calculation results as a loss function, and performing batch training of the neural network. The present invention not only introduces model uncertainty into the graph neural network, but also improves the adversarial robustness of the graph neural network, thereby improving the security and reliability of the model in practical applications and broadening the application prospects of the graph neural network.

Description

Technical Field

[0001] The present invention relates to the fields of Bayesian neural networks, graph neural networks, and adversarial defense in the direction of deep learning, and in particular to a training method for a Bayesian graph neural network model with noise resistance. Background Art

[0002] Graph neural networks (GNNs) are a type of neural network architecture in the field of deep learning that targets graph-structured data. Unlike conventional neural network models, GNNs use graph-structured data as input, where nodes are not isolated but interconnected. During training, GNNs aggregate and update the representation vectors of nodes and their neighbors to learn the connections between nodes and uncover the underlying properties of these connections. In recent years, GNNs have garnered widespread attention in fields such as recommendation systems, data mining, and biomedical research.

[0003] Bayesian neural networks are a specialized neural network architecture in the field of deep learning. Compared to traditional neural network approaches, Bayesian neural networks train a matrix of random variables and sample weight matrices from this matrix. As a result, even with the same input, the output of a Bayesian neural network can vary. This enables Bayesian neural networks to model data and uncertainty. However, Bayesian neural networks cannot handle graph-structured data, particularly lacking the ability to randomize the adjacency relationships between nodes.

[0004] Adversarial defense is a research area in the deep learning field that aims to enhance the robustness of models to adversarial noise. The Lipschitz constant is an important metric for measuring network robustness to adversarial noise and is utilized by many existing methods to improve the robustness of neural networks. While techniques exist to apply the Lipschitz constant to Bayesian neural networks, its application to graph neural networks is less clear.

[0005] Therefore, technicians in this field are committed to developing a training method for a Bayesian graph neural network model with noise resistance, designing a new Bayesian graph neural network with adversarial defense capabilities, and exploring the model's mechanism of action in terms of uncertainty and robustness. Summary of the Invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to simultaneously consider the randomness of the adjacency matrix and the randomness of the representation vector update to model the complete model uncertainty; how to ensure that the information transmission of node aggregation is not affected when using the Lipschitz constant to improve the robustness of the representation vector update.

[0007] To achieve the above object, the present invention provides a method for training a Bayesian graph neural network model with noise resistance, the method comprising the following steps:

[0008] Step 1: Update the representation vector of each node through the Bayesian update module;

[0009] Step 2: Aggregate the updated representation vectors of each node through the Bayesian aggregation module to obtain the final output vector;

[0010] Step 3: Calculate the expected upper bound of the spectral norm of the weight matrix and the loss function of the current layer;

[0011] Step 4: Calculate the cross entropy loss function of the output vector;

[0012] Step 5: The current layer loss function of each layer is summed and added to the cross entropy loss function to obtain a loss function, and batch training of the neural network is performed.

[0013] Furthermore, the step 1 also includes: obtaining a sampling weight matrix according to a trainable mean matrix and variance matrix based on a reparameterization technique.

[0014] Furthermore, the step 2 further includes:

[0015] Step 2.1, obtain a randomized adjacency matrix by sampling according to the deletion ratio;

[0016] Step 2.2: Add the self-loop to the adjacency matrix and normalize it to obtain a new adjacency matrix.

[0017] Step 2.3: Multiply the updated representation vector of each node by the new adjacency matrix and then add it to the original representation vector in proportion.

[0018] Furthermore, the step 2.2 includes: randomly sampling a set of 0-1 matrices at each layer of the aggregation module, and multiplying the obtained 0-1 matrix with the adjacency matrix to obtain a new adjacency matrix.

[0019] Furthermore, the process of calculating the expected upper bound of the spectral norm includes:

[0020] Calculate the spectral norm of the mean matrix;

[0021] Calculate the maximum value of the magnitude of the variance matrix column vector;

[0022] Calculate the maximum value of the magnitude of the row vector of the variance matrix;

[0023] Calculate the maximum absolute value of the difference between the sampled matrix and the mean matrix;

[0024] The above calculation results are summed to obtain the expected upper bound of the spectral norm.

[0025] Furthermore, the current layer loss function is obtained by summing the expected upper bound of the spectral norm and the total KL divergence.

[0026] Furthermore, step 1, step 2, and step 3 are repeated multiple times to achieve a sampling effect, the average of the results is taken as the final result, and the variance of the results is taken as the model uncertainty.

[0027] Furthermore, the calculation in step 4 is based on an iterative method and Monte Carlo sampling.

[0028] Furthermore, the model achieves node classification in a noise-free scenario by optimizing the cross entropy loss function and the KL divergence between the weight matrix and the prior distribution in the Bayesian update module.

[0029] Furthermore, the model improves the noise resistance of the model by optimizing the expected upper bound of the spectral norm of the weight matrix in the Bayesian update module.

[0030] Compared with the prior art, the present invention has the following beneficial technical effects:

[0031] (1) This paper transplants the Bayesian method from ordinary neural networks to graph neural networks, enabling graph-structured data to be effectively Bayesian processed, thus expanding the application scenarios of the Bayesian method.

[0032] (2) This invention makes both the update and aggregation phases of the graph neural network Bayesian, making them more theoretically interpretable.

[0033] (3) The spectral norm expectation upper bound regularization term added in this invention can effectively optimize the defense capability of the Bayesian graph neural network in the face of data perturbations and noise, making the model more robust. Therefore, this model can work more safely in more scenarios.

[0034] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of a preferred embodiment of the present invention;

[0036] Figure 21 is a schematic diagram of an algorithm of a single-sampling Bayesian update module according to a preferred embodiment of the present invention;

[0037] Figure 3 Schematic diagram of the algorithm of the single-sampling Bayesian aggregation module of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0039] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the present invention, comprising the following steps:

[0040] Step 1: Obtain the representation vector of each node through the Bayesian update module, such as Figure 2 As shown, the calculation steps of each layer are as follows:

[0041] (1) Using reparameterization technology, the sampling weight matrix is ​​obtained according to the trainable mean matrix and variance matrix;

[0042] (2) The spectral norm of the mean matrix and the expectation of the absolute value of the maximum element of the variance matrix are calculated by iterative method and Monte Carlo sampling respectively, and the sum of the maximum modulus of the row and column vectors of the variance matrix and the total KL divergence is used to obtain the current layer loss function; among them, the sum of the spectral norm of the mean matrix, the maximum modulus of the column vector of the variance matrix, the maximum modulus of the row vector of the variance matrix, and the maximum absolute value of the difference between the matrix obtained by sampling and the mean matrix is ​​the upper bound of the spectral norm expectation;

[0043] (3) Update the representation vector.

[0044] Step 2: Obtain the final output vector through the Bayesian aggregation module, such as Figure 3 As shown, the calculation steps of each layer are as follows:

[0045] (1) Obtain a randomized adjacency matrix based on deletion ratio sampling;

[0046] (2) normalize the new adjacency matrix after adding the self-loop;

[0047] (3) Multiply the representation vector by the critical matrix and add it to the original representation vector in a certain ratio.

[0048] Step 3: Repeat steps 1 and 2 enough times to achieve the sampling effect, take the average of the results as the final result, and take the variance of the results as the model uncertainty;

[0049] Step 4: Calculate the cross entropy loss function for each sampling result, sum it with the current layer loss function of each layer in step 1 to obtain the loss function, and average the multiple loss functions obtained by multiple samplings to perform batch training of the neural network.

[0050] This example applies to paper classification on the Cora dataset. The Cora dataset contains 2,708 scientific publications, 5,429 edges, and a total of seven categories. Each publication in the dataset is described by a word vector with a value of 0 / 1, indicating the absence / presence of the corresponding word in the dictionary. The dictionary contains 1,433 unique phrases. A node in the Cora dataset is a scientific publication, and an edge represents a citation relationship between two nodes. The initial feature at each stage is the word vector of the node.

[0051] The preferred embodiments of the present invention have been described in detail above. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible without inventive effort by those skilled in the art. Therefore, any technical solution that can be derived by one skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for training a Bayesian graph neural network model with noise resistance, characterized in that: The method is used to classify papers on the Cora dataset, and the method includes the following steps: Step 1: updating the representation vector of each node using a Bayesian update module, wherein the node corresponds to a scientific publication, the edge is a citation relationship, and the initial feature of the node is a word vector; Step 1 also includes: obtaining a sampling weight matrix based on a trainable mean matrix and variance matrix based on a reparameterization technique; Step 2: Aggregating the updated representation vectors of each node through the Bayesian aggregation module to obtain a final output vector; Step 2 also includes: Step 2.1, obtain a randomized adjacency matrix by sampling according to the deletion ratio; Step 2.2: Add the self-loop to the adjacency matrix and normalize it to obtain a new adjacency matrix. Step 2.3: Multiply the updated representation vector of each node by the new adjacency matrix and then add it to the original representation vector in proportion; Step 3: Calculate the expected upper bound of the spectral norm of the weight matrix and the current layer loss function; the calculation process of the expected upper bound of the spectral norm includes: Calculate the spectral norm of the mean matrix; Calculate the maximum value of the magnitude of the variance matrix column vector; Calculate the maximum value of the magnitude of the row vector of the variance matrix; Calculate the maximum absolute value of the difference between the sampled matrix and the mean matrix; Summing the above calculation results to obtain the expected upper bound of the spectral norm; The current layer loss function is obtained by summing the expected upper bound of the spectral norm and the total KL divergence; Step 4: Calculate the cross entropy loss function of the output vector; Step 5: The current layer loss function of each layer is summed and added to the cross entropy loss function to obtain a loss function, and batch training of the neural network is performed.

2. The method for training a Bayesian graph neural network model with noise immunity according to claim 1, wherein: The step 2.2 includes: randomly sampling a set of 0-1 matrices at each layer of the aggregation module, and multiplying the obtained 0-1 matrix with the adjacency matrix to obtain a new adjacency matrix.

3. The method for training a Bayesian graph neural network model with noise immunity according to claim 1, wherein: Repeat step 1, step 2, and step 3 multiple times to achieve a sampling effect, take the average of the results as the final result, and take the variance of the results as the model uncertainty.

4. The method for training a noise-resistant Bayesian graph neural network model according to claim 1, wherein: The calculation in step 4 is based on an iterative method and Monte Carlo sampling.

5. The method for training a Bayesian graph neural network model with noise immunity according to claim 1, wherein: The model achieves node classification in a noise-free scenario by optimizing the cross entropy loss function and the KL divergence between the weight matrix and the prior distribution in the Bayesian update module.

6. The method for training a noise-resistant Bayesian graph neural network model according to claim 1, wherein: The model improves the noise resistance of the model by optimizing the expected sum of the spectral norm upper bound of the weight matrix in the Bayesian update module.

Images