Random discarding method and system of graph neural network based on dynamic discarding

Abstract

The invention discloses a random discarding method and system for a graph neural network based on dynamic discarding, and relates to the technical field of artificial intelligence, and the method comprises the following steps: receiving a to-be-processed representative graph, carrying out the node feature extraction of the to-be-processed representative graph, obtaining a node receiving matrix, carrying out the normalization of the node receiving matrix, and obtaining a normalized matrix; extracting a feature vector matrix and a feature value diagonal matrix based on the normalized matrix, and calculating a message transfer matrix based on the feature vector matrix and the feature value diagonal matrix; the message passing matrix is input into a pre-established graph neural network model based on dynamic discarding, information discarding is carried out on the message passing matrix, a discarded message passing matrix is obtained through output, and the pre-established graph neural network model based on dynamic discarding is trained based on a random discarding stopping rule.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a random discarding method and system for a graph neural network based on dynamic discarding. Background Art

[0002] GNN was originally inspired by the success of convolutional neural networks (CNN) and recurrent neural networks (RNN) in image and sequence data processing. However, since these traditional neural network architectures are not suitable for processing non-Euclidean structured data, GNN has significant advantages in processing non-Euclidean structured data. GNN uses a message passing mechanism to generate feature representations for nodes by aggregating information about each node and its neighboring nodes. This feature representation has shown great potential in tasks such as node classification, link prediction, and graph clustering. In order to meet these challenges, researchers have proposed a variety of improvement methods. For example, as one of the typical representatives of GNN, the graph convolutional network (GCN) uses the adjacency matrix of the graph as a filter and updates the node representation through neighborhood aggregation.

[0003] Although GCN has made significant progress in graph representation learning, its training on large-scale graphs still faces multiple challenges. First, due to the recursive aggregation of neighbor information, GCN models are prone to overfitting problems, especially when the training data is limited. This phenomenon is particularly evident on small-scale graphs, causing the model to perform well on the training data but poorly on the test data. Second, as the number of GCN layers increases, the representations between nodes tend to become indistinguishable, a phenomenon known as oversmoothing. Oversmoothing makes it difficult for deep GCNs to capture high-order dependencies between nodes, causing the representations of all nodes to converge and lose sensitivity to the original input features. Finally, as the size of the graph increases, GCN will experience the problem of state spatiotemporal explosion, and its computational cost tends to increase exponentially with the expansion of the graph size. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a random discarding method and system for a graph neural network based on dynamic discarding.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a random discarding method of a graph neural network based on dynamic discarding, the method comprising the following steps:

[0006] Receiving a representative graph to be processed, extracting node features of the representative graph to be processed to obtain a node connection matrix, and normalizing the node connection matrix to obtain a normalized matrix;

[0007] Extracting an eigenvector matrix and an eigenvalue diagonal matrix based on the normalized matrix, and calculating a message passing matrix based on the eigenvector matrix and the eigenvalue diagonal matrix;

[0008] The message passing matrix is ​​input into a pre-established graph neural network model based on dynamic discarding, information is discarded on the message passing matrix, and the message passing matrix after discarding is output, wherein the pre-established graph neural network model based on dynamic discarding is trained based on the stopping random discarding rule.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the representative graph to be processed is G = {V, E}, where V = {v1, ..., v n} represents a set of n nodes, Represents the set of edges between nodes, the feature matrix of the node c is the dimension of node features.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a calculation process of the node connection matrix:

[0011]

[0012] Among them, A ij represents the i-th row and j-th column in matrix A, representing node v i With v j The relationship between mn Represents the element at row m and column n.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of normalizing the node connection matrix to obtain a normalized matrix:

[0014] The degree of a node is represented by d = {d1,…,d n}, d i Represents connection to node v i The sum of the weights of ii , which is a diagonal matrix, representing the degree of node i, defining the normalized matrix A of the adjacency matrix A sym The formula is as follows.

[0015]

[0016] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a calculation process of extracting the eigenvector matrix and the eigenvalue diagonal matrix based on the normalized matrix:

[0017] A sym =ΨΛΨ T

[0018] Ψ is A sym The eigenvector matrix of satisfies the above relationship, where Λ is the eigenvalue diagonal matrix.

[0019] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the message delivery matrix is ​​as follows:

[0020] The total message passing matrix is ​​expressed as The message transmitted from node i is represented as m1={m 11 ,…m 1k}∈i k×c ;

[0021] m 12 ,m 13 ,,m 14 , are all message passing matrices, representing the message matrices transmitted from node 1 to nodes 2, 3, and 4;

[0022] During the propagation process, the node representation is updated based on the node feature information and the messages from neighbors, which is formulated as follows:

[0023]

[0024] in Represents node v i In the hidden representation of layer l, N(i) represents the node v i The set of adjacent nodes, e i,j represents the edge from node j to node i; γ l and φ l Both represent differentiable activation functions, AGG represents clustering operations; the message matrix is ​​expressed as follows:

[0025]

[0026] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the message passing matrix into a pre-established dynamic discard-based graph neural network model is as follows:

[0027] The graph neural network based on dynamic discarding discards the message matrix with a dynamic discarding rate δ. There are δ|M| elements that will be discarded. For each element M in the message matrix i,j , according to the Bernoulli binomial distribution ε i，j ~Bernoulli(1-δ), generates an independent mask ε i,j To determine whether this element is retained, the message passing matrix after discarding is obtained by multiplying each element mask Reuse The factor pair To scale, the process is as follows:

[0028]

[0029] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the stopping random discarding rule is as follows:

[0030] The node updates its own hidden layer features by collecting the features of neighboring nodes, and a new embedding vector is generated at each step. By calculating the residual vector r i l+1 To determine whether random discarding should continue, r i l+1 The calculation formula is as follows:

[0031]

[0032] Where Ψ is A sym The eigenvector matrix, f * is the optimal combination coefficient, which is obtained by minimizing the following objective function. The formula is:

[0033]

[0034] When i l+1 The L1 norm of the original vector When the ratio of the L1 norm of exceeds the preset threshold η, the residual vector is considered to be important and has new information. The formula is

[0035]

[0036] If this ratio is lower than η, it means that the vector is mainly redundant information and fails to provide enough new information, so the node stops discarding.

[0037] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a random discarding system of a graph neural network based on dynamic discarding, comprising:

[0038] A feature extraction module is used to receive the representative graph to be processed, extract node features of the representative graph to be processed, obtain a node connection matrix, and normalize the node connection matrix to obtain a normalized matrix;

[0039] A matrix processing module, used for extracting an eigenvector matrix and an eigenvalue diagonal matrix based on a normalized matrix, and calculating a message passing matrix based on the eigenvector matrix and the eigenvalue diagonal matrix;

[0040] The information discarding module is used to input the message passing matrix into a pre-established graph neural network model based on dynamic discarding, perform information discarding on the message passing matrix, and output the message passing matrix after discarding, wherein the pre-established graph neural network model based on dynamic discarding is trained based on the stopping random discarding rule.

[0041] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the random discarding method of the graph neural network based on dynamic discarding as described above is adopted.

[0042] Beneficial effects of the present invention:

[0043] The present invention improves the random discarding method, adds a dynamic discarding rate and a stop discarding rule, and studies the impact of different dynamic discarding rate functions on the model. Sufficient experiments are conducted on different tasks. Compared with the existing random discarding method, the present invention can better alleviate the over-smoothing problem, improve the accuracy of the model, and improve the robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a schematic flow chart of the method of the present invention;

[0046] Figure 2 It is a schematic diagram of the transmission mode of the message matrix of the present invention;

[0047] Figure 3 It is a comparative schematic diagram of the present invention;

[0048] Figure 4 It is a schematic diagram of the experimental results of the present invention;

[0049] Figure 5 Schematic diagram of MADGap between the outputs of the last two layers of different model neural networks of the present invention;

[0050] Figure 6 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Embodiment 1:

[0053] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0054] Graph Neural Network: Graph Neural Network (GNN) is a general term for algorithms that use neural networks to learn graph structured data, extract and discover features and patterns in graph structured data, and meet the needs of graph learning tasks such as clustering, classification, prediction, segmentation, and generation.

[0055] like Figure 1 As shown, a random discarding method of a graph neural network based on dynamic discarding includes the following steps:

[0056] S101: receiving a representative graph to be processed, extracting node features of the representative graph to be processed to obtain a node connection matrix, and normalizing the node connection matrix to obtain a normalized matrix;

[0057] First, this application assumes that G = {V, E} represents a graph, where V = {v1, ..., v n} represents a set of n nodes, Represents the set of edges between nodes, the feature matrix of the node c is the dimension of the node feature. In order to conveniently describe the relationship between points, the adjacent matrix can be expressed as formula 1

[0058]

[0059] Among them A ij Represents node v i With v j The relationship between mn Represents the element in the mth row and nth column. The degree of the node can be expressed as d = {d1,…,d n}, d i Represents connection to node v i The sum of the weights, then this application can define a degree matrix D ii , which is a diagonal matrix, representing the degree of node i. Then define the normalized matrix A of the adjacency matrix A sym Its formula is shown in Formula 2.

[0060]

[0061] Ψ is A sym The eigenvector matrix of satisfies the relationship in Formula 3. Where Λ is the eigenvalue diagonal matrix.

[0062] A sym =ΨΛΨ T (3)

[0063] S102: extracting an eigenvector matrix and an eigenvalue diagonal matrix based on the normalized matrix, and calculating a message passing matrix based on the eigenvector matrix and the eigenvalue diagonal matrix;

[0064] The total message passing matrix can be expressed as M = {m1,…,m k}∈i k×c , where the message transmitted from node i can be expressed as m1 = {m 11 ,…m 1k}∈i k×c , the message matrix is ​​transmitted as follows Figure 2 shown.

[0065] m 12 ,m 13 ,,m 14 , are all message passing matrices, representing the message matrices transmitted from node 1 to nodes 2, 3, and 4.

[0066] Most existing GNN models adopt a message passing framework, that is, each node sends messages to its neighbors and receives messages from its neighbors. During the propagation process, the node representation is updated based on the node feature information and the messages from the neighbors, which can be formulated as shown in Formula 4.

[0067]

[0068] in Represents node v i In the hidden representation of layer l, N(i) represents the node v i The set of adjacent nodes, e i,j represents the edge from node j to node i; γ l and φ l Both represent differentiable activation functions, and AGG represents clustering operations. The message matrix can be expressed as Formula 5.

[0069]

[0070] S103: Input the message passing matrix into a pre-established graph neural network model based on dynamic discarding, discard information on the message passing matrix, and output the message passing matrix after discarding, wherein the pre-established graph neural network model based on dynamic discarding is trained based on the stopping random discarding rule.

[0071] First, the present application method is compared with DropEdge and DropNode. Figure 3 As shown in the figure, it can be seen that the method of the present application is different from the above two methods in that the present application discards random information in the message passing matrix M instead of discarding an edge and a node entirely.

[0072] Each 3×3 matrix represents a message transmission matrix transmitted from node 1. In Dropnode, if node 1 is discarded, the entire message matrix will be discarded. In Dropedge, if the edge is discarded, a column of message vectors will be discarded. The method of the present application can discard information in the message matrix according to the dynamic discard rate δ.

[0073] The graph neural network based on dynamic discarding discards the message matrix with a dynamic discarding rate δ. There are δ|M| elements that will be discarded. For each element M in the message matrix i,j , according to the Bernoulli binomial distribution ε i，j ~Bernoulli(1-δ), generates an independent mask ε i,j To determine whether this element is retained, the message passing matrix after discarding is obtained by multiplying each element mask Reuse The factor pair To scale, the process is as follows:

[0074]

[0075] The applied GNN model is implemented by perturbing the message matrix Instead of the original message matrix to propagate information. This only affects the training process. In fact, the present application does not need to explicitly generate a complete message matrix, because each row in the message matrix represents a different directed edge in the graph, and the method of the present application can be applied independently to each directed edge. And messages are passed in the model without increasing the time or space complexity. Since the number of aggregations performed is reduced, the number of floating-point calculations will be significantly reduced, and the accuracy and robustness will be improved.

[0076] After defining the random dropout rule, this application will explain when such dropout should be stopped. Due to random dropout, the model has improved generalization and stability compared to traditional models, but there is also the possibility that important information may not be fully learned. Adding a rule to stop random dropout can prevent the model from being over-smoothed.

[0077] In GNN, according to Formula 3, a node updates its own hidden layer features by aggregating the features of neighboring nodes. Each step generates a new embedding vector In order to ensure that this representation is informative, this application calculates the residual vector r i l+1 To determine whether random discarding should continue, r i l+1 The calculation formula is as shown in Formula 7

[0078]

[0079] Where Ψ is A as shown in Formula 3 sym The eigenvector matrix, f * The optimal combination coefficient is obtained by minimizing the following objective function, the formula is:

[0080]

[0081] When i l+1 The L1 norm of the original vector When the ratio of the L1 norm of exceeds the preset threshold η, the residual vector is considered to be important and has new information. The formula is

[0082]

[0083] If this ratio is lower than η, it means that the current vector is mainly redundant information and fails to provide enough new information, so the node stops discarding. Generally, the value of η is set to 10 -8 .

[0084] First, this application needs to prove the upper bound of the discard rate δ. Since the method of this application directly discards on the message matrix M, it is first necessary to ensure that at least one element in each row of the message matrix M is retained in the expectation, that is, the diversity of features. It is not desired that a feature be completely discarded during information transmission. The mathematical expression of this condition is:

[0085]

[0086] Secondly, it is necessary to ensure that at least one element in each column of the message matrix M is retained in the expectation, that is, the diversity of the topology. It is not desired that an edge be completely discarded during information transmission. The mathematical expression of this condition is:

[0087]

[0088] In order to ensure the diversity of information, the discard rate δ should satisfy:

[0089]

[0090] After proving the theoretical value of δ, this application needs to prove the advantage of using a dynamic dropout rate. Assume a simplified network where the output of each neuron before dropout is x and the output after dropout is x'. Under a fixed dropout rate δ, the expected and variance of the output x' are

[0091] Expectation: E(x')=(1-δ)E(x)

[0092] Variance: Var(x') = (1-δ) 2 Var(x)

[0093] Now, assume that the dynamically adjusted dropout rate δ(t) is a linearly decreasing function that decreases from δ0 to δ as the training progresses. T , where T is the total number of training times and t is the current number of training times. The specific formula is:

[0094]

[0095] This application sums the variances over all training cycles.

[0096] Total variance for a fixed drop rate: T×(1-δ) 2 Var(x)

[0097] Total variance of dynamic drop rate:

[0098] Since δ(t) decreases with the number of training times, the fixed drop rate δ = δ0. Therefore, according to the formula, the total variance accumulated by the dynamic drop rate during the entire training process will be less than the total variance of the fixed drop rate. Especially when δ0 is significantly greater than δ T When , the total variance accumulated by the dynamic dropout rate during the entire training process will be significantly smaller than the total variance of the fixed dropout rate.

[0099] When proving the overall variance above, this application uses a linear decreasing method for the sake of simplicity, but in actual experiments, there are many solutions for dynamically adjusting the discard rate.

[0100] Linear adjustment, the formula is:

[0101] Exponential decay, the formula is: δ(t) = δ0·e -kt

[0102] Where k is the decay rate, which controls the decay speed

[0103] Annealed cosine decay, the formula is:

[0104] In the subsequent experimental section, the best dynamic dropout rate function will be verified on different datasets. A fixed dropout rate provides a form of regularization, but it may not be applicable to all training stages. In the early stages of training, the model needs to explore a wider feature space to find effective representations; using a higher dropout rate at this time can promote this and prevent the model from relying on a few features and causing overfitting. As training progresses, lowering the dropout rate can help the model refine its learned features and improve its fit to the data. Dynamically adjusting the dropout rate can help the model perform better during training.

[0105] Specifically, the present invention is further described below by way of embodiments:

[0106] In order to verify the accuracy of the method and its effect on suppressing over-smoothing, this application will conduct experiments in four datasets: Cora, CiteSeer, PubMed, and Flicker. First, this application will verify the indicators of three dynamic discarding schemes: linear adjustment, exponential decay, and annealed cosine decay, and select the best scheme. Then, this application will use two commonly used graph convolution frameworks, GCN and GAT, to verify the accuracy of this application's method and compare it with other advanced methods. Finally, this application will verify the robustness and the effect of suppressing over-smoothing.

[0107] Dataset introduction and experimental parameters

[0108] Cora, CiteSeer, and PubMed datasets: The Cora, CiteSeer, and PubMed datasets contain papers in their respective fields, each represented by a term frequency-inverse document frequency (TF-IDF) vector. All three datasets are used for graph classification and node classification tasks, through which network models can learn how to predict the topic or category of a paper based on the content of the paper and the structure of the citation network. The nodes of each dataset are labeled to indicate the topic category of the paper, providing researchers with an ideal benchmarking platform to evaluate and compare the performance of different graph neural network models.

[0109] Flickr dataset: The Flickr dataset originates from the Flickr image sharing platform and is mainly used to study social network analysis and image classification tasks. This dataset includes a large number of social media user nodes, and the edges between nodes represent the social relationships between users. The labels of the nodes are classified based on the user's interests or activities. Table 1 shows the statistics of the dataset.

[0110] Table 1 Dataset statistics

[0111] Dataset Number of nodes Number of edges Number of special certificates Number of categories Cora 2078 5429 1443 7 CiteSeer 3327 4732 3703 6 PubMed 19717 44338 500 3 Flickr 89250 899756 500 7

[0112] This application will use the AdamW optimizer to train on the RTX 4070super GPU. The experimental environment is shown in Table 2:

[0113] Table 2 Experimental environment

[0114]

[0115] The training epoch is set to 200. The model of this application consists of a two-layer graph convolution module, and the hidden dimension number for two-layer is set to 16. According to Formula 12, the learning rate is 0.005, and the L2 regularization rate is 1×10 -4 , the value of η is set to 10 -8 The parameter settings are shown in Table 3.

[0116] Table 3 Parameter settings

[0117]

[0118] Bankbone backbone network: In this application, two mainstream GNNs are mainly considered as the backbone models: GCN and GAT.

[0119] Discarding plan selection

[0120] Linear attenuation, the formula is:

[0121] Exponential decay, the formula is: δ(t) = δ0ge -kt , where k is the decay rate, controlling the decay speed

[0122] Annealed cosine decay, the formula is:

[0123] GCN was selected as the backbone in the Cora and CiteSeer datasets respectively, and the initial drop rate δ0 was set to 0.9. The optimal adjustment scheme was selected by training the model and testing the accuracy of the model, as shown in Table 4.

[0124] Table 4 Effect of different attenuation models on model accuracy

[0125] Linear Falloff Exponential decay Annealed Cosine Decay Cora 0.82 0.815 0.823 CiteSeer 0.706 0.71 0.712

[0126] As shown in Table 4, annealed cosine decay shows the best performance, so the dynamic discard rate function in the subsequent experiments of this application will adopt annealed cosine decay.

[0127] The best discard upper bound δ0

[0128] According to the data set data given in Table 1, combined with formula 12, it can be obtained that the upper bound of the initial discard rate δ0 is approximately equal to 1. Then the upper bound of the discard rate δ0 is adjusted from 0.1 to 0.9 in different data sets for the two backbone networks with a step size of 0.1. The accuracy values ​​obtained by the upper bounds of the discard rate δ0 in the four data sets are shown in Table 5. This application plots the data of different data sets in two backbone networks into images as shown in Figure 4 As shown, the calculation accuracy obtains the best initial discard rate δ0 as shown in Table 6.

[0129] Table 5 Accuracy values ​​obtained by the upper bound δ0 of different discard rates in the dataset

[0130]

[0131] Table 6 The best initial dropout rate δ0 for different backbone networks under different data sets

[0132]

[0133] Compared with advanced solutions

[0134] Backbone network: In this application, two mainstream GNNs are mainly considered as backbone models: GCN and GAT. Some minor modifications are made on the basis of the method of this application. All these backbone models have random drop modules in different steps of their model implementation. For example, the GAT model performs random drop after self-attention calculation, and sets the number of attention heads to 8. In order to make a fair comparison, for different backbone network models, the random drop modules are unified in the same step. This application uniformly adopts the original input for the initial input, and adopts different drop methods at the starting point of the message propagation process, including the method of this application, DropNode and DropEdge.

[0135] Table 7 Comparison of the accuracy of the proposed method and other discarding methods in the GCN backbone network

[0136] Cora CiteSeer PubMed Flickr GCN 0.786 0.689 0.785 0.511 GCN-DropNode 0.816 0.723 0.799 0.519 GCN-DropEdge 0.795 0.696 0.792 0.515 GCN-Ours 0.829 0.713 0.802 0.523

[0137] Table 8 Comparison of the accuracy of the proposed method and other discarding methods in the GAT backbone network

[0138] Cora CiteSeer PubMed Flickr GAT 0.805 0.684 0.765 0.462 GAT-DropNode 0.809 0.691 0.776 0.423 GAT-DropEdge 0.807 0.709 0.772 0.508 GAT-Ours 0.816 0.724 0.781 0.516

[0139] Tables 7 and 8 summarize the accuracy results under GCN and GAT backbone networks. For the node classification task, the performance is measured by the accuracy on four public datasets.

[0140] The results show that the random drop method of the present application performs better than GCN and GAT without random drop in node classification. After comparing with the GCN and GAT methods without random drop, it is found that the effect of the present application method is different on different data sets, backbone models and tasks. For example, the random drop method on GAT achieved an average accuracy improvement of 5.6% on CiteSeer and an average accuracy improvement of 2.1% on PubMed. At the same time, the random drop method improved the accuracy of GCN on Cora by 2.1%, while it only improved the accuracy of GAT by 0.8%.

[0141] Comparison of different discarding schemes. The proposed method works well under various data sets, showing its strong adaptability to various scenarios. In general, there are 4 data sets under the node classification task. This application sets up 8 different experimental scenarios, which are all combinations of different backbone models and data sets, and a total of 32 groups of experiments are conducted. The experimental results show that the method of this application achieves the best effect in 7 experimental scenarios and the suboptimal effect in 1 experimental scenario.

[0142] Stable performance. The stable performance of the method of the present application on all datasets is also clearly demonstrated. Taking DropNode as a counterexample, the algorithm performs well in the CiteSeer dataset, but shows a significant decline on other datasets. A reasonable explanation is that the message matrix patterns retained by different masking methods are different from each other, and the discarding strategy of the present application has the highest consistency level. Therefore, compared with other methods, the method of the present application is more suitable for most scenarios.

[0143] Additional Experiments

[0144] Robustness analysis: The results are shown in Tables 9-12. The robustness of the discarding method was studied by adding random perturbations and measuring the ability of the discarding method to handle perturbed graphs. This application conducted experiments on four data sets: Cora, CiteSeer, PubMed, and Flickr. A certain proportion of edges were randomly added to these data sets for node classification. When the perturbation rate increased from 0% to 30%, the average decrease in accuracy of the method of this application was the smallest, and the anti-interference ability was the strongest. Under 30% perturbation, in the Cora, CiteSeer, PubMed, and Flickr data sets, the accuracy of the model decreased by 2.2%, 0.9%, 1.2%, and 1.8%, respectively, compared with the case without perturbation, which is much better than other models. This shows that the robustness and anti-interference ability of this method are the best among all models.

[0145] Table 9 Robustness test results and comparison in Cora dataset

[0146]

[0147] Table 10 Robustness test results and comparison in CiteSeer dataset

[0148]

[0149] Table 11 Robustness test results and comparison in PubMed dataset

[0150]

[0151] Table 12 Robustness test results and comparison in Flickr dataset

[0152]

[0153] Oversmoothing Analysis

[0154] Over-smoothing is a common problem in GNNs, which means that as the depth of the network increases, node representations become difficult to distinguish. In this section, the impact of various random dropout methods on this problem is evaluated, and the degree of over-smoothing is measured by MADGap. The MADGap value indicates the degree of activation difference between different layers of the model. Specifically, MADGap measures the mean absolute difference between the outputs of two adjacent neural network layers. Smaller values ​​in MADGap indicate nodes that are more difficult to distinguish, that is, over-smoothing is more likely to occur, and vice versa. The experiments were conducted on the Cora dataset with GCN as the backbone model. This application counts the MADGap between the outputs of the last two layers of neural networks of different models. The results are shown in Figure 5 As shown, it can be found that the method of the present application has a better effect in combating over-smoothing after adding the stop-drop rule. The method of the present application is improved by about 30% compared with the DropNode and DropEdge models. Adding the stop-drop rule will improve by about 10% compared to removing the stop-drop rule.

[0155] Embodiment 2: In the second aspect, as Figure 6 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a random discarding system of a graph neural network based on dynamic discarding, comprising:

[0156] The feature extraction module 11 is used to receive the representative graph to be processed, extract node features of the representative graph to be processed, obtain a node connection matrix, and normalize the node connection matrix to obtain a normalized matrix;

[0157] A matrix processing module 12, used for extracting an eigenvector matrix and an eigenvalue diagonal matrix based on the normalized matrix, and calculating a message passing matrix based on the eigenvector matrix and the eigenvalue diagonal matrix;

[0158] The information discarding module 13 is used to input the message passing matrix into a pre-established graph neural network model based on dynamic discarding, perform information discarding on the message passing matrix, and output the message passing matrix after discarding, wherein the pre-established graph neural network model based on dynamic discarding is trained based on the stopping random discarding rule.

[0159] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0160] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0161] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0162] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A random discarding method for graph neural networks based on dynamic discarding, characterized in that: The method comprises the following steps: Receiving a representative graph to be processed, extracting node features of the representative graph to be processed to obtain a node connection matrix, and normalizing the node connection matrix to obtain a normalized matrix; Extracting an eigenvector matrix and an eigenvalue diagonal matrix based on the normalized matrix, and calculating a message passing matrix based on the eigenvector matrix and the eigenvalue diagonal matrix; The message passing matrix is ​​input into a pre-established graph neural network model based on dynamic discarding, information is discarded on the message passing matrix, and the message passing matrix after discarding is output, wherein the pre-established graph neural network model based on dynamic discarding is trained based on the stopping random discarding rule.

2. The random discarding method of graph neural network based on dynamic discarding according to claim 1 is characterized in that: The representative graph to be processed is G = {V, E}, where V = {v1, ..., v n } represents a set of n nodes, Represents the set of edges between nodes, the feature matrix of the node c is the dimension of node features.

3. The random discarding method of graph neural network based on dynamic discarding according to claim 1 is characterized in that: The calculation process of the node connection matrix is: Among them, A ij represents the i-th row and j-th column in matrix A, representing node v i With v j The relationship between mn Represents the element at row m and column n.

4. The random discarding method of graph neural network based on dynamic discarding according to claim 3 is characterized in that: The process of normalizing the node connection matrix to obtain a normalized matrix is ​​as follows: The degree of a node is represented by d = {d1,...,d n }, d i Represents connection to node v i The sum of the weights of ii , which is a diagonal matrix, representing the degree of node i, defining the normalized matrix A of the adjacency matrix A sym The formula is as follows:

5. The random discarding method of graph neural network based on dynamic discarding according to claim 4 is characterized in that: The calculation process of extracting the eigenvector matrix and the eigenvalue diagonal matrix based on the normalized matrix is ​​as follows: A sym =PsLps T Ψ is A sym The eigenvector matrix of satisfies the above relationship, where Λ is the eigenvalue diagonal matrix.

6. The random discarding method of graph neural network based on dynamic discarding according to claim 1 is characterized in that: The message passing matrix is ​​as follows: The total message passing matrix is ​​expressed as M = {m1,...,m k }∈i k×c , where the message transmitted from node i is represented as m1 = {m 11 ,…m 1k }∈i k×c ; m 12 ,m 13 ,,m 14 , are all message passing matrices, representing the message matrices transmitted from node 1 to nodes 2, 3, and 4; During the propagation process, the node representation is updated based on the node feature information and the messages from neighbors, which is formulated as follows: in Represents node v i In the hidden representation of layer l, N(i) represents the node v i The set of adjacent nodes, e i,j represents the edge from node j to node i; γ l and φ l Both represent differentiable activation functions, AGG represents clustering operations; the message matrix is ​​expressed as follows:

7. The random discarding method of graph neural network based on dynamic discarding according to claim 1 is characterized in that: The process of inputting the message passing matrix into the pre-established dynamic drop-based graph neural network model is as follows: The graph neural network based on dynamic discarding discards the message matrix with a dynamic discarding rate δ. There are δ|M| elements that will be discarded. For each element M in the message matrix i,j , according to the Bernoulli binomial distribution ε i，j ~Bernoulli(1-δ), generates an independent mask ε i,j To determine whether this element is retained, the message passing matrix after discarding is obtained by multiplying each element mask Reuse The factor pair To scale, the process is as follows:

8. The random discarding method of graph neural network based on dynamic discarding according to claim 1 is characterized in that: The stop random drop rule is as follows: The node updates its own hidden layer features by collecting the features of neighboring nodes, and a new embedding vector is generated at each step. By calculating the residual vector To determine whether random discarding should continue, The calculation formula is as follows: Where Ψ is A sym The eigenvector matrix, f * is the optimal combination coefficient, which is obtained by minimizing the following objective function. The formula is: when The L1 norm of the original vector When the ratio of the L1 norm of exceeds the preset threshold η, the residual vector is judged to be important and has new information. The formula is If this ratio is lower than η, it means that the vector is mainly redundant information and fails to provide new information, so the node stops discarding it.

9. A random dropout system based on a graph neural network with dynamic dropout, characterized in that: The random discarding method of the graph neural network based on dynamic discarding according to any one of claims 1 to 8 is adopted, comprising: A feature extraction module is used to receive the representative graph to be processed, extract node features of the representative graph to be processed, obtain a node connection matrix, and normalize the node connection matrix to obtain a normalized matrix; A matrix processing module, used for extracting an eigenvector matrix and an eigenvalue diagonal matrix based on a normalized matrix, and calculating a message passing matrix based on the eigenvector matrix and the eigenvalue diagonal matrix; The information discarding module is used to input the message passing matrix into a pre-established graph neural network model based on dynamic discarding, perform information discarding on the message passing matrix, and output the message passing matrix after discarding, wherein the pre-established graph neural network model based on dynamic discarding is trained based on the stopping random discarding rule.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on a processor. When the processor loads and executes the computer program, the random discarding method of the graph neural network based on dynamic discarding described in any one of claims 1 to 8 is adopted.

Images