Deep map convolutional network over-smoothing suppression method and system

Abstract

The invention discloses a deep image convolutional network over-smoothing suppression method and system, and belongs to the technical field of deep learning and image data processing. According to the method, a bilateral constraint strategy is adopted, on one hand, partial columns of a node representation matrix are shielded through a random mask mechanism, the cumulative effect of high-order neighbor information is reduced, and personalized features of nodes are kept; and on the other hand, self-adaptive comparison constraints are introduced, the comparison loss weight is dynamically adjusted according to the similarity between the nodes, small constraints are applied to similar nodes, and strong distinguishing is applied to different nodes, so that the separability and the expression ability of the nodes are improved. According to the method, by optimizing a graph convolution propagation mechanism, the over-smoothing problem is suppressed while the information transmission effectiveness is ensured, and the classification performance and generalization ability of the deep GCN are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and graph data processing, and more specifically, relates to a method and system for suppressing over-smoothing of a deep graph convolutional network. Background Art

[0002] Graph convolutional networks have been widely used in tasks such as node classification, link prediction, and graph embedding due to their efficient feature extraction ability for graph-structured data. However, when existing deep GCNs are stacked with multiple layers, it is easy to cause the node representations to converge layer by layer, that is, the so-called over-smoothing problem, which further reduces the model's ability to capture the individual differences of nodes and results in a significant decline in classification performance. Traditional methods mostly focus on diluting neighbor information or directly stacking shallow features, but often cannot take into account the impacts of both the quality and quantity of neighbor information simultaneously, and have deficiencies such as fast convergence speed, information redundancy, or weakening of individual characteristics.

[0003] After retrieval, the Chinese patent publication number is CN 116596959 A, the publication date is August 15, 2023, and the invention creation name is: A three-dimensional human motion prediction method based on an optimized graph convolutional network. This patent uses a graph convolutional neural network for predicting human motion behaviors, and to solve the smoothing problem caused by excessive stacking of graph convolutional networks, it proposes to use the GCNII model to reduce the over-smoothing problem caused by excessive stacking of graph convolutional networks.

[0004] Another example is the Chinese patent publication number CN 118209820 A, the publication date is June 18, 2024, and the invention creation name is: including the following steps: Step 1: Construct a graph convolutional neural network model to extract features from the input graph data to achieve autonomous extraction of fault feature information; Step 2: Use a graph residual convolutional network to solve the problem of excessive smoothing in deep graph convolutional neural networks, and be able to better capture the feature information in graph data, thereby improving the accuracy and reliability of power grid fault detection.

[0005] In the above two patents, improvements have been made to the graph convolutional neural network to address the problem of over-smoothing that exists during its practical application. However, the above solutions still have certain limitations. Specifically: First, although CN116596959A introduces the GCNII model to alleviate the over-smoothing problem of deep GCNs, its improvements are mainly reflected in the structural design of residual connections and identity mappings. It still relies on a fixed propagation mechanism and lacks the ability to dynamically model the heterogeneity of node features and the diversity of neighbor information, making it unable to fully adapt to the complex topological structures in large-scale heterogeneous graphs. Second, the graph residual convolutional network (Res-GCN) proposed in CN118209820A has made improvements in alleviating information degradation, but the method still mainly focuses on shallow structures and lacks effective control over high-order neighbor information, which may cause the individual features of certain important nodes to be submerged during deep propagation, reducing the fine representation ability of the model.

[0006] Therefore, although the existing publicly disclosed technologies have alleviated the over-smoothing problem to a certain extent, there is still room for improvement in the collaborative optimization between information redundancy and node differential expression in dealing with deep graph structures. Summary of the Invention

[0007] 1. Problems to be Solved

[0008] The purpose of the present invention is to provide a method for suppressing over-smoothing in deep graph convolutional networks, which optimizes deep graph convolutional networks using a bilateral constraint mechanism, aiming to suppress the problem of over-smoothing of node features generated as the network depth increases, while taking into account the preservation of node individual information and the full utilization of high-order neighborhood information, and improving the performance and robustness of deep GCNs.

[0009] 2. Technical Solutions

[0010] To solve the above problems, the technical solutions adopted by the present invention are as follows:

[0011] First, the present invention provides a method for suppressing over-smoothing in deep graph convolutional networks, including the following steps:

[0012] S1. Data preprocessing;

[0013] S2. Random masking mechanism;

[0014] S3. Adaptive contrast constraint: Calculate the cosine similarity between nodes, and assign different contrast loss weights according to the similarity, imposing less constraint on similar nodes and stronger discrimination on dissimilar nodes to enhance node individual features;

[0015] S4. Joint optimization training: Weightedly fuse the graph convolution loss, random masking operation loss, and adaptive contrast loss, and train the model using the gradient descent or Adam optimization algorithm;

[0016] S5. Application and Inference: Apply the trained model to target tasks, including node classification, link prediction, or graph representation learning, and evaluate the over-smoothing suppression effect of the model.

[0017] Furthermore, in step S1, normalize the input graph data, construct the adjacency matrix and the node feature matrix, and perform linear dimensionality reduction or standardization on the node features; in step S2, at each layer of the graph convolutional network, randomly mask some columns of the node representation matrix. The masked columns are directly copied from the corresponding columns of the previous layer, while the unmasked columns perform graph convolution operations through the adjacency matrix to reduce the excessive superposition of high-order neighbor information.

[0018] Furthermore, when performing linear dimensionality reduction on the node feature matrix in step S1, use a multi-layer perceptron or other lightweight mappings, such as linear transformation, low-rank matrix factorization, etc., to map the original high-dimensional or sparse input to a representation space with a set dimension, so as to reduce the computational burden and overfitting risk during subsequent convolution.

[0019] Furthermore, the masking rate of the random masking mechanism gradually decays as the number of layers increases. The masking rate M(l) can be defined according to the following formula:

[0020] M(l) = 1 - log(λ / (l + 1))

[0021] where λ is a hyperparameter that controls the masking decay rate, and l is the current network layer number; when l is large, the masking rate decreases, which can balance the retention of high-order neighbor information by nodes and the need to suppress over-convergence.

[0022] Furthermore, the calculation of the adaptive contrast constraint weight W ij satisfies the following principle:

[0023] W ij = 1 - sim(i, j)

[0024] where sim(i, j) represents the cosine similarity between nodes i and j, and the range is [0, 1]. The higher the similarity, the stronger the similarity between the node pair, that is, W ij tends to 0 and the constraint is lighter. The lower the similarity, W ij tends to 1 and the constraint is stronger, which is used to amplify the feature differences between nodes.

[0025] Furthermore, the adaptive contrast loss adopts the InfoNCE class form, maximizes the similarity of positive sample pairs under augmented perspectives of adjacent layers or the same layer, and minimizes the similarity of negative sample pairs. The calculation formula of the adaptive contrast loss is as follows:

[0026]

[0027] where, is a positive sample pair, (h i , h j ) is a negative sample pair, τ is the temperature coefficient, w ii+ represents the adaptive weight of the positive sample pair , w ij represents the adaptive weight of the negative sample pair (h i , h j ).

[0028] Furthermore, in step 5, the conventional loss L main of the graph convolutional network is weighted and fused with the adaptive contrast loss L contrastive to form the final optimization objective:

[0029] L total = L main + λ c ·L contrastive

[0030] where λ c is the coefficient for balancing the proportion of the two types of losses; L main can be the cross-entropy or mean square error loss for node classification, link prediction, or other graph tasks.

[0031] Furthermore, the graph convolutional network structure can be a standard GCN, an SGC that only retains graph Laplacian propagation, or a GCNII with an initial residual connection. At the same time, in the high-order or deep network scenario, the present invention can improve steps S2 and S4 by combining any one of multi-task learning, graph attention mechanism, or hierarchical sampling technology to further suppress over-smoothing.

[0032] Further, in step S2, a graph attention mechanism is introduced to adjust the masking probability according to the attention weights between nodes, so that the information aggregation process focuses on important neighbor features, thereby enhancing node differences.

[0033] Further, in step S2, a hierarchical sampling strategy is combined to limit the number of neighbor samplings during each layer of propagation, reducing the aggregation of invalid high-order information.

[0034] Further, in step S4, a multi-task learning mechanism is introduced to jointly optimize the adaptive contrast loss and an auxiliary reconstruction task (such as adjacency matrix reconstruction or representation alignment) to enhance the model's ability to express individual features and structural relationships.

[0035] Second, the present invention also provides a system for suppressing over-smoothing of a deep graph convolutional network. This system is used for the method steps of the present invention. Specifically, the system includes a data processing module, a graph convolution and random masking module, an adaptive contrast constraint module, and a prediction evaluation module. Data interaction between the modules is carried out through standard interfaces. The overall system can be implemented either as a software solution or through an embedded hardware platform. Among them:

[0036] The data processing module is used to load the input original graph data, construct an adjacency matrix and a node feature matrix, and can perform necessary linear dimensionality reduction or normalization operations;

[0037] The random masking processing module, in combination with the graph convolution process, applies a dynamic masking rate to the column dimension of the node representation matrix at the l-th layer (l≥3) to retain or filter the corresponding feature columns;

[0038] The adaptive contrast learning module is used to calculate adaptive contrast weights based on the cosine similarity between nodes and apply an adaptive contrast loss to enhance node distinguishability in deep convolution;

[0039] The inference module jointly optimizes the graph convolution loss and the adaptive contrast loss after weighting, and performs a prediction task on unlabeled nodes or unknown links after training and outputs the results.

[0040] Third, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method steps of the present invention.

[0041] 3. Beneficial effects

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] A method for suppressing over-smoothing of a deep graph convolutional network according to the present invention optimizes the deep graph convolutional network by using a bilateral constraint mechanism, that is, simultaneously adopting two technical means of random masking and adaptive contrast constraint, and applying constraints to the columns and rows of the graph node representation matrix respectively, so as to not only control the amplitude of neighbor information aggregation, but also enhance the individual distinguishability between nodes, effectively suppressing the problem of over-smoothing of node features generated as the network depth deepens, while taking into account the preservation of node individual information and the full utilization of high-order neighborhood information.

[0044] In addition, the method of the present invention realizes the robustness of the model at different network depths by dynamically adjusting the masking ratio and the contrast constraint weight, and is applicable to a variety of graph neural network architectures (such as GCN, SGC, etc.), with high generality and scalability. Description of the drawings

[0045] Figure 1 Flow chart of the over-smoothing suppression method for the deep graph convolutional network of the present invention;

[0046] Figure 2 Module framework diagram of the system of the present invention; Specific implementation manners

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts fall within the scope of protection of the present invention.

[0048] Combined with Figure 1 , a method for suppressing over-smoothing of a deep graph convolutional network in this embodiment includes the following steps:

[0049] S1. Data preprocessing;

[0050] In this embodiment, the input graph data is preprocessed, including operations such as node feature normalization and missing value filling; subsequently, an initial node representation matrix X is constructed using a multi-layer perceptron or linear mapping. By mapping the original high-dimensional or sparse input to a representation space of a set dimension, the operation burden and overfitting risk during subsequent convolution are reduced, ensuring that the initial features have sufficient discriminability.

[0051] S2. Random masking mechanism processing;

[0052] In each layer of graph convolution, a matrix M(l) is generated according to a preset masking ratio, and the standard convolution operation is combined with partially retaining the information of the previous layer. The following update formula is used for the current node representation matrix H(l):

[0053]

[0054] where L represents the Laplacian matrix after normalization processing, is element-wise multiplication, and M(l) is a randomly generated masking matrix. The masking rate of the random masking mechanism gradually decays as the number of layers increases. In this embodiment, the masking rate is dynamically adjusted according to the number of network layers to ensure that part of the initial information is still retained in the deeper layers, thereby suppressing the over-smoothing phenomenon caused by deep stacking. The masking ratio of each column is adjusted according to the following formula:

[0055] M(l) = 1 - log(λ / (l + 1))

[0056] λ is a hyperparameter that controls the attenuation rate of the mask, and l is the current network layer number; when l is large, the mask rate decreases, which can balance the need for nodes to retain high-order neighbor information and prevent over-convergence. After adjustment, it can ensure that partial shallow-layer information is retained in the deep layer and prevent excessive information aggregation.

[0057] S3. Adaptive contrast constraint: Calculate the cosine similarity between nodes and assign different contrast loss weights according to the similarity. Apply less constraint to similar nodes and stronger discrimination to dissimilar nodes to enhance the individual characteristics of nodes.

[0058] Specifically, the adaptive contrast constraint weight W ij is calculated to satisfy the following principle:

[0059] W ij = 1 - sim(i, j), where sim(i, j) represents the cosine similarity between nodes i and j, and the range is [0, 1]. The higher the similarity, the stronger the similarity of the node pair, that is, W ij tends to 0 and the constraint is lighter. The lower the similarity, W ij tends to 1 and the constraint is stronger, which is used to amplify the feature differences between nodes.

[0060] For node pairs with relatively high cosine similarity, set W ij to a smaller value to reduce the "pulling closer" intensity in the contrast loss, thereby avoiding further over-convergence of already similar nodes.

[0061] For node pairs with relatively low cosine similarity, set W ij to increase relatively, which is used to increase its separation intensity in the contrast loss, maintain or amplify the difference degree, and prevent the node features from becoming consistent in the deep network.

[0062] Introduce contrast learning constraints at each layer or only at the final layer. This constraint uses the positive sample pairs of the same node in adjacent layers or different subgraphs as approximate positive examples, and at the same time uses the representations of other nodes as negative samples, and constructs an adaptive contrast loss function by calculating the cosine similarity sim():

[0063]

[0064] where, is the positive sample pair, (h i , h j ) is the negative sample pair, τ is the temperature coefficient, represents the adaptive weight of the positive sample pair w ij represents the adaptive weight of the negative sample pair (h i , h j ).

[0065] This embodiment achieves automatic balancing of constraint strength between high-similarity nodes by adaptively adjusting the contrast loss weight, thereby both shortening the distance between the same node in different layers and pushing away the features between different nodes, thereby improving the model's discrimination ability.

[0066] S4. Joint optimization training: The graph convolution loss, random mask operation loss and adaptive contrast loss are weighted and fused, and the model is trained using the gradient descent or Adam optimization algorithm to minimize the total loss function (including the weighted sum of the graph convolution reconstruction loss and the contrast loss) to ensure that the overall model maintains prediction accuracy and stability while suppressing over-smoothing.

[0067] This example uses the conventional loss L of the graph convolutional network main With the adaptive contrast loss L contrastive Perform weighted fusion to form the final optimization goal:

[0068] L total =L main +λ c ·L contrastive

[0069] Among them, λ c is the coefficient for balancing the proportion of the two types of losses; L main Cross entropy or mean squared error loss for node classification, link prediction, or other graph tasks.

[0070] It is worth noting that the method of this embodiment, in high-order or deep network scenarios, can optionally combine multi-task learning, graph attention mechanism or layered sampling technology to improve step S2 and step S4, further enhancing the scalability of the present invention and its adaptability to large-scale graph data.

[0071] Specifically, the graph attention mechanism is introduced in step S2 to adjust the mask probability according to the attention weights between nodes, so that the information aggregation process focuses on important neighbor features, thereby enhancing node differences.

[0072] In step S2, a stratified sampling strategy is combined to limit the number of neighbor samples during each layer of propagation and reduce invalid high-order information aggregation.

[0073] In step S4, a multi-task learning mechanism is introduced to jointly optimize the adaptive contrast loss with auxiliary reconstruction tasks (such as adjacency matrix reconstruction or representation alignment) to enhance the model's ability to express individual features and structural relationships.

[0074] S5. Applied reasoning: Apply the trained model to the target task, including node classification, link prediction, or graph representation learning, and evaluate the over-smoothing suppression effect of the model.

[0075] To verify the effectiveness of the method of the present invention in alleviating the over-smoothing problem of deep graph convolutional networks, the following publicly available datasets are selected for experimental evaluation: Cora: It contains 2,708 scientific publication nodes and 5,429 citation edges, divided into 7 categories; Citeseer: It contains 3,327 scientific literature nodes and 4,732 citation relationships, divided into 6 categories; Pubmed: It contains 19,717 nodes and 44,338 edges, divided into 3 categories.

[0076] On the above datasets, the "random masking + adaptive contrast constraint" method proposed by the present invention is compared with the following mainstream baseline models: GCN (basic graph convolutional network), GCNII (deep structure with initial residual), DropEdge (structural sparsification mechanism), TSC (contrastive learning model).

[0077] At a depth of 32 layers, the node classification accuracy of each method

[0078]

[0079]

[0080] Example 2

[0081] Combined with Figure 2 , an over-smoothing suppression system for a deep graph convolutional network in this embodiment, which is used to execute the method steps of Embodiment 1. Specifically, the system includes a data processing module, a graph convolution and random masking module, an adaptive contrast constraint module, and a prediction evaluation module. Data interaction is carried out between the modules through standard interfaces. The overall system can be implemented either as a software solution or through an embedded hardware platform, where:

[0082] The data processing module is used to load the input original graph data, construct an adjacency matrix and a node feature matrix, and can perform necessary linear dimensionality reduction or normalization operations;

[0083] The random masking processing module, combined with the graph convolution process, applies a dynamic masking rate to the column dimension of the node representation matrix of the l-th layer (l≥3) to retain or filter the corresponding feature columns;

[0084] The adaptive contrast learning module is used to calculate the adaptive contrast weight based on the cosine similarity between nodes and apply a contrast loss to enhance the node distinguishability in deep convolution;

[0085] The inference module jointly optimizes the weighted graph convolution loss and the adaptive contrast loss, and after training, performs a prediction task on unlabeled nodes or unknown links and outputs the results.

[0086] Example 3

[0087] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method steps of Embodiment 1.

[0088] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for suppressing over-smoothing of a deep graph convolutional network, characterized in that It includes the following steps: S1. Data preprocessing; S2. Random masking mechanism processing; S3. Adaptive contrast constraint: Calculate the cosine similarity between nodes, and assign different contrast loss weights according to the similarity. Apply less constraint to similar nodes and stronger discrimination to dissimilar nodes to enhance the individual characteristics of nodes; S4. Joint optimization training: Weightedly fuse the graph convolution loss, random masking operation loss, and adaptive contrast loss, and use the gradient descent or Adam optimization algorithm to train the model; S5. Application and inference: Apply the trained model to target tasks, including node classification, link prediction, or graph representation learning, and evaluate the over-smoothing suppression effect of the model.

2. The over-smoothing suppression method for a deep graph convolutional network according to claim 1, characterized in that: In step S1, the input graph data is normalized, the adjacency matrix and node feature matrix are constructed, and the node features are linearly reduced in dimension or standardized; in step S2, in each layer of the graph convolutional network, some columns of the node representation matrix are randomly masked. The masked columns are directly copied from the corresponding columns of the previous layer, while the unmasked columns perform graph convolution operations through the adjacency matrix to reduce the excessive superposition of high-order neighbor information.

3. A method for suppressing over-smoothing of a deep graph convolutional network according to claim 2, characterized in that: When linearly reducing the dimension of the node feature matrix in step S1, any one of the multi-layer perceptron, linear transformation, and low-rank matrix factorization is used to map the original high-dimensional or sparse input to the representation space of the set dimension to reduce the computational burden and overfitting risk during subsequent convolution.

4. A method for suppressing over-smoothing of a deep graph convolutional network according to claim 2, characterized in that: The masking rate of the random masking mechanism gradually decays as the number of layers increases. The masking rate M(l) can be defined according to the following formula: M(l) = 1 - log(λ / (l + 1)) where λ is the hyperparameter controlling the masking decay rate, and l is the current network layer number; when l is large, the masking rate decreases, which can balance the retention of high-order neighbor information by nodes and the need to suppress over-convergence.

5. A method for suppressing over-smoothing of a deep graph convolutional network according to any one of claims 1-4, characterized in that: Adaptive contrast constraint weight W ij The calculation rule is as follows: W ij = 1 - sim(i, j) Among them, sim(i, j) represents the cosine similarity between nodes i and j, with a range of [0, 1]. The higher the similarity, the stronger the similarity of the node pair, that is, W ij tends to 0, the constraint is lighter, the similarity is lower, and W ij tends to 1, the constraint is stronger, which is used to amplify the feature differences between nodes.

6. A method for suppressing over-smoothing of a deep graph convolutional network according to any one of claims 1-4, characterized in that: The adaptive contrast loss adopts the InfoNCE class form, maximizes the similarity of positive sample pairs under adjacent layers or augmented perspectives of the same layer, and minimizes the similarity of negative sample pairs. The calculation formula of the adaptive contrast loss is as follows: Among them, is a positive sample pair, (h i , h j ) is a negative sample pair, τ is the temperature coefficient, w ii+ represents the adaptive weight of the positive sample pair , w ij represents the adaptive weight of the negative sample pair (h i , h j ).

7. A method for suppressing over-smoothing of a deep graph convolutional network according to claim 6, characterized in that: In step 5, the conventional loss L of the graph convolutional network main is weighted and fused with the adaptive contrastive loss L contrastive to form the final optimization objective: L total = L main + λ c · L contrastive Among them, λ c is the coefficient for balancing the proportion of the two types of losses; L main is the cross-entropy or mean squared error loss for node classification, link prediction, or other graph tasks.

8. A method for suppressing over-smoothing of a deep graph convolutional network according to any one of claims 1-4, characterized in that: The graph convolutional network structure can be a standard GCN, an SGC that only retains graph Laplacian propagation, or a GCNII with an initial residual connection.

9. A deep graph convolutional network over-smoothing suppression system, characterized in that: This system is used to execute the method steps described in any one of claims 1-8. Specifically, this system includes a data processing module, a graph convolution and random masking module, an adaptive contrast constraint module, and a prediction and evaluation module. Data interaction is carried out between the modules through standard interfaces, where: The data processing module is used to load the input original graph data, construct the adjacency matrix and node feature matrix, and can perform necessary linear dimension reduction or standardization operations; The random masking processing module, combined with the graph convolution process, applies a dynamic masking rate to the column dimension of the node representation matrix of the l-th layer (l≥3) to retain or filter the corresponding feature columns; The adaptive contrast learning module is used to calculate the adaptive contrast weight based on the cosine similarity between nodes and apply the adaptive contrast loss to enhance the node discrimination in deep convolution; The inference module jointly optimizes the graph convolution loss and the adaptive contrast loss after weighting, and performs prediction tasks on unlabeled nodes or unknown links after training is completed, and outputs the results.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method steps described in any one of claims 1-8.

Images