A graph convolutional neural network regularization method and system based on distribution reshaping

Abstract

The application discloses a graph convolutional neural network regularization method and system based on distribution remodeling, and the method comprises ResNorm regularization processing on output node representation of each convolutional layer in a multi-layer graph convolutional neural network. The ResNorm regularization processing method considers the distribution of graph node degrees, that is, most nodes only have a small number of adjacent nodes, and a small number of nodes have a large number of adjacent nodes. The ResNorm regularization processing method remodels the distribution of variance into a normal distribution, and effectively improves the classification accuracy of low-degree nodes.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a regularization method and system for graph convolutional neural networks based on distribution reshaping. Background Technology

[0002] Regularization methods are commonly used to accelerate neural network training. BatchNorm was the first proposed regularization method and is widely used in computer vision, while LayerNorm is used in natural language processing. In the field of graph convolutional neural networks, GraphNorm adds a learnable parameter to preserve a portion of the mean, as the mean is considered to contain structural information. NodeNorm reduces node variance to address the problem of variance explosion.

[0003] BatchNorm and LayerNorm simply subtract the mean of the node representations; however, this is considered unsuitable for graph representation learning because the mean includes information about the topology. GraphNorm and NodeNorm address this issue. However, neither considers the distribution of graph node degree, where the vast majority of nodes have only a few adjacent nodes, while a minority of nodes have a large number of adjacent nodes. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a method and system for regularizing graph convolutional neural networks based on distribution reshaping, as well as a storage medium.

[0005] A first aspect of the present invention provides a regularization method for graph convolutional neural networks based on distribution reshaping, comprising: characterizing x of the output node of each convolutional layer in a multi-layer graph convolutional neural network. i The ResNorm regularization process is performed, and the specific expression for the ResNorm regularization process is as follows:

[0006]

[0007] Where, x i Let μ represent the output node representation of the i-th convolutional layer, which is a d-dimensional feature vector. i x represents i The mean, σ i x represents i The variance is given by , where a and e are the learnable parameter and the pre-defined parameter, respectively, and can be between 0 and 1.

[0008] A further technical solution of the present invention is: the ResNorm regularization processing can be further optimized as follows:

[0009]

[0010] Where σ max The variance σ represents the variance of all output nodes. i The maximum value, Let λ represent the mean of the r-th root of the variances of all nodes, and let λ represent a hyperparameter greater than 1.

[0011] A further technical solution of the present invention is: the representation matrix of each layer in the graph convolutional neural network after ResNorm regularization is:

[0012] H (l+1) =ResNorm(GCNLayer(H (l+1) ))

[0013] Where GCLNayer represents a graph convolutional layer, GCLNayer(H (l+1) ) = PH (l) W and P are propagation matrices, where W is the learnable parameter matrix and H is the propagation matrix. (l) It is the representation matrix of the l-th layer node.

[0014] A second aspect of the invention is a graph convolutional neural network regularization system based on distribution reshaping, the system being used to characterize x of the output node of each convolutional layer in a multi-layer graph convolutional neural network. i The ResNorm regularization process is performed, and the specific expression for the ResNorm regularization process is as follows:

[0015]

[0016] Where, x i Let μ represent the output node representation of the i-th convolutional layer, which is a d-dimensional feature vector. i x represents i The mean, σ i x represents i The variance of , where a is the learning parameter and e is the hyperparameter.

[0017] A further technical solution of the present invention is: the ResNorm regularization processing can be further optimized as follows:

[0018]

[0019] Where σ max The variance σ represents the variance of all output nodes. i The maximum value, Let λ represent the mean of the r-th root of the variances of all nodes, and let λ represent a hyperparameter greater than 1.

[0020] A further technical solution of the present invention is: the representation matrix of each layer in the graph convolutional neural network after processing by the system is:

[0021] H (l+1) =ResNorm(GCNLayer(H (l+1) ))

[0022] Where GCLNayer represents a graph convolutional layer, GCLNayer(H (l+1) ) = PH (l) W and P are propagation matrices, where W is the learnable parameter matrix and H is the propagation matrix. (l) It is the representation matrix of the l-th layer node.

[0023] A third aspect of the present invention provides a graph convolutional neural network regularization system based on distribution reshaping, comprising: a processor; and a memory, wherein the memory stores a computer-executable program, and when the processor executes the computer-executable program, the above-described graph convolutional neural network regularization method based on distribution reshaping is executed.

[0024] A fourth aspect of the invention is a storage medium having a program stored thereon, which, when executed by a processor, causes the processor to perform the above-described distribution-remodeled graph convolutional neural network regularization.

[0025] This invention provides a graph convolutional neural network regularization method and system based on distribution reshaping, as well as a storage medium. The method includes performing ResNorm regularization on the output node representation of each convolutional layer in a multi-layer graph convolutional neural network. The ResNorm regularization method considers the distribution of graph node degree, that is, most nodes have only a few adjacent nodes, while a few nodes have a large number of adjacent nodes. The ResNorm regularization method reshapes the variance distribution of the output node representation of the convolutional layer into a normal distribution, effectively improving the classification accuracy of low-degree nodes. Attached Figure Description

[0026] Figure 1 This is an architectural diagram of the computer device in an embodiment of the present invention;

[0027] Figure 2a This is a schematic diagram of the variance distribution of the output nodes of the convolutional layer before ResNorm regularization in an embodiment of the present invention.

[0028] Figure 2b This is a schematic diagram of the variance distribution of the output nodes of the convolutional layer after ResNorm regularization in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that in the description of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0030] The present invention provides the following embodiments regarding a regularization method and system for graph convolutional neural networks based on distribution reshaping, and a storage medium:

[0031] Based on Embodiment 1 of the present invention

[0032] This embodiment illustrates a regularization method for graph convolutional neural networks based on distribution reshaping. The method includes: representing x as the output node of each convolutional layer in a multi-layer graph convolutional neural network. i The ResNorm regularization process is performed. The specific expression for ResNorm regularization is as follows:

[0033]

[0034] Where, x i Let μ represent the output node representation of the i-th convolutional layer, which is a d-dimensional feature vector. i x represents i The mean, σ i x represents i The variance is given by , where a is the learning parameter and e is the hyperparameter, i.e., a and e are the learnable parameter and the pre-set parameter, respectively. In the preferred embodiment, both a and e are in the open interval from 0 to 1.

[0035] Furthermore, the ResNorm regularization process can be optimized as follows:

[0036]

[0037] Where σ max The variance σ represents the variance of all output nodes. i The maximum value, Let λ represent the mean of the r-th root of the variances of all nodes, and let λ represent a hyperparameter greater than 1.

[0038] Furthermore, the representation matrix of each layer in the graph convolutional neural network after ResNorm regularization is:

[0039] H (l+1) =ResNorm(GCNLayer(H (l+1) ))

[0040] Where GCLNayer represents a graph convolutional layer, GCLNayer(H (l+1) ) = PH(l) W and P are propagation matrices, where W is the learnable parameter matrix and H is the propagation matrix. (l) It is the representation matrix of the l-th layer node.

[0041] Based on Embodiment 2 of the present invention

[0042] The graph convolutional neural network regularization system based on distribution reshaping provided in Embodiment 2 of this invention is used to characterize x of the output node of each convolutional layer in a multi-layer graph convolutional neural network. i The ResNorm regularization process is performed. The specific expression for ResNorm regularization is as follows:

[0043]

[0044] Where, x i Let μ represent the output node representation of the i-th convolutional layer, which is a d-dimensional feature vector. i x represents i The mean, σ i x represents i The variance is given by , where a and e are parameters with fixed values.

[0045] Furthermore, the ResNorm regularization process can be further optimized as follows:

[0046]

[0047] Where σ max The variance σ represents the variance of all output nodes. i The maximum value, Let λ represent the mean of the r-th root of the variances of all nodes, and let λ represent a hyperparameter greater than 1.

[0048] Furthermore, the representation matrix of each layer in the graph convolutional neural network after system processing is as follows:

[0049] H (l+1) =ResNorm(GCNLayer(H (l+1) ))

[0050] Where GCLNayer represents a graph convolutional layer, GCLNayer(H (l+1) ) = PH (l) W and P are propagation matrices, where W is the learnable parameter matrix and H is the propagation matrix. (l) It is the representation matrix of the l-th layer node.

[0051] Based on Embodiment 3 of the present invention

[0052] The apparatus according to embodiments of the present invention can also be used by means of Figure 1 The architecture of the computing device shown is used to implement this.Figure 1 The architecture of the computing device is shown. For example... Figure 1 As shown, the computer system 101 includes a system bus 103, one or more CPUs 104, input / output 102, and memory 105. Memory 105 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including the distribution-reconstructed graph convolutional neural network regularization method described in any of the above embodiments. Figure 1 The architecture shown is merely exemplary and should be adjusted according to actual needs when implementing different devices. Figure 1 One or more components in.

[0053] Based on Embodiment 4 of the present invention

[0054] This invention also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements any of the distribution-remodeling-based graph convolutional neural network regularization methods described in the above embodiments. The method includes:

[0055] The output node representation x of each convolutional layer in a multi-layer graph convolutional neural network. i The ResNorm regularization process is performed. The specific expression for ResNorm regularization is as follows:

[0056]

[0057] Where, x i Let μ represent the output node representation of the i-th convolutional layer, which is a d-dimensional feature vector. i x represents i The mean, σ i x represents i The variance of , where a is the learning parameter and e is the hyperparameter.

[0058] The computer-readable storage medium of this invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be—but is not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0059] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0060] The program code contained on the storage medium can be transmitted using any suitable medium, including—but not limited to—wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0061] The technical solution of this invention involves applying ResNorm regularization to the output node representation of each convolutional layer in a multi-layer graph convolutional neural network. The ResNorm regularization method considers the distribution of graph node degrees, meaning that most nodes have only a few adjacent nodes, while a few nodes have a large number of adjacent nodes. The ResNorm regularization method reshapes the variance distribution of the convolutional layer output node representation into a distribution similar to a normal distribution. Figure 2a and 2b As shown, Figure 2a , Figure 2b The diagrams show the variance distribution before and after ResNorm regularization, which effectively improves the classification accuracy of low-degree nodes.

[0062] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A regularization method for graph convolutional neural networks based on distribution reshaping, characterized in that, include: Representation of the output node of each convolutional layer in a multi-layer graph convolutional neural network The ResNorm regularization process is performed, and the specific expression for the ResNorm regularization process is as follows: in, Indicates the first i The output node representation of each convolutional layer is a d-dimensional feature vector. express The mean, express variance e For hyperparameters, Represents the variance of all output nodes. The maximum value, Represent the variance of all nodes r The mean after raising it to the power of 1. This indicates a hyperparameter greater than 1; The input to the graph convolutional neural network is text or image data.

2. The graph convolutional neural network regularization method based on distribution reshaping according to claim 1, characterized in that, The representation matrix of each layer in the graph convolutional neural network after ResNorm regularization is: Where GCLRayer represents a graph convolutional layer. , P It is a propagation matrix. W It is a learnable parameter matrix. It is the first The representation matrix of the layer nodes.

3. A graph convolutional neural network regularization system based on distribution reshaping, characterized in that, The system is used to characterize the output nodes of each convolutional layer in a multi-layer graph convolutional neural network. The ResNorm regularization process is performed, and the specific expression for the ResNorm regularization process is as follows: in, Indicates the first i The output node representation of each convolutional layer is a d-dimensional feature vector. express The mean, express variance e For hyperparameters, Represents the variance of all output nodes. The maximum value, Represent the variance of all nodes r The mean after raising it to the power of 1. This indicates a hyperparameter greater than 1; The input to the graph convolutional neural network is text or image data.

4. The graph convolutional neural network regularization system based on distribution reshaping according to claim 3, characterized in that, The representation matrix of each layer in the graph convolutional neural network after processing by the system is as follows: Where GCLRayer represents a graph convolutional layer. , P It is a propagation matrix. W It is a learnable parameter matrix. It is the first The representation matrix of the layer nodes.

5. A graph convolutional neural network regularization system based on distribution reshaping, characterized in that, include: processor; And a memory, wherein the memory stores a computer-executable program, which, when executed by the processor, performs the graph convolutional neural network regularization method based on distribution reshaping as described in any one of claims 1 and 2.

6. A storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the graph convolutional neural network regularization method based on distribution reshaping as described in any one of claims 1 and 2.

Images