Paper classification method based on graph neural network with hybrid expert structure

Abstract

This paper classification method based on a graph neural network with a hybrid expert structure is proposed. The method includes: selecting a target graph dataset based on the current paper classification task and determining an initial graph neural network corresponding to the target graph dataset; determining a hybrid expert structure insertion method and a corresponding target hybrid expert structure based on the type of the initial graph neural network; inserting the target hybrid expert structure into the initial graph neural network according to the insertion method to generate a hybrid expert graph neural network; performing anti-overfitting processing on the training set nodes of the target graph dataset and training the hybrid expert graph neural network using the processed training set; and using the trained hybrid expert graph neural network to classify the text to be classified in the paper classification task. This method widens the graph neural network to improve the neural network's expressive power while reducing the time and space overhead associated with widening the network.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a paper classification method based on a graph neural network with a hybrid expert structure. Background Art

[0002] With the development of artificial intelligence (AI), graph neural networks (GNNs) have been widely used in fields such as natural language processing, image processing, and pharmaceutical medicine. GNNs are powerful tools for studying graph-structured data and can be used to accomplish downstream tasks such as paper classification. In other deep learning fields, increasing the number of parameters in neural network models has been found to enhance their expressive power and facilitate downstream tasks. Therefore, increasing the number of parameters in GNNs to improve their expressive power is attracting increasing attention.

[0003] In the related art, when increasing the number of parameters in graph neural networks, two approaches are typically adopted: deepening the graph neural network or widening the graph neural network. For example, deepening the graph neural network by adding graph convolutional layers. However, the deepening approach in the related art results in a high time cost, and the direct widening of the graph neural network in the related art generally results in a high space cost. Furthermore, increasing the number of parameters in the graph neural network makes it impossible for the graph neural network to complete large-scale and high-precision paper classification tasks.

[0004] Therefore, how to reasonably expand graph neural networks while reducing overhead so as to use them to complete more demanding paper classification tasks has become an urgent problem that needs to be solved. Summary of the Invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the first purpose of the present invention is to propose a paper classification method based on a graph neural network with a hybrid expert structure. The method determines the insertion method of the hybrid expert structure and the corresponding target hybrid expert structure according to the type of the graph neural network, and inserts the target hybrid expert structure into the graph neural network according to the insertion method to generate a hybrid expert graph neural network, thereby improving the expressive ability of the graph neural network by widening the graph neural network without increasing time and space overhead.

[0007] The second object of the present invention is to propose a paper classification system based on a graph neural network with a hybrid expert structure;

[0008] A third object of the present invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first embodiment of the present invention is to propose a paper classification method based on a graph neural network with a hybrid expert structure, the method comprising the following steps:

[0010] Select a target graph dataset based on the current paper classification task and determine the initial graph neural network corresponding to the target graph dataset;

[0011] Determining an insertion method of a hybrid expert structure and a corresponding target hybrid expert structure according to the type of the initial graph neural network, and inserting the target hybrid expert structure into the initial graph neural network according to the insertion method to generate a hybrid expert graph neural network;

[0012] Performing anti-overfitting processing on the training set nodes of the target graph dataset, and training the hybrid expert graph neural network using the processed training set;

[0013] The papers to be classified in the paper classification task are classified by using the trained hybrid expert graph neural network.

[0014] Optionally, in one embodiment of the present invention, the types of the initial graph neural network include: a graph neural network based on a message propagation paradigm and a graph neural network based on a decoupled architecture, and determining a method for inserting a hybrid expert structure according to the type of the initial graph neural network includes:

[0015] For the graph neural network based on the message propagation paradigm, the mixture of experts structure is inserted after each convolutional layer, and a residual connection is added after each mixture of experts structure;

[0016] For the graph neural network based on the decoupled architecture, the target position is found and the hybrid expert structure is inserted into the feature transformation stage, and the hybrid expert structure is inserted between two non-fully connected linear layers.

[0017] Optionally, in one embodiment of the present invention, the hybrid expert structure includes: a gate neural network and multiple expert neural networks, the input dimension and output dimension of the hybrid expert structure are the same, each of the expert neural networks uses a single-layer linear layer or a double-layer multi-layer perceptron (MLP), and the corresponding target hybrid expert structure is determined according to the type of the initial graph neural network, including:

[0018] According to the specific type information of the currently selected initial graph neural network, it is determined whether the corresponding expert neural network adopts the single-layer linear layer or the double-layer multi-layer perceptron (MLP).

[0019] Optionally, in one embodiment of the present invention, the gated neural network scores each of the expert neural networks according to the input to select a target expert neural network to perform forward calculation on the input, and the training of the hybrid expert graph neural network using the processed training set includes:

[0020] During the training process, a random dropout mechanism is used to score the gating neural network.

[0021] Optionally, in one embodiment of the present invention, after inserting the corresponding target hybrid expert structure into the message propagation paradigm-based graph neural network, a hybrid expert graph neural network based on the message propagation paradigm is generated, and the hybrid expert graph neural network based on the message propagation paradigm calculates the input data, including:

[0022] Conventional graph convolution calculation is performed using the following formula:

[0023]

[0024] Among them, GNN represents the graph convolution layer, h (l) represents the lth hidden layer representation of all nodes, G represents the graph structure, represents the representation of the i-th node after passing through the (l+1)-th graph convolution layer;

[0025] The output of the hybrid expert structure is obtained by the following formula:

[0026]

[0027] in,

[0028] Among them, W g are the parameters of the gate network, f j is the jth expert neural network selected, dropout() is the random deactivation function;

[0029] The node representation of the (l+1)th layer is obtained by combining the residual connection with the following formula:

[0030]

[0031] Among them, RES represents the residual network.

[0032] Optionally, in one embodiment of the present invention, the performing anti-overfitting processing on the training set nodes of the target graph dataset includes:

[0033] Pre-set the starting round and stabilization round of the training process;

[0034] Multiply the label of each node in the training set by a dynamic weight, and concatenate the label multiplied by the dynamic weight before the feature of the corresponding node, wherein the dynamic weight is 1 at the starting round and 0 at the stable round, and the dynamic weight increases linearly from the starting round to the stable round.

[0035] Optionally, in one embodiment of the present invention, the dynamic weight is calculated by the following formula:

[0036]

[0037] in,

[0038] Among them, w is the dynamic weight, t1 is the starting round, t2 is the stable round, xi is the feature of node i, yi is the label of node i, is the new feature obtained by concatenating the feature labels of node i, onehot(y i ) means converting the label into a one-hot representation.

[0039] To achieve the above objectives, the second embodiment of the present invention further proposes a multi-condition process industrial fault detection and diagnosis system based on deep transfer learning, including the following modules:

[0040] An insertion module is used to determine an insertion method of a hybrid expert structure and a corresponding target hybrid expert structure according to the type of the initial graph neural network, and insert the target hybrid expert structure into the initial graph neural network according to the insertion method to generate a hybrid expert graph neural network;

[0041] A training module, configured to perform anti-overfitting processing on the training set nodes of the target graph dataset, and train the hybrid expert graph neural network using the processed training set;

[0042] The classification module is used to classify the papers to be classified in the paper classification task through the trained hybrid expert graph neural network.

[0043] Optionally, in one embodiment of the present invention, the types of initial graph neural networks include: graph neural networks based on the message propagation paradigm and graph neural networks based on the decoupled architecture, and the insertion module is specifically used to: for the graph neural network based on the message propagation paradigm, insert the hybrid expert structure after each convolution layer, and add a residual connection after each hybrid expert structure; for the graph neural network based on the decoupled architecture, find the target position to insert the hybrid expert structure in the feature transformation stage, and insert the hybrid expert structure between two non-fully connected linear layers.

[0044] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects: the present invention expands the number of parameters of the graph neural network from the perspective of widening the graph neural network, avoiding the excessively high time complexity of the deep neural network, and uses a hybrid expert structure to widen the neural network. The generated hybrid expert graph neural network performs the paper classification task, which can effectively improve the model's expressiveness without generating excessively high space complexity, and reduce the time and space overhead brought by widening the network. In addition, the "label warm-up" strategy proposed by the present invention prevents overfitting, effectively avoiding overfitting caused by the technique of explicitly utilizing labels, so that the technique can be extended to multi-task graph learning problems. Furthermore, the hybrid expert graph neural network completed through training can perform larger-scale and higher-precision paper classification tasks, improving the accuracy and practicality of paper classification.

[0045] In order to implement the above-mentioned embodiments, the third aspect of the present invention further proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the paper classification method based on the graph neural network with a hybrid expert structure in the above-mentioned embodiment is implemented.

[0046] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0048] Figure 1 This is a flowchart of a paper classification method based on a graph neural network with a hybrid expert structure proposed in an embodiment of the present invention;

[0049] Figure 2 A flowchart of a hybrid expert structure insertion method for a graph neural network for a message propagation paradigm proposed in an embodiment of the present invention;

[0050] Figure 3 A flowchart of a hybrid expert structure insertion method for a graph neural network with a decoupled architecture proposed in an embodiment of the present invention;

[0051] Figure 4 A structural diagram of a paper classification system based on a graph neural network with a hybrid expert structure proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0053] The following describes a method and system for paper classification based on a graph neural network with a hybrid expert structure proposed in an embodiment of the present invention with reference to the accompanying drawings.

[0054] It should be noted that for neural networks, a larger number of parameters usually means better results, so the number of neural network parameters continues to expand. Since AlexNet, models such as ResNet (residual network), Bert (transformer-based bidirectional encoder representation technology), GPT-3 (text generation pre-training model) have pushed the number of parameters of neural networks to a new level time and time again. At present, there are two ways to expand the number of parameters of a neural network, one is deepening and the other is widening. The two methods of expanding the number of parameters, widening and deepening, are compared in the embodiments of the present application, and it is found that widening is better than deepening under certain circumstances.

[0055] In the field of graph learning, graph neural networks have become the most powerful tool. On most common benchmark graph datasets (such as Cora, CiteSeer, PubMed, ogbn-arxiv, etc.), graph neural networks with conventional parameter amounts can achieve good results, such as a model with 3-4 layers of graph convolutional layers and 512 hidden neurons. This makes the number of graph neural network parameters less concerned than in fields such as natural language processing and computer vision. However, in this embodiment, it was found that on certain benchmark graph datasets (such as ogbn-proteins, ogbn-mag, etc.), the number of model parameters also showed a significant positive correlation with the model effect. Therefore, for graph neural networks, expanding the number of parameters is also very important.

[0056] Currently, methods for expanding the number of parameters in graph neural networks in related technologies are basically focused on deepening graph neural networks. The main method is to add graph convolutional layers, including DeepGCN (deep graph convolutional network), DeeperGCN (deeper graph convolutional network), RevGNN (reversible graph neural network), etc. However, when graph neural networks are very deep, they usually face problems such as gradient vanishing, over-smoothing, and high time and space overhead. Although some research in related technologies has been able to avoid the problems of gradient vanishing and over-smoothing to a certain extent, the high time overhead cannot be solved. For example, training a 1000-layer RevGNN on an NVIDIA V100 card can even take 13.5 days.

[0057] In terms of widening graph neural networks, most researchers only directly widen graph neural networks, but no systematic research has been conducted on widening graph neural networks. This results in high space overhead for widening graph neural networks.

[0058] To this end, this application proposes a paper classification method based on a graph neural network with a hybrid expert structure. This application uses a mixture of experts (MoE) structure to widen the model width and achieve excellent results. Replacing a single network with multiple expert networks can implicitly widen the neural network and increase the number of parameters. Therefore, in the embodiments of the present invention, a hybrid expert structure is added to the graph neural network to expand the number of parameters of the graph neural network model, so that it can better solve the paper classification task.

[0059] Figure 1 This is a flowchart of a paper classification method based on a graph neural network with a hybrid expert structure proposed in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0060] Step S101: select a target graph dataset according to the current paper classification task, and determine the initial graph neural network corresponding to the target graph dataset.

[0061] The graph neural networks targeted by this invention are neural networks applied to graph-structured data. Graph-structured data refers to network-like graphs composed of nodes and their connections, such as social networks and paper citation networks. Graph neural networks typically consist of multiple graph convolutional layers, the basic structural unit of graph neural networks. Each graph convolutional layer typically updates a node's hidden vector by aggregating the features or hidden vectors of neighboring nodes. Downstream tasks are completed based on the node vector generated by the last graph convolutional layer.

[0062] The target graph dataset can be any existing benchmark graph dataset, such as the ogbn-proteins dataset and the PPI dataset. This application selects a benchmark graph dataset that is more suitable for the paper classification task to be performed as the target graph dataset.

[0063] In embodiments of the present invention, the paper classification task can be to label a large number of existing papers according to predefined category labels. The paper classification task can first perform text recognition and then classify the large amount of text according to corresponding rules, for example, by classifying each paper according to its field.

[0064] It should be noted that, in other embodiments of the present invention, the hybrid expert graph neural network proposed in the present invention can also be applied to other tasks. For example, it can also be used to classify a large number of received emails and filter out spam. For another example, the paper classification can also be used to classify a large amount of news information and recommend content that may be of interest to users. Among them, the main logic of the present invention in executing various downstream tasks is similar. The hybrid expert graph neural network is generated according to the specific circumstances of the task to be executed, and then the trained hybrid expert graph neural network is used to execute various downstream tasks. The specific application scenarios are not limited here. The implementation method of executing each downstream task can refer to the paper classification method of the present invention.

[0065] Specifically, this application first selects a target graph dataset and an existing graph neural network (i.e., an initial graph neural network) corresponding to the dataset based on the current paper classification task.

[0066] It should be noted that in some other embodiments of the present application, graph neural networks can also be used to perform tasks in other fields, such as relationship extraction, entity recognition, and multi-hop reading in natural language processing, compound screening, protein interaction point detection, etc. in the biomedical field, and information recommendation and risk control in the industrial field. In the embodiments of the present application, the required graph data set and the initial graph neural network to be optimized can be selected based on the tasks in different fields to be performed and the specific type of the current task (for example, whether the current paper classification task is multi-task node classification or single-task node classification).

[0067] For example, when the received task is to predict protein function in a multi-label binary classification system, the ogbn-proteins dataset is selected as the target graph dataset, and on the graph neural network ranking list corresponding to the dataset, the corresponding graph neural network is selected as the initial graph neural network based on the specific characteristics and constraints of the current task, such as the limitation of training time.

[0068] Step S102: Determine the insertion method of the hybrid expert structure and the corresponding target hybrid expert structure according to the type of the initial graph neural network, and insert the target hybrid expert structure into the initial graph neural network according to the insertion method to generate a hybrid expert graph neural network.

[0069] The hybrid expert architecture consists of a series of expert neural networks and a gate neural network. All expert networks have the same input and output dimensions. The gate network selects experts based on the input to perform forward computations on the input and then aggregates the experts' outputs. Replacing a single network with multiple expert networks implicitly widens the neural network and increases the number of parameters. The hybrid expert graph neural network architecture can be integrated into any graph neural network structure, requiring only modifications to the existing structure.

[0070] Specifically, this application adds a hybrid expert structure MoE to the selected initial graph neural network. Since the input and output dimensions of MoE are the same, it will not affect other structures of the original initial graph neural network. The insertion method of the hybrid expert structure and the corresponding target hybrid expert structure are determined according to the type of the initial graph neural network.

[0071] In one embodiment of the present application, the types of initial graph neural networks include: a graph neural network based on the message propagation paradigm and a graph neural network based on the decoupling architecture. In addition, the hybrid expert structure includes: a gate neural network and multiple expert neural networks. The input dimension and output dimension of the hybrid expert structure are the same, and each expert neural network uses a single-layer linear layer or a double-layer multi-layer perceptron (MLP). MLP is a feedforward artificial neural network model that can map multiple input data sets to a single output data set.

[0072] Specifically, the graph neural network based on the message propagation paradigm is composed of several graph convolutional layers. Each graph convolutional layer updates the hidden vector of the node by aggregating the features or hidden vectors of neighboring nodes. The downstream task (such as node classification task) is completed according to the node vector generated by the last graph convolutional layer. The decoupled graph neural network divides feature propagation and feature transformation into two independent stages, and the former is performed in the preprocessing stage. First, data preprocessing is performed, and node features are propagated on the graph without parameters. Then, the feature transformation stage is entered, and each node sends the preprocessed features to the feedforward neural network to obtain output. This application determines the insertion method of the hybrid expert structure and the specific structure of the inserted hybrid expert layer for these two types of graph neural networks.

[0073] In this embodiment, the method of inserting the hybrid expert structure is determined according to the type of the initial graph neural network, including:

[0074] For graph neural networks based on the message propagation paradigm, a mixture of experts structure is inserted after each convolutional layer, and a residual connection is added after each mixture of experts structure;

[0075] For graph neural networks based on decoupled architecture, the target position is found and the hybrid expert structure is inserted in the feature transformation stage, and the hybrid expert structure is inserted between two non-fully connected linear layers.

[0076] Furthermore, in an embodiment of the present invention, determining a corresponding target hybrid expert structure according to the type of the initial graph neural network includes:

[0077] Based on the specific type information of the currently selected initial graph neural network, it is determined whether the corresponding expert neural network uses a single-layer linear layer or a double-layer multilayer perceptron (MLP). Furthermore, in an embodiment of the present invention, after inserting the hybrid expert structure, during the training and subsequent practical application of the generated hybrid expert graph neural network, the gate neural network scores each expert neural network based on the input to select the target expert neural network for forward calculation of the input. Specifically, the gate network of the MoE uses a single-layer linear layer. When selecting the expert network, only one expert is selected, that is, the expert with the highest score in the gate network, and the output of this expert is used as the output of the MoE. During the training process, the phenomenon of expert allocation polarization sometimes occurs, that is, most of the input is allocated to a small number of experts. As training progresses, these experts will be more fully trained, becoming more powerful and scoring higher, and the expert allocation will become further polarized. To avoid this phenomenon, the embodiment of the present invention adopts a dropout (random deactivation) mechanism for scoring the gate network, so that experts who do not have the highest scores also have the opportunity to be selected. That is, in an embodiment of the present invention, the hybrid expert graph neural network is trained using the processed training set, including:

[0078] During the training process, a random dropout mechanism is used to score the gated neural network.

[0079] Based on the above content, the following describes the specific implementation methods of how to insert hybrid experts into graph neural networks based on the message propagation paradigm and graph neural networks based on the decoupled architecture.

[0080] In one embodiment of the present application, for the first type of graph neural network, that is, the graph neural network based on the message propagation paradigm, the way to insert MoE is to insert it after the graph convolution layer. In order to avoid problems such as gradient disappearance caused by adding new layers, residual connections are usually added after MoE. In order to more clearly explain the hybrid expert structure insertion method of the graph neural network based on the message propagation paradigm in the present invention, the following is Figure 2 As an illustration. Figure 2 This is a flow chart of a hybrid expert structure insertion method based on a graph neural network with a message propagation paradigm provided by an embodiment of the present invention. Figure 2 As shown, the method includes:

[0081] Step S201, calculating layer convolution.

[0082] Among them, the conventional graph convolution calculation is performed by the following formula:

[0083]

[0084] Among them, GNN represents the graph convolution layer, h (l)represents the lth hidden layer representation of all nodes, G represents the graph structure, represents the representation of the i-th node after passing through the (l+1)-th graph convolution layer;

[0085] Step S202: Calculate the output of the hybrid expert structure.

[0086] Among them, the output of the hybrid expert structure is obtained by the following formula:

[0087]

[0088] in,

[0089] Among them, W g are the parameters of the gate network, f j is the jth expert neural network selected, dropout() is the random deactivation function;

[0090] Step S203: Acquire node representation by combining residual connections.

[0091] Among them, the node representation of the (l+1) layer can be obtained by combining the residual connection with the following formula:

[0092]

[0093] Among them, RES represents the residual network, where the residual network is usually a single linear layer.

[0094] To facilitate understanding of the implementation process of the above method in practical applications, two examples are described below. As the first example, GAT (graph attention network) + BoT (bag of tricks on graph neural network) + NGNN (neural network in graph neural network) on the ogbn-proteins dataset can be applied to the hybrid expert structure of the graph neural network based on the message propagation paradigm. During the application process, NGNN needs to be replaced by adding a MoE layer after each graph convolution layer, where MoE uses a total of 16 experts, and the dropout (random inactivation) is set to 0 when selecting experts. Afterwards, it is determined that the corresponding expert neural network is a two-layer MLP, and the vector dimension of the middle layer is set to twice the dimension of the input / output layer. The activation function uses a conventional function, adding GeLU (Gaussian error linear function) between the two layers of the MLP, and adding ReLU (rectified linear function) after the two layers. Based on this, most of the other hyperparameters of the original model were retained, except for increasing the number of training rounds from 1200 to 2000, increasing the input dropout (random inactivation) from 0.1 to 0.2, and reducing the learning rate from 0.008 to 0.001.

[0095] As a second example, on the PPI dataset, the present invention applies the hybrid expert graph neural network to the currently top-ranked model, Graph Convolutional Network II (GCNII). The GCNII model consists of 9 layers of graph convolutional layers. The present invention adds a MoE layer after the middle 7 layers of GCNII. A total of 4 experts are used, and the dropout (random deactivation) of the selected experts is set to 0.2. The best expert network structure is a single linear layer. Except for reducing the learning rate from 0.001 to 0.0001, all other hyperparameters are the same as the original model.

[0096] In one embodiment of the present application, for the second type of graph neural network, that is, the decoupled graph neural network, it is necessary to find a suitable position to insert the MoE (mixed expert) in the feature transformation stage. Usually, the MoE is suitable for insertion between two non-fully connected linear layers, because the MoE itself is similar to a fully connected linear layer, and if it is put together with the same type, the effect may not be obvious. In order to more clearly explain the hybrid expert structure insertion method of the graph neural network based on the decoupled architecture in the present invention, the following is Figure 3 As an illustration. Figure 3 This is a flow chart of a hybrid expert structure insertion method based on a decoupled graph neural network architecture provided by an embodiment of the present invention. Figure 3 As shown, the method includes:

[0097] Step S301, performing feature propagation of data preprocessing;

[0098] Step S302, performing feature transformation;

[0099] Step S303: insert the hybrid expert graph neural network into the feature transformation and continue the feature transformation.

[0100] For example, in the ogbn-mag dataset, the present invention applies the SeHGNN (simple and efficient heterogeneous graph neural network) + ComplEx (complex graph data pre-training) model to the hybrid expert structure of the graph neural network based on the decoupled architecture. In the feature transformation stage, the SeHGNN model uses two 1-dimensional convolution operations on the preprocessed features and labels respectively. Then, in the feature transformation process, two MoEs are inserted at appropriate positions, respectively between the two 1-dimensional convolutions for features and the two 1-dimensional convolutions for labels, and the expert network structure is determined to be a single-layer linear layer. Four experts can be used, and the dropout (random inactivation) of the selected experts is set to 0. Based on this, the number of rounds in each training stage can be increased from 300 to 500, and the other hyperparameters are all based on the original model.

[0101] Step S103: Anti-overfitting processing is performed on the training set nodes of the target graph data set, and the hybrid expert graph neural network is trained using the processed training set.

[0102] It's worth noting that a common technique for graph neural network models is to explicitly use labels and propagate them using graph convolutions. The Bag of Tricks (BoT) system on graph neural networks investigates this technique. The approach involves first dividing the training set nodes into two categories: label-providing nodes and label-predicting nodes. For label-providing nodes, their labels are concatenated with their features and fed into the graph neural network. For label-predicting nodes, their labels are used only for loss calculation and backpropagation. During the validation and testing phases, the labels of all training set nodes are concatenated with their features and fed into the model. This technique allows the graph neural network to learn how to leverage the labels of nearby nodes to make predictions about the node. While this technique has shown promising results on single-task datasets, it has been largely unsuccessful on multi-task datasets. This is because multi-task datasets often have many tasks, often far more than the number of feature dimensions (for example, the ogbn-proteins dataset has 112 tasks but only 8 dimensions of features). This technique can cause the model to overfocus on label information and ignore features, leading to overfitting.

[0103] To this end, the present invention performs anti-overfitting processing on the training set nodes of the target graph data set. In one embodiment of the present invention, anti-overfitting processing is performed through a "label warm-up" strategy. The "label warm-up" strategy aims to extend the technique of explicit label propagation to the multi-task node classification problem. To avoid overfitting, the embodiment of the present invention multiplies the labels of the training set nodes by a certain weight before splicing them into the features. This strategy is set with a "starting round" and a "stabilizing round". The weight is 0 before the "starting round", increases linearly from the "starting round" to the "stabilizing round", and is fixed to 1 after the "stabilizing round". This allows the model to first focus on utilizing feature information, and first converge to a local optimal point that can make good use of the features. After that, it gradually begins to pay attention to label information, and converges to a local optimal point near the first local optimal point that can make good use of both feature and label information.

[0104] Specifically, in an embodiment of the present invention, anti-overfitting processing is performed on the training set nodes of the target graph dataset, including:

[0105] Pre-set the starting round and stabilization round of the training process;

[0106] Multiply the label of each node in the training set by the dynamic weight, and concatenate the label after multiplying the dynamic weight before the feature of the corresponding node. The dynamic weight is 1 at the beginning of the round and 0 at the stable round. The dynamic weight increases linearly from the beginning of the round to the stable round.

[0107] And, in an embodiment of the present invention, the dynamic weight is calculated by the following formula:

[0108]

[0109] in,

[0110] Among them, w is the dynamic weight, t1 is the starting round, t2 is the stable round, x i is the feature of node i, y i is the label of node i, is the new feature obtained by concatenating the feature labels of node i, onehot(y i ) means converting the label into a one-hot representation and then concatenating the representations of all tasks in sequence.

[0111] For example, we applied the label warm-up strategy to two multi-task node classification datasets, ogbn-proteins and PPI. On ogbn-proteins, the "starting round" and "stabilization" values ​​were set to 1400 and 1800, respectively. On PPI, both the "starting round" and "stabilization" values ​​were set to 2000.

[0112] Step S104: classify the papers to be classified in the paper classification task through the trained hybrid expert graph neural network.

[0113] Specifically, the trained hybrid expert graph neural network executes the received tasks to classify a large number of papers to be classified in the paper classification task. For example, the large number of paper texts to be classified in the paper classification task are divided into papers in different technical field categories such as "machine learning", "computer vision" and "natural language processing".

[0114] In one embodiment of the present invention, before the trained hybrid expert graph neural network performs the paper classification task, the hybrid expert graph neural network can also be fine-tuned in combination with the data set of the current paper classification task.

[0115] In order to more fully illustrate the effect of the paper classification method based on the graph neural network with a hybrid expert structure of the present invention, experimental verification is also carried out in the embodiments of the present invention.

[0116] Specifically, on the ogbn-proteins ranking list, the present invention is applied to the GAT+BoT+NGNN (graph attention network + bag of tricks on graph neural network + neural network in graph neural network) model, which was originally ranked second. The ROC-AUC score (true and false positive and negative rate score) of the GAT+BoT+NGNN model on the test set is 88.09%. If the "hybrid expert graph neural network" technology is directly applied without changing the dimension of the middle layer of the expert network, the score can reach 88.32%. If the "label preheating" strategy is further used, the score can reach 88.48%. If the dimension of the middle layer of the expert network is further expanded to 2 times, the score can reach 88.57% when the labels are not explicitly used, and the score can reach 88.68% when the labels are explicitly used using the "label preheating" strategy. In contrast, if the labels are directly used explicitly without the "label preheating" strategy, the score will drop to 86.96%.

[0117] On the ogbn-mag rankings, this invention was applied to the SeHGNN+ComplEx (Simple and Efficient Heterogeneous Graph Neural Network + Complex Graph Data Pre-training) model, which previously ranked first. The original model had an accuracy of 57.19% on the test set. After applying the "Hybrid Expert Graph Neural Network" architecture, the test set accuracy reached 57.39%.

[0118] In the PPI rankings, the present invention is applied to the GCNII (Graph Neural Network II) model, which was originally ranked second. The test set F1 score of the original model is 99.56%. If the "hybrid expert graph neural network" structure or the "label preheating" strategy is applied separately, no significant improvement can be achieved. This may be because the graph of this dataset is relatively small. It is an inductive learning dataset consisting of 24 graphs, and the average number of nodes in the graph is only 2372.7. And the node features are only 50-dimensional, so this dataset does not have very high requirements for the expressive ability of the model. However, if the "hybrid expert graph neural network" structure or the "label preheating" strategy is applied at the same time, the test set F1 score can reach 99.63. This is because the explicit use of 121-dimensional labels can bring rich new information, which makes the model's stronger expressive ability useful.

[0119] In summary, the paper classification method based on the graph neural network with a hybrid expert structure in the embodiment of the present invention expands the number of parameters of the graph neural network from the perspective of widening the graph neural network, avoiding the excessive time complexity of the deep neural network, and uses the hybrid expert structure to widen the neural network. The generated hybrid expert graph neural network performs the paper classification task, which can effectively improve the model's expressiveness without generating excessive space complexity, reducing the time and space overhead brought by widening the network. In addition, the "label warm-up" strategy proposed by this method prevents overfitting, effectively avoiding overfitting caused by the technique of explicitly utilizing labels, so that this technique can be extended to multi-task graph learning problems. Furthermore, the hybrid expert graph neural network completed through training can perform larger-scale and higher-precision paper classification tasks, improving the accuracy and practicality of paper classification.

[0120] Figure 4 This is a schematic diagram of the structure of a paper classification system based on a graph neural network with a hybrid expert structure provided in this embodiment. Figure 4 As shown, the paper classification system based on the graph neural network with a hybrid expert structure includes: a determination module 100, an insertion module 200, a training module 300 and a classification module 400. Among them,

[0121] Determination module 100, for selecting a target graph dataset according to the current paper classification task and determining an initial graph neural network corresponding to the target graph dataset;

[0122] An insertion module 200 is used to determine an insertion method of the hybrid expert structure and a corresponding target hybrid expert structure according to the type of the initial graph neural network, and insert the target hybrid expert structure into the initial graph neural network according to the insertion method to generate a hybrid expert graph neural network;

[0123] A training module 300 is used to perform anti-overfitting processing on the training set nodes of the target graph dataset and train the hybrid expert graph neural network using the processed training set;

[0124] The classification module 400 is used to classify the papers to be classified in the paper classification task through the trained hybrid expert graph neural network.

[0125] Optionally, in one embodiment of the present invention, the types of initial graph neural networks include: graph neural networks based on the message propagation paradigm and graph neural networks based on the decoupled architecture, and the insertion module is specifically used to: for the graph neural network based on the message propagation paradigm, insert the hybrid expert structure after each convolution layer, and add a residual connection after each hybrid expert structure; for the graph neural network based on the decoupled architecture, find the target position to insert the hybrid expert structure in the feature transformation stage, and insert the hybrid expert structure between two non-fully connected linear layers.

[0126] It should be noted that the above explanation of the embodiment of the paper classification method based on the graph neural network with a hybrid expert structure is also applicable to the system of this embodiment. Please refer to the relevant description of the above embodiment and will not be repeated here.

[0127] In summary, the paper classification system based on the graph neural network with a hybrid expert structure in the embodiment of the present invention expands the number of parameters of the graph neural network from the perspective of widening the graph neural network, avoiding the excessive time complexity of the deep neural network, and uses the hybrid expert structure to widen the neural network. The generated hybrid expert graph neural network performs the paper classification task, which can effectively improve the model's expressiveness without generating excessive space complexity, reducing the time and space overhead brought by widening the network. In addition, the system effectively avoids overfitting caused by the technique of explicitly utilizing labels, so that the technique can be extended to multi-task graph learning problems. Furthermore, the hybrid expert graph neural network completed through training can perform larger-scale and higher-precision paper classification tasks, improving the accuracy and practicality of paper classification.

[0128] In order to implement the above embodiments, the present invention also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the paper classification method based on a graph neural network with a hybrid expert structure as described in any of the above embodiments.

[0129] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0130] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0131] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0132] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0133] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0134] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0135] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0136] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A paper classification method based on a graph neural network with a hybrid expert structure, characterized by: The following steps are involved: Select a target graph dataset based on the current paper classification task and determine the initial graph neural network corresponding to the target graph dataset; Determining an insertion method of a hybrid expert structure and a corresponding target hybrid expert structure according to the type of the initial graph neural network, and inserting the target hybrid expert structure into the initial graph neural network according to the insertion method to generate a hybrid expert graph neural network; Performing anti-overfitting processing on the training set nodes of the target graph dataset, and training the hybrid expert graph neural network using the processed training set; The papers to be classified in the paper classification task are classified by using the trained hybrid expert graph neural network.

2. The method according to claim 1, characterized in that The types of the initial graph neural network include: a graph neural network based on a message propagation paradigm and a graph neural network based on a decoupled architecture. The method of inserting the hybrid expert structure according to the type of the initial graph neural network includes: For the graph neural network based on the message propagation paradigm, the mixture of experts structure is inserted after each convolutional layer, and a residual connection is added after each mixture of experts structure; For the graph neural network based on the decoupled architecture, the target position is found and the hybrid expert structure is inserted into the feature transformation stage, and the hybrid expert structure is inserted between two non-fully connected linear layers.

3. The method according to claim 2, characterized in that The hybrid expert structure includes: a gate neural network and multiple expert neural networks. The input dimension and output dimension of the hybrid expert structure are the same. Each of the expert neural networks uses a single-layer linear layer or a double-layer multi-layer perceptron (MLP). The corresponding target hybrid expert structure is determined according to the type of the initial graph neural network, including: According to the specific type information of the currently selected initial graph neural network, it is determined whether the corresponding expert neural network adopts the single-layer linear layer or the double-layer multi-layer perceptron (MLP).

4. The method according to claim 3, characterized in that The gate neural network scores each of the expert neural networks according to the input to select a target expert neural network to perform forward calculation on the input, and the hybrid expert graph neural network is trained using the processed training set, including: During the training process, a random dropout mechanism is used to score the gating neural network.

5. The method according to claim 3, characterized in that After inserting the corresponding target hybrid expert structure into the graph neural network based on the message propagation paradigm, a hybrid expert graph neural network based on the message propagation paradigm is generated. The hybrid expert graph neural network based on the message propagation paradigm calculates the input data, including: Conventional graph convolution calculation is performed using the following formula: Among them, GNN represents the graph convolution layer, h (l) represents the lth hidden layer representation of all nodes, G represents the graph structure, represents the representation of the i-th node after passing through the (l+1)-th graph convolution layer; The output of the hybrid expert structure is obtained by the following formula: in, Among them, W g are the parameters of the gate network, f j is the jth expert neural network selected, dropout() is the random deactivation function; The node representation of the (l+1)th layer is obtained by combining the residual connection with the following formula: Among them, RES represents the residual network.

6. The method according to claim 1, characterized in that The anti-overfitting processing of the training set nodes of the target graph dataset includes: Pre-set the starting round and stabilization round of the training process; Multiply the label of each node in the training set by a dynamic weight, and concatenate the label multiplied by the dynamic weight before the feature of the corresponding node, wherein the dynamic weight is 1 at the starting round and 0 at the stable round, and the dynamic weight increases linearly from the starting round to the stable round.

7. The method according to claim 6, characterized in that The dynamic weight is calculated by the following formula: in, Among them, w is the dynamic weight, t1 is the starting round, t2 is the stable round, x i is the feature of node i, y i is the label of node i, is the new feature obtained by concatenating the feature labels of node i, onehot(y i ) means converting the label into a one-hot representation.

8. A paper classification system based on a graph neural network with a hybrid expert structure, characterized by: include: A determination module is used to select a target graph dataset based on the current paper classification task and determine the initial graph neural network corresponding to the target graph dataset; An insertion module is used to determine an insertion method of a hybrid expert structure and a corresponding target hybrid expert structure according to the type of the initial graph neural network, and insert the target hybrid expert structure into the initial graph neural network according to the insertion method to generate a hybrid expert graph neural network; A training module, configured to perform anti-overfitting processing on the training set nodes of the target graph dataset, and train the hybrid expert graph neural network using the processed training set; The classification module is used to classify the papers to be classified in the paper classification task through the trained hybrid expert graph neural network.

9. The system according to claim 8, characterized in that The types of the initial graph neural network include: a graph neural network based on a message propagation paradigm and a graph neural network based on a decoupled architecture. The insertion module is specifically used to: For the graph neural network based on the message propagation paradigm, the mixture of experts structure is inserted after each convolutional layer, and a residual connection is added after each mixture of experts structure; For the graph neural network based on the decoupled architecture, the target position is found and the hybrid expert structure is inserted into the feature transformation stage, and the hybrid expert structure is inserted between two non-fully connected linear layers.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the paper classification method based on a graph neural network with a hybrid expert structure as described in any one of claims 1 to 7.

Images