General graph task method based on large language model

Abstract

The invention relates to a general graph task method based on a large language model (LLM), which comprises the following steps of: firstly, serializing graph structure data into input suitable for the LLM, and injecting the structure information of a graph into a model through two codes, namely, RWPE (Random Walk Position Element) and Laplacian Position Element (LAPE) based on Laplacian decomposition to finish a general serialization process. Then, according to different tasks, different serializations are carried out, including subgraph segmentation and arrangement in a graph classification task and local domain sampling in a node classification task; after completely serialized data is obtained, knowledge injection is carried out in a mode of freezing and fine tuning a pre-trained large model, finally, the framework obtains excellent performance in the graph task field (graph classification and node classification), the framework effect is verified on multiple data sets, meanwhile, an interpretability experiment and a small sample scene verification experiment are carried out, and the method is suitable for large-scale application. The superiority of the frame is further proved.

Description

Technical Field

[0001] The invention relates to a general graph task method based on a large language model, belonging to the technical field of pattern recognition in computer intelligence science. Background Art

[0002] As a widely used and deeply influential data structure, graphs occupy an important position in data analysis and processing in various fields today. In the real world, there are many application scenarios of graph data. For example, in social networks, nodes usually represent social users and edges usually represent the friendships between users. By analyzing social network data, users' behavioral preferences can be obtained, so as to make more accurate social recommendations for users. In protein networks, nodes represent proteins and edges usually represent the interaction between proteins. By analyzing protein network data, the interaction relationship between different proteins can be studied, which is helpful to study the molecular mechanism of diseases and discover new drug targets from a systematic perspective. With the advent of the big data era, the scale of graph data is becoming larger and larger, and the potential information contained is becoming richer and richer. Therefore, accurately and efficiently mining graph data and rationally using the information in the data can effectively promote industrial development, improve the quality of life and increase economic benefits.

[0003] Recently, with the rise of pre-trained large models, people's attention to the general methods of basic models has reached a climax. For example, SAM (Segment Anything Model) in computer vision obtains a high level of understanding of image content by learning the underlying features in large-scale unlabeled or weakly labeled data sets, enabling it to effectively identify and segment various types of objects; ChatGPT in natural language processing is based on the Transformer architecture and has been trained with a large amount of text data. It can not only generate smooth, coherent and contextually logical conversations, but also complete a variety of complex tasks such as code writing, problem answering, and text creation. The successful application of ChatGPT shows how pre-trained large models can bring the interactivity and practicality of artificial intelligence to a new level, which makes it possible to construct large graph models through domain migration.

[0004] At present, some cutting-edge research is also focusing on how to use LLM (Large Language Model) to solve related tasks in the graph field. Some researchers have designed corresponding prompt texts for various tasks with implicit graph structures, and obtained reasoning results only based on the output text generated by the large model. Their work has demonstrated the powerful potential of large language models in the graph field. However, they did not take into account the complex attribute characteristics and topological characteristics of the graph. The method they proposed is only effective for simple graph tasks such as logical sorting, maximum flow, and Hamiltonian circuits. Therefore, (1) the complex attribute characteristics and topological characteristics of the graph are not considered, and (2) most graph tasks do not have corresponding text features. These two key problems make it difficult for existing large language models to be directly applied to complex graph tasks. Summary of the invention

[0005] To solve the above problems, the present invention proposes a general graph task method based on a large language model, which can utilize the capabilities of the large language model to complete complex graph classification and node classification tasks, and has acceptable efficiency while ensuring accuracy.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A general graph task method based on a large language model, the method comprising the following steps:

[0008] Step 1: Given graph structure data G = (V, A, E), where V = v1, v2, v3, …, v n is a node set, n = |V|, E is an edge set, and the corresponding adjacency matrix A∈{0,1} n×n , when there is an edge connection between node i and node j, A ij =1, otherwise A ij = 0. First, the graph structure data is generally serialized using the improved random walk positional encoding RWPE (Random Walk Positional Embedding) or the Laplace decomposition-based positional encoding LAPE (Laplacian Positional Embedding). The construction and serialization process of the two positional encodings are as follows.

[0009] Step 1-1, RWPE position encoding construction, given a graph G containing n nodes m , the RWPE position encoding is defined using a k-step random walk:

[0010]

[0011] Where RW=AD -1 is the random walk operator, p irepresents the random walk position encoding of the i-th node, represents the position encoding of the mth graph. Here, only the probability of each node wandering to itself is considered. When k is large enough, RWPE provides a unique node representation for each node.

[0012] Step 1-2, LAPE positional encoding construction, specifically, first define the factorization of the graph Laplace as Among them I n is the n×n identity matrix, D is the n×n degree matrix, Λ and U represent the eigenvalue and characteristic quantity respectively. Given a graph G containing n nodes m , LAPE position coding uses k-neighborhood to define:

[0013]

[0014] p i represents the position code of the i-th node, represents the Laplace positional encoding of the m-th graph.

[0015] Step 2: Based on the two position codes constructed in step 1 as input, the data is further processed using a task-based serialization process, which includes the following sub-steps:

[0016] Step 2-1: Serialize data for graph classification tasks and give a set of m graphs. and the position code P obtained by formula (2) and formula (4) = {P1, P2, ..., P m}, the initial node embedding X = {X′1,X′2,…,X′ can be obtained through MLP mapping m}, where X′ m =MLP(X m )+MLP(P m ), Represents the number of nodes in the mth graph. Then, the graph collection is sorted to obtain the serialized features

[0017] Step 2-2, serialize the data for the node classification task and give the graph and its corresponding position code P m After that, the same operation as step 1 can be performed to obtain the initial node embedding Then, for node v, sample its k neighbors Where d(u,v) represents the shortest distance between node v and node u, and K represents the K-hop neighborhood. For node v, its k-hop neighborhood representation can be expressed as:

[0018]

[0019] in represents k-times graph message passing, and For each node v, the K-neighborhood representation of each node can be obtained by the above formula (5): Then The nodes can be arranged to obtain the serialization features

[0020] Step 3, model freezing and fine-tuning. LLAMA2 is selected as the pre-trained large model. In order to retain a large amount of pre-trained knowledge of the multi-head attention layer (MHA) and the feed-forward network layer (FFN), MHA and FFN are frozen during fine-tuning, and other parts of the model are fine-tuned. After obtaining the serialized feature S in step 2, the fine-tuning process can be expressed as:

[0021]

[0022] where LN(·) represents RMSNorm, l=1,...,L represents the number of LLAMA2 modules used, when l=1, ( is the result of step 2).

[0023] Step 4: Fine-tune downstream tasks. For graph classification tasks and node classification tasks, design different modules for classification, which includes the following sub-steps:

[0024] Step 4-1, graph classification, in order to ensure that each graph does not affect each other during the calculation process, a mask matrix M consisting of only 0 and 1 is constructed, and the graph attention calculation is performed using the inductive method. m When they all belong to the same graph, the corresponding part of the mask matrix M is set to 1, otherwise it is set to 0. Subsequently, the MHA module of the model in formula (6) can be updated as:

[0025]

[0026] Where Q represents the query value, K represents the key value, and V represents the key value. is the projection matrix, d q represents the hidden layer dimension, M is the mask matrix, and finally, global average pooling is selected as the readout layer, which is expressed as follows:

[0027]

[0028] in This is the result when l=L.

[0029] Step 4-2, node classification, after obtaining the serialized feature S, attention calculation is performed on each hop of information, and the readout layer is represented as follows:

[0030]

[0031]

[0032] Where W a represents the transformation matrix, S k represents the k-hop representation of each node, α k represents the weight coefficient matrix, and exp() represents the natural exponential function.

[0033] Step 5: After step 4 is completed, the cross entropy loss function is used for model training, which can be specifically expressed as:

[0034]

[0035] Where L ce represents the cross entropy loss, and Y i They represent the predicted value and label respectively. In node classification, N represents the number of nodes, and in graph classification tasks, N represents the number of graphs.

[0036] After the model training is completed, the trained model can be used to perform different tasks (graph classification tasks, node classification tasks).

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] The present invention surpasses the latest methods in complex graph classification and node classification tasks by transferring the powerful generalization ability of large language models to graph tasks; it can process a wide range of data, including citation networks, commodity purchase relationship networks, co-author networks, online forum networks, protein networks and molecular compounds; when the amount of data is small (10% of the normal data amount), the method proposed in the present invention can improve the accuracy by more than 7%. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The figure is a flow chart of the method of the present invention.

[0040] Figure 2 Schematic diagram of the general graph task method based on a large language model provided by the present invention.

[0041] Figure 3Schematic diagram of the serialization process based on graph classification task of the present invention.

[0042] Figure 4 It is a schematic diagram of the serialization process based on the node classification task of the present invention.

[0043] Figure 5 Schematic diagram of the interpretability experimental results of the present invention.

[0044] Figure 6 It is a schematic diagram of the experimental results of a small sample scenario of the present invention. DETAILED DESCRIPTION

[0045] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0046] Embodiment: A general graph task method based on a large language model proposed in the present invention is as follows Figure 1 and Figure 2 Specifically, the present invention comprises the following steps:

[0047] Step 1: Given graph structure data G = (V, A, E), where V = v1, v2, v3, …, v n is a node set, n = |V|, E is an edge set, and the corresponding adjacency matrix A∈{0,1} n×n , when there is an edge connection between node i and node j, A ij =1, otherwise A ij = 0. First, the graph structure data is generally serialized using the improved random walk positional encoding RWPE (Random Walk Positional Embedding) or the Laplace decomposition-based positional encoding LAPE (Laplacian Positional Embedding). The construction and serialization process of the two positional encodings are as follows.

[0048] Step 1-1, RWPE position encoding construction, given a graph G containing n nodes m , the RWPE position encoding is defined using a k-step random walk:

[0049]

[0050] Where RW=AD -1 is the random walk operator, p i represents the random walk position encoding of the i-th node, represents the position encoding of the mth graph. Here, only the probability of each node wandering to itself is considered. When k is large enough, RWPE provides a unique node representation for each node.

[0051] Step 1-2, LAPE positional encoding construction, specifically, first define the factorization of the graph Laplace as Among them I n is the n×n identity matrix, D is the n×n degree matrix, Λ and U represent the eigenvalue and characteristic quantity respectively. Given a graph G containing n nodes m , LAPE position coding uses k-neighborhood to define:

[0052]

[0053] p i represents the position code of the i-th node, represents the Laplace positional encoding of the m-th graph.

[0054] Step 2: Based on the two position codes constructed in step 1 as input, the data is further processed using a task-based serialization process, which includes the following sub-steps:

[0055] Step 2-1: Serialize data for graph classification tasks and give a set of m graphs. and the position code P obtained by formula (2) and formula (4) = {P1, P2, ..., P m}, the initial node embedding X = {X′1,X′2,…,X′ can be obtained through MLP mapping m}, where X ′ m =MLP(X m )+MLP(P m ), represents the number of nodes in the mth graph. Then, Figure 3 As shown, sort the graph collection to obtain serialized features

[0056] Step 2-2, serialize the data for the node classification task and give the graph and its corresponding position code P m After that, the same operation as step 1 can be performed to obtain the initial node embedding Then, if Figure 4 As shown, for node v, sample its k neighbors Where d(u,v) represents the shortest distance between node v and node u, and K represents the K-hop neighborhood. For node v, its k-hop neighborhood representation can be expressed as:

[0057]

[0058] in represents k-times graph message passing, and For each node v, the K-neighborhood representation of each node can be obtained by the above formula (5): Then The nodes can be arranged to obtain the serialization features

[0059] Step 3: Model freezing and fine-tuning. LLAMA2 is selected as the pre-trained large model. In order to retain a large amount of pre-trained knowledge of the multi-head attention layer (MHA) and the feed-forward network layer (FFN), MHA and FFN are frozen during fine-tuning, and other parts of the model are fine-tuned to obtain the serialized features in step 2. After that, the fine-tuning process can be expressed as:

[0060]

[0061] where LN(·) represents RMSNorm, l=1,...,L represents the number of LLAMA2 modules used, when l=1, ( is the result of step 2).

[0062] Step 4: Fine-tune downstream tasks. For graph classification tasks and node classification tasks, design different modules for classification, which includes the following sub-steps:

[0063] Step 4-1, graph classification, in order to ensure that each graph does not affect each other during the calculation process, a mask matrix M consisting of only 0 and 1 is constructed, and the graph attention calculation is performed using the inductive method. m When they all belong to the same graph, the corresponding part of the mask matrix M is set to 1, otherwise it is set to 0. Subsequently, the MHA module of the model in formula (6) can be updated as:

[0064]

[0065] Where Q represents the query value, K represents the key value, and V represents the key value. is the projection matrix, d q represents the hidden layer dimension, M is the mask matrix, and finally, global average pooling is selected as the readout layer, which is expressed as follows:

[0066]

[0067] in This is the result when l=L.

[0068] Step 4-2, node classification, after obtaining the serialized feature S, attention calculation is performed on each hop of information, and the readout layer is represented as follows:

[0069]

[0070] Where W a represents the transformation matrix, S K represents the k-hop representation of each node, α k represents the weight coefficient matrix, and exp() represents the natural exponential function.

[0071] Step 5: After step 4 is completed, the cross entropy loss function is used for model training, which can be specifically expressed as:

[0072]

[0073] Where L ce represents the cross entropy loss, and Y i They represent the predicted value and label respectively. In node classification, N represents the number of nodes, and in graph classification tasks, N represents the number of graphs.

[0074] After the model training is completed, the trained model can be used to perform different tasks (graph classification tasks, node classification tasks).

[0075] The general graph task method based on a large language model provided by the present invention has an evaluation score in graph classification tasks and graph node classification tasks compared with other comparison methods GCN, GAT, GraphSAGE, SGC, APPNP, DAGNN, BM-GCN, FavardGNN, NAGphormer, Exphormer, Specformer, GIN, SAGPool, DiffPool, GraphCON, KP-GIN, GraphTrans, Graph VIT, and GraphMLP-Mixer under graph classification and node classification datasets (shown in Tables 1 and 2). The bold parts in the tables represent the best scores (shown in Tables 3 and 4).

[0076] Table 1 Node classification dataset

[0077]

[0078] Table 2 Image classification dataset

[0079]

[0080] Table 3 Comparative experiments on node classification tasks

[0081]

[0082]

[0083] As shown in Table 3, the model of the present invention has achieved a relatively obvious leading effect on all datasets, whether on small-scale datasets or large-scale datasets. This verifies that the proposed method can complete the corresponding complex graph node classification task through graph serialization operations, which is consistent with the design motivation.

[0084] Table 4 Comparative experiments on node classification tasks

[0085]

[0086] As shown in Table 4, in the graph classification task, whether on a small-scale dataset or a large-scale dataset, the method proposed in the present invention has achieved a relatively obvious leading effect on all datasets, which verifies that the method can complete the corresponding complex graph classification tasks through graph serialization operations.

[0087] like Figure 5 As shown in Figure 1, the method of the present invention is compared with the same model without freezing and random initial weights. As can be seen from the figure, the fine-tuning method adopted by the present invention performs better than the method without freezing and without pre-training, which proves the effectiveness of the fine-tuning method and further illustrates that graph data mining tasks can benefit from the knowledge of LLMS.

[0088] like Figure 6 As shown, in order to demonstrate the ability of the method in a small sample scenario, the present invention sets a mask to control the number of samples while ensuring the class balance of the samples as much as possible. 0.1 means that only 10% of the data is used for training. The sampling rate range is set to 0.9 to 0.1, with an interval of 0.1. As shown in the figure, the method of the present invention has obvious advantages, especially when the amount of data is only 10%, there is a 7% improvement in accuracy.

[0089] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A general graph task method based on a large language model (LLM), characterized in that The method comprises the following steps: Step 1: Given graph structure data G = (V, A, E), where V = v1, v2, v3, …, v n is the node set, n = |V| represents the number of nodes, E is the edge set, and the corresponding adjacency matrix A∈{0,1} n×n , when there is an edge connection between node i and node j, A ij =1, otherwise A ij = 0, first use the improved random walk positional encoding RWPE (Random Walk Positional Embedding) or the Laplace decomposition-based positional encoding LAPE (Laplacian Positional Embedding) to universally serialize the graph structure data; Step 2: Based on the two positional encodings constructed in step 1, the data is further processed using a task-based serialization process. Step 3: Model freezing and fine-tuning, select LLAMA2 as the pre-trained large model. Step 4: Fine-tune downstream tasks. For graph classification tasks and node classification tasks, design different modules for classification. Step 5: After step 4 is completed, the cross entropy loss function is used for model training. After the model training is completed, the trained model can be used to perform different tasks (graph classification tasks, node classification tasks, etc.) service).

2. The general graph task method based on a large language model (LLM) according to claim 1, characterized in that: In step 1, the two position encoding construction and serialization processes are as follows: Step 1-1, RWPE position encoding construction, given a graph G containing n nodes m , the RWPE position encoding is defined using a k-step random walk: Where RW=AD -1 is the random walk operator, p i represents the random walk position encoding of the i-th node, represents the position encoding of the mth graph. Here, only the probability of each node wandering to itself is considered. When k is large enough, RWPE provides a unique node representation for each node. Step 1-2, LAPE positional encoding construction, specifically, first define the factorization of the graph Laplace as Among them I n is the n×n identity matrix, D is the n×n degree matrix, Λ and U represent the eigenvalue and characteristic quantity respectively. Given a graph G containing n nodes m , LAPE position coding uses k-neighborhood to define: p i represents the position code of the i-th node, represents the Laplace positional encoding of the m-th graph.

3. The general graph task method based on a large language model (LLM) according to claim 2, characterized in that Step 2: Based on the two position codes constructed in step 1 as input, the data is further processed using a task-based serialization process, which includes the following sub-steps: Step 2-1: Serialize data for graph classification tasks and give a set of m graphs. and the position code P obtained by formula (2) and formula (4) = {P1, P2, ..., P m }, and obtain the initial node embedding X = {X′1,X′2,…,X′ through MLP mapping m }, where X′ m =MLP(X m )+MLP(P m ), Represents the number of nodes in the mth graph. Then, the graph collection is sorted to obtain the serialized features. Step 2-2, serialize the data for the node classification task and give the graph and its corresponding position code P m Then, the same operation as step 1 is performed to obtain the initial node embedding Then, for node v, sample its k neighbors Where d(u,v) represents the shortest distance between node v and node u, K represents the K-hop neighborhood, and for node v, its k-hop neighborhood is expressed as: in represents k-times graph message passing, and For each node v, the K-neighborhood representation of each node is obtained by the above formula (5): Then Nodes are arranged to obtain serialized features 4. The general graph task method based on a large language model (LLM) according to claim 3, characterized in that Step 3: Model freezing and fine-tuning. LLAMA2 is selected as the pre-trained large model. After obtaining the serialized feature S in step 2, the fine-tuning process is expressed as: where LN(·) represents RMSNorm, e=1,...,L represents the number of LLAMA2 modules used, and when l=1, ( is the result of step 2).

5. The general graph task method based on a large language model (LLM) according to claim 4, characterized in that: Step 4: Fine-tune downstream tasks. For graph classification tasks and node classification tasks, design different modules for classification, which includes the following sub-steps: Step 4-1, graph classification, in order to ensure that each graph does not affect each other during the calculation process, a mask matrix M consisting of only 0 and 1 is constructed, and the graph attention calculation is performed using the inductive method. m When they all belong to the same graph, the corresponding part of the mask matrix M is set to 1, otherwise it is set to 0. Then, the MHA module of the model in formula (6) is updated as: Where Q represents the query value, K represents the key value, and V represents the key value. is the projection matrix, d q represents the hidden layer dimension, M is the mask matrix, and finally, global average pooling is selected as the readout layer, which is expressed as follows: in is the result when l=L, Step 4-2, node classification, after obtaining the serialized feature S, attention calculation is performed on each hop of information, and the readout layer is represented as follows: Where W a represents the transformation matrix, S k represents the k-hop representation of each node, α k represents the weight coefficient matrix, and exp() represents the natural exponential function.

6. The general graph task method based on a large language model (LLM) according to claim 5, characterized in that: Step 5: After step 4 is completed, the cross entropy loss function is used for model training, which is specifically expressed as: Where L ce represents the cross entropy loss, and Y i They represent the predicted value and label respectively. In node classification, N represents the number of nodes, and in graph classification tasks, N represents the number of graphs.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the general graph task method based on a large language model (LLM) as described in any one of claims 1 to 6 are implemented.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the general graph task method based on a large language model (LLM) as described in any one of claims 1 to 6.

Images