Text graph node classification method based on pre-trained language model and deep prompt

Abstract

The application discloses a text graph node classification method based on a pre-training language model and a deep prompt, belongs to the field of computer graph representation learning, and comprises the following steps: constructing a framework TegPPT; adopting frozen pre-training language model parameters and optimizing continuous prompt and classification head parameters to optimize the framework TegPPT; and specifically, using the pre-training language model as a basic model to execute a node classification task of a text attribute graph, jointly encoding the text attribute and the topological structure of the graph through a serialization module, exciting the underlying knowledge of the pre-training language model through continuous prompting, improving the ability of the model to understand and express the node text attribute and the complex relationship between nodes, and ensuring that the pre-training space and the downstream task space are fully aligned through deep tuning technology, so that the generalization ability of the model and the performance on the node classification task are optimized. The application can not only better capture the graph topological structure and the text information, but also avoid the potential negative transfer risk in the prior art.

Description

Technical Field

[0001] This invention relates to the field of computer graph learning technology, and in particular to a text graph node classification method based on pre-trained language models and deep cues. Background Technology

[0002] Text attribute graphs are a special type of graph structure that contains not only topological information but also textual information about the nodes. Node classification, as a key downstream task in text attribute graph representation learning, has attracted considerable attention from researchers.

[0003] In existing technologies, language models are commonly used to encode text information, and the generated node embeddings are then input into a graph neural network model for training, achieving joint modeling of node text information and graph topology. Although these methods perform well in some cases, they ignore the topological correlation between adjacent nodes when generating text embeddings, fail to mutually enhance node representations through underlying semantics, and overlook the potential problem of suboptimal performance interactions between modules during joint training.

[0004] Therefore, a new method for classifying text attribute graph nodes is needed. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a text graph node classification method based on a pre-trained language model and deep prompts, which can not only better capture graph topology and text information, but also avoid the potential negative transfer risk in existing studies.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A text graph node classification method based on a pre-trained language model and deep prompts includes the following:

[0008] The framework TegPPT was built, which includes text attribute graph serialization and depth hint optimization.

[0009] The text attribute graph serialization includes: constructing a text sequence based on the text description of any target node in the text attribute graph; selecting neighboring nodes of the target node using a multiple random walk strategy, and further integrating the text descriptions of the neighboring nodes as auxiliary information into the text sequence; and optimizing the length of the generated text sequence using a large language model.

[0010] The deep prompting optimization includes: the text sequence is processed by the word embedding layer of the pre-trained language model to form a high-dimensional word embedding matrix; on this basis, the continuous prompts are randomly initialized, and the word embedding matrix and the continuous prompts are input into the pre-trained language model to achieve parameter fine-tuning; a linear classification head is defined to guide the fine-tuning of the parameters of the pre-trained language model to adapt to the node classification task of the text attribute graph.

[0011] The TegPPT framework was optimized by freezing the parameters of the pre-trained language model and optimizing the parameters of the continuous prompts and classification heads.

[0012] A further improvement to the technical solution of the present invention lies in the fact that the text attribute graph serialization specifically includes the following steps:

[0013] Step 1.1.1: Taking any node in the text attribute graph as the target, from the target node n... i Text description Constructing the initial sequence

[0014]

[0015] Step 1.1.2: Perform neighbor sampling using a multiple random walk strategy, sorting the neighbor nodes by the frequency of the walks, and then sampling at the target node n. i The set of all neighboring nodes N i Select the first K nodes to obtain the sampled walk path N. i ′:

[0016] N′ i ={n j |n j ∈sort(N i ,Π),j=1,2,…,K} (2)

[0017] Where Π=(π1,π2,...,π) N ) represents the system steady-state vector; N represents the number of nodes in the text attribute graph;

[0018] Based on the stationary distribution theory of Markov chains, a system will reach a steady-state distribution after multiple random walks. Therefore, the steady-state probability π can be used to determine the stationary distribution. i Approximate description of node n i The frequency of access; the iterative process of Π is described as follows:

[0019]

[0020]

[0021] When ∥Π (k+1) -Π (k)When ∥1 < ε, the system is considered to have reached a steady-state distribution, where k is the number of iterations; ∥·∥1 represents the L1 norm of the vector; the preset threshold ε = 10. -8 ; I represents a unit vector; α = 0.85 is the equilibrium parameter controlling the random walk; P is the transition matrix; i,j Indicates from node n i To its neighbor node n j Transition probability:

[0022]

[0023] Where A represents the adjacency matrix of the text attribute graph; A ij =1 indicates node n i With n j If there is an edge between them, then A is not an edge between them. ij =0; the indicator function F(x) takes the value 1 when x is true, and 0 otherwise; deg(n i ) is a node n in an undirected graph. i The degree, correspondingly in a directed graph, should be deg out-degree. out (n i Replaced by;

[0024] Step 1.1.3: Integrate the neighbor node text descriptions as auxiliary information into the text sequence. First, perform an aggregation operation on the neighbor node text sequence, represented as:

[0025]

[0026] Where, N′ i For target node n i The sampled neighbor set; Represents neighbor node n j A text sequence; ∑ represents the aggregation operation;

[0027] Next, the aggregated text sequence is embedded into the text sequence framework of the target node. middle:

[0028]

[0029] Where Φ(·) represents the embedding function; It is a node description that comprehensively considers both topological structure and textual information;

[0030] Step 1.1.4: Optimize the length of the generated text sequence using a large language model to improve text quality;

[0031] Let L max L is the maximum input sequence length of the pre-trained language model. sequenceLet S be the length of the current sequence. If the text sequence is too long, a large language model is needed to optimize the abbreviation; otherwise, no adjustment is needed. The optimization strategy is expressed as:

[0032]

[0033] To ensure that the abbreviated sequence S′ retains the semantic information contained in the original text to the greatest extent, a semantic similarity measurement method, BERTScore, is introduced to guide the iterative process, thereby evaluating and optimizing the abbreviation results.

[0034] A further improvement to the technical solution of this invention lies in the following: In step 1.1.4, the semantic similarity measurement method BERTScore guides the iterative process, specifically including the following:

[0035] First, the BERT model is used to generate the context embedding matrices for each text sequence:

[0036] "′

[0037] E S =[e1,e2,...,e n ], E S′ =[e1,e2,...,e m (9)

[0038] Among them, e i and e′ j These are the context embeddings of the i-th and j-th words in S and S′, respectively; each element M in the similarity matrix M... ij Indicates embedding e i and e′ j Cosine similarity between them; M ij The larger the value, the higher the similarity between words i and j; M ij Represented as:

[0039]

[0040] Next, to comprehensively measure the semantic similarity between two text sequences, matching scores from different perspectives are introduced:

[0041]

[0042]

[0043] Precision evaluated the accuracy of the alignment match between the abbreviated sequence S′ and the original sequence S. This means that for word i in S, find the word j in S′ that best matches it; conversely, Recall evaluates the alignment and matching of the original sequence S to the abbreviated sequence S′.

[0044] Finally, the fine-grained similarity score BERTScore between the two text sequences is calculated:

[0045]

[0046] By maximizing BERTScore, subtle contextual information and deep semantic relationships between text sequences are effectively captured, ensuring that abbreviated sequences retain the semantic content of the original sequences as much as possible.

[0047] A further improvement to the technical solution of this invention lies in the following steps: the depth-based feedback optimization specifically includes the following steps:

[0048] Step 1.2.1: Input the constructed text sequence S′ into the word embedding layer of the pre-trained language model for processing, generating a high-dimensional word embedding matrix:

[0049]

[0050] Where Embed(·) represents the embedding function, where n is the length of sequence S′ and d is the embedding dimension;

[0051] Step 1.2.2, initialize the sequential prompts for each level:

[0052]

[0053] Where D is the number of consecutive input prompts, and L is the length of consecutive prompts in each level;

[0054] Step 1.2.3: Embed the word X and its corresponding... Layered continuous prompts Input them together into the Transformer layer. Continuous suggestions can start from any layer s and continue for D layers.

[0055] The input transmission process for each layer is represented as follows:

[0056]

[0057] Step 1.2.4: Finally, define a linear classification head to guide the fine-tuning of the pre-trained model parameters to adapt to the node classification task of the text attribute graph.

[0058] After all Transformer layers have finished processing, the output X of the last layer is... (last) The final category label is calculated using a linear classification head. The calculation of the classification head is represented as follows:

[0059]

[0060] in, and Here, represents the weights and biases of the linear layer, and k is the number of target classes.

[0061] A further improvement to the technical solution of this invention lies in the following: the optimization of the TegPPT framework by freezing the pre-trained language model parameters and optimizing the continuous prompts and classification head parameters specifically includes the following:

[0062] Freeze the parameters of the pre-trained language model and define a joint loss function to optimize the framework TegPPT:

[0063]

[0064] in, It is the average loss function for the classification task. It is the L2 regularization function for continuous prompts and classification head parameters;

[0065] Average loss function for classification tasks Represented as:

[0066]

[0067] Where N represents the number of nodes in the text attribute graph, y ij Let represent the true label of the j-th class of the i-th instance, and That is the corresponding predicted probability;

[0068] L2 regularization function for continuous prompts and classification head parameters Represented as:

[0069]

[0070] Where λ is the regularization coefficient; since the parameters of the pre-trained language model are frozen, the gradient is only applied to the trainable parameters, namely the continuous cue H, the class head weight W, and the bias b; adding It can prevent overfitting and improve the model's generalization ability.

[0071] The technological advancements achieved by this invention due to the adoption of the above technical solutions are as follows:

[0072] 1. This invention constructs a novel collaborative learning paradigm, TegPPT, which eliminates the need for explicit graph neural network aggregation. This paradigm innovatively employs a pre-trained language model as the foundational model for text attribute graph representation learning. This model captures and integrates the topological structure of the text attribute graph with textual information, achieving joint encoding of graph topology and text attributes. This unified encoding eliminates explicit graph neural network aggregation and effectively overcomes the negative transfer problem encountered in traditional joint training paradigms. While improving the accuracy of node classification tasks on text attribute graphs, it also significantly reduces computational complexity, making it more efficient and scalable in practical applications.

[0073] 2. This invention constructs and merges the text sequences of each node and its neighbors through text attribute graph serialization, which not only encodes the text features of the central node itself, but also embeds the structural relationships with neighboring nodes.

[0074] 3. This invention, through the design of reasonable and efficient continuous prompts, fully mobilizes the underlying knowledge of the pre-trained language model with high accuracy and parameter efficiency, thereby improving the model's ability to understand and express the textual attributes of nodes and the complex relationships between nodes.

[0075] 4. This invention employs a deep prompting optimization technique that combines continuous prompts. With only a few parameters needing fine-tuning, it can ensure that the representation space of the pre-trained language model is highly aligned with the node classification task space, significantly improving the model's performance on node classification tasks. It also ensures that the pre-training space is fully aligned with the downstream task space, which not only optimizes the model's generalization ability but also significantly improves the model's performance on node classification tasks. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a flowchart of a text graph node classification method based on a pre-trained language model and deep prompts provided in an embodiment of the present invention;

[0078] Figure 2 This is a framework diagram of a text graph node classification method based on a pre-trained language model and deep hints provided in an embodiment of the present invention. Detailed Implementation

[0079] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0080] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments:

[0081] like Figure 1-2As shown, a text graph node classification method based on a pre-trained language model and deep prompts specifically includes the following steps:

[0082] Step 1: Construct the framework TegPPT, which includes text attribute graph serialization and depth-based feedback optimization;

[0083] Step 1.1, the text attribute graph serialization includes: for any target node in the text attribute graph, constructing a text sequence based on its text description; selecting neighboring nodes of the target node using a multiple random walk strategy, and further integrating the text descriptions of the neighboring nodes as auxiliary information into the text sequence; optimizing the length of the generated text sequence using a large language model; specifically including the following steps:

[0084] Step 1.1.1: Taking any node in the text attribute graph as the target, from the target node n... i Text description Constructing the initial sequence

[0085]

[0086] Step 1.1.2: Perform neighbor sampling using a multiple random walk strategy, sorting the neighbor nodes by the frequency of the walks, and then sampling at the target node n. i The set of all neighboring nodes N i Select the first K nodes to obtain the sampled walk path N. i ′:

[0087] N i ′={n j |n j ∈sort(N i ,Π),j=1,2,…,K} (2)

[0088] Where Π=(π1,π2,...,π) N Let π be the system's steady-state vector, and N be the number of nodes in the text attribute graph. Based on the stationary distribution theory of Markov chains, the system will reach a steady-state distribution after many random walks. Therefore, the steady-state probability π is used... i Approximate description of node n i The frequency of access. The iterative process of Π can be described as:

[0089]

[0090]

[0091] When ∥Π (k+1) -Π (k)When ∥1 < ε, the system is considered to have reached a steady-state distribution, where k is the iteration number, ∥·∥1 represents the L1 norm of the vector, and the preset threshold ε = 10. -8 I represents a unit vector, and α = 0.85 is the equilibrium parameter controlling the random walk. Let P be the transition matrix. i,j Indicates from node n i To its neighbor node n j Transition probability:

[0092]

[0093] Where A represents the adjacency matrix of the text attribute graph, A ij =1 indicates node n i With n j If there is an edge between them, then A is not an edge between them. ij =0. The indicator function F(x) takes the value 1 when x is true, and 0 otherwise. deg(n i ) is a node n in an undirected graph. i The degree, correspondingly in a directed graph, should be deg out-degree. out (n i ) replaced.

[0094] Step 1.1.3: Integrate the neighbor node text descriptions as auxiliary information into the text sequence. First, perform an aggregation operation on the neighbor node text sequence, represented as:

[0095]

[0096] Where, N i ' is the target node n i The sampled neighbor set, Represents neighbor node n j The text sequence, where ∑ represents the aggregation operation.

[0097] Next, the aggregated text sequence is embedded into the text sequence framework of the target node. middle:

[0098]

[0099] Where Φ(·) represents the embedding function, It is a node description that takes into account both topological structure and textual information.

[0100] Step 1.1.4: Optimize the length of the generated text sequence using a Large Language Model (LLM) to improve text quality. Let L... max L is the maximum input sequence length of the pre-trained language model. sequenceLet S be the length of the current sequence. If the text sequence is too long, LLM optimization and abbreviation are needed; otherwise, no adjustment is required. The optimization strategy can be expressed as:

[0101]

[0102] To ensure that the abbreviated sequence S′ retains the semantic information contained in the original text to the greatest extent, a semantic similarity measurement method, BERTScore, is introduced to guide the iterative process and thus evaluate and optimize the abbreviation results.

[0103] First, the BERT model is used to generate the context embedding matrices for each text sequence:

[0104] "′

[0105] E S =[e1,e2,...,e n ], E S′ =[e1,e2,...,e m (9)

[0106] Among them, e i and e′ j These are the context embeddings of the i-th and j-th words in S and S′, respectively. Each element M in the similarity matrix M... ij Indicates embedding e i and e′ j Cosine similarity between them. M ij A larger value indicates a higher similarity between words i and j. M ij It can be represented as:

[0107]

[0108] Next, to comprehensively measure the semantic similarity between two text sequences, matching scores from different perspectives are introduced:

[0109]

[0110]

[0111] Precision evaluated the accuracy of the alignment match between the abbreviated sequence S′ and the original sequence S. This means that for word i in S, find the word j in S′ that best matches it. Conversely, Recall evaluates the alignment and matching of the original sequence S to the abbreviated sequence S′. Finally, the fine-grained similarity score BERTScore between the two text sequences is calculated:

[0112]

[0113] By maximizing BERTScore, subtle contextual information and deep semantic relationships between text sequences are effectively captured, ensuring that abbreviated sequences retain the semantic content of the original sequences as much as possible.

[0114] Step 1.2, the deep prompt optimization includes: the text sequence is processed by the word embedding layer of the pre-trained language model to form a high-dimensional word embedding matrix; based on this, continuous prompts are randomly initialized, and the word embedding matrix and continuous prompts are input together into the pre-trained language model to achieve parameter fine-tuning; a linear classification head is defined to guide the fine-tuning of the pre-trained language model parameters to adapt to the node classification task of the text attribute graph; specifically, it includes the following steps:

[0115] Step 1.2.1: Input the constructed text sequence S′ into the word embedding layer of the pre-trained language model for processing, generating a high-dimensional word embedding matrix:

[0116]

[0117] Where Embed(·) represents the embedding function, where n is the length of the sequence S′ and d is the embedding dimension.

[0118] Step 1.2.2, initialize the sequential prompts for each level:

[0119]

[0120] Where D is the number of consecutive input prompts and L is the length of consecutive prompts in each level.

[0121] Step 1.2.3: Embed the word X and its corresponding... Layered continuous prompts Input them together into the Transformer layer. Continuous suggestions can start from any layer s and continue to layer D.

[0122] The input transmission process of each layer can be represented as:

[0123]

[0124] Step 1.2.4 finally defines a linear classification head to guide the fine-tuning of the pre-trained model parameters to adapt to the node classification task of the text attribute graph. After all Transformer layers have been processed, the output X of the last layer is... (last) The final category label is calculated using a linear classification head.

[0125] The calculation of the classification head can be expressed as:

[0126]

[0127] in, and Here, represents the weights and biases of the linear layer, and k is the number of target classes.

[0128] Step 2 involves optimizing the TegPPT framework by freezing the pre-trained language model parameters and optimizing the continuous prompts and classification head parameters. This includes the following:

[0129] Freeze the parameters of the pre-trained language model and define a joint loss function to optimize the framework TegPPT:

[0130]

[0131] in, It is the average loss function for the classification task. It is the L2 regularization function for continuous prompts and classification header parameters.

[0132] Average loss function for classification tasks

[0133]

[0134] Where N represents the number of nodes in the text attribute graph, y ij Let represent the true label of the j-th class of the i-th instance, and That is the corresponding predicted probability.

[0135] L2 regularization function for continuous prompts and classification head parameters

[0136]

[0137] Where λ is the regularization coefficient. Since the parameters of the pre-trained language model are frozen, the gradient is applied only to the trainable parameters: the continuous cue H, the classifier head weights W, and the bias b. Adding... It can prevent overfitting and improve the model's generalization ability.

[0138] In summary, this invention first constructs the framework TegPPT, including: 1) Text attribute graph serialization. For any target node in the text graph, a text sequence is constructed based on its text description, and the text descriptions of neighboring nodes are incorporated as auxiliary information. Through information interaction among neighboring nodes, the local topological information of the target node is effectively integrated, promoting the unified encoding of text and topological structure; 2) Deep cue optimization. Continuous cues are applied deeply to multiple layers of the pre-trained model, not just the input layer; through layer-by-layer information transmission, the pre-trained language model is guided to transfer knowledge from the pre-training space to the node classification task, ultimately converting the high-dimensional output into a probability distribution of predefined categories, thereby generating node category labels; finally, the framework TegPPT is optimized by freezing the parameters of the pre-trained language model and optimizing the parameters of continuous cues and classification heads. This invention not only better captures graph topological structure and textual information, but also avoids the potential negative transfer risk in existing research.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text graph node classification method based on a pre-trained language model and deep prompts. characterized in that it comprises the following contents: constructing a framework TegPPT including textual attribute graph serialization and deep prompt tuning; said textual attribute graph serialization comprises: constructing a text sequence based on the text description for any target node in the textual attribute graph; selecting neighbor nodes of the target node by adopting a multiple random walk strategy, and further integrating the text descriptions of the neighbor nodes into the text sequence as auxiliary information; optimizing the length of the generated text sequence by using a large language model; said deep prompt tuning comprises: processing the text sequence through a word embedding layer of a pre-trained language model to form a high-dimensional word embedding matrix; on this basis, performing random initialization of continuous prompts, and inputting the high-dimensional word embedding matrix together with the continuous prompts into the pre-trained language model to implement parameter fine-tuning; defining a linear classification head to guide the fine-tuning of the parameters of the pre-trained language model so as to adapt to the node classification task of the textual attribute graph; optimizing the framework TegPPT by freezing the parameters of the pre-trained language model and optimizing the parameters of the continuous prompts and the classification head.

2. The text graph node classification method based on a pre-trained language model and deep prompts according to claim 1, characterized in that, said textual attribute graph serialization specifically comprises the following steps: Step 1.1.1: Taking any node in the text attribute graph as the target, start from the target node... Text description Constructing the initial sequence : (1) Step 1.1.2: Perform neighbor sampling using a multiple random walk strategy, sorting the neighbor nodes by the frequency of the walks, and then sampling at the target node. set of all neighboring nodes Select before 1 node, get the target node Sampling Neighbor Set : (2) in This is the system's steady-state vector; The number of nodes contained in the text attribute graph; Based on the stationary distribution theory of Markov chains, a system will reach a steady-state distribution after multiple random walks. Therefore, the steady-state probability can be used... Approximate description of nodes Frequency of access; The iterative process is described as follows: (3) (4) When satisfied When the system reaches a steady-state distribution, it is considered to have reached a steady-state distribution. It is the number of iterations; Representing vectors Norm; Preset threshold ; Represents a unit vector; It is the balance parameter that controls the random walk; Here is the transition matrix; Indicates from node to its neighboring nodes Transition probability: (5) in, An adjacency matrix representing a text attribute graph; Represents a node and There are edges between them, otherwise... Indicator Function exist The value is 1 if true, and 0 otherwise. It is a node in an undirected graph The degree, correspondingly in a directed graph, should be the out-degree. Replaced; step 1.1.3, integrating the text descriptions of neighbor nodes into the text sequence as auxiliary information, and first performing an aggregation operation on the text sequence of neighbor nodes, which is expressed as: (6) in For the target node The set of sampled neighbors; Representing neighboring nodes A text sequence; Indicates aggregation operation; Next, the aggregated text sequence is embedded into the text sequence framework of the target node. middle: (7) in, Indicates an embedded function; It is a node description that comprehensively considers both topological structure and textual information; step 1.1.4, optimizing the length of the generated text sequence by using a large language model to improve text quality; set up The maximum input sequence length for the pre-trained language model. For the current sequence The length of the text sequence is important; if the text sequence is too long, a large language model is needed to optimize the abbreviation, otherwise no adjustment is needed. The optimization strategy is expressed as: (8) To ensure the abbreviated sequence To preserve the semantic information contained in the original text to the greatest extent, a semantic similarity measurement method is introduced. This guides the iterative process, thereby evaluating and optimizing the abbreviation results.

3. The text graph node classification method based on a pre-trained language model and deep hints according to claim 2, characterized in that, In step 1.1.4, the semantic similarity measurement method The guiding iterative process includes the following: first, using the BERT model to respectively generate context embedding matrices of the text sequence: (9) in, and They are and Middle The and the first Context embedding of each word; similarity matrix Each element in Indicates embedding and Cosine similarity between them; Larger means word and The higher the similarity; Represented as: (10) then, in order to comprehensively measure the semantic similarity between two text sequences, matching scores from different perspectives are introduced: (11) (12) The abbreviation sequence was evaluated. To the original sequence The accuracy of alignment matching. Indicates for words in ,turn up The word that best matches it on the contrary, The evaluation is of the original sequence. To the abbreviation sequence Alignment matching cases; Finally, fine-grained similarity scores between the two text sequences are calculated. : (13) By maximizing It effectively captures subtle contextual information and deep semantic relationships between text sequences, ensuring that the abbreviated sequence retains the semantic content of the original sequence as much as possible.

4. The text graph node classification method based on a pre-trained language model and deep prompts according to claim 1, characterized in that, said deep prompt tuning specifically comprises the following steps: Step 1.2.1, the constructed text sequence The word embeddings are processed by the word embedding layer of the pre-trained language model to generate a high-dimensional word embedding matrix. (14) in Represents an embedded function, where It is a sequence Length, It is the embedded dimension; step 1.2.2, initializing the continuous prompts of each layer: (15) in This refers to the number of consecutive prompts. It is the length of the continuous prompts in each layer; Step 1.2.3, embed the words and corresponding Layered continuous prompts Enter together Within a layer, continuous prompts can be generated from any layer. Begin, Continue layer; the input transfer process of each layer is expressed as: (16) step 1.2.4, finally defining a linear classification head to guide the fine-tuning of the parameters of the pre-trained model so as to adapt to the node classification task of the textual attribute graph; In all After the layer processing is complete, the output of the last layer. The final category label is calculated using a linear classification head. The calculation of the classification head is expressed as follows: (17) in and These are the weights and biases of the linear layer, respectively. It represents the number of target categories.

5. The text graph node classification method based on a pre-trained language model and deep prompts according to claim 1, characterized in that, said optimizing the framework TegPPT by freezing the parameters of the pre-trained language model and optimizing the parameters of the continuous prompts and the classification head specifically comprises the following contents: freezing the parameters of the pre-trained language model, and defining a joint loss function to optimize the framework TegPPT: (18) in, It is the average loss function for the classification task. It is a continuous prompt and classification header parameter. Regularization function; Average loss function for classification tasks Represented as: (19) in, This indicates the number of nodes in the text attribute graph. Indicates the first The first instance The actual label of the class, and That is the corresponding predicted probability; Continuous prompts and classification header parameters Regularization function Represented as: (20) in It is the regularization coefficient; since the parameters of the pre-trained language model are frozen, the gradient is only applied to the trainable parameters, i.e., continuous prompts. Category Head Weights and bias join in It can prevent overfitting and improve the model's generalization ability.

Images