Text graph node classification method based on pre-training language model and depth prompt

By using pre-trained language model and depth prompt methods in text attribute graph node classification, the topological structure and text information are integrated, and the problem of neglected topological correlation between nodes in the existing technology is solved, and efficient node classification and model are realized Improve generalization ability.

CN119046730AActive Publication Date: 2024-11-29YANSHAN UNIV

Patent Information

Application Number
CN202411147640.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-29
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

The prior art ignores the topological correlation between adjacent nodes in the text attribute graph node classification, cannot effectively enhance node characterization, and there is potential negative migration risk.

Method used

Using text graph node classification method based on pre-trained language model and depth prompt, the text attribute graph serialization and depth prompt tuning are built, the topology structure and text information are integrated, and the text is optimized using multiple random walk strategies and BERTScore. Sequence, freeze pretrained language model parameters and optimize continuous prompts and classification header parameters.

Benefits of technology

The joint encoding of graph topology structure and text information is realized, the accuracy of node classification tasks is improved, the computational complexity is reduced, the negative migration risk is avoided, and the generalization ability and parameter efficiency of the model are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046730A_ABST
    Figure CN119046730A_ABST
Patent Text Reader

Abstract

The invention discloses a text graph node classification method based on a pre-training language model and depth prompt, which belongs to the field of computer graph representation learning and comprises the following steps: constructing a framework TegPPT; optimizing the framework TegPPT by freezing pre-training language model parameters and optimizing continuous prompt and classification head parameters; executing a node classification task of a text attribute graph by using a pre-training language model as a basic model, jointly coding a topological structure of text attributes and the graph through a serialization module, and exciting underlying knowledge of the pre-training language model through continuous prompt; the ability of the model in understanding and expressing node text attributes and complex relationships among nodes is improved, the deep tuning technology ensures that a pre-training space is fully aligned with a downstream task space, and the generalization ability of the model and the performance of a node classification task are optimized. According to the method, a graph topological structure and text information can be better captured, and meanwhile, the potential negative migration risk in existing research is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer graph learning technology, and in particular to a text graph node classification method based on a pre-trained language model and deep prompts. Background Art

[0002] The text attribute graph is a special graph structure that contains not only the topological structure information of the graph, but also the text information of the nodes. Node classification, as one of the key downstream tasks of text attribute graph representation learning, has attracted the attention of many researchers.

[0003] In the prior art, language models are commonly used to encode text information, and the generated node embeddings are input into the graph neural network model for training to achieve joint modeling of node text information and graph topology. Although these methods perform well in some cases, they ignore the topological correlation between adjacent nodes when generating text embeddings, fail to mutually enhance node representations through underlying semantics, and ignore the problem of potential suboptimal performance interactions between modules in joint training.

[0004] Therefore, a new text attribute graph node classification method is needed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a text graph node classification method based on a pre-trained language model and deep cues, which can not only better capture the graph topology and text information, but also avoid the potential negative transfer risks in existing research.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A text graph node classification method based on pre-trained language model and deep hints, including the following contents:

[0008] Build TegPPT, a framework including text attribute graph serialization and deep hint tuning;

[0009] The text attribute graph serialization includes: for any target node in the text attribute graph, constructing a text sequence based on its text description; using multiple random walk strategies to select neighbor nodes of the target node, and further integrating the neighbor node text descriptions into the text sequence as auxiliary information; using a large language model to optimize the length of the generated text sequence;

[0010] The deep prompt tuning includes: processing the text sequence through the word embedding layer of the pre-trained language model to form a high-dimensional word embedding matrix; on this basis, performing random initialization of continuous prompts, and inputting the word embedding matrix and the continuous prompts into the pre-trained language model together to achieve parameter fine-tuning operation; defining a linear classification head to guide the fine-tuning of the parameters of the pre-trained language model to adapt to the node classification task of the text attribute graph;

[0011] The framework TegPPT is optimized by freezing the pre-trained language model parameters and optimizing the continuous prompt and classification head parameters.

[0012] A further improvement of the technical solution of the present invention is that the text attribute graph serialization specifically includes the following steps:

[0013] Step 1.1.1, take any node in the text attribute graph as the target, and start from the target node n i Text description of Constructing the initial sequence

[0014]

[0015] Step 1.1.2: Use multiple random walk strategies to sample neighbors, sort by the frequency of neighbor nodes passed by the walk, and i The set of all neighbor nodes N i Select the first K nodes from the sampled walking path N i ′:

[0016] N′ i ={n j ∣n j ∈sort(N i ,Π),j=1,2,…,K} (2)

[0017] Where π=(π1,π2,...,π N ) is the system steady-state vector; N is the number of nodes in the text attribute graph;

[0018] Based on the stationary distribution theory of Markov chain, the system will reach a steady-state distribution after multiple random walks, so the steady-state probability π is used i Approximate description node n i The frequency of being visited; the iterative process of Π is described as:

[0019]

[0020]

[0021] When ∥Π is satisfied (k+1) -Π (k)When ∥1<ε, the system is considered to have reached a steady-state distribution, where k is the number of iterations; ∥·∥1 represents the L1 norm of the vector; the preset threshold ε=10 -8 ; I represents the unit vector; α = 0.85 is the equilibrium parameter controlling the random walk; is the transfer matrix; P i,j Represents slave node n i To its neighbor node n j Transition probability:

[0022]

[0023] Among them, A represents the adjacency matrix of the text attribute graph; A ij =1 indicates node n i With n j There is an edge between them, otherwise A ij =0; the indicator function F(x) takes the value 1 when x is true, otherwise it takes the value 0; deg(n i ) is a node n in an undirected graph i The degree of, correspondingly, in a directed graph, should be out-degree deg out (n i )

[0024] Step 1.1.3, integrate the neighbor node text description into the text sequence as auxiliary information. First, perform aggregation operation on the neighbor node text sequence, which is expressed as:

[0025]

[0026] Among them, N′ i The target node n i The set of sampled neighbors of ; Represents neighbor node n j A text sequence; ∑ represents an aggregation operation;

[0027] Then embed the aggregated text sequence into the text sequence framework of the target node middle:

[0028]

[0029] Where Φ(·) represents the embedding function; It is a node description that comprehensively considers topological structure and text information;

[0030] Step 1.1.4, use a large language model to optimize the length of the generated text sequence to improve the text quality;

[0031] Let L max is the maximum input sequence length of the pre-trained language model, L sequenceis the length of the current sequence S. If the text sequence is too long, a large language model is needed to optimize the abbreviation. Otherwise, no adjustment is required. The optimization strategy is expressed as:

[0032]

[0033] In order to ensure that the abbreviated sequence S′ can retain the semantic information contained in the original text to the greatest extent, a semantic similarity measurement method BERTScore is introduced to guide the iterative process and then evaluate and optimize the abbreviation results.

[0034] A further improvement of the technical solution of the present invention is that in step 1.1.4, the semantic similarity measurement method BERTScore guides the iteration process, which specifically includes the following contents:

[0035] First, use the BERT model to generate the context embedding matrix of the text sequence:

[0037] E S =[e1,e2,...,e n ], E S′ =[e1′,e2′,...,e m ′] (9)

[0038] Among them, e i and e′ j are the context embeddings of the i-th and j-th words in S and S′ respectively; each element M in the similarity matrix M ij Indicates embedding i and e′ j The cosine similarity between ij The larger the value, the higher the similarity between words i and j. ij It is expressed as:

[0039]

[0040] Next, in order to comprehensively measure the semantic similarity between two text sequences, matching scores from different angles are introduced:

[0041]

[0042]

[0043] Precision evaluates the accuracy of the alignment of the abbreviation sequence S′ to the original sequence S. It means that for word i in S, find the word j in S′ that best matches it. Conversely, Recall evaluates the alignment of the original sequence S to the abbreviated sequence S′.

[0044] Finally, the fine-grained similarity score BERTScore between the two text sequences is calculated:

[0045]

[0046] By maximizing the BERTScore, the subtle contextual information and deep semantic relationships between text sequences are effectively captured, ensuring that the abbreviated sequence retains the semantic content of the original sequence as much as possible.

[0047] A further improvement of the technical solution of the present invention is that the depth prompt tuning specifically includes the following steps:

[0048] Step 1.2.1, input the constructed text sequence S′ into the word embedding layer of the pre-trained language model to generate a high-dimensional word embedding matrix:

[0049]

[0050] Where, Embed(·) represents the embedding function, where n is the length of the sequence S′ and d is the embedding dimension;

[0051] Step 1.2.2, initialize the continuous prompts for each layer:

[0052]

[0053] Where D is the number of layers of continuous prompts, and L is the length of continuous prompts in each layer;

[0054] Step 1.2.3, embed the word X and the corresponding Continuous prompt of layer They can be input into the Transformer layer together. Continuous prompts can start from any layer s and continue for D layers.

[0055] The input transmission process of each layer is expressed as:

[0056]

[0057] Step 1.2.4, finally define the linear classification head to guide the fine-tuning of the pre-trained model parameters to adapt to the node classification task of the text attribute graph;

[0058] After all Transformer layers are processed, the output of the last layer is X (last) The final class label is calculated through a linear classification head The calculation of the classification head is expressed as:

[0059]

[0060] in, and are the weights and biases of the linear layer, respectively, and k is the number of target categories.

[0061] A further improvement of the technical solution of the present invention is that the framework TegPPT is optimized by freezing the pre-trained language model parameters and optimizing the continuous prompt and classification head parameters, which specifically includes the following contents:

[0062] Freeze the pre-trained language model parameters and define a joint loss function to optimize the framework TegPPT:

[0063]

[0064] in, is the average loss function for the classification task, is the L2 regularization function of the continuous prompt and classification head parameters;

[0065] Average loss function for classification tasks It is expressed as:

[0066]

[0067] Where N represents the number of nodes in the text attribute graph, y ij represents the true label of the jth class of the i-th instance, and is the corresponding predicted probability;

[0068] L2 regularization function for continuous hint and classification head parameters It is expressed as:

[0069]

[0070] Among them, λ is the regularization coefficient; since the parameters of the pre-trained language model are frozen, the gradient is only for the trainable parameters, namely the continuous hint H, the classification head weight W and the bias b; adding It can prevent overfitting and improve the generalization ability of the model.

[0071] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:

[0072] 1. The present invention constructs a novel collaborative learning paradigm TegPPT, which does not require the introduction of explicit graph neural network aggregation. This paradigm innovatively adopts a pre-trained language model as the basic model for text attribute graph representation learning. The model captures and integrates the topological structure and text information of the text attribute graph, realizes the joint encoding of the graph topological structure and text attributes, achieves unified encoding, eliminates explicit graph neural network aggregation, and effectively overcomes the negative transfer problem that occurs in the traditional joint training paradigm; while improving the accuracy of node classification tasks on text attribute graphs, it also greatly reduces the computational complexity, making it more efficient and scalable in practical applications.

[0073] 2. The present invention constructs and fuses the text sequences of each node and its neighbors through text attribute graph serialization, which not only encodes the text features of the central node itself, but also embeds the structural relationship with the neighboring nodes.

[0074] 3. The present invention fully mobilizes the underlying knowledge of the pre-trained language model with high accuracy and parameter efficiency by designing reasonable and efficient continuous prompts, thereby improving the model's ability to understand and express node text attributes and complex relationships between nodes.

[0075] 4. The present invention adopts a deep prompt tuning technology combined with continuous prompts, which only requires fine-tuning of a small number of parameters to ensure that the representation space of the pre-trained language model is highly aligned with the node classification task space, thereby significantly improving the performance of the model on the node classification task; ensuring that the pre-training space is fully aligned with the downstream task space, which not only optimizes the generalization ability of the model, but also significantly improves the performance of the model on the node classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0077] Figure 1 It is a flowchart of a text graph node classification method based on a pre-trained language model and deep prompts provided in an embodiment of the present invention;

[0078] Figure 2 It is a framework structure diagram of a text graph node classification method based on a pre-trained language model and deep prompts provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0080] The present invention is further described in detail below with reference to the accompanying drawings and embodiments:

[0081] like Figure 1-2As shown, a text graph node classification method based on a pre-trained language model and deep hints specifically includes the following steps:

[0082] Step 1: Build a framework TegPPT including text attribute graph serialization and depth hint tuning;

[0083] Step 1.1, the text attribute graph serialization includes: for any target node in the text attribute graph, constructing a text sequence based on its text description; using multiple random walk strategies to select neighbor nodes of the target node, and further integrating the neighbor node text descriptions into the text sequence as auxiliary information; using a large language model to optimize the length of the generated text sequence; specifically including the following steps:

[0084] Step 1.1.1, take any node in the text attribute graph as the target, and start from the target node n i Text description of Constructing the initial sequence

[0085]

[0086] Step 1.1.2: Use multiple random walk strategies to sample neighbors, sort by the frequency of neighbor nodes passed by the walk, and i The set of all neighbor nodes N i Select the first K nodes from the sampled walking path N i ′:

[0087] N i ′={n j ∣n j ∈sort(N i ,Π),j=1,2,…,K} (2)

[0088] Where π=(π1,π2,...,π N ) is the system steady-state vector, and N is the number of nodes in the text attribute graph. Based on the stationary distribution theory of Markov chains, the system will reach a steady-state distribution after multiple random walks. Therefore, the steady-state probability π is used. i Approximate description node n i The frequency of being visited. The iterative process of Π can be described as:

[0089]

[0090]

[0091] When ∥Π is satisfied (k+1) -Π (k)When ∥1<ε, the system is considered to have reached a steady-state distribution, where k is the number of iterations, ∥·∥1 represents the L1 norm of the vector, and the preset threshold ε=10 -8 ; I represents the unit vector, α=0.85 is the equilibrium parameter controlling the random walk; is the transfer matrix, P i,j Represents slave node n i To its neighbor node n j Transition probability:

[0092]

[0093] Among them, A represents the adjacency matrix of the text attribute graph, A ij =1 indicates node n i With n j There is an edge between them, otherwise A ij = 0. The indicator function F(x) takes the value 1 when x is true, and 0 otherwise. deg(n i ) is a node n in an undirected graph i The degree of, correspondingly, in a directed graph, should be out-degree deg out (n i )

[0094] Step 1.1.3, integrate the neighbor node text description into the text sequence as auxiliary information. First, perform aggregation operation on the neighbor node text sequence, which is expressed as:

[0095]

[0096] Among them, N i ′ is the target node n i The set of sampled neighbors, Represents neighbor node n j ∑ represents the aggregation operation.

[0097] Then embed the aggregated text sequence into the text sequence framework of the target node middle:

[0098]

[0099] where Φ(·) represents the embedding function, It is a node description that comprehensively considers topological structure and text information.

[0100] Step 1.1.4, use the large language model (LLM) to optimize the length of the generated text sequence to improve the text quality. Let L max is the maximum input sequence length of the pre-trained language model, L sequenceis the length of the current sequence S. If the text sequence is too long, LLM needs to be optimized and abbreviated. Otherwise, no adjustment is required. The optimization strategy can be expressed as:

[0101]

[0102] In order to ensure that the abbreviated sequence S′ can retain the semantic information contained in the original text to the greatest extent, a semantic similarity measurement method, BERTScore, is introduced to guide the iterative process and further evaluate and optimize the abbreviation results.

[0103] First, use the BERT model to generate the context embedding matrix of the text sequence:

[0105] E S =[e1,e2,...,e n ], E S′ =[e1′,e2′,...,e m ′] (9)

[0106] Among them, e i and e′ j are the context embeddings of the i-th and j-th words in S and S′ respectively. Each element M in the similarity matrix M ij Indicates embedding i and e′ j The cosine similarity between M ij The larger the value, the higher the similarity between words i and j. ij It can be expressed as:

[0107]

[0108] Next, in order to comprehensively measure the semantic similarity between two text sequences, matching scores from different angles are introduced:

[0109]

[0110]

[0111] Precision evaluates the accuracy of the alignment of the abbreviation sequence S′ to the original sequence S. It means that for word i in S, find the word j in S' that best matches it. Conversely, Recall evaluates the alignment of the original sequence S to the abbreviation sequence S'. Finally, the fine-grained similarity score BERTScore between the two text sequences is calculated:

[0112]

[0113] By maximizing the BERTScore, the subtle contextual information and deep semantic relationships between text sequences are effectively captured, ensuring that the abbreviated sequence retains the semantic content of the original sequence as much as possible.

[0114] Step 1.2, the deep prompt tuning includes: the text sequence is processed by the word embedding layer of the pre-trained language model to form a high-dimensional word embedding matrix; on this basis, the continuous prompt is randomly initialized, and the word embedding matrix and the continuous prompt are input into the pre-trained language model together to achieve parameter fine-tuning operation; a linear classification head is defined to guide the fine-tuning of the parameters of the pre-trained language model to adapt to the node classification task of the text attribute graph; specifically, the following steps are included:

[0115] Step 1.2.1, input the constructed text sequence S′ into the word embedding layer of the pre-trained language model to generate a high-dimensional word embedding matrix:

[0116]

[0117] Here, Embed(·) represents the embedding function, where n is the length of the sequence S′ and d is the embedding dimension.

[0118] Step 1.2.2, initialize the continuous prompts for each layer:

[0119]

[0120] Where D is the number of layers of input continuous prompts and L is the length of continuous prompts in each layer.

[0121] Step 1.2.3, embed the word X and the corresponding Continuous prompt of layer The continuous prompts can start from any layer s and continue for D layers.

[0122] The input transmission process of each layer can be expressed as:

[0123]

[0124] Step 1.2.4, finally define the linear classification head to guide the fine-tuning of the pre-trained model parameters to adapt to the node classification task of the text attribute graph. After all the Transformer layers are processed, the output X of the last layer is (last) The final class label is calculated through a linear classification head

[0125] The calculation of the classification head can be expressed as:

[0126]

[0127] in, and are the weights and biases of the linear layer, respectively, and k is the number of target categories.

[0128] Step 2: Optimize the framework TegPPT by freezing the pre-trained language model parameters and optimizing the continuous prompt and classification head parameters. Specifically, it includes the following:

[0129] Freeze the pre-trained language model parameters and define a joint loss function to optimize the framework TegPPT:

[0130]

[0131] in, is the average loss function for the classification task, is the L2 regularization function for the continuous hint and classification head parameters.

[0132] Average loss function for classification tasks

[0133]

[0134] Where N represents the number of nodes in the text attribute graph, y ij represents the true label of the jth class of the i-th instance, and is the corresponding predicted probability.

[0135] L2 regularization function for continuous hint and classification head parameters

[0136]

[0137] Where λ is the regularization coefficient. Since the parameters of the pre-trained language model are frozen, the gradient is only for the trainable parameters, namely the continuous hint H, the classification head weight W and the bias b. It can prevent overfitting and improve the generalization ability of the model.

[0138] In summary, the present invention first constructs the framework TegPPT, including: 1) text attribute graph serialization. For any target node in the text graph, a text sequence is constructed based on its text description, and the text description of the neighboring node is integrated into it as auxiliary information. Through the information interaction of the neighboring nodes, the local topological information of the target node is effectively integrated to promote the unified encoding of the text and topological structure; 2) deep prompt tuning. Continuous prompts are deeply applied to multiple layers of the pre-trained model, not just the input layer; through layer-by-layer information transmission, the pre-trained language model is guided to transfer the knowledge of the pre-trained space to the node classification task, and finally the high-dimensional output is converted into the probability distribution of predefined categories, thereby generating node category labels; finally, the framework TegPPT is optimized by freezing the pre-trained language model parameters and optimizing the continuous prompts and classification head parameters. The present invention can not only better capture the graph topology and text information, but also avoid the potential negative transfer risks in existing research.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text graph node classification method based on pre-trained language model and deep hint, characterized in that: Includes the following: Build TegPPT, a framework including text attribute graph serialization and deep hint tuning; The text attribute graph serialization includes: for any target node in the text attribute graph, constructing a text sequence based on its text description; using multiple random walk strategies to select neighbor nodes of the target node, and further integrating the neighbor node text descriptions into the text sequence as auxiliary information; using a large language model to optimize the length of the generated text sequence; The deep prompt tuning includes: processing the text sequence through the word embedding layer of the pre-trained language model to form a high-dimensional word embedding matrix; on this basis, performing random initialization of continuous prompts, and inputting the word embedding matrix and the continuous prompts into the pre-trained language model together to achieve parameter fine-tuning operation; defining a linear classification head to guide the fine-tuning of the parameters of the pre-trained language model to adapt to the node classification task of the text attribute graph; The framework TegPPT is optimized by freezing the pre-trained language model parameters and optimizing the continuous prompt and classification head parameters.

2. According to claim 1, a text graph node classification method based on a pre-trained language model and deep hints is characterized in that: The text attribute graph serialization specifically includes the following steps: Step 1.1.1, take any node in the text attribute graph as the target, and start from the target node n i Text description of Constructing the initial sequence Step 1.1.2: Use multiple random walk strategies to sample neighbors, sort by the frequency of neighbor nodes passed by the walk, and i The set of all neighbor nodes N i Select the first K nodes from the sampled walking path N′ i : N′ i {n j ∣n j ∈sort(N i ,Π),j=1,2,…,K} (2) Where π=(π1,π2,...,π N ) is the system steady-state vector; N is the number of nodes in the text attribute graph; Based on the stationary distribution theory of Markov chain, the system will reach a steady-state distribution after multiple random walks, so the steady-state probability π is used i Approximate description node n i The frequency of being visited; the iterative process of Π is described as: When ∥Π is satisfied (k+1) -Π (k) When ∥1<ε, the system is considered to have reached a steady-state distribution, where k is the number of iterations; ∥·∥1 represents the L1 norm of the vector; the preset threshold ε=10 -8 ; I represents the unit vector; α = 0.85 is the equilibrium parameter controlling the random walk; is the transfer matrix; P i,j Represents slave node n i To its neighbor node n j Transition probability: Among them, A represents the adjacency matrix of the text attribute graph; A ij =1 indicates node n i With n j There is an edge between them, otherwise A ij =0; the indicator function F(x) takes the value 1 when x is true, otherwise it is 0; deg(n i ) is a node n in an undirected graph i The degree of, correspondingly, in a directed graph, should be out-degree deg out (n i ) Step 1.1.3, integrate the neighbor node text description into the text sequence as auxiliary information. First, perform aggregation operation on the neighbor node text sequence, which is expressed as: Among them, N′ i The target node n i The set of sampled neighbors of ; Represents neighbor node n j A text sequence; ∑ represents an aggregation operation; Then embed the aggregated text sequence into the text sequence framework of the target node middle: Where Φ(·) represents the embedding function; It is a node description that comprehensively considers topological structure and text information; Step 1.1.4, use a large language model to optimize the length of the generated text sequence to improve the text quality; Let L max is the maximum input sequence length of the pre-trained language model, L sequence is the length of the current sequence S. If the text sequence is too long, a large language model is needed to optimize the abbreviation. Otherwise, no adjustment is required. The optimization strategy is expressed as: In order to ensure that the abbreviated sequence S′ can retain the semantic information contained in the original text to the greatest extent, a semantic similarity measurement method BERTScore is introduced to guide the iterative process and then evaluate and optimize the abbreviation results.

3. According to claim 1, a text graph node classification method based on a pre-trained language model and depth hints is characterized in that: In step 1.1.4, the semantic similarity metric BERTScore guides the iterative process, which specifically includes the following: First, use the BERT model to generate the context embedding matrix of the text sequence: THE S =[e1,e2,...,e n ],THE S′ =[e′1,e′2,...,e′ m ] (9) Among them, e i and e′ j are the context embeddings of the i-th and j-th words in S and S′ respectively; each element M in the similarity matrix M ij Indicates embedding i and e′ j The cosine similarity between ij The larger the value, the higher the similarity between words i and j. ij It is expressed as: Next, in order to comprehensively measure the semantic similarity between two text sequences, matching scores from different angles are introduced: Precision evaluates the accuracy of the alignment of the abbreviation sequence S′ to the original sequence S. It means that for word i in S, find the word j in S′ that best matches it. Conversely, Recall evaluates the alignment of the original sequence S to the abbreviated sequence S′. Finally, the fine-grained similarity score BERTScore between the two text sequences is calculated: By maximizing the BERTScore, the subtle contextual information and deep semantic relationships between text sequences are effectively captured, ensuring that the abbreviated sequence retains the semantic content of the original sequence as much as possible.

4. According to claim 1, a text graph node classification method based on a pre-trained language model and depth hints is characterized in that: The depth hint tuning specifically includes the following steps: Step 1.2.1, input the constructed text sequence S′ into the word embedding layer of the pre-trained language model to generate a high-dimensional word embedding matrix: Where, Embed(·) represents the embedding function, where n is the length of the sequence S′ and d is the embedding dimension; Step 1.2.2, initialize the continuous prompts for each layer: Where D is the number of layers of continuous prompts, and L is the length of continuous prompts in each layer; Step 1.2.3, embed the word X and the corresponding l-layer continuous prompt H l They can be input into the Transformer layer together. Continuous prompts can start from any layer s and continue for D layers. The input transmission process of each layer is expressed as: Step 1.2.4, finally define the linear classification head to guide the fine-tuning of the pre-trained model parameters to adapt to the node classification task of the text attribute graph; After all Transformer layers are processed, the output of the last layer is X (last) The final class label is calculated through a linear classification head The calculation of the classification head is expressed as: in, and are the weights and biases of the linear layer, respectively, and k is the number of target categories.

5. According to claim 1, a text graph node classification method based on a pre-trained language model and depth hints is characterized in that: The optimization of the framework TegPPT by freezing the pre-trained language model parameters and optimizing the continuous prompt and classification head parameters specifically includes the following contents: Freeze the pre-trained language model parameters and define a joint loss function to optimize the framework TegPPT: in, is the average loss function for the classification task, is the L2 regularization function of the continuous prompt and classification head parameters; Average loss function for classification tasks It is expressed as: Where N represents the number of nodes in the text attribute graph, y ij represents the true label of the jth class of the i-th instance, and is the corresponding predicted probability; L2 regularization function for continuous hint and classification head parameters It is expressed as: Among them, λ is the regularization coefficient; since the parameters of the pre-trained language model are frozen, the gradient is only for the trainable parameters, namely the continuous hint H, the classification head weight W and the bias b; adding It can prevent overfitting and improve the generalization ability of the model.

Citation Information

Patent Citations

  • Characterization model of text attribute graph, pre-trained self-supervision method and model framework of node representation update

    CN117312490A

  • Node classification method and device based on decoupling graph, equipment, medium and product

    CN117407529A

  • Customer data information processing method and device and electronic equipment

    CN117764595A

  • Social media rumor detection method based on active learning iteration

    CN117992651A

Cited By

  • Text task processing control method and device based on large language model

    CN119623649A

  • Multimodal drug molecule prediction method based on knowledge prompt

    CN119763718A

  • Text attribute graph self-supervised learning method, system and device and medium

    CN119962612A

  • A Self-Supervised Learning Method, System, Device and Medium for Text Attribute Graph

    CN119962612B

  • Different matching text attribute graph node classification method and system based on optimal transmission

    CN120973943A