Text attribute graph representation model, pre-trained self-supervised method, model framework for node representation update

Abstract

The application aims to provide a text attribute graph representation model, a pre-training self-supervised method, and a node representation updating model framework, the method comprising: modeling a hierarchical text attribute graph by decoupling words in a text attribute graph from a basic graph structure and modeling the words as nodes in the original graph structure; further capturing semantic information and structural information in original data by performing self-supervised tasks on different levels of the hierarchical text attribute graph; in a model training stage, updating the representation of the nodes by using a mean aggregator, using the updated node representation as input, and continuing pre-training by using the self-supervised task, so as to obtain the representation of the nodes and words in the hierarchical text attribute graph, and complete the node classification and link prediction tasks in the graph.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, and in particular to a text attributed graph representation model, a pre-training self-supervised method, and a node representation updating model framework. BACKGROUND

[0002] Text attributed graphs are ubiquitous in real life, for example, in the paper citation network, each paper is attached with text content, and in the social network, each user can be described by text description. The research on learning technology of text attributed graphs has attracted considerable attention in the fields of graph learning, information retrieval and natural language processing, reflecting the importance of understanding and analyzing text information in the context of graph-based structure.

[0003] The research on text attributed graphs is generally divided into three paradigms: 1) only using language models; 2) only using graph models; 3) using a combination of language models and graph models. Current research will encounter some problems, such as ignoring the relationship between nodes or words, scalability limitations, and lack of generality. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a text attributed graph representation model, a pre-training self-supervised method, and a node representation updating model framework. The new text attributed graph representation model and pre-training self-supervised method provided by the embodiments of the present application can simultaneously optimize the learning of the representation of the initial nodes and the representation of the word nodes. Further, the embodiments of the present application provide an aggregator-based node updating model framework. This method introduces multiple self-supervised pre-training tasks to deeply mine the semantic information and structural information of the hierarchical text attributed graph. After a certain number of training rounds, the aggregator in the framework updates the initial node features using the updated node representations, which can better learn the initial node and word node representations.

[0005] To achieve the above purpose, the embodiments of the present application provide a text attributed graph representation model, which is applied to representation learning of input graphs containing initial text features at nodes. The method comprises: taking a text attributed graph as input, decoupling the initial text contained in the nodes in the graph to form a large corpus, and then constructing edges between words according to the co-occurrence frequency of words in the corpus; after obtaining the underlying word network, constructing edges between initial nodes and word nodes based on the inclusion relationship between nodes and text, connecting the upper initial network and the underlying word network to form a hierarchical text attributed graph; and mapping the node features of the constructed hierarchical text attributed graph to a hidden space by a heterogeneous graph neural network for representation.

[0006] In one embodiment, the step of constructing the underlying word network from the decoupled text comprises: for the text decoupled from the nodes, merging them into a large corpus, tokenizing the corpus by a tokenizer, obtaining the set of all words, and regarding all different words as nodes of the graph; calculating the PMI value (a measure of the degree of co-occurrence between words) of each pair of words in each sliding window by applying the sliding window technique, and constructing a word-word edge between the word pairs with a PMI value greater than 0 to obtain the underlying word network.

[0007] In one embodiment, the step of constructing the hierarchical text attribute graph on the basis of the word network comprises: for the input text attribute graph, first obtaining the word sequence contained at each node by a tokenizer to obtain the containment relationship between the nodes and the words; after the word network is constructed, constructing a node-word edge by the obtained containment relationship, connecting the constructed word network with the original network structure, and constructing the hierarchical text attribute graph.

[0008] The embodiments of the present specification also provide a self-supervised method for pre-training and a model framework for updating node representation, which are applied to train a representation model of the text attribute graph, and the device comprises a plurality of self-supervised pre-training task units: a link prediction unit that captures structural information in the data by predicting edges present in the hierarchical text attribute graph; a node attribute reconstruction unit that captures semantic information in the data by masking part of the basic attributes of the initial nodes and then reconstructing the masked features using a heterogeneous graph auto-encoding model; an important word reconstruction unit that captures semantic information in the important words by masking part of the basic attributes of the important words and then reconstructing the masked features using a heterogeneous graph auto-encoding model; and an important word identification unit that captures the correspondence between the nodes and the important words by predicting the attributes of the node-word edges to determine whether they are important words. The optimization objectives of the units of each optimization device correspond to different levels of the hierarchical text attribute graph.

[0009] In one embodiment, a pre-training method of a model, the method comprising: a link prediction unit that captures structural information in the hierarchical text attribute graph by using a heterogeneous graph neural network to input the scores between node pairs and then training the model using a cross-entropy loss function, by negative sampling a certain amount of node-node edges that do not exist in the hierarchical text attribute graph as negative samples and using the existing edges as positive samples.

[0010] In an embodiment, a pre-training method of a model, the method comprising: a node attribute reconstruction unit, inputting an original graph structure and a masked feature matrix by masking initial features of original nodes in a hierarchical text attribute graph into a graph autoencoder to obtain reconstructed node features; re-masking hidden vectors of the masked feature nodes before inputting into a decoder in the autoencoder; and finally training the model to learn semantic information in the hierarchical text attribute graph through a shrinkage cosine loss.

[0011] In an embodiment, a pre-training method of a model, the method comprising: an important word reconstruction unit, inputting an original graph structure and a masked feature matrix by masking initial features of word nodes in a hierarchical text attribute graph into a graph autoencoder to obtain reconstructed word node features; re-masking hidden vectors of the masked feature nodes before inputting into a decoder in the autoencoder; and finally training the model to learn semantic information in the hierarchical text attribute graph through a shrinkage cosine loss.

[0012] In an embodiment, a pre-training method of a model, the method comprising: an important word identification unit, labeling node-word edges according to a corresponding relationship between important words and initial nodes, outputting vectors of node pairs through a heterogeneous graph neural network, taking a concatenation form of the vectors as a representation of the edges, mapping the representation of the edges into a probability of predicting whether it is an important word through a mapping head, and finally training the model to learn the corresponding relationship between the nodes and the important words through a cross-entropy function.

[0013] In an embodiment, a model framework for updating node representations, the method comprising: an aggregator unit, aggregating features of initial nodes and word nodes within one hop around a certain type of node, and setting different hyperparameters on different types of edges to obtain an updated feature matrix; and a multi-task loss unit, combining losses of the above self-supervised tasks into a multi-task loss by setting different hyperparameters to train the model.

[0014] As can be seen from the technical solutions provided by the embodiments of the present specification, the purpose of the embodiments of the present specification is to provide a text attribute graph representation model, a self-supervised pre-training method, and a model framework for updating node representations. The new text attribute graph representation model provided by the embodiments of the present specification can effectively realize the interaction between a single word and node information and make more use of initial text information. Further, the embodiments of the present specification provide a self-supervised pre-training method and a model framework for updating node representations. The training method provides multi-level supervision signals for the model under the self-supervised setting and can deeply mine structural information and semantic information in a hierarchical text attribute graph. The results of experiments also prove that the model and the training method and device provided by the present specification can effectively learn a generalized text attribute graph representation. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings required to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some of the embodiments described in the present specification, and other drawings can also be obtained by those of ordinary skill in the art without creative labor under the premise of the drawings.

[0016] Figure 1 is a self-supervised task diagram provided by the present specification for different levels of hierarchical text attribute graph;

[0017] Figure 2 is a text attribute graph modeling method, pre-training task and overall model framework diagram provided by the present specification; DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present specification will be described clearly and completely below in combination with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only some of the embodiments of the present specification, not all. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor should be within the scope of the present application.

[0019] A text attribute graph representation model provided by the present specification is applied to representing the initial nodes and the contained text of the text attribute graph. The method can include the following steps.

[0020] Step S10: For the input text attribute graph, the original text contained in the initial node of the text attribute graph is decoupled to construct a large corpus. Different words in the corpus are regarded as word nodes, and then the PMI value of each word pair in each sliding window is calculated by using the sliding window technology. The word-word edge is constructed between the word pairs with positive PMI values, and the underlying word network is obtained.

[0021] Step S12: For the input text attribute graph, the word sequence contained in each word node is obtained by using a word segmenter. According to the relationship between the node and the word (the relationship can be divided into two types: containing or not containing), the underlying word network and the initial node graph are connected by constructing node-word edges to construct a hierarchical text attribute graph.

[0022] Step S14: The hierarchical text attribute graph is represented and learned by using a heterogeneous graph neural network. The features of the initial node and the word node are mapped to different hidden spaces, and different parameter matrices are set on different types of edges to complete different information transmission and aggregation.

[0023] For the convenience of the present application, please refer to Figure 2 . Figure 2 The method of modeling text attributed graph as hierarchical text attributed graph is given in part 1) of the present application. Current text attributed graph research usually uses graph- independent methods such as bag-of-words, pre-trained bert to convert the text information carried at the node into digital features and as the basic attributes of the node. Such practice compresses the information of the original text and does not make full use of it. At the same time, the fixed representation of the text may not be suitable for some scenarios. For example, “transformer” refers to a device used to reduce or increase the power supply voltage, while in the academic community it represents a model. Such methods ignore the relationship between words in the text.

[0024] First, in order to enhance the modeling of text information within the text attributed graph node, the method of the present application constructs a hierarchical text attributed graph containing original nodes and word nodes. More specifically, the word nodes are decoupled from the corpus composed of text information from all initial nodes. Then the edges between nodes are constructed based on the occurrence of words in nodes (node-word edges) and the co-occurrence of words in the entire corpus (word-word edges), so that the method can capture the subtle differences of the text at a more fine-grained level.

[0025] The present application proposes to use multiple self-supervised tasks to guide the heterogeneous graph neural network to learn semantic information and structural information in the hierarchical text attributed graph. Please refer to Figure 1 : For the initial network level, we use link prediction and node attribute reconstruction to capture the basic semantic information and structural information contained in the data; for the connection part of the initial network and the word network, we use the important word identification task to obtain the correspondence between the node and the important word, and ensure that the important word provides more important information for the node; for the underlying word network, we use important word reconstruction to learn the semantic information of the word node.

[0026] In the present embodiment, please refer to Figure 2 , the hierarchical text attributed graph constructed in 1) is used as input, and the multiple loss of multiple self-supervised tasks in 2) is used as optimization target to guide the heterogeneous graph neural network model to perform representation learning on the node.

[0027] In the present embodiment, for the hierarchical text attributed graph, first the embedding of a single word is generated by the pre-trained Bert model as the initial feature of the word node; then according to the text information contained in the node, the average value of the text embedding of all words of the node is obtained by using the pre-trained Bert model as the feature of the initial node.

[0028] By the characterization model provided in the embodiment, a hierarchical text attributed graph with initial nodes and words is constructed, and a bridge is built for information interaction. The embodiment enables a single word to participate in the training process, and the original node can obtain information from the related nodes and its own words. The text attributed graph task is evaluated on tasks such as node classification and link prediction. The experimental results show that the method provided in the specification can effectively model the semantic and structural information of the text attributed graph and better learn the generalized representation.

[0029] The specification also provides a self-supervised method for pre-training, which is applied to training the representation model, and the method comprises the following steps.

[0030] Step S20: The topology of the hierarchical text attributed graph is predicted to guide the model to learn the structural information in the data.

[0031] Step S22: The features of part of the initial nodes are masked, and then the masked node features are reconstructed using a graph autoencoder, and finally a scaled cosine function is used to guide the model to perform representation learning.

[0032] Step S24: The features of part of the important word nodes are masked, and then the masked word node features are reconstructed using a graph autoencoder, and finally a scaled cosine function is used to guide the model to perform representation learning.

[0033] Step S26: The attributes of the node-word edges are predicted to guide the model to learn the belonging relationship between the nodes and the important words.

[0034] Step S28: The corresponding hyperparameters are set for the several different self-supervised tasks of steps S20-S26, and the obtained multiple losses are used to guide the model to learn the hierarchical text attributed graph.

[0035] In the embodiment, the training tasks correspond to different levels of training signals. Please refer to Figure 1 , wherein the initial network level corresponds to the link prediction unit and the node attribute reconstruction unit; the connection level between the initial network and the low-level word network corresponds to the important word identification unit; and the word network level corresponds to the important word reconstruction unit.

[0036] Please refer to Figure 2 . In the embodiment, the text is decoupled in the basic text attributed graph to construct a word network level, which enhances the modeling of the text information in the text attributed graph node, enabling us to capture the subtle differences of the text at a more fine-grained level.

[0037] In the embodiment, the pre-training method of the text attributed graph representation model comprises a link prediction unit. Specifically, in the design of the link prediction task, we regard it as a binary classification problem and train the model through negative sampling: first, we regard the edges existing in the graph as positive samples and regard some edges not existing in the graph as negative samples; second, for each node pair (u, v) in the graph, we calculate their scores based on the representations h u and h v . The score of the initial node and the word node pair of the negative sample: e u,v_i represents; wherein is the dot product, and can be any other function for calculating similarity. By marking the positive sample as 1 and the negative sample as 0, we can optimize the R-GNN using the following loss function:

[0038]

[0039] wherein σ represents an activation function, v i ~ P n (v) represents a negative sampling distribution.

[0040] In the embodiment, the pre-training method of the text attributed graph representation model comprises node attribute reconstruction. Specifically, first, the initial nodes are sampled, then a mask token [MASK], a learnable vector, is used, and the feature matrix of the word node remains unchanged; the masked feature matrix X and Z are input into the graph encoder to obtain the hidden variable matrix H, and the hidden variable of the masked node is re-masked before being input into the graph decoder; finally, H is input into the graph decoder to obtain the reconstructed matrix W, and the model is trained through the scaling cosine function:

[0041]

[0042]

[0043]

[0044] wherein γ represents a scale hyperparameter; x i represents the basic attribute of the original node; z i represents the basic attribute of the word node; V paper represents a set of original nodes; represents a set of original nodes that need to be masked; V word represents a set of word nodes; w i represents the reconstructed attribute of the original node.

[0045] In the embodiment, the pre-training method of the text attribute graph representation model includes important word reconstruction. Specifically, first, important word nodes are sampled, then a masking token [MASK], a learnable vector, and an initial node feature matrix are used without change; the masked feature matrix Z and X are input into the graph encoder to obtain the hidden variable matrix H, and the hidden variable of the masked node is re-masked before being input into the graph decoder; finally, H is input into the graph decoder to obtain the reconstruction matrix W, and the model is trained by a scaling cosine function:

[0046]

[0047] wherein z i represents the basic attribute of the word node; w i represents the reconstructed word node attribute; represents a set of key words.

[0048] In the embodiment, the pre-training method of the text attribute graph representation model includes important word recognition. Specifically, since each node contains different important words, we design an important word recognition task targeting at judging important words: first, we mark the node-word edges differently according to whether the word is an important word of the node, 1 representing important and 0 representing unimportant; second, we concatenate the representation h u and h m of each node pair (u, m) as the representation h u,m of the edge; then we input the representation of the edge into the projection head to predict the label of the edge, and then use the following loss function for optimization:

[0049]

[0050] y′ u,m = projection-head(h u,m )

[0051]

[0052] wherein y′ u,m represents the probability of being predicted as an important word edge.

[0053] In the embodiment, the training target of the text attribute graph representation model in the training process is:

[0054]

[0055] wherein λ1, λ2, and λ3 are hyperparameters.

[0056] In an embodiment, the pre-training method of the text attribute graph representation model comprises: randomly sampling a part of edges that do not exist in the graph as negative samples and edges that exist in the graph as positive samples, calculating scores between node pairs as scores of the edges based on the heterogeneous graph neural network, and predicting whether the edges exist.

[0057] This embodiment corresponds to the link prediction training target, which is not described in detail here.

[0058] In an embodiment, the pre-training method of the text attribute graph representation model comprises: randomly masking the features at the initial nodes, and reconstructing the features of the masked nodes based on the heterogeneous graph autoencoder.

[0059] This embodiment corresponds to the node attribute reconstruction training target, which is not described in detail here.

[0060] In an embodiment, the pre-training method of the text attribute graph representation model comprises: randomly masking the features at the word nodes, and reconstructing the features of the masked nodes based on the heterogeneous graph autoencoder.

[0061] This embodiment corresponds to the important word reconstruction training target, which is not described in detail here.

[0062] In an embodiment, the pre-training method of the text attribute graph representation model comprises: marking the node-word edges based on the relationship between the important words and the initial nodes, outputting vector representations of node pairs based on the heterogeneous graph neural network, and taking the concatenated vector representations as representations of the edges, and inputting the labels of the edges into a mapping head to predict the labels.

[0063] This embodiment corresponds to the important word recognition training target, which is not described in detail here.

[0064] The present specification also provides a node updating model framework for training the representation model, the method comprising: after a certain number of rounds of training using the multi-task guided heterogeneous graph neural network of the above embodiments, aggregating the features of the initial nodes and the word nodes within one hop around a certain type of node through two average value aggregators, setting adaptive parameters to control the contribution of different types of nodes to the target node, and finally inputting the updated feature matrix into the heterogeneous graph neural network for training again.

[0065] The specification also provides a model training apparatus applied to a text attribute graph representation model, as described in the above embodiments. Since the principle of solving the problem of the model training apparatus is similar to that of the model training method, the implementation of the model training apparatus can be referred to the implementation of the model training method, and the repeated parts will not be described herein. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and contemplated. The apparatus can specifically include: a link prediction unit that randomly samples a part of edges that do not exist in the graph as negative samples, and edges that exist as positive samples, calculates the score between node pairs as the score of the edge based on the heterogeneous graph neural network, and predicts whether the edge exists; a node attribute reconstruction unit that randomly masks the features at the initial nodes, reconstructs the features of the masked nodes based on the heterogeneous graph autoencoder; an important word reconstruction unit that randomly masks the features at the word nodes, reconstructs the features of the masked nodes based on the heterogeneous graph autoencoder; an important word identification unit that labels the node-word edges based on the belonging relationship between the important words and the initial nodes, outputs the vector representation of the node pair based on the heterogeneous graph neural network, and takes the concatenated vector representation as the representation of the edge, and predicts the label of the edge in the input to a mapping head; and a node updating unit that updates the features of nodes of a certain type through two average value aggregators, and inputs the obtained feature matrix to the heterogeneous graph neural network again for training.

[0066] During the experiment, we choose R-GAT as the basic model. After training for 2000 rounds, we update the features of papers and words with the aggregator, and then we input the normalized updated features into the same R-GAT. During the whole process, we use AdamW Optimizer and cosine learning rate decay to train the model, and minimize the loss L without preheating; we set the number of layers of R-GAT to 2, the dimension of the hidden layer to 768, use the layernorm way, the learning rate to 0.001, and the total number of training rounds to 4000 rounds.

[0067] During the experiment, we select a benchmark dataset ogbn-arxiv as the basic dataset, which represents the citation network between all computer science (CS) arxiv papers in the MAG index. The original text of each node and its title and abstract is an arxiv article, and each directed edge represents that one article cites another article. In addition, all papers have information about the year of publication.

[0068] Table 1: Basic information of ogbn-arxiv dataset

[0069]

[0070] During the experiment, we suggest to split the dataset into four parts according to the publication date of the paper to adapt to the pre-training setting, where 41% of the papers are published before 2017; 13% of the papers are published in 2017; 17% of the papers are published in 2018; and 29% of the papers are published since 2019. In order to make our experimental results more persuasive and generalizable, we consider two settings: 1) Directly learning: In this setting, we choose all papers to participate in the pre-training phase; in the fine-tuning phase, we suggest training on papers published before 2017, validating on papers published in 2018, and testing on papers published since 2019; 2) Inductive learning: In this case, we pre-train on papers published before 2016, validate on papers published in 2017, and test on papers published since 2019.

[0071] Table 2: Hierarchical text property graph information constructed

[0072]

[0073] During the experiment, we also consider two downstream tasks to examine the generality of our method. They are: 1) Topic prediction: This task is about predicting the topic area of arXiv CS papers, which is manually labeled by the paper authors and arXiv hosts. Formally, this task can be formulated as a 40-class classification problem; 2) Important word identification: This task aims to identify important words according to the corresponding relationship between paper words. Formally, it can be regarded as a binary classification problem. We conduct experiments on the constructed hierarchical text property graph and use the implementation method stated in this specification as the basis for training the model. On the topic prediction task, we report the accuracy; on the important word identification task, we report the ROC / AUC, and each experimental result is the average of 10 predictions, and the standard deviation is reported. The NRUP in the figure is the method provided in this specification.

[0074] Table 3: Comprehensive performance of different models. The best performance of each indicator is in bold.

[0075]

[0076] It can be seen that compared with the selected baseline model, our NRUP achieves better or competitive performance, proving the effectiveness and portability of our method.

[0077] At the same time, in this scene example, in order to verify the synergistic effect of each level of self-supervised task, we design an ablation experiment for different self-supervised tasks, and the results are as shown in Table 4:

[0078] Table 4. Results of partial ablation experiments.

[0079]

[0080] In this ablation experiment, we remove the part of the node update in the model framework, and train the model using different tasks in the inductive learning setting, and report the performance on the test dataset. From the table, we can see that the task in a specific scenario can bring benefits to the basic task, and our multi-loss NRUP achieves the best performance.

[0081] In this scenario example, the optimization of word representation is a feature of our joint modeling method based on hierarchical text attributed graph. To verify that the representation of words is indeed optimized during training, we conduct experiments, and the results are shown in Table 5:

[0082] Table 5: Results of word representation

[0083] (Word embedding represents the paper representation generated by the word node representation, Paper embedding represents the representation output by RGAT)

[0084]

[0085] We use two basic self-supervised tasks in the inductive learning setting to verify that the word embedding is indeed optimized; we average the optimized word representation obtained after training 2000 rounds of a certain task as the representation of the downstream paper, and then train a linear classifier on the downstream data to predict the field of the paper. The experimental results show that the word node representation is optimized as the paper node, which is the reason why our update framework is effective.

[0086] At the same time in this scenario example, in order to verify the role of the main components in the node update framework, we designed an ablation experiment on whether to contain the node representation update component and whether to perform normalization, and the results are shown in Table 6:

[0087] Table 6. Results of ablation experiments to explore the node representation update component.

[0088]

[0089] We explore the impact of the update architecture, and the table shows the results of whether to use the update architecture for pre-training. Without the update component, we use certain self-supervised tasks for end-to-end pre-training, and keep the best pre-trained model to generate the representation of the downstream dataset; while in our architecture, we update the features in the middle of pre-training, and keep the best model in the later stage of training. The performance on the downstream dataset shows that our framework is effective.

[0090] The normalization plays a crucial role in updating the pre-training framework, which restores the updated feature matrix to the normal distribution and eliminates the influence of distribution shift. The table results show the results of updating the features without normalization and using different normalization methods. We find that the effect of normalizing the feature matrix is significantly better than not normalizing. At the same time, the effect of separately normalizing the feature matrix of the paper and the word is better than that of the overall normalization. In short, normalization brings improvement.

[0091] The specification also provides a text attribute graph representation device, which comprises:

[0092] The hierarchical text attribute graph modeling unit, the modeling method comprises decoupling the text at the node and constructing the underlying word network, connecting the word network with the initial network to obtain the hierarchical text attribute graph.

[0093] The model training unit is used for self-supervised representation learning based on the hierarchical text attribute graph, including a link prediction unit, a node attribute reconstruction unit, an important word reconstruction unit, and an important word identification unit.

[0094] The node representation updating unit is used to update the initial node features during the training process, aggregate the features of the one-hop initial node and word node neighbors through the mean value aggregator, and replace the initial features at the node with the aggregated and normalized features to continue the training.

[0095] In this embodiment, only the differences from the foregoing embodiments are described, and the other contents can be explained by referring to the contents of the foregoing embodiments, which will not be described here.

[0096] A node representation updating model framework, characterized by being applied to representation learning of a hierarchical text attribute graph, the model framework comprising:

[0097] An optimization device based on multi-task loss, mainly used for model optimization in the pre-training stage, the optimization device comprising: a link prediction unit, a node attribute reconstruction unit, a key word reconstruction unit, and a key word identification unit.

[0098] Link prediction unit: for the original node level, the core goal is to learn the structure information in the original level, by calculating the score e u,v Optimize the model, the formula is as follows:

[0099]

[0100] Node attribute reconstruction unit: for the original node level, the core goal is to learn the semantic information in the original level, and the model is optimized by minimizing the error of the initial feature matrix X and the reconstruction matrix W, the formula is as follows:

[0101]

[0102] Keyword reconstruction unit: for the word node level, the core goal is to learn the semantic information of the word node, and the model is optimized by minimizing the error of the initial feature matrix Z and the reconstruction matrix W, the formula is as follows:

[0103]

[0104] Keyword recognition unit: for the link part between the original node and the word node, the core goal is to identify the node that contributes more to the original node, and the label y u,v between the node pair (u, v) is predicted, and the model is optimized, the formula is as follows:

[0105]

[0106] y′ u,m =projection-head(h u,m )

[0107]

[0108] Based on the encoding module of the heterogeneous graph neural network, in the encoding module, information can be calculated and aggregated for each relationship in the heterogeneous graph by the heterogeneous graph neural network RGNN, and messages from different relationship types can be aggregated for each node. Unlike GNN, RGNN saves a parameter matrix for each relationship type, and the message propagated on the relationship type is affected by the parameter matrix; the features of the initial nodes and word nodes in the hierarchical text attribute graph can be mapped to the latent space by RGNN, which is based on the semantic information of the structure information and the features of the nodes themselves, and each node has a corresponding representation vector at the node;

[0109] Based on the aggregator node update device, after training for a certain number of times, the features of the initial nodes and word nodes around a node are aggregated by the aggregator, and different hyperparameters are set for the initial nodes and word nodes in the aggregation process to distinguish the contribution degree of different nodes, and finally the updated features of the node are obtained. In the device, there are aggregators for initial nodes and word nodes, which are distinguished by different hyperparameters;

[0110] Based on the coding model and the training module of the optimization device, the module is first characterized by the original hierarchical text attribute graph by the coding model to obtain the representation matrix of the text attribute graph, and then the loss of different units is calculated by the multi-task loss optimization device, and the multi-loss in the multi-task loss optimization device is taken as the optimization target, and the formula is as follows:

[0111]

[0112] Finally, the optimization device composed of AdamW and other components optimizes the heterogeneous graph neural network parameters in the coding module.

[0113] The above-mentioned embodiments of the device or module, etc. can be specifically implemented by a computer chip or entity, or by a product with certain functions. For the convenience of description, the above device is described as various modules respectively described in function. Of course, in the implementation of the present application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules implementing the same function can be combined to realize the functions of the modules, etc. The above-described device embodiment is only illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division mode, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0114] Those skilled in the art also know that in addition to implementing the controller in the form of pure computer readable program code, the controller can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps to achieve the same function. Therefore, such a controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0115] The present application can be described in the general context of computer-executable instructions, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, and the like that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0116] Those skilled in the art can clearly understand the application by the description of the above embodiments that the application can be implemented by means of software and necessary universal hardware platforms. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to make a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the application.

[0117] The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. The application can be used in many general or special computer system environments or configurations. For example, personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, small computers, large computers, distributed computing environments including any of the above systems or devices, and the like.

[0118] Although the application is described through the embodiments, those skilled in the art know that the application has many modifications and changes without departing from the spirit of the application, and it is expected that the appended claims include these modifications and changes without departing from the application.

Claims

1. A method for generating a representation model of a text attribute graph, characterized in that, include: Using a text attribute graph as input, the original text carried at the initial nodes in the text attribute graph is decoupled and segmented. Then, edges are constructed between words based on the co-occurrence frequency in the original text, thereby building a word network. The original text, after decoupling, includes word nodes corresponding to each word. The original text attribute graph is combined with the word network to form a hierarchical text attribute graph. For the input text attribute graph, all the original texts are decoupled to construct a corpus. The corpus is segmented to obtain all the words contained in the original text. The decoupled individual words are used as word nodes, and edges are constructed between words according to the contribution frequency in the corpus to obtain the decoupled word network. Based on the original text attribute graph, edges between the initial nodes and word nodes are constructed according to the inclusion relationship, thereby obtaining a hierarchical text attribute graph. The upper layer of the hierarchical text attribute graph is the original text attribute graph, and the lower layer is a word network. Attributes are defined for word nodes and initial nodes. For attributes at word nodes, the representation generated by a pre-trained language model is used as its feature. For attributes at initial nodes, the average value is obtained by aggregating the features of all one-hop word node neighbors around them. A representation model of a text attribute graph is generated by learning the representation of a hierarchical text attribute graph through a heterogeneous graph neural network. The heterogeneous graph neural network simultaneously learns the representations of the nodes in the text attribute graph and the word nodes in the original text.

2. The method as described in claim 1, characterized in that, The step "generating a representation model of the text attribute graph by learning the representation of the hierarchical text attribute graph through a heterogeneous graph neural network" includes: For the input of hierarchical text attribute graph, the corresponding encoding module of heterogeneous graph neural network is set according to the relation type, so that the information transmission has corresponding weight matrices on different types of edges; In the heterogeneous graph neural network, the initial node and word node representations are updated simultaneously based on the optimization objective; ultimately, the optimized node representations and word representations can be obtained.

3. A representation model for a text attribute graph, employing the method described in claim 1, characterized in that, The representation model includes a hierarchical text attribute graph, wherein the upper layer of the hierarchical text attribute graph is the original text attribute graph, and the lower layer of the hierarchical text attribute graph is a word network. For the attributes at word nodes, the representation generated by a pre-trained language model is used as its feature; for the attributes at the initial node, the feature is obtained by aggregating the features of all its surrounding one-hop word node neighbors and taking the average value.

4. A self-supervised method for pre-training, used for self-supervised training of a text attribute graph representation model as described in claim 3, characterized in that, The method includes: Link prediction: Edges existing in the hierarchical text attribute graph are taken as positive samples, and edges not existing in the hierarchical text attribute graph are randomly negatively sampled as negative samples; the problem is treated as a binary classification problem, and the score of each node pair is calculated using a similarity-based function of a heterogeneous graph neural network, and the prediction error is measured by the cross-entropy function; Node attribute reconstruction: The basic attributes of the initial nodes are randomly masked, and learnable vectors are used as their initial features; the original graph structure and the masked feature matrix are taken as input, and the initial features of the masked nodes are reconstructed through a graph autoencoder, and the reconstruction loss is measured by a scaling cosine function. Important word reconstruction: The basic properties of word nodes are randomly masked and a learnable vector is used as its initial feature. The original graph structure and the masked feature matrix are taken as input, and the initial features of the masked nodes are reconstructed by a graph autoencoder. The reconstruction loss is then measured by a scaling cosine function. Important word identification: Based on different types of edge labels, the representation of each node pair is output through a heterogeneous graph neural network, and the representations of the node pairs are concatenated to form the edge representation; the problem is treated as a binary classification problem, and the prediction error is finally measured by the cross-entropy function.

5. The method as described in claim 4, characterized in that, In the node attribute reconstruction task, the latent space vectors at the masked nodes are masked again before being input into the graph decoder.

6. A multi-task loss optimization device, characterized in that, The apparatus, applied to the representation model of the text attribute graph as described in claim 4, comprises: Link prediction units predict edges that do not exist in the negative sampling hierarchical text attribute graph; The node attribute reconstruction unit masks some of the basic attributes of the initial nodes and reconstructs the original features through a heterogeneous graph autoencoder, and then learns the reconstruction error through a cosine loss function. The keyword reconstruction unit masks the basic attributes of some important word nodes and reconstructs the original features through a heterogeneous graph autoencoder, and then learns the reconstruction error through a cosine loss function. The keyword recognition unit marks the node-word edges based on whether they are keywords of the initial node. The edge representation is obtained by concatenating the node pair vectors output by the heterogeneous graph neural network. Finally, the cross-entropy function is used to learn the word node and initial node representations in the hierarchical text attribute graph based on the labels.

7. A model framework for node representation updates, employing the method as described in claim 1, characterized in that, The model framework, applied to representation learning of hierarchical text attribute graphs, includes: The optimization device based on multi-task loss is mainly used for model optimization in the pre-training stage. The optimization device includes: a link prediction unit, a node attribute reconstruction unit, a keyword reconstruction unit, and a keyword recognition unit. Link prediction unit: For the original node level, its core objective is to learn the structural information in the original level by calculating the score e between node pairs (u,v). u，v The model is optimized using the following formula: Node attribute reconstruction unit: Targeting the original node level, its core objective is to learn the semantic information in the original level. It optimizes the model by minimizing the error between the initial feature matrix X and the reconstruction matrix W, as shown in the following formula: Keyword Reconstruction Unit: Targeting the word node level, its core objective is to learn the semantic information of word nodes. It optimizes the model by minimizing the error between the initial feature matrix Z and the reconstruction matrix W, as shown in the following formula: Keyword recognition unit: This unit focuses on the link between the original node and the word node. Its core objective is to identify the node that contributes significantly to the original node, and to predict the label y between node pairs (u,v). u，v The model is optimized using the following formula: y′ u，m =projection-head(h u,m ) The encoding module based on the heterogeneous graph neural network (RGNN) can calculate and aggregate information for each relation in the heterogeneous graph, and aggregate messages from different relation types for each node. The RGNN stores a parameter matrix for each relation type, and messages propagated on that relation type are influenced by this parameter matrix. The RGNN can map the features of initial nodes and word nodes in the hierarchical text attribute graph into the latent space. This is based on the structural information of the graph and the semantic information of its own features to represent the initial nodes and word nodes, with each initial node and word node having a corresponding representation vector. The aggregator-based node update device, after training for a certain number of rounds, aggregates the features of the initial nodes and word nodes around a certain node through the aggregator. During the aggregation process, different hyperparameters are set for the initial nodes and word nodes to distinguish the contribution of different nodes, and finally the updated features of the node are obtained. The node update device has aggregators for initial nodes and aggregators for word nodes, which are distinguished by different hyperparameters. The training module based on the encoding model and optimization device first uses the encoding model to represent the original hierarchical text attribute map to obtain the representation matrix of the text attribute map. Then, the multi-task loss optimization device calculates the loss of different units and uses the multiple loss in the multi-task loss optimization device as the optimization objective. The formula is as follows. Finally, the optimization device optimizes the parameters of the heterogeneous graph neural network in the encoding module.

Images