A heterogeneous graph representation learning method fusing text attributes

Abstract

The application discloses a kind of heterogeneous graph representation learning methods of fusing text attribute, belong to graph data processing technical field, the learning method will text representation method and graph representation learning method be combined, based on transformer and graph neural network model is constructed.Firstly, the text feature representation based on text coding strategy is carried out using existing node text attribute, and the representation vector of text is integrated into the heterogeneous graph.And then, the missing attribute of node is completed based on the mean aggregation of neighbor node and the method of node feature transformation.Finally, the heterogeneous graph of completed attribute is input into existing graph neural network model, and the best node vector is generated through end-to-end training strategy.The application can deeply fuse node text attribute and thus improve the performance of graph neural network model.

Description

Technical Field

[0001] This invention belongs to the field of graph data processing technology, and specifically relates to a heterogeneous graph representation learning method that integrates text attributes. Background Technology

[0002] Graph data is a widely existing form of data in real life. From macroscopic data like the internet, knowledge graphs, and social networks, to microscopic data like proteins and compound molecules, graph structures can be used to model and represent data. Heterogeneous graphs can accurately and effectively model the rich semantic information and complex network relationships in the real world.

[0003] With the development of artificial intelligence, a large amount of real-world graph data has accumulated. How to deeply analyze and mine this graph data has become a research hotspot in data mining. The research methods for graph data are roughly as follows:

[0004] (1) Shallow Graph Embedding Model: The purpose of graph embedding is to project nodes in a graph onto a low-dimensional vector space, using low-dimensional dense vectors to represent any node in the network, thus making it more flexible for application in different data mining tasks; at the same time, in the low-dimensional vector space, the representation of nodes can reflect the relationships between nodes and retain the feature information of nodes. Most current methods are based on existing deep models and combine network features to learn the feature representation of nodes or edges, such as DeepWalk, which combines random walk and skip-gram models to learn node representations; LINE adds second-order similarity to first-order neighbor similarity to obtain node representations with rich information; SDNE uses a deep autoencoder to extract nonlinear features of the network structure to generate node representations; metapath2vec takes heterogeneous graphs as the research object, generates random node sequences under the guidance of meta-paths, and then inputs the sequences into the skip-gram model to generate node representations. Similar to metapath2vec, HERec proposed a type constraint strategy to filter node sequences and capture complex semantics in heterogeneous graphs.

[0005] (2) Graph Neural Network Model: Graph neural networks are deep graph embedding models that extend deep learning to graph-structured data. They aggregate neighbor information according to an information propagation framework, which consists of two steps: 1) propagating information according to the topological relationships between nodes; 2) aggregating node information to update node representations. Graph neural network models can be divided into spectral domain and spatial domain models. Spectral domain-based graph neural networks rely on the eigenvalue decomposition of the Laplacian matrix, such as GCN and FAGCN, which aggregate neighborhood feature information by designing different kernel functions. Spatial domain-based methods directly define convolutions on the graph to aggregate feature information from the neighborhood of each node. For example, GraphSAGE and GAT design different functions to aggregate neighborhood feature information. The above-mentioned graph neural networks are only applicable to homogeneous graphs. Some recent studies have extended graph neural networks to heterogeneous graphs. For example, HAN first learns the importance of neighboring nodes under different meta-paths through node-level attention; then it learns the importance of different meta-paths through semantic-level attention, and finally aggregates the feature information of neighboring nodes under different meta-paths based on importance. HetGNN first samples neighbors through random walks with restart, and then aggregates different types of node information in the neighborhood through a Long Short-Term Memory (LSTM) network and an attention mechanism. The MAGNN model first maps the feature information of different types of nodes into the same space, then aggregates the information under the meta-path, and finally completes the aggregation of node vectors under multiple meta-paths.

[0006] The above research demonstrates that deep learning methods have achieved good results in graph data representation learning. However, shallow graph embedding models and graph neural network models in graph data representation learning both suffer from the problem of not fully mining the textual information of nodes, resulting in the loss of semantic information. Therefore, how to achieve the integration of textual attributes into graph nodes and the scientific and accurate representation and completion of attributes, as well as the mutual enhancement between textual attribute representation, completion, and model design, has become a problem that current research needs to solve.

[0007] Existing network representation learning methods primarily focus on improving model performance. However, with the deepening research into heterogeneous graph neural networks, researchers have discovered that complete node attributes are a necessary prerequisite for algorithm operation. Existing research uses simple manual imputation methods (such as average imputation and one-hot vector imputation) to fill in missing attributes. These methods separate attribute completion from the graph representation learning process, ignoring the importance of accurate attributes for downstream tasks. Therefore, it is difficult to guarantee model performance using simply imputed attributes, especially when integrating node text attribute representation into graph neural networks. Traditional heterogeneous graph neural networks construct node attributes using one-hot vectors, resulting in semantic sparsity. For nodes with missing attributes, attribute completion is done by summing the mean, which introduces noise and degrades model performance. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a heterogeneous graph representation learning method that integrates text attributes. Based on heterogeneous graph representation learning, it deeply represents the text attributes of nodes in the heterogeneous graph, achieving a deep fusion technology between text attributes and the heterogeneous graph neural network model.

[0009] The technical solution of the present invention is as follows:

[0010] A heterogeneous graph representation learning method that incorporates text attributes includes the following steps:

[0011] Step 1: Obtain the text data T of the dataset nodes, pre-train the text data, and use the existing node text attributes to perform text feature representation based on the text encoding strategy to obtain the feature vector X of the text representation. T Represent it as a NumPy matrix and save it. Construct a heterogeneous graph based on the node relationships in the dataset to obtain the adjacency matrix A between nodes, and save the feature vector X. T As attributes of nodes with text attributes, they are incorporated into the heterogeneous graph;

[0012] Step 2: Using the adjacency matrix A, find nodes with text attributes in the neighborhood of the missing attribute node. Then, use mean aggregation and a fully connected network to transform the node features to complete the missing attribute, resulting in a complete node attribute matrix X. S ;

[0013] Step 3: Combine the adjacency matrix A and the node attribute matrix X of the complete heterogeneous graph with complete attributes. S The input is fed into an existing graph neural network model, and the optimal node representation vector Z is generated through end-to-end training of the node classification task.

[0014] Furthermore, the specific process of step 1 is as follows:

[0015] Step 1.1: Obtain the required data, construct the dataset, and use a web crawler algorithm to crawl the content in the current dataset as the text data of the nodes T = (W1, W2…W…). n ), W1, W2…W n Represents the words in text T;

[0016] Step 1.2: Construct an adjacency matrix A for each node in the text dataset according to the relationship between the nodes. The adjacency matrix A is a square matrix of the number of nodes * the number of nodes. Each element in the square matrix represents the edge relationship between the nodes. If there is a relationship between the nodes, the corresponding position is 1, otherwise it is 0. Thus, a heterogeneous graph with an adjacency matrix structure is constructed.

[0017] Step 1.3: Perform text preprocessing on the crawled text data T, including removing stop words and converting English uppercase to lowercase;

[0018] Step 1.4: Input the preprocessed text data T into the encoder. The Transformer model pre-trained for the machine translation task encodes the text attributes of the nodes and outputs the encoded representation vector. The encoder consists of 6 identical stacked layers. Each layer has two sub-layers: one sub-layer is a multi-head self-attention mechanism, and the other sub-layer is a feedforward network layer. Residual connections are used between the two sub-layers, and then layer normalization is performed. The output of each sub-layer is LayerNorm(x+Sublayer(x)), where Sublayer(x) is the function function of each layer; x represents the text feature.

[0019] Step 1.5: Input the encoder output and other linguistic forms of the text into the decoder, and obtain the text representation vector X through machine translation task pre-training. T =g(W1,W2…W n The decoder is also composed of six identical layers stacked together; in addition to the two sub-layers in the encoder, the decoder inserts a third sub-layer that performs a masked multi-head attention mechanism on the encoder's output; where g(·) is the text vectorization encoding method Transformer; T It is a 384-dimensional vector, which is represented as a 384-dimensional matrix in NumPy format containing the number of nodes with text attributes, and stored therein, with each row of the matrix representing a node.

[0020] Furthermore, the specific process of step 2 is as follows:

[0021] Step 2.1: Locate the node v that is missing text attributes. - Neighbor node v with text attribute in the neighborhood + ;

[0022] Step 2.2: Complete missing text attributes by aggregating the text attributes of neighboring nodes using the mean;

[0023] Node v with missing text attributes - By connecting neighbor nodes v with text attributes + Complete its own attributes; the calculation formula is as follows:

[0024]

[0025] in, These are the node attributes that need to be completed. It is node v - For text attributes in the neighborhood of nodes with text attributes, agg(·) is the completion method for mean aggregation. It is node v - The set of nodes in the neighborhood containing text information;

[0026] Step 2.3: Perform feature transformation on the text attributes to construct a text attribute representation vector for each node;

[0027] The node attributes are updated based on downstream tasks by performing feature transformation on node attributes through a fully connected network, specifically defined as follows:

[0028]

[0029] MLP is a two-layer fully connected network that performs feature transformation on the completed target node attributes to adapt them to downstream tasks. A represents node v. - and The adjacency matrix between them, D -1 Represents node v - The reciprocal of the degree, Represents node v - The text attribute matrix of the neighborhood containing text information nodes;

[0030] Step 2.4: Obtain the complete heterogeneous graph;

[0031] By combining step 2.3, we obtain the text attributes of nodes with missing text attributes and nodes with text attributes v. + The text attributes construct a complete attribute matrix. Finally, the adjacency matrix A from step 1.2 is combined to form a complete heterogeneous graph G = (A, X). S ).

[0032] Furthermore, the specific process of step 3 is as follows:

[0033] Step 3.1: Divide the nodes in the constructed heterogeneous graph G into training, validation, and test sets according to the specified proportions; and simultaneously input them into the graph neural network model:

[0034]

[0035] in, Let Z represent the graph neural network model, where Z represents the node representation obtained by the graph neural network. Then, Z is transformed to the dimension of the node labels using an MLP (Multi-Level Processing) to obtain...

[0036] Step 3.2: Perform classification prediction.

[0037]

[0038] Among them, L prediction Represents classification loss, Y represents the model's predicted value, and Y represents the node's label. Y is a matrix of the number of nodes multiplied by the number of node label categories, and f represents the cross-entropy loss function, which depends on the specific downstream task;

[0039] Step 3.3: Optimize the model by minimizing the training set cross-entropy loss. The cross-entropy loss function formula is as follows:

[0040]

[0041] in, and Y(x i These represent the predicted label probability distribution and the actual label probability distribution of nodes, respectively.

[0042] Step 3.4: Optimize the parameters by reducing the cross-entropy loss function in Step 3.3 through backpropagation end-to-end operation to update the trainable parameters in the graph neural network and the fully connected network until the classification accuracy on the validation set no longer increases and the model converges.

[0043] Step 3.5: After the model converges, verify the model performance using the test set, and obtain the optimal node representation vector Z.

[0044] The beneficial technical effects of this invention are as follows:

[0045] This invention constructs a Transformer model based on an encoder and decoder to encode node text attributes, creating attribute vectors with rich semantic information for nodes with text attributes. It proposes a method for completing missing node attributes based on mean aggregation of neighboring nodes and node feature transformation. Completing missing node attributes through mean aggregation and node feature transformation reduces noise from traditional methods (mean completion) and overcomes the separation of attribute completion from model training, thereby improving overall model performance. An end-to-end training strategy is designed to obtain node representation vectors, enabling the learning of representation vectors suitable for downstream tasks. This invention combines text representation methods with graph representation learning methods, fully exploring the semantic information of node text in heterogeneous graphs to enhance the expressive power of heterogeneous graph neural network models. This invention focuses on integrating text attributes into graph nodes, emphasizing scientifically accurate attribute representation and completion, achieving mutual reinforcement between text attribute representation, completion, and model design. The proposed method, combined with a graph neural network model, exhibits higher accuracy and stronger adaptability in node classification tasks. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method of the present invention;

[0047] Figure 2This is a flowchart representing the text of the present invention;

[0048] Figure 3 A flowchart for completing the node text attributes of this invention;

[0049] Figure 4 This is a flowchart of the end-to-end training graph neural network model in this invention. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0051] Text representation learning is the technical basis of this invention. This invention focuses on deep representation of node text information and integrates the obtained representation vectors into graph neural network representation learning to improve the representation capability of the graph neural network model.

[0052] This invention, based on graph data, aims to improve the performance of heterogeneous graph neural network models. The core problems it addresses include: First, the acquisition and representation of node text data. Second, text attribute completion for nodes with missing text attributes in heterogeneous graphs. Third, the design of an end-to-end training strategy for generating node representation vectors.

[0053] Key technical issue 1: Acquisition and representation of node text data

[0054] We crawled the titles of paper nodes from the DBLP dataset and the plot descriptions of movie nodes from the IMDB dataset as the text data for the nodes. The obtained text data T = (W1, W2…W…) n By fully extracting the semantic information of the text through text representation methods, a representation vector X is obtained. T =g(W1,W2…W n ), where g(·) is the text vectorization encoding method (Transformer). W1, W2…W n Represents the words in text T.

[0055] Key technical issue 2: Text attribute completion for nodes with missing text attributes in heterogeneous graphs

[0056] This invention achieves missing attribute completion for nodes based on the mean aggregation of neighboring nodes and node feature transformation. Text attribute completion involves adding missing text attributes to nodes in the network. Specifically, for nodes v that lack text attributes... - By connecting neighbor nodes v with text attributes + Complete your own attributes in, These are the node attributes that need to be completed. It is node v -For nodes with text attributes in their neighborhood, agg(·) is the completion method (mean aggregation). It is node v - The set of nodes in the neighborhood containing text information.

[0057] Key technical issue 3: Design of training strategy for end-to-end generation of node representation vectors

[0058] This invention combines a proposed heterogeneous graph representation learning method that integrates text attributes with a graph neural network and optimizes it end-to-end based on downstream tasks. Attributes of all nodes. The complete attribute matrix X S The graph's topology is input into the graph neural network model: in, This represents a graph neural network model, where Z is obtained through an MLP. Let Y represent the model's predicted value, Y represent the node label, and f represent the loss function, which depends on the specific downstream task. End-to-end model optimization is achieved by minimizing the loss.

[0059] like Figure 1 As shown, this invention follows a main framework of "node text representation - text attribute completion - end-to-end training". The framework first utilizes a transformer to perform text feature representation based on a text encoding strategy, and then integrates the text representation vector into a heterogeneous graph. Next, it completes the missing node attributes using a method based on the mean aggregation of neighboring nodes and node feature transformation. Finally, the heterogeneous graph with completed attributes is input into an existing graph neural network model, and node vectors are generated through an end-to-end training strategy.

[0060] This invention proposes a heterogeneous graph representation learning method that integrates node text attributes (HGNN-ETA). The framework includes three key parts: node text attribute encoding based on attention mechanism, node attribute completion based on neighbor aggregation mechanism, and end-to-end heterogeneous graph model construction and optimization.

[0061] In heterogeneous graphs, favorable node attributes can improve the overall performance of graph neural network models. Node text attributes in heterogeneous graphs possess important features (e.g., paper nodes in the DBLP dataset and movie nodes in the IMDB dataset), containing rich semantic information. Therefore, this invention integrates text information into the representation learning of heterogeneous graphs. This invention encodes node text attributes using a Transformer model pre-trained on a machine translation task. The encoder's input is the node text attributes, and the decoder's input is the encoder's output and other linguistic forms of the text. The text representation vector is obtained through pre-training on a machine translation task.

[0062] The specific process of node text attribute encoding is as follows: Figure 2As shown, the dataset acquisition method depends on the specific application scenario, including existing historical data of the recommendation system and current customer data acquired in real time by the system. The dataset in this embodiment includes two datasets: DBLP and IMDB. DBLP is an existing dataset, while IMDB is a self-constructed dataset. First, the text of the datasets (DBLP and IMDB) required for the experiment is obtained using a web crawler algorithm. Second, a heterogeneous graph is constructed based on the relationships between nodes in the dataset. Then, the text in the constructed heterogeneous graph is extracted. Finally, the node text attributes are encoded using a Transformer model based on an encoder and decoder to obtain the final text representation vector. The specific process is as follows:

[0063] Step 1.1: Use a web crawler algorithm to crawl the titles of paper nodes in the DBLP dataset and the plot descriptions of movie nodes in the IMDB dataset as the text data T = (W1, W2…W…). n W1, W2…W n Represents the words in text T;

[0064] Step 1.2: Construct an adjacency matrix A for the paper, author, conference, and paper keyword nodes in the DBLP dataset according to the relationships between the nodes (author writes paper, paper is published in conference, paper contains keywords); and for the movie, director, and actor nodes in the IMDB dataset according to the relationships between the nodes (director shoots movie, actor stars in movie). The adjacency matrix A is a square matrix of the number of nodes * the number of nodes. Each element in the square matrix represents the edge relationship between the nodes. If there is a relationship between the nodes, the corresponding position is 1, and otherwise it is 0. This constructs a heterogeneous graph.

[0065] Step 1.3: Preprocess the crawled text data (paper titles in DBLP, movie plots in IMDB) by removing stop words and converting English uppercase to lowercase.

[0066] Step 1.4: Input the cleaned text data T into the encoder. The Transformer model, pre-trained for the machine translation task, encodes the text attributes of the nodes and outputs the encoded representation vector. The encoder consists of six identical stacked layers. Each layer has two sub-layers: one is a multi-head self-attention mechanism, and the other is a feedforward network layer. Residual connections are used between the two sub-layers, and then layer normalization is performed. That is, the output of each sub-layer is LayerNorm(x+Sublayer(x)); where Sublayer(x) is the function function of each layer; x represents the text feature.

[0067] Step 1.5: Input the encoder output and other linguistic forms of the text into the decoder, and obtain the text representation vector X through machine translation task pre-training.T =g(W1,W2…W n The decoder is also composed of six identical layers stacked together. In addition to the two sub-layers in the encoder, the decoder inserts a third sub-layer that performs a multi-head attention mechanism with a mask on the encoder's output. Here, g(·) is the Transformer text vectorization encoding method. T It is a 384-dimensional vector, which is represented as a 384-dimensional matrix in NumPy format containing the number of nodes with text attributes, and stored therein. Each row of the matrix represents a node.

[0068] The Transformer model, based on an encoder and decoder, encodes node text attributes, constructing attribute vectors with rich semantic information for nodes possessing text attributes. This method fully mines the text attribute information of nodes, and incorporating the resulting text representation vectors into the model improves its performance.

[0069] In heterogeneous graphs, some types of nodes lack textual information (e.g., author and conference nodes in DBLP, director and actor nodes in IMDB). Through the information transmission mechanism of graph neural networks, these types of nodes can aggregate the textual information of their neighboring nodes to complete their own textual attributes. Node textual attribute completion is implemented in two steps: (1) aggregating neighboring information by mean, and (2) transforming node features.

[0070] The specific process for completing node text attributes is as follows: Figure 3 As shown, this invention first completes missing text attributes by aggregating the text attributes of neighboring nodes using the mean. Then, it constructs a text attribute representation vector for each node by performing feature transformation on the text attributes, thereby obtaining a complete heterogeneous graph. The specific process is as follows:

[0071] Step 2.1: Locate the node v that is missing text attributes. - Neighbor node v with text attribute in the neighborhood + ;

[0072] In heterogeneous graphs, some types of nodes lack textual information (e.g., author and conference nodes in DBLP, director and actor nodes in IMDB). These types of nodes are missing attributes, but they are directly connected to nodes with attributes. For example, author nodes in the DBLP dataset are connected to paper nodes with attributes, and director and actor nodes in IMDB are connected to movie nodes. This invention aims to find nodes with textual attributes in the neighborhood of nodes that are missing textual attributes.

[0073] Step 2.2: Complete missing text attributes by aggregating the text attributes of neighboring nodes using the mean;

[0074] Node v with missing text attributes -By connecting neighbor nodes v with text attributes + Complete its own attributes, and the calculation is as follows:

[0075]

[0076] in, These are the node attributes that need to be completed. It is node v - For text attributes in the neighborhood of nodes with text attributes, agg(·) is the completion method for mean aggregation. It is node v - The set of nodes in the neighborhood containing text information;

[0077] Step 2.3: Perform feature transformation on the text attributes to construct a text attribute representation vector for each node;

[0078] Updating node attributes based on downstream tasks by performing feature transformation on node attributes through a fully connected network can be specifically defined as:

[0079]

[0080] MLP is a two-layer fully connected network that performs feature transformation on the completed target node attributes to adapt them to downstream tasks. Represented as a node with missing nodes v - The constructed attribute, A represents node v - and The adjacency matrix between them, D -1 Represents node v - The reciprocal of the degree, Indicates v - The text attribute matrix of the neighborhood contains text information nodes. This is a matrix consisting of the number of nodes in the network that lack text attributes, multiplied by the size of the MLP output layer.

[0081] Step 2.4: Obtain the complete heterogeneous graph.

[0082] By combining step 2.3, we obtain the text attributes of nodes with missing text attributes and nodes with text attributes v. + The text attributes construct a complete attribute matrix. X i X represents the text attribute used for filling in missing nodes. j The attributes of nodes with text attributes in the dataset are represented by the adjacency matrix A obtained in step 1.2, which together form a complete heterogeneous graph G = (A, X). S ).

[0083] This invention combines a proposed heterogeneous graph representation learning method that integrates text attributes with a graph neural network to achieve end-to-end optimization based on downstream tasks. The complete heterogeneous graph obtained by completing text attributes is combined with existing graph neural network models (HAN and GAT) to achieve end-to-end optimized training based on node classification tasks.

[0084] The specific process of training a graph neural network model end-to-end is as follows: Figure 4 As shown, the attributes of all nodes The complete attribute matrix X S A complete heterogeneous graph is constructed together with the graph adjacency matrix A. This complete heterogeneous graph is then input into a graph neural network model (HAN, GAT) for classification prediction. The network parameters are optimized by minimizing the cross-entropy loss function to obtain the node representation vector Z. The specific process is as follows:

[0085] Step 3.1: Divide the nodes in the constructed heterogeneous graph G into training, validation, and test sets according to the proportions, and input them together into the graph neural network model:

[0086]

[0087] in, Let Z represent the graph neural network model, where Z represents the node representation obtained by the graph neural network. Then, Z is transformed to the dimension of the node labels using an MLP (Multi-Level Processing) to obtain...

[0088] Step 3.2: Perform classification prediction.

[0089]

[0090] Among them, L prediction Represents classification loss, Y represents the model's predicted value, and Y represents the node's label. Y is a matrix of the number of nodes multiplied by the number of node label categories, and f represents the cross-entropy loss function, which depends on the specific downstream task;

[0091] Step 3.3: Optimize the model by minimizing the cross-entropy loss; the formula for the cross-entropy loss function is as follows:

[0092]

[0093] in, and Y(x i These represent the predicted label probability distribution and the actual label probability distribution of nodes, respectively.

[0094] Step 3.4: Optimize the parameters by reducing the cross-entropy loss function in Step 3.3 through backpropagation end-to-end operation to update the trainable parameters in the graph neural network and the fully connected network until the classification accuracy on the validation set no longer increases and the model converges.

[0095] Step 3.5: After the model converges, verify the model performance using the test set and obtain the optimal node representation vector Z.

[0096] To fully demonstrate the feasibility and superiority of this invention, relevant experimental studies were conducted. The experiments used two publicly available datasets (DBLP and IMDB).

[0097] The proposed HGNN-ETA framework was combined with two graph neural network models (HAN and GAT) to construct models for comparative experiments; GAT was combined to construct the GAT-ETA model, and HAN was combined with HAN to construct the HAN-ETA model.

[0098] Comparative experiments were conducted with three shallow embedding models (Deepwalk, Metapath2vec, and herec) and five graph neural network models (GCN, GAT, AM-GCN, NSHE, and HAN).

[0099] The effectiveness of the proposed method on node classification tasks was verified by comparing it with eight other methods on two publicly available datasets. The experimental results are shown in Table 1. During the experiments, nodes in the constructed heterogeneous graph G were used as the training set at proportions of 20%, 40%, 60%, and 80%, respectively; 10% of the nodes were used as the validation set; and the remaining nodes were used as the test set. The performance of the model was verified using the test set, with macio-f1 and micro-f1 as evaluation metrics. Table 1 clearly shows that GAT-ETA and HAN-ETA significantly outperform GAT and HAN, with HAN-ETA showing the best performance, demonstrating the effectiveness and superiority of the proposed method.

[0100] Table 1. Results of the node classification experiment (%)

[0101]

[0102]

[0103] This invention combines text representation methods with graph representation learning methods, constructing a model based on transformers and graph neural networks. First, it utilizes existing node text attributes to perform text feature representation based on a text encoding strategy, and integrates the text representation vector into a heterogeneous graph. Then, it uses methods based on the mean aggregation of neighboring nodes and node feature transformation to complete missing node attributes. Finally, the heterogeneous graph with completed attributes is input into an existing graph neural network model, and node vectors are generated through an end-to-end training strategy. This invention can deeply integrate node text attributes, thereby improving the performance of graph neural network models.

[0104] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A heterogeneous graph representation learning method that integrates text attributes, characterized in that, Specifically, the steps include the following: Step 1: Obtain the text data of the dataset nodes The text data is pre-trained, and text feature representation based on text encoding strategy is performed using existing node text attributes to obtain the feature vector of the text representation. The data is represented as a NumPy matrix and saved. A heterogeneous graph is constructed based on the node relationships in the dataset to obtain the adjacency matrix between nodes. and save the feature vector As attributes of nodes with text attributes, they are incorporated into the heterogeneous graph; Step 2: Using the adjacency matrix Nodes with text attributes in the neighborhood of nodes with missing attributes are found. The missing attributes are then filled in using mean aggregation and node feature transformation via a fully connected network, resulting in a complete node attribute matrix. The specific process is as follows: Step 2.1: Locate the nodes with missing text attributes. Neighboring nodes with text attributes ; Step 2.2: Complete missing text attributes by aggregating the text attributes of neighboring nodes using the mean; Nodes with missing text attributes By connecting neighbor nodes with text attributes Complete its own attributes; the calculation formula is as follows: ； in, These are the node attributes that need to be completed. It is a node The text attributes of nodes with text attributes in the neighborhood. It is a completion method for mean aggregation. It is a node The set of nodes in the neighborhood containing text information; Step 2.3: Perform feature transformation on the text attributes to construct a text attribute representation vector for each node; The node attributes are updated based on downstream tasks by performing feature transformation on node attributes through a fully connected network, specifically defined as follows: ； MLP is a two-layer fully connected network that performs feature transformation on the completed target node attributes to adapt them to downstream tasks. Represents a node and The adjacency matrix between them Represents a node The reciprocal of the degree, Represents a node The text attribute matrix of the neighborhood containing text information nodes; Step 2.4: Obtain the complete heterogeneous graph; By combining step 2.3, we can determine the text attributes of nodes with missing text attributes and nodes with text attributes. The text attributes construct a complete attribute matrix. , The text attribute used to indicate missing node completion. This represents the attributes of nodes with text attributes in the dataset, and finally, the adjacency matrix is ​​combined. Construct a complete heterogeneous graph ; Step 3: Obtain the adjacency matrix of the complete heterogeneous graph with complete attributes. And a complete node attribute matrix The input is fed into an existing graph neural network model, and the optimal node representation vector Z is generated through end-to-end training of the node classification task.

2. The heterogeneous graph representation learning method integrating text attributes according to claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: Obtain the required data, construct the dataset, and use a web crawler algorithm to extract the content from the current dataset as the text data of the nodes. , Representing text Vocabulary in the text; Step 1.2: Construct adjacency matrices between nodes in the text dataset according to the relationships between them. Adjacency matrix It is a square matrix with the number of nodes multiplied by the number of nodes. Each element in the square matrix represents the edge relationship between nodes. If there is a relationship between nodes, the corresponding position is 1, otherwise it is 0, thus constructing a heterogeneous graph with an adjacency matrix structure. Step 1.3: Transfer the crawled text data Perform text preprocessing, including removing stop words and converting English uppercase to lowercase; Step 1.4: Transfer the preprocessed text data The input encoder, based on a Transformer model pre-trained for machine translation tasks, encodes the text attributes of nodes and outputs encoded representation vectors. The encoder consists of six identical stacked layers; each layer has two sub-layers: one is a multi-head self-attention mechanism, and the other is a feedforward network layer. Residual connections are used between the two sub-layers, followed by layer normalization. The output of each sub-layer is... ,in These are the functional functions of each layer; Representing text features; Step 1.5: Input the encoder output and other linguistic forms of the text into the decoder, and obtain the text representation vector through machine translation task pre-training. ; The decoder is also composed of six identical layers stacked together; in addition to the two sub-layers in the encoder, the decoder inserts a third sub-layer that performs a multi-head attention mechanism with masking on the encoder's output; among which, It is the Transformer text vectorization encoding method; It is a 384-dimensional vector, which is represented as a 384-dimensional matrix in NumPy format containing the number of nodes with text attributes, and stored therein, with each row of the matrix representing a node.

3. The heterogeneous graph representation learning method integrating text attributes according to claim 2, characterized in that, The specific process of step 3 is as follows: Step 3.1: Divide the nodes in the constructed heterogeneous graph G into training, validation, and test sets according to the specified proportions; and simultaneously input them into the graph neural network model: Z ； in, This represents a graph neural network model. This represents the node representation obtained from the graph neural network, and then... The dimension of the node label is obtained by transforming it through MLP. ; Step 3.2: Perform classification prediction. ； in, Represents classification loss, Y represents the model's predicted value, and Y represents the node's label. Y is a matrix consisting of the number of nodes multiplied by the number of node label categories. The cross-entropy loss function depends on the specific downstream task. Step 3.3: Optimize the model by minimizing the training set cross-entropy loss. The cross-entropy loss function formula is as follows: ； in, and These are the node predicted label probability distribution and the node true label probability distribution, respectively. Step 3.4: Optimize the parameters by reducing the cross-entropy loss function in step 3.3 through backpropagation end-to-end operation to update the trainable parameters in the graph neural network and the fully connected network until the classification accuracy on the validation set no longer increases and the model converges. Step 3.5: After the model converges, verify the model performance using the test set, and obtain the optimal node representation vector Z.

Images