Keyword generation method for heterogeneous information networks based on contrastive learning pre-training

Abstract

This invention discloses a keyword generation method for heterogeneous information networks based on contrastive learning pre-training, comprising the following steps: encoding text into low-dimensional vectors using a text encoder to generate a text representation; encoding the structural features, heterogeneous features, and self-supervised information of the heterogeneous information network using a graph encoder to obtain a graph representation; aligning the text representation and graph representation through contrastive learning pre-training; introducing automatically generated, learnable, and continuous cue vectors to provide the identified natural language statements to the text encoder, and comparing the natural language statements with the structural and heterogeneous feature representations generated by the graph encoder to generate weights for classification, fusing them to obtain a single representation; and using the obtained single representation to generate keywords for the heterogeneous information network. This invention achieves significantly superior generation performance in keyword generation tasks for heterogeneous information networks.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph network technology in natural language processing, and in particular to a keyword generation method for heterogeneous information networks based on contrastive learning pre-training. Background Technology

[0002] Heterogeneous information networks are ubiquitous. Interactions between users and items in social networks, knowledge graphs, and search and recommendation systems can be modeled as networks with various types of nodes and edges. Textual heterogeneous information networks are networks with textual information, such as the titles and abstracts of paper nodes in academic networks, which can provide productive auxiliary information for downstream tasks. Most current work on heterogeneous information networks neglects this textual information and maps graph nodes to low-dimensional representations based solely on structural information. To fill this gap, some models for mining heterogeneous information networks propose integrating textual information into node representations. They primarily design a framework that combines the structural and textual information of nodes to generate a single node representation.

[0003] The text network embedding models mentioned above face several limitations. First, they can only classify nodes using pre-trained labels; in other words, they are unsuitable for few-shot learning settings. In few-shot learning, we need to transfer the pre-trained model to classify nodes with unseen labels during the testing phase. In practice, typically only a few labels are available, posing a significant challenge to maintaining performance. Second, previous methods using textual information were originally designed for homogeneous information networks, and no work has yet attempted to address the few-shot learning problem on heterogeneous textual information networks.

[0004] To address the few-shot learning problem, natural language processing research (e.g., ChatGPT) has proposed cue learning, which re-engineers downstream tasks to resemble pre-trained tasks. Whether or not fine-tuning is used, cue learning facilitates the rapid application of prior knowledge to new tasks, thereby enhancing few-shot learning. Recently, cue learning has also been employed in multimodal scenarios for aligning image and text data. However, currently, no cue learning-based techniques have been used to process both graph and text data.

[0005] In view of this, we propose a keyword generation method for heterogeneous information networks based on contrastive learning pre-training. This method applies cue learning to graph data, solves the few-sample learning problem on heterogeneous textual information networks, and obtains more efficient and accurate keyword generation results for heterogeneous information networks. Summary of the Invention

[0006] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention discloses a keyword generation method for heterogeneous information networks based on contrastive learning pre-training. The method employs a text encoder to encode text information; a graph encoder to encode structural and heterogeneous features as well as self-supervised information; a contrastive learning mechanism to align text and network representations; and a learnable continuous vector-based cue learning framework to address the few-shot problem in heterogeneous text information networks.

[0007] A keyword generation method for heterogeneous information networks based on contrastive learning pre-training, the method comprising:

[0008] Step 1: Use a text encoder to encode the text into a low-dimensional vector to generate a text representation;

[0009] Step 2: Use a graph encoder to encode the structural features, heterogeneous features, and self-supervised information of the heterogeneous information network to obtain a graph representation;

[0010] Step 3: Align the text representation and the graph representation through comparative learning during pre-training.

[0011] Step 4: Introduce automatically generated learnable and continuous cue vectors, provide the labeled natural language statements to the text encoder, and compare the natural language statements with the structural and heterogeneous feature representations generated by the graph encoder to generate weights for classification, and fuse them to obtain a single representation;

[0012] Step 5: Use the obtained single representation to perform the keyword generation task for heterogeneous information networks.

[0013] Specifically, the text encoder uses the Sentence-BERT model to generate fixed-size text representations.

[0014] Specifically, step 2 includes the following steps:

[0015] Step 201: Sample the heterogeneous subgraph. For a given node, the subgraph surrounding the node needs to be sampled first.

[0016] Step 202: An autoencoder is used to capture the structural information of the subgraph. Given the adjacency matrix A of the subgraph, it will first be processed by the encoder to generate a multi-layered latent representation. Then, the decoder reverses the above process to obtain the reconstructed output. Autoencoders aim to minimize the reconstruction error of the input and output, enabling nodes with similar structures to have similar representations, with a loss function L. structure The calculation formula is as follows:

[0017]

[0018] Where B is the penalty sparsity imposed on non-zero elements to mitigate the sparsity problem, and e represents bitwise multiplication. Indicates a regularization operation;

[0019] Step 203: Explore the heterogeneous features of the heterogeneous information network by grouping nodes of the same type together and applying Bi-LSTM to each group to model type-specific features, given a type T. j Node group Representation of node v The calculation is as follows:

[0020]

[0021] Where Bi-LSTM{v} represents applying Bi-LSTM to the type grouping of node v. Represents a node group Quantity;

[0022] Then, an attention mechanism is applied to aggregate all groups of types to generate a representation h for a given node. v ,

[0023]

[0024]

[0025] Where δ represents the activation function, using LeakyReLU, u∈R d It is the weight parameter, u T This represents the transpose of u. Let {T} represent the node v, and {T} represent the set of types, α v,j Indicates attention weight;

[0026] Step 204: Based on the self-supervised information pre-trained subgraph, two pre-training tasks are introduced: mask node modeling task and edge reconstruction task, to achieve node-level and edge-level graph exploration.

[0027] Specifically, in the masked node modeling task, nodes are sorted according to their ranking, and a predetermined proportion of nodes are randomly selected and labeled with [MASK]. The sorted nodes are then fed into the Transformer encoder. The representation generated by Bi-LSTM is used as the identifier, and the sorting information is used as the position vector. The hidden layers learned by the Transformer encoder are then processed. It will be fed into the feedforward layer to predict the target node, mathematically represented as:

[0028]

[0029] p v=softmax(W MNM z v ),

[0030] Among them, z v This is the output of the feedforward layer. `Feedforward()` indicates the output from the feedforward layer, `softmax()` represents the activation function, and `W`... MNM ∈V v ×d is the weight shared with the input node representation matrix for classification, V v d is the number of nodes in the subgraph, d is the dimension of the hidden layer vector, and p is the number of nodes in the subgraph. v This is the predicted distribution of v across all nodes, using one-hot labels during training. and prediction Cross-entropy and loss function L MNM The calculation is as follows:

[0031]

[0032] Among them, y i and p i It is y i and p i The i-th component, y i p represents a set of labels i A set representing predicted probabilities;

[0033] The edge reconstruction task involves sampling positive and negative edges in the subgraph. Positive edges are those that actually exist in the original subgraph, while negative edges do not. Given a merged set N of positive and negative edges... S The score for edge reconstruction is calculated using the inner product between a pair of nodes. It calculates the score, h v Let be the representation of node v, e be the inner product, and h be the inner product. u The representation of node u uses the binary cross-entropy between predicted and true edges to calculate the loss function L for edge reconstruction. ER :

[0034]

[0035] N S Represents the number of node pairs, BinaryCrossEntropy() represents the binary cross-entropy, e uv Let (u,v) represent the actual scores of nodes u and v, and (u,v) represent the edge connecting nodes u and v.

[0036] Furthermore, the sampling strategy for sampling the subgraph around a node employs a random walk with restart, which iteratively traverses the neighborhood of a given node v and returns to the starting node v with a certain probability. In order to sample nodes with higher importance, the random walk strategy first reaches nodes with higher ranking. In order to make the graph encoder heterogeneous, the traversal is restricted to sampling all types of nodes.

[0037] Specifically, the contrastive learning is used to align text representations and graph representations during training. The learning objective is designed as a contrastive loss function that, given a set of text-subgraph pairs, maximizes the similarity score of matching text-subgraph pairs while minimizing the score of non-matching text-subgraph pairs.

[0038] In the contrastive learning process, given a node v, the node representation learned by the graph encoder is H, and the weight vector generated by the text encoder is denoted as H. Where K represents the number of categories, each weight wi is learned from the cue, and the predicted probability can be calculated as follows:

[0039]

[0040] Where τ is the learned temperature hyperparameter, and <·,·> represent the similarity score. <w i H> represents the text weight vector w i The similarity score between nodes represents the vector H.

[0041] Specifically, the introduction of automatically generated, learnable, and continuous cue vectors in step 4 involves replacing discrete text words with continuous vectors learned end-to-end from the data. The cue P input to the text encoder is designed as follows:

[0042] P = [V1][V2][V] M [CLASS]

[0043] Where [CLASS] represents the node's category label, [V M [] is a word vector with the same dimension as the word representation in the training phase. M is a hyperparameter representing the number of consecutive text vectors in the prompt. After inputting the consecutive prompts P into the text encoder Text(·), the classification weight vectors representing the node concepts can be obtained. The predicted probability is calculated as follows:

[0044]

[0045] Among them, each prompt P i The category markers in Text(P) are replaced by the word vector representation of the i-th category name. i ) indicates that a prompt will be made for P. i The vector obtained after being fed into the text encoder.

[0046] Preferably, to obtain a more accurate cue vector in step 4, a residual connection based on the text encoder and the graph encoder is used to utilize the context subgraph of a given node. The text representation of the category label and the node representation in the subgraph are input into the text-subgraph self-attention layer to help the text features find the most relevant context node of a given node.

[0047] Obtain the output D of the text-subgraph comparator e Then, the text features are updated using residual connections.

[0048] Text(P)←Text(P)+λD e

[0049] Here, λ is a learnable parameter used to control the degree of residual connectivity.

[0050] Preferably, λ is initialized to 10. -4 A very small value, so that prior linguistic knowledge from text features can be preserved to the greatest extent possible.

[0051] Compared with existing methods, the advantages of this invention are: it proposes a prompting learning framework to utilize textual information in heterogeneous textual networks while simultaneously addressing few-shot learning issues; and it introduces a graph encoder that captures the structure and heterogeneous features of the heterogeneous network while preserving the node-level and edge-level self-supervised information of the network subgraphs. Therefore, this invention's heterogeneous network keyword generation method based on contrastive learning pre-training achieves significantly superior generation performance in the heterogeneous network keyword generation task. Attached Figure Description

[0052] Figure 1 A flowchart illustrating an embodiment of the present invention is shown;

[0053] Figure 2 A schematic diagram of the pre-training framework in an embodiment of the present invention is shown;

[0054] Figure 3 A schematic diagram of the prompting learning optimization framework according to an embodiment of the present invention is shown. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0056] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0057] Prerequisites:

[0058] Let G = (V, E, T) denote a heterogeneous information network, where V and E represent the node set and edge set, respectively; T V and T E These represent the set of node types and the set of edge types, respectively. A heterogeneous information network is |T V |>1 and / or|T E The network is 1.

[0059] like Figure 1 The present invention provides a keyword generation method for heterogeneous information networks based on contrastive learning pre-training, the method comprising:

[0060] Step 1: Use a text encoder to encode the text into a low-dimensional vector to generate a text representation;

[0061] Step 2: Use a graph encoder to encode the structural features, heterogeneous features, and self-supervised information of the heterogeneous information network to obtain a graph representation;

[0062] Step 3: Align the text representation and the graph representation through comparative learning during pre-training.

[0063] Step 4: Introduce automatically generated learnable and continuous cue vectors, provide the labeled natural language statements to the text encoder, and compare the natural language statements with the structural and heterogeneous feature representations generated by the graph encoder to generate weights for classification, and fuse them to obtain a single representation;

[0064] Step 5: Use the obtained single representation to generate keywords for the heterogeneous information network.

[0065] The method of this invention mainly includes a text encoder and a graph encoder, which encode text and network subgraphs into low-dimensional vectors, respectively. In the embodiment, Sentence-Bert (SBERT) is used as the text encoder to generate text representations; for the graph encoder, the subgraph to be processed is first sampled, and all types of nodes are forced to be sampled to ensure heterogeneity. Then, an autoencoder mechanism is applied to explore structural features, and Bi-LSTM is applied to nodes grouped by type to characterize the heterogeneity of the graph.

[0066] Two graph pre-training tasks, masked node modeling and edge reconstruction, are introduced to leverage self-supervised information at the node and edge levels. A contrastive learning framework is then introduced to align these two representations. Specifically, given a pair of text and subgraphs, they are considered matched if they both belong to a given node. The contrastive learning framework maximizes the similarity score of matched text-subgraph pairs and minimizes the similarity score of mismatched text-subgraph pairs.

[0067] The pre-trained model needs to be transferred to downstream tasks to adapt to few-shot settings. Specifically, during the optimization phase, for each new classification task, classification weights are generated by providing natural language statements describing the categories of interest to the text encoder and comparing them with the structural and heterogeneous feature representations generated by the network encoder. How then should we design prompts that are crucial for downstream tasks? Even subtle changes to the words in the prompts can affect the model's performance. This embodiment does not use handwritten prompts like "a paper of [CLASS] domain," but instead introduces automatically generated, learnable, and continuous prompt vectors. This automated prompting mechanism can bring more task-relevant and efficient transfer effects to the pre-trained model.

[0068] The specific technical solution is as follows.

[0069] Text Encoder

[0070] The pre-training framework in this embodiment is as follows: Figure 2 As shown, it consists of two encoders: a text encoder and a graph encoder. The text encoder maps natural language text to low-dimensional representation vectors. The Sentence-BERT (SBERT) model is used to generate fixed-size text representations.

[0071] Map Encoder

[0072] Graph encoders map network data into low-dimensional representations.

[0073] Heterogeneous graph sampling

[0074] For a given node, the subgraph surrounding the node is first sampled, and then the subgraph is processed by a graph encoder to generate a node representation. After sampling the subgraph, the nodes in the subgraph are ranked using a centrality metric that evaluates the importance of the nodes.

[0075] A sampling strategy of random walk with restart is employed. It iteratively traverses the neighborhood of a given node v and returns to the starting node v with a certain probability. To sample nodes of higher importance, the walk strategy first reaches nodes with higher ranking. To make the encoder heterogeneous, the traversal is restricted to sampling nodes of all types.

[0076] Structural modules

[0077] First, an autoencoder is used to capture the structural information of the subgraph. Given the adjacency matrix A of the subgraph, it is first processed by the encoder to generate a multi-layered latent representation. Then, the decoder reverses the above process to obtain the reconstructed output. An autoencoder aims to minimize the reconstruction error between the input and output, so that nodes with similar structures have similar representations. Mathematically,

[0078]

[0079] Where B is the penalty sparsity imposed on non-zero elements to mitigate the sparsity problem, and e represents bitwise multiplication. This indicates a regularization operation.

[0080] Heterogeneous modules

[0081] To explore the heterogeneous features of the network, nodes of the same type are first grouped together. This operation may disrupt the structure of the subgraphs; however, the previously employed autoencoder has already preserved structural features. Then, a Bi-LSTM is applied to each group to model type-specific features. Bi-LSTM is able to capture the interactions of node features and has extensive sequence representation capabilities. Given type T... j Node group Representation of node v The calculation is as follows:

[0082]

[0083] Where Bi-LSTM{v} represents applying Bi-LSTM to the type grouping of node v. Represents a node group Quantity;

[0084] Then, an attention mechanism is applied to aggregate all groups of types to generate a representation h for a given node. v .

[0085]

[0086]

[0087] Where δ represents the activation function, using LeakyReLU, u∈R d It is the weight parameter, u T This represents the transpose of u. Let {T} represent the node v, and {T} represent the set of types, α v,j This represents the attention weight.

[0088] Self-supervised pre-training

[0089] Furthermore, subgraphs are pre-trained based on self-supervised information. Specifically, two pre-training tasks are introduced: Masked Node Modeling (MNM) and Edge Reconstruction (ER), to achieve node-level and edge-level graph exploration.

[0090] For the masked node modeling task, we sort the nodes according to their ranking and randomly select 15% of the nodes to be labeled with [MASK]. The sorted nodes are then fed into the Transformer encoder, where the representation generated by Bi-LSTM is used as the label representation, and the ranking information is used as the position vector. The hidden layers learned by the Transformer encoder... It will be fed into the feedforward layer to predict the target node, mathematically.

[0091]

[0092] p v =softmax(W MNM z v ), (6)

[0093] Among them, z v This is the output of the feedforward layer. `Feedforward()` indicates the output from the feedforward layer, `softmax()` represents the activation function, and `W`... MNM ∈V v ×d is the weight shared with the input node representation matrix for classification, V v d is the number of nodes in the subgraph, d is the dimension of the hidden layer vector, and p is the number of nodes in the subgraph. v This is the predicted distribution of v across all nodes, using one-hot labels during training. and prediction Cross-entropy and loss function L MNM The calculation is as follows:

[0094]

[0095] Among them, y i and p i It is y i and p i The i-th component, y i p represents a set of labels i A set representing predicted probabilities.

[0096] The edge reconstruction task involves sampling positive and negative edges in the subgraph. Positive edges are those that actually exist in the original subgraph, while negative edges do not. In practice, |N can be set. S| = 6, and the number of positive edges and negative edges are the same. Given the merge set N of positive and negative edges. S The score for edge reconstruction is calculated using the inner product between a pair of nodes. It calculates the score, h v Let be the representation of node v, e be the inner product, and h be the inner product. u The representation of node u uses the binary cross-entropy between predicted and true edges to calculate the loss function L for edge reconstruction. ER :

[0097]

[0098] |N S | represents the number of node pairs, BinaryCrossEntropy() represents the binary cross-entropy, e uv Let (u,v) represent the actual scores of nodes u and v, and (u,v) represent the edge connecting nodes u and v.

[0099] Pre-training through comparative learning

[0100] This embodiment aligns the representation spaces of text and graphs during training, and its learning objective is designed as a contrastive loss function. Specifically, given a set of text-subgraph pairs, this embodiment aims to maximize the similarity score of matching text-subgraph pairs while minimizing the score of mismatched text-subgraph pairs. For example, given a subgraph of a node, if the text information is a summary of that node, then the text-subgraph pair is matched; otherwise, it is not matched because the text information is irrelevant to that node. The similarity score is calculated using cosine similarity.

[0101] In contrastive learning environments, high-quality negative samples can help improve model performance. Therefore, in training batches, the text and subgraphs used are selected from nodes with the same label to make them difficult to distinguish.

[0102] Figure 3 This demonstrates a cue-based learning optimization framework. This embodiment can be applied to experimental environments with few samples. The pre-trained model, when faced with samples of new label types, can predict whether a node's subgraph matches the text description. This is achieved by comparing the node representations generated by the graph encoder with the classification weights generated by the text encoder. The text description can be used to specify the node category of interest, even if that category has few samples. Given a node v, the node representation learned by the graph encoder is H, and the weight vector generated by the text encoder is... Where K represents the number of categories. Each weight wi is learned from a cue, such as "a paper of [CLASS]domain", where "CLASS" can be a specific category name, such as "Information Retrieval", "database", or "data mining". For ease of downstream tasks, the cue can also be designed as "The two nodes are [CLASS]", which is a binary label such as "connected" and "unconnected". Mathematically, the prediction probability can be calculated as follows:

[0103]

[0104] Where τ is the learned temperature hyperparameter, and <·,·> represent the similarity score. <w i H> represents the text weight vector w i The similarity score between nodes represents the vector H.

[0105] Continuous prompts

[0106] Traditional cue learning methods employ expert-designed manual cues. This embodiment replaces discrete text words with continuous vectors that can be learned end-to-end from the data. Specifically, the cue P input to the text encoder should be designed as follows:

[0107] P = [V1][V2][V] M [CLASS], (10)

[0108] Where [CLASS] represents the node's category label, [V M `P` is a word vector with the same dimension as the word representations during training, and `M` is a hyperparameter representing the number of consecutive text vectors in the prompt. After inputting the consecutive prompts `P` into the text encoder `Text(·)`, the classification weight vectors representing the node concepts are obtained. Mathematically, the predicted probability is calculated as...

[0109]

[0110] Among them, each prompt P i The category markers in Text(P) are replaced by the word vector representation of the i-th category name. i ) indicates that a prompt will be made for P. i The vector obtained after being fed into the text encoder.

[0111] Residual connection

[0112] Considering the context nodes of a given node, such as the author node of a paper node, helps the text encoder become more accurate. Therefore, to further inform the pre-trained language model, a residual connection based on the text encoder and the graph encoder is used to leverage the context subgraph of a given node. First, the text representation of the category label and the node representation in the subgraph are input into a text-subgraph self-attention layer to help the text features find the most relevant context nodes for a given node. The output D of the text-subgraph contrastor is then obtained. e Then, the text features are updated through residual connections, as shown below.

[0113] Text(P)←Text(P)+λD e (12)

[0114] Here, λ is a learnable parameter used to control the degree of residual connectivity. λ is initialized to a small value of 10. -4 This is so that prior linguistic knowledge derived from text features can be preserved to the greatest extent possible.

[0115] To optimize the text vectors, training is performed to minimize the standard classification loss function based on cross-entropy. Gradients are backpropagated through the text encoder Text(·) to leverage the rich knowledge encoded in the parameters. The choice of continuous text vectors also allows for a full exploration of the word representation space, thereby improving the learning of task-relevant text.

[0116] This embodiment considers a real-world dataset, namely OAG. OAG is an academic network with four types of nodes, selecting titles and abstracts as text, and classifying the corresponding paper nodes into five categories: (1) information retrieval, (2) database, (3) data mining, (4) machine learning, and (5) natural language processing.

[0117] The dataset was divided into 80% training dataset, 10% validation dataset, and 10% test dataset. Table 1 summarizes the information of the dataset.

[0118] Table 1: Data set statistics.

[0119]

[0120] All representation vectors have a fixed dimension of 512. For the text encoder, the vocabulary size is 49,152, with each text sequence having a fixed size of 77, including [SOS] and [EOS] labels. Text vectors during optimization are initialized using a zero-mean Gaussian distribution with a standard deviation of 0.02. The number of text words is set to 8 during training. Training is performed using stochastic gradient descent with an initial learning rate of 0.002, decaying using cosine annealing. The maximum number of training epochs is set to 200. To mitigate potentially explosive gradients in early training iterations, a warm-up technique is used to fix the learning rate at 1e-5 in the first training epoch.

[0121] Experiments were conducted using an Intel(R) Xeon(R) Platinum 8268 CPU and a Tesla V100 to run pre-training and downstream tasks.

[0122] This embodiment evaluates the performance of the method and benchmark model of the present invention on the keyword generation task. ACC and Macro-F1 scores are used as evaluation metrics (averaged over five trials).

[0123] Table 2 shows the experimental results of the keyword generation task; the highest score is shown in bold.

[0124]

[0125] This embodiment applies to a task not previously considered in graph evaluation: keyword generation. The method of this invention is able to generate keywords because it possesses autoregressive generation capabilities within a pre-training and fine-tuning framework. In practice, keywords from paper nodes in the OAG are used as ground-truth labels for training. Fine-tuning begins with the text [CLS][MASK], where [CLS] is the first word of the keyword, and the next step predicts new words within the [MASK] tags. This process stops when [SEP] is returned. Traditional text network representation methods cannot generate keywords. Therefore, for comparison, the KeyBERT model, which generates keywords based on BERT representations, is used in the experiments. It takes the description of each node as input. The F1 score of the Top 1 and Top 3 keyword predictions is used as the evaluation metric.

[0126] Table 2 shows the results of the keyword generation task. The method of this invention is superior to KeyBERT because it combines textual information with subgraph information, while KeyBERT can only utilize textual information.

[0127] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A keyword generation method for heterogeneous information networks based on contrastive learning pre-training, characterized in that, Includes the following steps: Step 1: Use a text encoder to encode the text into a low-dimensional vector to generate a text representation; Step 2: Use a graph encoder to encode the structural features, heterogeneous features, and self-supervised information of the heterogeneous information network to obtain a graph representation; Step 3: Align the text representation and the graph representation through comparative learning during pre-training. Step 4: Introduce automatically generated, learnable, and continuous cue vectors, provide the labeled natural language statements to the text encoder, and compare the natural language statements with the structural and heterogeneous feature representations generated by the graph encoder to generate weights for classification, and fuse them to obtain a single representation; Step 5: Use the obtained single representation to generate keywords for the heterogeneous information network; In step 4, the introduction of automatically generated, learnable, and continuous cue vectors involves replacing discrete text words with continuous vectors learned end-to-end from the data, which are then input into the text encoder as cue vectors. Designed as follows: in, The category label representing the node. It is a word vector with the same dimension as the word representation during the training phase. This is a hyperparameter representing the number of consecutive text vectors in the prompt, which will be used to indicate consecutive prompts. Input to text encoder Then, the classification weight vector representing the node concept can be obtained, and the predicted probability is calculated as follows: Each prompt The category marker in the first The word vectors of each category name represent the replacement. This indicates that a prompt will be made. The vector obtained after being fed into the text encoder; In step 4, to obtain a more accurate cue vector, a residual connection between the text encoder and the graph encoder is used to utilize the context subgraph of a given node. The text representation of the category label and the node representation in the subgraph are input into the text-subgraph self-attention layer to help the text features find the most relevant context node for a given node. Obtain the output of the text-subgraph comparator Then, the text features are updated using residual connections. in It is a learnable parameter used to control the degree of residual connectivity.

2. The keyword generation method for heterogeneous information networks based on contrastive learning pre-training according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 201: Sample the heterogeneous subgraph. For a given node, the subgraph surrounding the node needs to be sampled first. Step 202: Use an autoencoder to capture the structural information of the subgraph, given the adjacency matrix of the subgraph. It will first be processed by the encoder to generate multi-layered latent representations, and then the decoder will reverse the above process to obtain the reconstructed output. An autoencoder aims to minimize the reconstruction error of the input and output, enabling nodes with similar structures to have similar representations. The loss function... The calculation formula is as follows: Here, B is a penalty sparsity imposed on non-zero elements to mitigate the sparsity problem. This indicates digit-wise multiplication. This represents a regularization operation; Step 203: Explore the heterogeneous features of the heterogeneous information network by grouping nodes of the same type together and applying Bi-LSTM to each group to model type-specific features, given a type. Node group ,node The representation of The calculation is as follows: in, This indicates that Bi-LSTM will be applied to the node. In terms of type grouping, Represents a node group Quantity; Then, an attention mechanism is applied to aggregate all groups of types to generate a representation of a given node. , in, This represents the activation function, using LeakyReLU. These are weight parameters. express transpose, It is a node The expression, A collection representing a type, Indicates attention weight; Step 204: Based on the self-supervised information pre-trained subgraph, two pre-training tasks are introduced: mask node modeling task and edge reconstruction task, to achieve node-level and edge-level graph exploration.

3. The keyword generation method for heterogeneous information networks based on contrastive learning pre-training according to claim 2, characterized in that, The masked node modeling task involves sorting nodes according to their ranking, randomly selecting a preset proportion of nodes and replacing them with the [MASK] identifier, and then feeding the sorted nodes into the Transformer encoder. The representation generated by Bi-LSTM serves as the identifier, and the sorting information is used as the position vector. This is then processed by the hidden layers learned by the Transformer encoder. The data is fed into the feedforward layer to predict the target node, mathematically represented as: in, It is the output of the feedforward layer. Indicates a feedforward layer, This represents the activation function. These are the weights used for classification, shared with the input node representation matrix. It is the number of nodes in the subgraph. It is the dimension of the hidden layer vector. yes The predicted distribution across all nodes, during training, uses one-hot labels. and prediction Cross-entropy and loss function The calculation is as follows: in, and yes and The One portion, A collection of labels. A set representing predicted probabilities; The edge reconstruction task involves sampling positive and negative edges in the subgraph. Positive edges are those that actually exist in the original subgraph, while negative edges do not. A merged set of positive and negative edges is then defined. The score for edge reconstruction is calculated using the inner product between a pair of nodes. , It is to calculate the score. It is a node The expression, It is the inner product. It is a node The representation uses the binary cross-entropy between predicted edges and true edges to compute the loss function for edge reconstruction. : ， Indicates the number of node pairs. Represents the binary cross-entropy. Represents a node and nodes The actual score, Represents a node and nodes The connecting edges.

4. The keyword generation method for heterogeneous information networks based on contrastive learning pre-training according to claim 2, characterized in that, The sampling strategy for the subgraph surrounding a node employs a random walk with restart, iteratively traversing a given node. The neighborhood of the node, and with a certain probability, the starting node is returned. To sample nodes of higher importance, the random walk strategy is made to reach the nodes with higher order first. To make the graph encoder heterogeneous, the traversal is restricted to sampling all types of nodes.

5. The keyword generation method for heterogeneous information networks based on contrastive learning pre-training according to claim 1, characterized in that, The contrastive learning described herein is used to align text representations and graph representations during training. The learning objective is designed as a contrastive loss function that, given a set of text-subgraph pairs, maximizes the similarity score of matching text-subgraph pairs while minimizing the score of non-matching text-subgraph pairs.

6. The keyword generation method for heterogeneous information networks based on contrastive learning pre-training according to claim 5, characterized in that, In the comparative learning process described above, given a node The node representation learned by the graph encoder is as follows The weight vector generated by the text encoder is represented as ,in Represents the number of categories, each with a weight. All learning comes from prompts, and the predicted probability is calculated as follows: in, It is the learned temperature hyperparameter. Represents the similarity score. Represents the text weight vector And node representation vector The similarity score.

7. The keyword generation method for heterogeneous information networks based on contrastive learning pre-training according to claim 1, characterized in that, The text encoder uses the Sentence-BERT model to generate fixed-size text representations.

8. The keyword generation method for heterogeneous information networks based on contrastive learning pre-training according to claim 7, characterized in that, Will Initialize to .

Images