Graph embedding method based on mutual information enhancement

Abstract

The invention provides a graph embedding method based on mutual information enhancement, and the method comprises the steps: carrying out the graph coding and graph representation learning of a target document and a target vocabulary through a self-similarity method, and obtaining a potential topic; performing mutual information enhancement on the potential topic and the context information in the target document through a pre-constructed mutual information enhancement dynamic topic model to obtain reconstructed document distribution; and performing parameter optimization on the mutual information enhancement dynamic topic model through a clustering regularizer to obtain a target function, and performing loss optimization on the reconstruction document distribution through the target function to obtain a nonlinear dependency relationship between a target document and a potential topic. According to the method, the document is coded and subjected to mutual information enhancement through the constructed mutual information enhancement dynamic topic model, and potential topic distribution and document embedding expression are enhanced.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a graph embedding method based on mutual information enhancement. Background Technology

[0002] Dynamic topic modeling is a technique in Natural Language Processing (NLP) designed to model the evolution of latent topics in a document over time. Existing dynamic topic models can be broadly categorized into two types: probabilistic dynamic topic modeling, which typically infers topic distribution through variational inference or Gibbs sampling; and neural network dynamic topic modeling methods, which utilize deep neural structures to learn topic representations through backpropagation.

[0003] However, most current neural dynamic topic models are based on representing documents as word sequences. This sequence modeling approach ignores the potential non-linear dependencies and intrinsic information value between candidate topic words, making it easy for the center word to capture irrelevant topics when capturing local information, resulting in inconsistent topic representations. Therefore, it is essential to design a graph embedding method based on mutual information enhancement. Summary of the Invention

[0004] The purpose of this invention is to provide a graph embedding method based on mutual information enhancement, which encodes and enhances the mutual information of documents through a constructed mutual information enhanced dynamic topic model, thereby enhancing the latent topic distribution and document embedding representation.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A graph embedding method based on mutual information enhancement includes the following steps:

[0007] Latent topics are obtained by learning graph encoding and graph representation for target documents and target vocabulary using the self-similarity method;

[0008] By enhancing the mutual information of potential topics and contextual information in target documents through a pre-built mutual information enhancement dynamic topic model, the reconstructed document distribution is obtained;

[0009] The parameters of the mutual information-enhanced dynamic topic model are optimized by a clustering regularizer to obtain the objective function. The loss of the reconstructed document distribution is then optimized using the objective function to obtain the nonlinear dependency relationship between the target document and the potential topic.

[0010] Optionally, graph encoding and graph representation learning are performed on the target document and target vocabulary using a self-similarity method to obtain latent topics, including:

[0011] Each word in the target vocabulary is mapped to a vector to obtain a word embedding matrix;

[0012] Based on the word embedding matrix, a semantic similarity matrix between word pairs is constructed by combining the set of words in the target vocabulary where the words in the target document appear.

[0013] Word pairs with a cosine similarity higher than a preset threshold in the semantic similarity matrix are selected and connected to obtain a self-similarity graph;

[0014] By learning graph representations of self-similar graphs through graph isomorphic networks, latent topics can be obtained.

[0015] Optionally, the formula for calculating cosine similarity is:

[0016] ;

[0017] in, and These are word embeddings for two words respectively. It is an L2 paradigm.

[0018] Optionally, mutual information enhancement is performed on latent topics and contextual information in the target document using a pre-built mutual information-enhanced dynamic topic model to obtain a reconstructed document distribution, including:

[0019] The encoder of the dynamic topic model enhanced by mutual information encodes the target document according to a normal distribution to obtain variational distribution parameters;

[0020] The document-topic distribution is obtained by performing parameterized resampling on the variational distribution parameters.

[0021] Maximize the lower bound of mutual information on the context information, and optimize the alignment of the document-topic distribution based on the maximized lower bound of mutual information.

[0022] The decoder of the dynamic topic model enhanced by mutual information performs a multinomial distribution sampling reconstruction operation on the optimized and aligned document-topic distribution to obtain the reconstructed document distribution.

[0023] Optionally, the expression for maximizing the lower bound of mutual information is:

[0024] ;

[0025] in, Embedded in sentences, For the document-topic distribution within time slice t, These are the positive and negative samples within time slice t.

[0026] Optionally, the parameters of the mutual information-enhanced dynamic topic model are optimized using a clustering regularizer to obtain an objective function. The loss of the reconstructed document distribution is then optimized using this objective function to obtain the nonlinear dependencies between the target documents and potential topics, including:

[0027] Construct the first loss function based on the reconstruction loss and KL divergence;

[0028] The second loss function is obtained by minimizing mutual information through a clustering regularizer to enhance the transmission cost of the dynamic topic model.

[0029] The objective function is obtained by weighted fusion of the first loss function and the second loss function.

[0030] According to specific embodiments provided by the present invention, the following technical effects are disclosed: The graph embedding method based on mutual information enhancement provided by the present invention includes: performing graph encoding and graph representation learning on a target document and a target vocabulary using a self-similarity method to obtain latent topics; enhancing the mutual information of the latent topics and contextual information in the target document using a pre-constructed mutual information enhanced dynamic topic model to obtain a reconstructed document distribution; optimizing the parameters of the mutual information enhanced dynamic topic model using a clustering regularizer to obtain an objective function; and optimizing the loss of the reconstructed document distribution using the objective function to obtain a non-linear dependency relationship between the target document and the latent topics. This method enhances the latent topic distribution and document embedding representation by encoding and enhancing the mutual information of the document using the constructed mutual information enhanced dynamic topic model. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of the graph embedding method of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 1 As shown, this invention provides a graph embedding method based on mutual information enhancement, comprising the following steps:

[0036] Step 100: Learn latent topics by performing graph encoding and graph representation on the target document and target vocabulary using the self-similarity method;

[0037] Step 200: Enhance the mutual information of potential topics and contextual information in target documents using a pre-built mutual information-enhanced dynamic topic model to obtain the reconstructed document distribution;

[0038] Step 300: Optimize the parameters of the mutual information-enhanced dynamic topic model using a clustering regularizer to obtain the objective function, and then optimize the loss of the reconstructed document distribution using the objective function to obtain the nonlinear dependency relationship between the target document and the potential topic.

[0039] Specifically, in step 100, in order to effectively capture the dependencies between words, this embodiment employs a self-similarity method to perform graph encoding on the pre-trained word embeddings. For the target document... and target vocabulary list Using a pre-trained GloVe word vector model, each word in the vocabulary is... Mapped to vector And generate a word embedding matrix. For any document Take the set of words that appear in the vocabulary and construct a semantic similarity matrix S between word pairs within the document. The elements of this matrix... Word vectors and The cosine similarity is calculated using the following formula:

[0040] ;

[0041] in, and Each represents a word and Word embedding, L2 similarity is used. When a word's similarity score is low, it indicates that the word is unlikely to serve as a good potential candidate topic in the current document. Therefore, a threshold hyperparameter is introduced to control the degree of word similarity retention to construct a self-similarity graph, retaining only word pairs with similarity scores higher than the threshold. In the self-similarity graph, nodes represent words appearing in the document, edges represent semantic relationships between word pairs, and edge weights are given by similarity scores. Similarity filtering effectively filters out word pairs with weak semantic connections and reduces unnecessary noisy edges in the graph, retaining only significant semantic relationships. Finally, a Graph Isomorphism Network (GIN) is used to learn graph representations from the self-similarity graph to obtain potential topics.

[0042] Specifically, the mutual information-enhanced dynamic topic model in step 200 is generally divided into two parts: an encoder for inferring the document topic distribution and a decoder for reconstructing the document distribution. The topic distribution for each document... They all depend on a logistic normal distribution ,in It is the mean. It is a diagonal covariance matrix. In this embodiment, the document is represented as... The data is then input into the encoder for encoding, where... Document graph embedding for time slice t, Given the bag-of-words vectors for time slice t, and then outputting the parameters of the variational distribution, the encoder's encoding process can be represented as follows: After encoding, the mean of each Gaussian distribution is generated. and standard deviation ,in Indicates the dimension of the input. Indicates the number of hidden units. It is an activation function. The linear layer in the encoder uses KanLinear, which, compared to traditional MLPs, can more efficiently approximate complex multivariable functions, giving the model powerful nonlinear expressive capabilities. Then, a reparameterization technique is used to perform parameterized resampling on the variational distribution parameters, expressed as:

[0043] ;

[0044] Where ⊙ represents the Frobenius inner product, Then use The function models the document-topic distribution as .

[0045] It's important to note that mutual information measures the relevance between two objects. The sentence embeddings obtained through the Sentence-BERT model exhibit high mutual information with the document representation and the latent topic distribution. Specifically, maximizing the lower bound of mutual information preserves the rich contextual information contained in the sentence embeddings, thereby encouraging the document-topic distribution to resonate more fully with the richer semantic space. To indicate alignment, thereby improving the quality of document and topic representation, the expression is:

[0046] ;

[0047] in, Embedded in sentences, For the document-topic distribution within time slice t, Within time slice t and Positive and negative samples.

[0048] Furthermore, the decoder utilizes document-topic distribution. The topic-word distribution matrix at time slice t Through multinomial distribution The sampled reconstruction document indicates that, This represents the learnable weights of the decoder network. In this embodiment, to improve parameter efficiency, different time slices are used... Decomposed into W-word embeddings and Theme embedding, the expression is: ; and use temperature hyperparameters Theme in Time Slice t The correlation between them is modeled, and the expression for the decoder's decoding process is: .

[0049] Specifically, the objective function in step 300 is obtained by weighted fusion of two loss functions, and the expression of the first loss function is:

[0050] ;

[0051] in, It is the bag-of-words vector of the document. It is a topic-word distribution matrix. It is the cross-entropy loss, used to measure the difference between the word distribution generated by the model and the bag-of-words of the real document. This is the posterior distribution (variational distribution) of the encoder output. The first term is the reconstruction loss, which measures the model's ability to reconstruct the input text to ensure that the model learns a reasonable topic structure. The second term is the KL divergence between the prior distribution and the variational distribution, which makes it close to the standard normal distribution to avoid overfitting and improve the model's generalization ability.

[0052] The second loss function is obtained by minimizing the transmission cost of the Euclidean distance between word embeddings and topic embeddings in the mutual information-enhanced dynamic topic model through a clustering regularizer, and its expression is:

[0053] ;

[0054] in, It is a soft allocation matrix obtained through the Sinkhorn algorithm, used to represent the optimal transmission probability from words to topics.

[0055] The final expression for the overall objective function after fusion is:

[0056] ;

[0057] ;

[0058] in and These are the hyperparameters of the model.

[0059] This invention provides an example embodiment using four publicly available datasets: two long-text academic paper datasets and two short-text news datasets. These datasets undergo the same preprocessing, such as stop word removal and lemmatization, to ensure they function correctly on traditional models and guarantee experimental fairness. The Sentence-BERT model is used to encode the documents. During the self-similarity graph construction process, the threshold hyperparameter... This determines whether semantic edges of word pairs are preserved, thus affecting the sparsity of the graph structure. This example demonstrates... A grid search is performed within the specified range, and the optimal value is selected based on the topic coherence metric on the validation set. Temperature parameter Set to 0.2. Furthermore, the weight coefficients of each regularization term in the loss function are respectively set to... , This example also comprehensively evaluates the model's performance in terms of topic coherence and topic diversity to measure the performance of the proposed method in semantic consistency and topic discriminability. Finally, the example is compared with several topic models, including: a classic probabilistic dynamic topic model (DTM), a neural model extending DTM through neural variational inference (NDTM), a neural dynamic topic model with pre-trained word embeddings (DETM), a clustering-based topic discovery model using document embeddings (BERTopic), a neural dynamic topic model based on attention mechanisms (DSNTM), and a novel neural unlinked dynamic topic model (CFDTM). The results for each model with 50 topics are shown in Table 1.

[0060] Table 1 Comparison of results for topics with 50 topics

[0061]

[0062] The beneficial effects of this invention are as follows:

[0063] 1) By constructing a document graph through word embedding and retaining only highly similar word pairs, semantically irrelevant words are filtered out, the inherent dependencies between topic words are strengthened, and global contextual information is introduced through sentence embedding and mutual information between document / topic distribution, which improves semantic distinguishability while avoiding topic degradation.

[0064] 2) A graph isomorphic network was used to learn document graph embeddings, which improved topic consistency while preserving the semantic association of candidate topic words and improved the efficiency of capturing complex semantic structures.

[0065] 3) By using clustering regularization and combining it with the Sinkhorn algorithm, word-topic embedding alignment was optimized, reducing topic overlap and noise impact.

[0066] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0067] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A graph embedding method based on mutual information enhancement, characterized in that, Includes the following steps: Latent topics are obtained by learning graph encoding and graph representation for target documents and target vocabulary using the self-similarity method; The potential topics and contextual information in the target document are enhanced by a pre-built mutual information-enhanced dynamic topic model to obtain the reconstructed document distribution; The parameters of the mutual information-enhanced dynamic topic model are optimized by a clustering regularizer to obtain an objective function. The loss of the reconstructed document distribution is then optimized using the objective function to obtain the nonlinear dependency relationship between the target document and the potential topic.

2. The graph embedding method based on mutual information enhancement according to claim 1, characterized in that, By learning graph encoding and graph representation for target documents and target vocabulary using the self-similarity method, latent topics are obtained, including: Each word in the target vocabulary is mapped to a vector to obtain a word embedding matrix; Based on the word embedding matrix, a semantic similarity matrix between word pairs is constructed by combining the set of words in the target vocabulary in which the words of the target document appear. Word pairs with a cosine similarity higher than a preset threshold in the semantic similarity matrix are selected and connected to obtain a self-similarity graph; The latent topics are obtained by learning graph representations of the self-similar graph through a graph isomorphic network.

3. The graph embedding method based on mutual information enhancement according to claim 2, characterized in that, The formula for calculating the cosine similarity is: ； in, and These are word embeddings for two words respectively. It is an L2 paradigm.

4. The graph embedding method based on mutual information enhancement according to claim 1, characterized in that, The reconstructed document distribution is obtained by enhancing the mutual information of the latent topics and the contextual information in the target document using a pre-built mutual information-enhanced dynamic topic model, including: The target document is encoded using a normal distribution through the encoder of the mutual information-enhanced dynamic topic model to obtain variational distribution parameters; The variational distribution parameters are subjected to parameterized resampling to obtain the document-topic distribution; The mutual information lower bound is maximized on the context information, and the document-topic distribution is optimized and aligned based on the maximized mutual information lower bound. The decoder of the mutual information-enhanced dynamic topic model performs a multinomial distribution sampling reconstruction operation on the optimized and aligned document-topic distribution to obtain the reconstructed document distribution.

5. The graph embedding method based on mutual information enhancement according to claim 4, characterized in that, The expression for maximizing the lower bound of mutual information is: ； in, Embedded in sentences, For the document-topic distribution within time slice t, These are the positive and negative samples within time slice t.

6. The graph embedding method based on mutual information enhancement according to claim 1, characterized in that, The parameters of the mutual information-enhanced dynamic topic model are optimized using a clustering regularizer to obtain an objective function. The reconstructed document distribution is then optimized using this objective function to obtain the nonlinear dependency relationship between the target document and the latent topic, including: Construct the first loss function based on the reconstruction loss and KL divergence; The second loss function is obtained by minimizing the transmission cost of the mutual information-enhanced dynamic topic model through the clustering regularizer; The first loss function and the second loss function are weighted and fused to obtain the objective function.

Images