A graph contrastive topic discovery method based on dynamic semantic expansion

Abstract

This invention provides a graph-based contrastive topic discovery method based on dynamic semantic augmentation, belonging to the field of natural language processing. Addressing the problem of extremely sparse short text data on social media and the dynamic evolution of topics over time, this method first acquires a time-constrained hybrid corpus and uses a large language model to perform spatiotemporal alignment and semantic completion on the original short text. Second, it constructs a bipolar word co-occurrence graph, calculates mutual information between words based on the hybrid corpus, and constructs topological graph structures representing positive and negative associations respectively. Finally, it performs graph-based contrastive neural topic modeling, fusing text features with graph topological priors, introducing a contrastive learning mechanism in the latent space for constraint, and jointly optimizing the objective function to output topic distribution. This invention transforms text augmentation into topological graph bridging, effectively filling semantic gaps and accurately delineating topic boundaries, making it suitable for topic mining of highly sparse short text streams.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and data mining, specifically to a graph comparison topic discovery method based on dynamic semantic expansion for highly sparse and highly dynamic short text streams. Background Technology

[0002] The essence of topic discovery lies in mining hidden potential topics from massive document collections, each topic typically explained by a semantically related sequence of words. With the widespread adoption of mobile internet and smart devices, social media platforms such as Weibo and Twitter have generated massive amounts of user-generated content. These data streams contain highly valuable public opinion dynamics. However, social media data is generally characterized by short length, casual language, and a large amount of noise, leading to a serious "data sparsity" problem.

[0003] Traditional topic models (such as LDA) and early variational autoencoder (VAE) topic models heavily rely on word co-occurrence frequencies. When processing short texts, due to a lack of sufficient contextual information, they often exhibit "semantic gaps," extracting meaningless, noisy topics. In recent years, to alleviate sparsity, some studies have utilized large language models (LLMs) to semantically augment short texts. However, existing augmentation methods typically employ static strategies, ignoring the dynamic characteristics of topic evolution over time. Large models are prone to "data skipping" when generating derived vocabulary, introducing future knowledge into the understanding of historical texts, leading to lags and biases in temporal analysis.

[0004] Furthermore, existing topic models combining graph neural networks often only consider positive co-occurrence associations between words, ignoring mutual exclusion and negative associations. This results in blurred boundaries between different topics in highly overlapping semantic spaces, making the models prone to overfitting. Therefore, for sparse data streams such as social media, there is an urgent need for a topic discovery method that can effectively expand local semantics without disrupting temporal patterns and accurately delineate topic boundaries. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a graph-based contrastive topic discovery method based on dynamic semantic augmentation. This invention utilizes a time-constrained large language model to perform dynamic semantic compensation on sparse text, and simultaneously introduces contrastive learning by constructing a positive and negative bipolar word co-occurrence graph, effectively alleviating the data sparsity problem and enhancing the clarity of topic boundaries.

[0006] The technical content of this invention is as follows:

[0007] Step 1: Obtain a time-constrained hybrid corpus; clean the original short text data stream and extract the top k words with the highest weighted term frequency-inverse document frequency (TF-IDF) weights from each short text as the core anchor set; inject the core anchor set and the timestamps corresponding to the short texts into a pre-trained large language model, constraining the large language model to generate derived background texts only for that time node; use a sentence vector model to calculate the similarity between the core anchor set and the derived background texts. If the similarity is greater than a set gating threshold, then concatenate the derived background texts with the original short texts to generate a dense hybrid corpus.

[0008] Step 2: Construct a bipolar word co-occurrence graph; Set a sliding window on the hybrid corpus to count word frequencies and co-occurrence frequencies between words, and then calculate point mutual information (PMI) between all words; Based on the set positive and negative thresholds, connect word pairs with PMI greater than the positive threshold to form a positive co-occurrence graph, and connect word pairs with PMI less than the negative threshold to form a negative co-occurrence graph, thereby extracting global semantic topology information from the expanded corpus.

[0009] Step 3: Perform topic inference based on a graph contrastive neural topic model; input the text features from the hybrid corpus into the encoder of a variational autoencoder to infer the latent variable representation and the predicted topic distribution of the text; simultaneously, use two sets of graph convolutional networks to aggregate global vocabulary features on the positive and negative co-occurrence maps respectively, and decode and output the word-topic distribution guided by the positive and negative maps; construct a contrastive loss in the latent vector space to maximize the similarity between the predicted topic distribution and the topic distribution guided by the positive map, and minimize the similarity between the predicted topic distribution and the topic distribution guided by the negative map; jointly train the model with the reconstruction loss, KL divergence, and the contrastive loss, and output the topic distribution of the text to be analyzed after training.

[0010] This invention utilizes a large language model to generate structured and enhanced derived text, transforming "text expansion" into "topological graph bridging," thus filling the semantic gaps caused by the sparsity of short texts. Simultaneously, it mandates the injection of timestamps into the generated prompts, strictly avoiding the "data leakage" and "semantic transgression" problems in time series analysis. This invention innovatively proposes a contrastive learning mechanism based on positive and negative bipolar co-occurrence graphs. It not only utilizes the positive graph to strengthen synonym associations but also leverages the negative graph to capture mutually exclusive semantics, forcing the model to bring positive samples closer and push away negative samples in the latent space, significantly improving the model's ability to delineate topic boundaries under extremely sparse and noisy data. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the overall process of a graph comparison topic discovery method based on dynamic semantic expansion according to the present invention.

[0012] Figure 2 This is a schematic diagram of the graph-contrast neural theme model structure of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0014] In one embodiment of the present invention, a graph comparison topic discovery method based on dynamic semantic expansion includes three core stages, and the specific implementation process is as follows:

[0015] Step 1, Dynamic semantic augmentation of the text:

[0016] First, obtain the raw short text data stream. ,in For text content, This is the timestamp corresponding to the text. After cleaning and denoising the text, calculate the TF-IDF weights of the words, and extract the top words with the highest weights from each text. Each word as a semantic anchor set .

[0017] To perform semantic completion for spatiotemporal alignment, the semantic anchor set is... With timestamp Inject the preset prompt word (Prompt) and call the Large Language Model (LLM) to generate derived background text for a specific time point. The generation process of a large language model can be represented as a conditional probability distribution:

[0018] Subsequently, a dual-gating filtering mechanism is introduced. A pre-trained Sentence-BERT model is used to map the core anchor set and derived background text into dense sentence vectors. and Calculate the cosine similarity between the two. :

[0019] when ( The generated text is considered valid when a set threshold (e.g., 0.5) is reached. The valid derived background text is then concatenated with the original short text to form a dense hybrid corpus. .

[0020] Step 2, Constructing a bipolar word co-occurrence map:

[0021] For mixed corpora Global word segmentation and stop word filtering are performed to extract high-frequency and effective words and construct a global vocabulary. Set the size to [size] on the mixed corpus. A sliding window to count individual words probability of occurrence and word pairs Co-occurrence probability within the same window .

[0022] Point-to-point mutual information (PMI) is used to quantify the true semantic relevance between words. The calculation formula is as follows:

[0023] Set positive threshold With negative threshold The global word network is split into a bipolar graph:

[0024] (1) Forward co-occurrence diagram :when At that time, at the word node and Establish edges between them, and denote their adjacency matrix as follows: , is used to characterize strongly related synonymous or co-occurring contexts.

[0025] (2) Negative co-occurrence graph :when At that time, at the word node and Edges are established between them using the absolute value of PMI as the weight, and their adjacency matrix is ​​denoted as follows. These are used to represent mutually exclusive or extremely low co-occurrence semantics. Derived words generated by the large language model serve as bridging nodes, reconstructing the semantic connectivity of sparse data at the topological level.

[0026] Step 3, Graph Comparison Neural Topic Modeling Stage:

[0027] Extracting statistical features from mixed corpora Dense features of pre-trained BERT At the encoder, the two are concatenated and input into a multilayer perceptron (MLP) to infer the Gaussian parameters (mean) of the latent topic distribution. With log variance ):

[0028] The latent vector representation of the document is obtained through sampling using the reparameterization trick. The topic distribution of the document is obtained via the Softmax function. :

[0029] At the decoder end, two sets of graph convolutional networks (GCNs) are used in parallel to process global vocabulary node features. Through the adjacency matrix and Information is transmitted between the two sides, and word-topic distributions incorporating positive topological priors are generated respectively. Word-topic distribution with negative topological prior :

[0030] A contrastive loss is further constructed in the latent vector space. Positive sample features are calculated based on the decoder output. Features of negative samples Using an InfoNCE-type loss function to narrow down the latent vectors. The cosine distance to positive sample features is used to extrapolate the cosine distance to negative sample features: (in For batch size, (This represents the vector after L2 normalization).

[0031] Finally, by integrating the variational lower bound and the contrastive loss, a joint optimization objective function is constructed. :

[0032] In the formula, For the reconstruction loss of text word frequency, To constrain the KL divergence of the latent vector distribution, To compare the weight coefficients of the loss, the optimizer is used to update the model parameters through backpropagation. After the model converges, the accurate topic distribution corresponding to each text can be output.

Claims

1. A graph-based contrastive topic discovery method based on dynamic semantic expansion, characterized in that, Includes the following steps: Step 1: Obtain a time-constrained hybrid corpus; use a large language model to perform spatiotemporal alignment semantic completion on the original short text to generate an enhanced hybrid corpus; Step 2: Construct a bipolar word co-occurrence graph; calculate the inter-point mutual information between words based on the hybrid corpus, and construct topological graph structures representing positive and negative associations respectively; Step 3: Perform graph contrastive neural topic modeling; construct an inference network that integrates text features and graph topological priors, and introduce a contrastive learning mechanism to constrain the latent space; Step 4: Obtain the topic distribution of the text to be processed by jointly optimizing the objective function.

2. The graph comparison topic discovery method based on dynamic semantic expansion according to claim 1, characterized in that, The acquisition of the time-constrained hybrid corpus specifically includes: Extract the word with the highest weighted term frequency-inverse document frequency weight from the original short text as a semantic anchor; The semantic anchors and the timestamps of the original text are injected into the prompt words, and the large language model is called to generate derived background text that is semantically aligned with the time node. The similarity between semantic anchors and derived background text is calculated using sentence vector models, and texts below a set threshold are removed. The derived background text, filtered through gating, is concatenated with the original short text to form a dense hybrid corpus.

3. The graph comparison topic discovery method based on dynamic semantic expansion according to claim 1, characterized in that, The step of constructing a bipolar word co-occurrence map specifically includes: On a hybrid corpus, word frequency and co-occurrence frequency are statistically analyzed using a sliding window, and the point-to-point mutual information (PMI) among all words is calculated. Set positive and negative thresholds. If the PMI value is greater than the positive threshold, establish an edge in the positive co-occurrence graph; if the PMI value is less than the absolute value of the negative threshold, establish an edge in the negative co-occurrence graph with the absolute value as the weight.

4. The graph comparison topic discovery method based on dynamic semantic expansion according to claim 1, characterized in that, The specific process of graph-contrast neural topic modeling is as follows: The encoder part concatenates statistical and semantic features from the hybrid corpus and uses a variational autoencoder to infer the latent topic vector of the document. ; Decoder section: Two sets of graph convolutional networks are used in parallel to aggregate global lexical features on the positive and negative co-occurrence maps, respectively, and output a positive word-topic distribution incorporating topological priors. With negative word-topic distribution .

5. The graph comparison topic discovery method based on dynamic semantic expansion according to claim 4, characterized in that, The specific formula for introducing contrastive learning constraints in the latent vector space is as follows: in, This indicates a topic representation guided by a positive word-topic distribution. This indicates a topic representation guided by negative word-topic distribution.

6. The graph comparison topic discovery method based on dynamic semantic expansion according to claim 1, characterized in that, The joint objective function is composed of the reconstruction loss, the KL divergence of the variational distribution, and the weighted sum of the contrastive loss as described in claim 5.

Images