Short Text Topic Modeling Method Based on Dynamic Clustering and Word Embedding Enhancement

Abstract

This invention discloses a short text topic modeling method based on dynamic clustering and word embedding enhancement. First, short text data is collected to obtain a short text stream. Then, the short text stream is clustered using the FastStream clustering method, and pseudo-documents are constructed based on the clustering results. Next, a word embedding matrix and a word co-occurrence matrix are formed using a word embedding model pre-trained on a large corpus. Subsequently, the topic distribution of the pseudo-documents and its word distribution are modeled using Dirichlet distribution to obtain a topic model. Then, the topic model is trained on the pseudo-documents using the Gibbs sampling method, updating the topic assignment of each word in the pseudo-documents and the topic distribution parameters of the pseudo-documents until the topic model converges. Finally, new short texts are acquired for topic inference to obtain the topic distribution. This invention utilizes the combination of FastStream clustering and word embedding techniques to effectively improve the topic modeling performance of short text data by creating pseudo-document views and enhancing word embeddings.

Description

Technical Field

[0001] This invention belongs to the technical field of text analysis, specifically relating to a short text topic modeling method based on dynamic clustering and word embedding enhancement. Background Technology

[0002] In the digital age, short text data, such as social media messages, news headlines, and online comments, has become one of the main ways of information exchange. Due to its limited content and rapid updates, short text, while providing rich information, also poses challenges to traditional text analysis techniques; especially in topic modeling and cluster analysis, traditional methods often struggle to capture the deep semantic relationships within short text, affecting the effective organization and summarization of the data. Therefore, researching and developing new technologies that can effectively process short text data is of great significance for supporting applications such as trend analysis and user preference mining.

[0003] In existing technologies, short text clustering typically employs traditional clustering algorithms such as K-means or hierarchical clustering. These methods perform well when handling small-scale, static datasets; however, their efficiency drops significantly when dealing with real-time data streams and large-scale data processing, and they lack flexibility and scalability in dynamic environments. Furthermore, although pre-trained word embedding models, such as Word2Vec and BERT, have made significant progress in capturing lexical semantics, directly applying these models to short text clustering still has limitations. These models are insufficient in capturing complex semantic relationships between short texts, making it difficult to effectively improve the quality and semantic consistency of clustering.

[0004] Specifically, existing methods face the following main problems when processing short text analysis:

[0005] 1. Low processing efficiency: Traditional methods require pre-setting the number of clusters and performing multiple iterations on the entire dataset, which is extremely unfavorable for large-scale data stream scenarios that require real-time processing;

[0006] 2. Insufficient adaptability and flexibility: Existing methods are difficult to adjust and update in a timely manner when dealing with dynamic changes and real-time data streams, which affects the real-time performance and accuracy of clustering results;

[0007] 3. Clustering quality issues: Due to the high-dimensional sparsity and semantic complexity of short texts, the clustering results of traditional methods often lack accuracy and semantic consistency.

[0008] To address the aforementioned challenges, there is an urgent need to develop a novel technology that can effectively handle the dynamic characteristics of short texts, improve processing efficiency, and significantly optimize clustering quality. This invention proposes a short text topic modeling method based on dynamic clustering and word embedding enhancement, aiming to overcome the limitations of existing technologies and provide more accurate and efficient technical support for applications such as real-time data analysis, trend prediction, and user preference mining. Summary of the Invention

[0009] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a short text topic modeling method based on dynamic clustering and word embedding enhancement. This method combines FastStream clustering and word embedding techniques to effectively improve the topic modeling performance of short text data by creating pseudo-document views and enhancing word embedding.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] The primary objective is to provide a short text topic modeling method based on dynamic clustering and word embedding enhancement, comprising the following steps:

[0012] Step 1, Data Preprocessing: Collect short text data and perform text cleaning to obtain a short text stream;

[0013] Step 2, Pseudo-document construction: Use the FastStream clustering method to cluster the short text stream and construct pseudo-documents based on the clustering results;

[0014] Step 3, word embedding generation: Using a word embedding model pre-trained on a large corpus, words in the pseudo-document are mapped to a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix;

[0015] Step 4, Pseudo-document topic modeling: The topic distribution and word distribution of pseudo-documents are modeled using the Dirichlet distribution to obtain the topic model;

[0016] Step 5, Topic Distribution Inference: Train the topic model on the pseudo-documents using the Gibbs sampling method, update the topic assignment of each word in the pseudo-documents and the topic distribution parameters of the pseudo-documents, until the topic model converges;

[0017] Step 6: Short text topic prediction: Obtain new short texts and generate their word embedding representations, input them into the trained topic model for topic inference, and obtain the topic distribution.

[0018] As a preferred technical solution, the pseudo-document construction steps are as follows:

[0019] Let the short text stream be represented as Where, d tThe short text arriving at time t; the parameters for initializing the FastStream clustering method; the parameters include the document set D, the vocabulary V, and the word frequency n for each document. wd and the word frequency n of each cluster wz The document collection is used to store documents assigned to clusters.

[0020] Each short text in the short text stream is treated as a document and sequentially input into the document set, and the arrived document d is used. t Update word relationship graph G t = (V, E), where E is the set of edges representing word co-occurrence relations; edge weight e x,t The co-occurrence frequency f(x,t) of word x and document t is represented by e. x,t ∈E, if f(x,t)≥∈, where ∈ is the word co-occurrence frequency threshold in the word relation graph;

[0021] The topic is discovered through a word relationship graph. Words with frequencies higher than a set frequency threshold are selected as seed nodes, and word weights are calculated based on their co-occurrence frequencies with other words, as shown in the following formula:

[0022]

[0023] Among them, c w The word weight of word w, For topic k m The number of words contained For topic k m The total number of occurrences of all words in the text, k m To represent the m-th topic;

[0024] For each document d t Calculate the probability that document d will be assigned to an existing cluster z. t If the probability of being assigned to an existing cluster z is greater than the probability of being assigned to any other possible cluster, or if it is greater than the minimum acceptance probability of cluster z, then the document will be assigned to an existing cluster z.

[0025] The document d t The probability of being assigned to an existing cluster z is calculated as follows:

[0026]

[0027] Among them, z d For document d t The assigned cluster z, Assign a set to the current topic for all documents; α is one of the hyperparameters of the Dirichlet distribution, used to control the prior probability of generating new clusters; β is another hyperparameter of the Dirichlet distribution, used to control the prior distribution of the probability of words in the vocabulary appearing in the topic; m z n is the number of documents in cluster z; wz n is the total number of occurrences of word w in cluster z, that is, the total number of times word w appears in all documents assigned to cluster z; z The total number of occurrences of all words in cluster z, where i and j represent the position or order of the words in the document or cluster;

[0028] For document d that cannot be assigned to an existing cluster t Calculate the probability of creating a new cluster. If the probability of creating a new cluster is greater than the probability threshold, a new cluster will be created and the document will be assigned to the new cluster.

[0029] The probability of creating a cluster is calculated using the following formula:

[0030]

[0031] Among them, z new This is a new cluster;

[0032] After clustering is completed, pseudo-documents are generated from the clustering results. Each pseudo-document contains all the short texts in its respective cluster, forming a new document set D'.

[0033] As a preferred technical solution, when creating a new cluster, a cluster activity weight is introduced and the weight decay is calculated using Newton's law of cooling:

[0034]

[0035] Among them, l z Let z be the cluster activity weight of the z-th cluster. The initial active weights are defined when the z-th cluster is created, κ is a constant used to control the decay rate of the active weights, and t c At the current time, u z This represents the last update time for the z-th cluster.

[0036] For clusters with an activity weight of 0, calculate the probability of merging with other clusters and decide whether to merge or exist independently based on the following formula:

[0037]

[0038] Among them, z c Let z represent the c-th cluster being processed, and z represent clusters that can be merged. selfLet c represent the non-merging, independently existing clusters, and n represent the cluster currently being processed. wc Let w represent the total number of occurrences of word w in cluster c.

[0039] As a preferred technical solution, the word embedding generation step is as follows:

[0040] Preprocess the text in the pseudo-document, including removing non-text characters, standardizing the format, and splitting the text into individual words;

[0041] A word embedding model pre-trained on a large-scale corpus maps each word in the pseudo-document to a high-dimensional vector space, generating a high-dimensional vector representation of each word to form a word embedding matrix;

[0042] Each word embedding vector in the word embedding matrix is ​​applied to each word in the pseudo-document, and the similarity between words is calculated to form a word co-occurrence matrix.

[0043] As a preferred technical solution, the similarity between the words is calculated using cosine similarity, with the following formula:

[0044]

[0045] Where, sim(v) i ,v j ) represents the similarity between word i and word j, v i Let v be the word embedding vector of word i. j Let be the word embedding vector of word j.

[0046] As a preferred technical solution, the pseudo-document topic modeling steps are as follows:

[0047] The topic distribution of pseudo-documents is represented using the Dirichlet distribution: θ d ~Dir(α), where θ d Let α represent the topic distribution of pseudo-document d, where α is the parameter of the Dirichlet distribution;

[0048] For each word n ​​in pseudo-document d, according to the topic distribution θ of the pseudo-document... d Choose a topic for each word z dn ~θ d , z dn The topic of word n ​​in pseudo-document d;

[0049] The word distribution for each topic is generated using the Dirichlet distribution, denoted as: φ k ~Dir(β), where φ k Let β represent the word distribution of topic k, where β is the parameter of the Dirichlet distribution;

[0050] The topic z is selected based on each word n ​​in pseudo-document d. dnSubstituting the word distribution into the word distribution yields the word distribution for this topic. This then generates the distribution of each word n ​​in pseudo-document d.

[0051] Finally, a topic model is built using pseudo-documents enhanced with word embeddings, and represented using a likelihood function:

[0052]

[0053] in, Let w represent the likelihood function of the entire topic model, z represent the topic, l represent the pseudo-document set, and α and β are hyperparameters.

[0054] As a preferred technical solution, the inferred topic distribution specifically refers to:

[0055] The Gibbs sampling method is used to randomly initialize a topic for each word in the pseudo-document;

[0056] The conditional probability of each word being assigned to each topic is calculated using the following formula:

[0057]

[0058] Wherein, p(z) dn =k|w dn ,z (-dn) Let z(α,β) be the conditional probability that word n ​​in pseudo-document d is assigned to topic k. dn For the topic of word n ​​in pseudo-document d, w dn The distribution of word n ​​in pseudo-document d; Excluding word n, the number of words in pseudo-document d that are assigned to topic k; This represents the total number of words in pseudo-document d, excluding word n. Z represents the number of words belonging to topic k and being word w, excluding word n. (-dn) V represents the topic of all words in pseudo-document d except for word n, where V is the vocabulary and K is the number of topics in the topic model.

[0059] The topic of each word is updated using a multinomial distribution based on the conditional probability of each word being assigned to each topic, and then fed into the topic model for learning.

[0060] Continue to calculate the conditional probability for each word, update the topic and input it into the topic model for learning, until the predetermined number of iterations is reached or the model converges;

[0061] Obtain the topic assignment results when the predetermined number of iterations or model convergence is reached, and obtain the topic distribution of the pseudo-document.

[0062] The second objective is to provide a short text topic modeling system based on dynamic clustering and word embedding enhancement, including a data preprocessing module, a pseudo-document construction module, a word embedding generation module, a topic modeling module, a distribution inference module, and a subject prediction module;

[0063] The data preprocessing module is used to collect short text data and perform text cleaning to obtain a short text stream.

[0064] The pseudo-document construction module is used to cluster short text streams using the FastStream clustering method and construct pseudo-documents based on the clustering results;

[0065] The word embedding generation module is used to map words in pseudo-documents to a high-dimensional vector space using a word embedding model pre-trained on a large corpus to form a word embedding matrix and a word co-occurrence matrix;

[0066] The topic modeling module is used to model the topic distribution and word distribution of pseudo-documents using the Dirichlet distribution, thereby obtaining a topic model.

[0067] The distribution inference module is used to train the topic model on pseudo-documents using the Gibbs sampling method, and to update the topic assignment of each word in the pseudo-documents and the topic distribution parameters of the pseudo-documents until the topic model converges.

[0068] The subject prediction module is used to acquire new short texts and generate their word embedding representations, which are then input into a trained topic model for topic inference to obtain the topic distribution.

[0069] The third objective is to provide an electronic device, comprising:

[0070] At least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0071] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the short text topic modeling method based on dynamic clustering and word embedding enhancement described above.

[0072] The fourth objective is to provide a computer-readable storage medium storing a program, characterized in that, when the program is executed by a processor, it implements the aforementioned short text topic modeling method based on dynamic clustering and word embedding enhancement.

[0073] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0074] 1. Real-time Processing Capability: This invention utilizes FastStream clustering technology, combining real-time processing capabilities with the analysis of large-scale short text data, breaking through the bottlenecks of traditional methods. FastStream clustering technology, through incremental updates, can quickly adjust clustering results and update topic models as data continuously arrives, avoiding the high-cost operation of global recalculation. This innovation makes this invention particularly suitable for scenarios requiring immediate response, such as social media monitoring, online news analysis, and real-time market trend tracking, significantly improving system processing efficiency and response speed. Furthermore, this real-time processing capability has a clear competitive advantage in dealing with sudden events and rapidly changing information environments.

[0075] 2. Dynamic Adaptability: This invention introduces a dynamic adjustment mechanism to flexibly respond to continuous changes in data streams, particularly effectively addressing the "concept drift" problem. In practical applications, the characteristics and topic distribution of data streams may change significantly over time. This invention enables the model to automatically adapt to these changes without manual intervention by updating pseudo-documents and cluster centers in real time. Specifically, the model updates parameters through incremental learning, allowing it to promptly capture new data patterns and trend changes. This dynamic adaptability ensures the model's stability and accuracy in long-term use, enabling it to maintain efficient operation in complex and ever-changing environments. It is suitable for dynamic, information-intensive fields such as real-time market analysis and personalized recommendation systems.

[0076] 3. Effective Solution to Data Sparsity Problem: To address the high-dimensional sparsity problem of short text data, this invention introduces pseudo-document technology. By aggregating semantically similar short texts into pseudo-documents, the information density of the data is improved. Pseudo-document technology not only effectively reduces data sparsity but also improves the training effect of the model by performing topic modeling at a richer semantic level. Specifically, pseudo-documents enhance the completeness of semantic representation by aggregating similar texts, enabling the topic model to more accurately identify and capture potential topics in short texts. This method greatly improves the semantic consistency and representativeness of generated topics, overcoming the shortcomings of traditional methods in short text processing.

[0077] 4. Reducing Ineffective and Redundant Clustering: This invention optimizes the clustering management process by introducing cluster activity weights and a weight decay mechanism based on Newton's law of cooling. The cluster activity weights dynamically assess the activity level of each cluster, ensuring system resources are concentrated on meaningful clusters, while the weight decay mechanism ensures that inactive or redundant clusters are removed promptly. This optimization process not only reduces ineffective and redundant clustering but also effectively improves the overall system efficiency and resource utilization. By reducing unnecessary computational burden, this invention significantly improves performance in large-scale data processing, adapting to application environments with limited resources and high-efficiency processing requirements. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is an overall flowchart of the short text topic modeling method based on dynamic clustering and word embedding enhancement in this embodiment of the invention.

[0080] Figure 2 This is a schematic diagram of the short text topic modeling system based on dynamic clustering and word embedding enhancement in an embodiment of the present invention.

[0081] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0082] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0083] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0084] like Figure 1 As shown, this embodiment of the short text topic modeling method based on dynamic clustering and word embedding enhancement includes the following steps:

[0085] Step 1: Data preprocessing: Collect short text data and perform text cleaning to obtain a short text stream.

[0086] Step 2, Pseudo-document construction: Use the FastStream clustering method to cluster the short text stream and construct pseudo-documents based on the clustering results.

[0087] Specifically, the steps for building a pseudo-document are as follows:

[0088] 2.1. Let the short text stream be represented as... Where, d t This represents the short text arriving at time t; the parameters for initializing the FastStream clustering method include the document set D (used to store documents assigned to clusters), the vocabulary V, and the word frequency n for each document. wd and the word frequency n of each cluster wz wait.

[0089] 2.2 Treat each short text in the short text stream as a document and input them sequentially into the document set, then use the arrived document d t Update word relationship graph G t = (V, E), where E is the set of edges representing word co-occurrence relations; edge weight e x,t The co-occurrence frequency f(x,t) of word x in a document is represented by e. x,t ∈E, if f(x,t)≥∈, where ∈ is the word co-occurrence frequency threshold in the word relationship graph, used to determine whether the word co-occurrence relationship is valid.

[0090] 2.3. Identify topic words through a word relationship graph, select words with frequencies higher than a set frequency threshold as seed nodes, and calculate word weights based on their co-occurrence frequencies with other words, using the following formula:

[0091]

[0092] Among them, c w The word weight of word w, For topic k m The number of words contained For topic k m The total number of occurrences of all words in the text, k m To represent the m-th topic, the core of discovering topic words through word relationship graphs lies in utilizing the co-occurrence relationship between words, combined with word frequency and weight calculations, to identify those words that appear repeatedly in multiple documents and have strong co-occurrence relationships as topic words.

[0093] 2.4. For each document d t Calculate the probability that document d will be assigned to an existing cluster z. t If the probability of being assigned to an existing cluster z is greater than the probability of being assigned to any other possible cluster, or if it exceeds the minimum acceptance probability of cluster z, then the document will be assigned to an existing cluster z.

[0094] For each document d t The probability that it is assigned to an existing cluster z is:

[0095]

[0096] Among them, z d For document d t The assigned cluster z, Assign a set to the current topic for all documents; α is one of the hyperparameters of the Dirichlet distribution, used to control the prior probability of generating new clusters; β is another hyperparameter of the Dirichlet distribution, used to control the prior distribution of the probability of words in the vocabulary appearing in the topic; m z n is the number of documents in cluster z; wz n is the total number of occurrences of word w in cluster z, that is, the total number of times word w appears in all documents assigned to cluster z; z The total number of occurrences of all words in cluster z is given by , where i and j represent the position or order of the words in the document or cluster.

[0097] 2.5. For documents d that cannot be assigned to an existing cluster. t The probability of creating a new cluster is calculated. If the probability of creating a new cluster is greater than the probability threshold, a new cluster will be created and the document will be assigned to the new cluster.

[0098] For document d that cannot be assigned to an existing cluster t The probability of creating a new cluster is:

[0099]

[0100] Among them, z new This is a new cluster;

[0101] 2.6 After clustering, pseudo-documents are generated from the clustering results. Each pseudo-document contains all the short texts in its cluster, forming a new document set D'.

[0102] Furthermore, to avoid generating too many micro-clusters, this application introduces cluster activity weights when creating new clusters and calculates weight decay using Newton's law of cooling:

[0103]

[0104] Among them, l z Let z be the cluster activity weight of the z-th cluster. The initial active weights are defined when the z-th cluster is created, κ is a constant used to control the decay rate of the active weights, and t c At the current time, u z This represents the last update time for the z-th cluster.

[0105] For clusters with an activity weight of 0, calculate the probability of merging with other clusters and decide whether to merge or exist independently based on the following formula:

[0106]

[0107] Among them, z c Let z represent the c-th cluster being processed, and z represent clusters that can be merged. self Let c represent the non-merging, independently existing clusters, and n represent the cluster currently being processed. wc Let w represent the total number of occurrences of word w in cluster c. By identifying and reassigning documents in inappropriate clusters through a series of rules, the model's accuracy and topic concentration are improved.

[0108] Through the above process, FastStream can process short text streams in real time, dynamically generate and update pseudo-documents, and achieve efficient short text stream clustering.

[0109] Step 3, word embedding generation: Using a word embedding model pre-trained on a large corpus, words in the pseudo-documents are mapped to a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix.

[0110] Furthermore, word embeddings are used to enhance pseudo-document co-occurrence information, thereby improving the effectiveness of short text topic modeling. The word embedding generation steps are as follows:

[0111] 3.1 Preprocess the text in the pseudo-document, including removing non-text characters, standardizing the format, and splitting the text into individual words;

[0112] 3.2. Use a word embedding model (such as Word2Vec or GloVe) pre-trained on a large-scale corpus to map each word in the pseudo-document into a high-dimensional vector space, generating a high-dimensional vector representation of each word to form a word embedding matrix;

[0113] 3.3. Apply the word embedding vector from the word embedding matrix to each word in the pseudo-document, and calculate the word similarity to form a word co-occurrence matrix. The word similarity is calculated using cosine similarity, with the following formula:

[0114]

[0115] Where, sim(v) i ,v j ) represents the similarity between word i and word j, v i Let v be the word embedding vector of word i. j Let be the word embedding vector of word j.

[0116] In this embodiment, the word embedding model uses the Word2Vec model, and the pre-training process is as follows:

[0117]

[0118] Among them, wO It is the target word, w I It is a context word, v O and v I These are the word embedding vectors of the target word and the context word, respectively, where ω′ is any word in the vocabulary V, and v ω′ Let ω′ be the word embedding vector of the word in the vocabulary.

[0119] Step 3 improves the effectiveness of topic modeling by implicitly adding word co-occurrence information to the pseudo-documents. Through these steps, word embeddings are effectively applied in pseudo-document topic models, solving the sparsity problem of short text data and improving the performance and effectiveness of short text topic modeling.

[0120] Step 4: Pseudo-document topic modeling: The topic distribution and word distribution of pseudo-documents are modeled using the Dirichlet distribution to obtain the topic model.

[0121] Specifically, the steps for pseudo-document topic modeling are as follows:

[0122] 4.1 Obtain the topic distribution of the pseudo-documents, represented by the Dirichlet distribution: θ d ~Dir(α), where θ d Let α represent the topic distribution of pseudo-document d, where α is the parameter of the Dirichlet distribution;

[0123] 4.2. Representing the topic distribution of pseudo-documents using the Dirichlet distribution: θ d ~Dir(α), where θ d Let α represent the topic distribution of pseudo-document d, where α is the parameter of the Dirichlet distribution;

[0124] 4.3 For each word n ​​in pseudo-document d, according to the topic distribution θ of the pseudo-document... d Choose a topic for each word z dn ~θ d , z dn The topic of word n ​​in pseudo-document d;

[0125] 4.4. The word distribution for each topic is generated using the Dirichlet distribution, represented as: φ k ~Dir(β), where φ k Let β represent the word distribution of topic k, where β is the parameter of the Dirichlet distribution;

[0126] 4.5. Select the topic z based on each word n ​​in pseudo-document d. dn Substituting the word distribution into the word distribution yields the word distribution for this topic. This then generates the distribution of each word n ​​in pseudo-document d.

[0127] 4.6 Finally, a topic model is built using pseudo-documents enhanced with word embeddings, and represented using a likelihood function:

[0128]

[0129] in, Let w represent the likelihood function of the entire topic model, z represent the topic, l represent the pseudo-document set, and α and β are hyperparameters.

[0130] This step involves representing words in each pseudo-document using word embeddings and modeling the topic distribution of the pseudo-documents and the word distribution of each topic using Dirichlet distributions, thus forming a complete topic modeling process. This process, by introducing pre-trained word embeddings and pseudo-document topic models, enhances the semantic analysis capabilities of short text data, effectively solves the data sparsity problem, and improves the effectiveness of topic modeling.

[0131] Step 5, Topic Distribution Inference: Train the topic model on the pseudo-documents using the Gibbs sampling method, update the topic assignment of each word in the pseudo-documents and the topic distribution parameters of the pseudo-documents, until the topic model converges.

[0132] 5.1 Use the Gibbs sampling method to randomly initialize a topic for each word in the pseudo-document;

[0133] 5.2 Calculate the conditional probability of each word being assigned to each topic. The formula is:

[0134]

[0135] Wherein, p(z) dn =k|w dn ,z (-dn) Let z(α,β) be the conditional probability that word n ​​in pseudo-document d is assigned to topic k. dn For the topic of word n ​​in pseudo-document d, w dn The distribution of word n ​​in pseudo-document d; Excluding word n, the number of words in pseudo-document d that are assigned to topic k; This represents the total number of words in pseudo-document d, excluding word n. Z represents the number of words belonging to topic k and being word w, excluding word n. (-dn) V represents the topic of all words in pseudo-document d except for word n, where V is the vocabulary and K is the number of topics in the topic model.

[0136] 5.3. Update the topic of each word using a multinomial distribution based on the conditional probability of each word being assigned to each topic, and input the result into the topic model for learning;

[0137] 5.4 Continue to calculate the conditional probability for each word, update the topic and input it into the topic model for learning, that is, iterate through steps 5.2-5.4 until the predetermined number of iterations is reached or the model converges;

[0138] 5.5 Obtain the topic allocation results when the predetermined number of iterations or model convergence is reached, and obtain the topic distribution of the pseudo-document.

[0139] The iterative Gibbs sampling process allows the model to progressively optimize parameter estimation and eventually reach a stable state. Through this inference and parameter estimation method, the model can accurately capture the topic distribution in pseudo-documents, thereby achieving efficient topic modeling for short text data. This process combines pre-trained word embedding vectors and parameter estimation of the Dirichlet distribution, enhancing the model's ability to capture and understand semantic information and significantly improving the effectiveness of topic modeling.

[0140] Step 6: Short text topic prediction: Obtain new short texts and generate their word embedding representations, input them into the trained topic model for subject inference, and obtain the topic distribution.

[0141] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0142] Based on the same ideas as the short text topic modeling method based on dynamic clustering and word embedding enhancement in the above embodiments, this invention also provides a short text topic modeling system based on dynamic clustering and word embedding enhancement. This system can be used to execute the aforementioned short text topic modeling method based on dynamic clustering and word embedding enhancement. For ease of explanation, the structural diagram of the embodiment of the short text topic modeling system based on dynamic clustering and word embedding enhancement only shows the parts related to the embodiments of this invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0143] like Figure 2 As shown, another embodiment of the present invention provides a short text topic modeling system based on dynamic clustering and word embedding enhancement, including a data preprocessing module, a pseudo-document construction module, a word embedding generation module, a topic modeling module, a distribution inference module, and a topic prediction module;

[0144] The data preprocessing module is used to collect short text data and perform text cleaning to obtain a short text stream.

[0145] The pseudo-document building module is used to cluster short text streams using the FastStream clustering method and build pseudo-documents based on the clustering results;

[0146] The word embedding generation module is used to map words in pseudo-documents to a high-dimensional vector space using a word embedding model pre-trained on a large corpus, forming a word embedding matrix and a word co-occurrence matrix.

[0147] The topic modeling module is used to model the topic distribution and word distribution of pseudo-documents using the Dirichlet distribution, thus obtaining a topic model;

[0148] The distribution inference module is used to train the topic model on pseudo-documents using the Gibbs sampling method, updating the topic assignment of each word in the pseudo-documents and the topic distribution parameters of the pseudo-documents until the topic model converges.

[0149] The topic prediction module is used to acquire new short texts and generate their word embedding representations, which are then input into a trained topic model for subject inference to obtain the topic distribution.

[0150] It should be noted that the short text topic modeling system based on dynamic clustering and word embedding enhancement of the present invention corresponds one-to-one with the short text topic modeling method based on dynamic clustering and word embedding enhancement of the present invention. The technical features and beneficial effects described in the embodiments of the short text topic modeling method based on dynamic clustering and word embedding enhancement are applicable to the embodiments of the short text topic modeling system based on dynamic clustering and word embedding enhancement. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0151] Furthermore, in the implementation of the short text topic modeling system based on dynamic clustering and word embedding enhancement in the above embodiments, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the short text topic modeling system based on dynamic clustering and word embedding enhancement is divided into different program modules to complete all or part of the functions described above.

[0152] Please see Figure 3 In one embodiment, an electronic device is provided that implements a short text topic modeling method based on dynamic clustering and word embedding enhancement. The electronic device may include a first processor, a first memory, and a bus, and may also include a computer program, such as a short text topic modeling program, stored in the first memory and executable on the first processor.

[0153] The first memory includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the first memory can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the first memory can include both internal and external storage units of the electronic device. The first memory can be used not only to store application software and various types of data installed on the electronic device, such as the code of a short text topic modeling program, but also to temporarily store data that has been output or will be output.

[0154] In some embodiments, the first processor may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules (such as short text topic modeling programs) stored in the first memory and calls data stored in the first memory to perform various functions of the electronic device and process data.

[0155] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0156] The short text topic modeling program stored in the first memory of the electronic device is a combination of multiple instructions, which, when run in the first processor, can achieve the following:

[0157] Step 1, Data Preprocessing: Collect short text data and perform text cleaning to obtain a short text stream;

[0158] Step 2, Pseudo-document construction: Use the FastStream clustering method to cluster the short text stream and construct pseudo-documents based on the clustering results;

[0159] Step 3, word embedding generation: Using a word embedding model pre-trained on a large corpus, words in the pseudo-document are mapped to a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix;

[0160] Step 4, Pseudo-document topic modeling: The topic distribution and word distribution of pseudo-documents are modeled using the Dirichlet distribution to obtain the topic model;

[0161] Step 5, Topic Distribution Inference: Train the topic model on the pseudo-documents using the Gibbs sampling method, update the topic assignment of each word in the pseudo-documents and the topic distribution parameters of the pseudo-documents, until the topic model converges;

[0162] Step 6: Short text topic prediction: Obtain new short texts and generate their word embedding representations, input them into the trained topic model for topic inference, and obtain the topic distribution.

[0163] Furthermore, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0164] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A short text topic modeling method based on dynamic clustering and word embedding enhancement, characterized in that, Includes the following steps: Step 1, Data Preprocessing: Collect short text data and perform text cleaning to obtain a short text stream; Step 2, Pseudo-document construction: Use the FastStream clustering method to cluster the short text stream, and construct pseudo-documents based on the clustering results. The steps are as follows: Let the short text stream be represented as Where, d t The short text arriving at time t; the parameters for initializing the FastStream clustering method; the parameters include the document set D, the vocabulary V, and the word frequency n for each document. wd and the word frequency n of each cluster wz The document collection is used to store documents assigned to clusters. Each short text in the short text stream is treated as a document and sequentially input into the document set, and the arrived document d is used. t Update word relationship graph G t = (V, E), where E is the set of edges representing word co-occurrence relations; edge weights The word x represents the document d. t The co-occurrence frequency f(x,d) t ),Right now if f(x,d t )≥∈, where ∈ is the word co-occurrence frequency threshold in the word relationship graph; The topic is discovered through a word relationship graph. Words with frequencies higher than a set frequency threshold are selected as seed nodes, and word weights are calculated based on their co-occurrence frequencies with other words, as shown in the following formula: Among them, c w The word weight of word w, For topic k m The number of words contained For topic k m The total number of occurrences of all words in the text, k m To represent the m-th topic; For each document d t Calculate the probability that document d will be assigned to an existing cluster z. t If the probability of being assigned to an existing cluster z is greater than the probability of being assigned to any other possible cluster, or if it is greater than the minimum acceptance probability of cluster z, then the document will be assigned to an existing cluster z. The document d t The probability of being assigned to an existing cluster z is calculated as follows: in, For document d t The assigned cluster z, Assign a set to the current topic for all documents; α is one of the hyperparameters of the Dirichlet distribution, used to control the prior probability of generating new clusters; β is another hyperparameter of the Dirichlet distribution, used to control the prior distribution of the probability of words in the vocabulary appearing in the topic; m z n is the number of documents in cluster z; wz n is the total number of occurrences of word w in cluster z, that is, the total number of times word w appears in all documents assigned to cluster z; z The total number of occurrences of all words in cluster z, where i and j represent the position or order of the words in the document or cluster; For document d that cannot be assigned to an existing cluster t Calculate the probability of creating a new cluster. If the probability of creating a new cluster is greater than the probability threshold, a new cluster will be created and the document will be assigned to the new cluster. The formula for calculating the probability of creating a cluster is: Among them, z new This is a new cluster; After clustering, pseudo-documents are generated from the clustering results. Each pseudo-document contains all the short texts in its respective cluster, forming a new document set D. ' ; Step 3, word embedding generation: Using a word embedding model pre-trained on a large corpus, words in the pseudo-document are mapped to a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix; Step 4, Pseudo-document topic modeling: The topic distribution and word distribution of pseudo-documents are modeled using the Dirichlet distribution to obtain the topic model; Step 5, Topic Distribution Inference: Train the topic model on the pseudo-documents using the Gibbs sampling method, update the topic assignment of each word in the pseudo-documents and the topic distribution parameters of the pseudo-documents, until the topic model converges; Step 6: Short text topic prediction: Obtain new short texts and generate their word embedding representations, input them into the trained topic model for topic inference, and obtain the topic distribution.

2. The short text topic modeling method based on dynamic clustering and word embedding enhancement as described in claim 1, characterized in that, When creating a new cluster, cluster activity weights are introduced and weight decay is calculated using Newton's law of cooling. Among them, l p The cluster activity weight of the p-th cluster among all clusters. The initial active weights are defined when the p-th cluster is created, κ is a constant used to control the decay rate of the active weights, and t c At the current time, u p This represents the time when the p-th cluster was last updated. For clusters with an activity weight of 0, calculate the probability of merging with other clusters and decide whether to merge or exist independently based on the following formula: Among them, z c z represents the c-th cluster being processed with an activity weight of 0. inactive z represents the clusters that can be merged. self Let c represent a cluster that does not merge and exists independently, and let n represent the cluster currently being processed with an activity weight of 0. wc Let w represent the total number of occurrences of word w in cluster c.

3. The short text topic modeling method based on dynamic clustering and word embedding enhancement as described in claim 1, characterized in that, The word embedding generation steps are as follows: Preprocess the text in the pseudo-document, including removing non-text characters, standardizing the format, and splitting the text into individual words; A word embedding model pre-trained on a large-scale corpus maps each word in the pseudo-document to a high-dimensional vector space, generating a high-dimensional vector representation of each word to form a word embedding matrix; Each word embedding vector in the word embedding matrix is ​​applied to each word in the pseudo-document, and the similarity between words is calculated to form a word co-occurrence matrix.

4. The short text topic modeling method based on dynamic clustering and word embedding enhancement as described in claim 3, characterized in that, The similarity between the words is calculated using cosine similarity, and the formula is: Where, sim(v) u ,v o ) represents the similarity between word u and word o, v u Let v be the word embedding vector of word u. o Let be the word embedding vector of word o.

5. The short text topic modeling method based on dynamic clustering and word embedding enhancement as described in claim 1, characterized in that, The pseudo-document topic modeling steps are as follows: The topic distribution of pseudo-documents is represented using the Dirichlet distribution: θ d ~Dir(α), where, θ d Let α represent the topic distribution of pseudo-document d, where α is the parameter of the Dirichlet distribution; For each word n ​​in pseudo-document d, according to the topic distribution θ of the pseudo-document... d Choose a topic for each word z dn ~θ d , z dn The topic of word n ​​in pseudo-document d; The word distribution for each topic is generated using the Dirichlet distribution, denoted as: φ k ~Dir(β), where φ k Let β represent the word distribution of topic k, where β is the parameter of the Dirichlet distribution; The topic z is selected based on each word n ​​in pseudo-document d. dn Substituting the word distribution into the word distribution yields the word distribution for this topic. This then generates the distribution of each word n ​​in pseudo-document d. Finally, a topic model is built using pseudo-documents enhanced with word embeddings, and represented using a likelihood function: in, Let w represent the likelihood function of the entire topic model, z represent the topic, l represent the pseudo-document set, and α and β are hyperparameters.

6. The short text topic modeling method based on dynamic clustering and word embedding enhancement as described in claim 5, characterized in that, The inferred topic distribution is specifically as follows: The Gibbs sampling method is used to randomly initialize a topic for each word in the pseudo-document; The conditional probability of each word being assigned to each topic is calculated using the following formula: Wherein, p(z) dn =k|w dn ,z (-dn) Let z(α,β) be the conditional probability that word n ​​in pseudo-document d is assigned to topic k. dn For the topic of word n ​​in pseudo-document d, w dn The distribution of word n ​​in pseudo-document d; Excluding word n, the number of words in pseudo-document d that are assigned to topic k; This represents the total number of words in pseudo-document d, excluding word n. Z represents the number of words belonging to topic k and being word w, excluding word n. (-dn) V represents the topic of all words in pseudo-document d except for word n, where V is the vocabulary and K is the number of topics in the topic model. The topic of each word is updated using a multinomial distribution based on the conditional probability of each word being assigned to each topic, and then fed into the topic model for learning. Continue to calculate the conditional probability for each word, update the topic and input it into the topic model for learning, until the predetermined number of iterations is reached or the model converges; Obtain the topic assignment results when the predetermined number of iterations or model convergence is reached, and obtain the topic distribution of the pseudo-document.

7. A short text topic modeling system based on dynamic clustering and word embedding enhancement, characterized in that, The system includes a data preprocessing module, a pseudo-document construction module, a word embedding generation module, a topic modeling module, a distribution inference module, and a subject prediction module; The data preprocessing module is used to collect short text data and perform text cleaning to obtain a short text stream. The pseudo-document construction module is used to cluster short text streams using the FastStream clustering method and construct pseudo-documents based on the clustering results. The steps are as follows: Let the short text stream be represented as Where, d t The short text arriving at time t; the parameters for initializing the FastStream clustering method; the parameters include the document set D, the vocabulary V, and the word frequency n for each document. wd and the word frequency n of each cluster wz The document collection is used to store documents assigned to clusters. Each short text in the short text stream is treated as a document and sequentially input into the document set, and the arrived document d is used. t Update word relationship graph G t = (V, E), where E is the set of edges representing word co-occurrence relations; edge weights The word x represents the document d. t The co-occurrence frequency f(x,d) t ),Right now if f(x,d t )≥∈, where ∈ is the word co-occurrence frequency threshold in the word relationship graph; The topic is discovered through a word relationship graph. Words with frequencies higher than a set frequency threshold are selected as seed nodes, and word weights are calculated based on their co-occurrence frequencies with other words, as shown in the following formula: Among them, c w The word weight of word w, For topic k m The number of words contained For topic k m The total number of occurrences of all words in the text, k m To represent the m-th topic; For each document d t Calculate the probability that document d will be assigned to an existing cluster z. t If the probability of being assigned to an existing cluster z is greater than the probability of being assigned to any other possible cluster, or if it is greater than the minimum acceptance probability of cluster z, then the document will be assigned to an existing cluster z. The document d t The probability of being assigned to an existing cluster z is calculated as follows: in, For document d t The assigned cluster z, Assign a set to the current topic for all documents; α is one of the hyperparameters of the Dirichlet distribution, used to control the prior probability of generating new clusters; β is another hyperparameter of the Dirichlet distribution, used to control the prior distribution of the probability of words in the vocabulary appearing in the topic; m z n is the number of documents in cluster z; wz n is the total number of occurrences of word w in cluster z, that is, the total number of times word w appears in all documents assigned to cluster z; z The total number of occurrences of all words in cluster z, where i and j represent the position or order of the words in the document or cluster; For document d that cannot be assigned to an existing cluster t Calculate the probability of creating a new cluster. If the probability of creating a new cluster is greater than the probability threshold, a new cluster will be created and the document will be assigned to the new cluster. The formula for calculating the probability of creating a cluster is: Among them, z new This is a new cluster; After clustering, pseudo-documents are generated from the clustering results. Each pseudo-document contains all the short texts in its respective cluster, forming a new document set D. ' ; The word embedding generation module is used to map words in pseudo-documents to a high-dimensional vector space using a word embedding model pre-trained on a large corpus to form a word embedding matrix and a word co-occurrence matrix; The topic modeling module is used to model the topic distribution and word distribution of pseudo-documents using the Dirichlet distribution, thereby obtaining a topic model. The distribution inference module is used to train the topic model on pseudo-documents using the Gibbs sampling method, and to update the topic assignment of each word in the pseudo-documents and the topic distribution parameters of the pseudo-documents until the topic model converges. The subject prediction module is used to acquire new short texts and generate their word embedding representations, which are then input into a trained topic model for topic inference to obtain the topic distribution.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the short text topic modeling method based on dynamic clustering and word embedding enhancement as described in any one of claims 1-6.

9. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the short text topic modeling method based on dynamic clustering and word embedding enhancement as described in any one of claims 1-6.

Images