Short text topic modeling method based on dynamic clustering and word embedding enhancement

Abstract

The invention discloses a short text topic modeling method based on dynamic clustering and word embedding enhancement. The short text topic modeling method comprises the following steps: firstly, collecting short text data to obtain a short text stream; clustering the short text streams by using a FastStream clustering method, and constructing a pseudo document according to a clustering result; then forming a word embedding matrix and a word co-occurrence matrix by utilizing a word embedding model pre-trained by a large corpus; then modeling topic distribution and word distribution of the pseudo document by using Dirichlet distribution to obtain a topic model; then training the topic model on the pseudo document by using a Gibbs sampling method, and updating topic distribution of each word in the pseudo document and topic distribution parameters of the pseudo document until the topic model is converged; and finally, obtaining a new short text to carry out topic inference to obtain topic distribution. According to the method, the FastStream clustering technology and the word embedding technology are combined, and the topic modeling performance of the short text data is effectively improved by creating the pseudo document view and enhancing word embedding.

Description

Technical Field

[0001] The present invention belongs to the technical field of text analysis, and in particular relates to a short text topic modeling method based on dynamic clustering and word embedding enhancement. Background Art

[0002] In the digital age, short text data such as social media messages, news headlines, and online comments have become one of the main ways of information exchange. Short texts, due to their limited content and rapid updates, provide rich information while also bringing challenges to traditional text analysis techniques; especially in topic modeling and cluster analysis, traditional methods often have difficulty capturing the deep semantic relationships of short texts, which affects the effective organization and summary of data. Therefore, researching and developing new technologies that can effectively process short text data is of great significance to support applications such as trend analysis and user preference mining.

[0003] In the prior art, short text clustering usually uses traditional clustering algorithms, such as K-means or hierarchical clustering. These methods perform well when processing small-scale, static data sets. However, when faced with real-time data streams and large-scale data processing, their efficiency is significantly reduced, and they lack flexibility and scalability in dynamic environments. In addition, although pre-trained word embedding models, such as Word2Vec and BERT, have made significant progress in capturing lexical semantics, there are still certain limitations in directly applying these models to short text clustering; these models are insufficient in capturing the complex semantic connections between short texts, and it is difficult to effectively improve the quality of clustering and semantic consistency.

[0004] Specifically, existing methods face the following main problems when dealing with short text analysis:

[0005] 1. Low processing efficiency: Traditional methods require pre-setting the number of clusters and performing multiple iterations on the entire data set, which is extremely unfavorable for large-scale data stream scenarios that require real-time processing;

[0006] 2. Lack of adaptability and flexibility: Existing methods are difficult to adjust and update models in a timely manner when dealing with dynamic changes and real-time data streams, which affects the real-time and accuracy of clustering results;

[0007] 3. Clustering quality issues: Due to the high-dimensional sparsity and semantic complexity of short texts, the clustering results of traditional methods often lack accuracy and semantic consistency.

[0008] In order to meet the above challenges, it is urgent to develop a new technology that can effectively deal with the dynamic characteristics of short texts, improve processing efficiency, and significantly optimize clustering quality. This paper proposes a short text topic modeling method based on dynamic clustering and word embedding enhancement, aiming to address the limitations of existing technologies and provide more accurate and efficient technical support for applications such as real-time data analysis, trend prediction, and user preference mining. Summary of the invention

[0009] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a short text topic modeling method based on dynamic clustering and word embedding enhancement, which combines FastStream clustering technology with word embedding technology to effectively improve the topic modeling performance of short text data by creating pseudo-document views and enhancing word embedding.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] The first object is to provide a short text topic modeling method based on dynamic clustering and word embedding enhancement, comprising the following steps:

[0012] Step 1: Data preprocessing: collect short text data and perform text cleaning to obtain short text streams;

[0013] Step 2: Pseudo-document construction: Use the FastStream clustering method to cluster the short text stream and construct a pseudo-document based on the clustering results;

[0014] Step 3: Word embedding generation: Use the word embedding model pre-trained by a large corpus to map the words in the pseudo document into a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix;

[0015] Step 4: Pseudo-document topic modeling: Dirichlet distribution is used to model the topic distribution and word distribution of pseudo-documents to obtain a topic model;

[0016] Step 5, topic distribution inference: Use the Gibbs sampling method to train the topic model on the pseudo document, update the topic assignment of each word in the pseudo document and the topic distribution parameters of the pseudo document until the topic model converges;

[0017] Step 6: Short text topic prediction: Get a new short text and generate its word embedding representation, input it into the trained topic model for topic inference, and obtain the topic distribution.

[0018] As a preferred technical solution, the pseudo document construction steps are:

[0019] Assume that the short text stream is represented as Among them, d tRepresents the short text arriving at time t; initializes the parameters of the FastStream clustering method; the parameters include the document set D, the vocabulary V, the word frequency n of each document wd And the word frequency n of each cluster wz ; The document collection is used to store documents assigned to clusters;

[0020] Treat each short text in the short text stream as a document and input it into the document collection in sequence, and use the document d t Update word relationship graph G t =(V,E), where E is the edge set of word co-occurrence relations; edge weight e x,t Represents the co-occurrence frequency of word x and document t, i.e., e x,t ∈E,if f(x,t)≥∈,∈ is the word co-occurrence frequency threshold in the word relationship graph;

[0021] The topic is discovered through the word relationship graph. Words with a frequency higher than the set word frequency threshold are selected as seed nodes, and the word weight is calculated based on the co-occurrence frequency with other words. The formula is as follows:

[0022]

[0023] Among them, c w is the word weight of word w, For the theme m The number of words contained in For the theme m The total number of occurrences of all words in k m To represent the mth topic;

[0024] For each document d t , calculate the probability of being assigned to the existing cluster z, when document d t When the probability of being assigned to the existing cluster z exceeds the probability of all other possible assignments to the existing clusters, or exceeds the minimum acceptance probability of cluster z, the document is assigned to the existing cluster z;

[0025] The document d t The probability of being assigned to an existing cluster z is calculated as:

[0026]

[0027] Among them, z d For document d t The cluster z to which it is assigned, Assign a set of current topics to all documents; α is one of the hyperparameters of the Dirichlet distribution, which is used to control the prior probability of generating new clusters; β is another hyperparameter of the Dirichlet distribution, which is used to control the prior distribution of the probability of words in the vocabulary appearing in the topic; m z is the number of documents in cluster z; n wz is the total number of occurrences of word w in cluster z, that is, the total number of occurrences of word w in all documents assigned to cluster z; n z is the total number of occurrences of all words in cluster z, i and j represent the position or order of the word in the document or cluster;

[0028] For documents that cannot be assigned to existing clusters, t , calculate the probability of creating a new cluster. If the probability of creating a new cluster is greater than the new probability threshold, a new cluster will be created and the documents will be assigned to the new cluster.

[0029] The calculation formula for the probability of creating a cluster is:

[0030]

[0031] Among them, z new is the new cluster;

[0032] After clustering, pseudo documents are generated based on the clustering results. Each pseudo document contains all the short texts in the cluster to which it belongs, forming a new document set D'.

[0033] As a preferred technical solution, when creating a new cluster, the cluster activity weight is introduced and the weight decay is calculated by Newton's cooling law:

[0034]

[0035] Among them, l z is the cluster activity weight of the z-th cluster, is the initial active weight when the zth cluster is created, κ is a constant used to control the decay rate of the active weight, and t c is the current time point, u z The time when the zth cluster was last updated;

[0036] For clusters with a cluster activity weight of 0, calculate the probability of merging with other clusters and decide whether to merge or exist separately according to the following formula:

[0037]

[0038] Among them, z c represents the cth cluster being processed, z represents the cluster that can be merged, and z selfrepresents the independent clusters that are not merged, c represents the cluster currently being processed, and n wc represents the total number of occurrences of word w in cluster c.

[0039] As a preferred technical solution, the word embedding generation step is:

[0040] Preprocess the text in the pseudo-documents, including removing non-text characters, unifying the format, and segmenting the text into individual words;

[0041] Use a word embedding model pre-trained with a large-scale corpus to map each word in the pseudo document into a high-dimensional vector space, generating a high-dimensional vector representation of each word to form a word embedding matrix;

[0042] Each word embedding vector in the word embedding matrix is ​​applied to each word in the pseudo document, and the similarity between words is calculated to form a word co-occurrence matrix.

[0043] As a preferred technical solution, the similarity between the words is calculated using cosine similarity, and the formula is:

[0044]

[0045] Among them, sim(v i ,v j ) is the similarity between word i and word j, v i is the word embedding vector of word i, v j is the word embedding vector of word j.

[0046] As a preferred technical solution, the pseudo-document topic modeling step is:

[0047] Use Dirichlet distribution to represent the topic distribution of pseudo documents: θ d ~Dir(α), where θ d represents the topic distribution of pseudo document d, α is the parameter of Dirichlet distribution;

[0048] For each word n ​​in the pseudo document d, according to the topic distribution θ of the pseudo document d Choose a topic for each word dn ~θ d , z dn is the topic of word n ​​in pseudo document d;

[0049] The Dirichlet distribution is used to generate the word distribution of each topic, expressed as: k ~Dir(β), where φ k represents the word distribution of topic k, β is the parameter of Dirichlet distribution;

[0050] Select topic z for each word n ​​in pseudo document d dnSubstitute into the word distribution to get the word distribution of the topic Then generate the distribution of each word n ​​in the pseudo document d

[0051] Finally, the topic model is established using pseudo documents enhanced by word embedding, and represented by the likelihood function:

[0052]

[0053] in, represents the likelihood function of the entire topic model, w represents the word embedding vector, z represents the topic, l represents the pseudo document set, and α and β are hyperparameters.

[0054] As a preferred technical solution, the inferred topic distribution is specifically:

[0055] Use Gibbs sampling to randomly initialize a topic for each word in the pseudo document;

[0056] Calculate the conditional probability of each word being assigned to each topic, as follows:

[0057]

[0058] Among them, p(z dn =k|w dn ,z (-dn) ,α,β) is the conditional probability that word n ​​in pseudo document d is assigned to topic k, z dn is the topic of word n ​​in pseudo document d, w dn is the distribution of word n ​​in pseudo document d; is the number of words assigned to topic k in pseudo document d without including word n; is the total number of all words in pseudo document d without including word n, is the number of words belonging to topic k and being word w without including word n, z (-dn) represents the topics of all words in pseudo document d except word n, V is the vocabulary, and K is the number of topics in the topic model;

[0059] Use the conditional probability of each word being assigned to each topic to update the topic of each word through multinomial distribution and input it into the topic model for learning;

[0060] Continue to calculate the conditional probability for each word, update the topic and input it into the topic model for learning until the predetermined number of iterations is reached or the model converges;

[0061] The topic assignment result is obtained when a predetermined number of iterations is reached or the model converges, and the topic distribution of the pseudo document is obtained.

[0062] The second object is to provide a short text topic modeling system based on dynamic clustering and word embedding enhancement, including a data preprocessing module, a pseudo document construction module, a word embedding generation module, a topic modeling module, a distribution inference module and a subject prediction module;

[0063] The data preprocessing module is used to collect short text data and perform text cleaning to obtain a short text stream;

[0064] The pseudo document construction module is used to cluster the short text stream using the FastStream clustering method and construct a pseudo document according to the clustering result;

[0065] The word embedding generation module is used to map the words in the pseudo document into a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix using a word embedding model pre-trained by a large corpus;

[0066] The topic modeling module is used to model the topic distribution and word distribution of the pseudo document using Dirichlet distribution to obtain a topic model;

[0067] The distribution inference module is used to train the topic model using the Gibbs sampling method on the pseudo document, and update the topic assignment of each word in the pseudo document and the topic distribution parameters of the pseudo document until the topic model converges;

[0068] The subject prediction module is used to obtain new short texts and generate their word embedding representations, which are input into the trained topic model for topic inference to obtain topic distribution.

[0069] The third object is to provide an electronic device, comprising:

[0070] at least one processor; and a memory in communication with the at least one processor; wherein,

[0071] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned short text topic modeling method based on dynamic clustering and word embedding enhancement.

[0072] The fourth purpose is to provide a computer-readable storage medium storing a program, characterized in that when the program is executed by a processor, the above-mentioned short text topic modeling method based on dynamic clustering and word embedding enhancement is implemented.

[0073] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0074] 1. Real-time processing capability: The present invention utilizes FastStream clustering technology to combine real-time processing capability with the analysis of large-scale short text data, breaking through the bottleneck of traditional methods. FastStream clustering technology can quickly adjust clustering results and update topic models as data continues to arrive through incremental updates, avoiding the high-cost operation of global recalculation. This innovation makes the present invention particularly suitable for scenarios that require immediate response, such as social media monitoring, online news analysis, and real-time market trend tracking, which significantly improves the processing efficiency and response speed of the system. At the same time, this real-time processing capability has obvious competitive advantages in responding to emergencies and rapidly changing information environments.

[0075] 2. Dynamic adaptability: The present invention can flexibly respond to continuous changes in data streams, especially effectively solve the "concept drift" problem by introducing a dynamic adjustment mechanism. In practical applications, the characteristics and topic distribution of data streams may change significantly over time. The present invention enables the model to automatically adapt to these changes without manual intervention by updating pseudo-documents and cluster centers in real time. Specifically, the model updates parameters through incremental learning, allowing it to capture new data patterns and trend changes in a timely manner. This dynamic adaptability ensures the stability and accuracy of the model in long-term use, and can maintain efficient operation in complex and changing environments. It is suitable for dynamic information-intensive fields such as real-time market analysis and personalized recommendation systems.

[0076] 3. Effective solution to the problem of data sparsity: In response to the high-dimensional sparsity problem of short text data, the present invention introduces pseudo-document technology to improve the information density of data by aggregating semantically similar short texts into pseudo-documents. Pseudo-document technology not only effectively reduces the sparsity of data, but also improves the training effect of the model by performing topic modeling at a richer semantic level. Specifically, pseudo-documents enhance the integrity of semantic representation by aggregating similar texts, enabling topic models to more accurately identify and capture potential topics in short texts. This method greatly improves the semantic consistency and representativeness of generated topics, and solves the shortcomings of traditional methods in short text processing.

[0077] 4. Reduce invalid and redundant clusters: The present invention optimizes the cluster management process by introducing cluster activity weights and a weight decay mechanism based on Newton's cooling law. The cluster activity weights are used to dynamically evaluate the activity of each cluster cluster to ensure that system resources are concentrated on meaningful clusters, while the weight decay mechanism ensures that cluster clusters that have been inactive or redundant for a long time can be removed in a timely manner. This optimization process not only reduces invalid and redundant clusters, but also effectively improves the overall operating efficiency and resource utilization of the system. By reducing unnecessary computational burdens, the present invention significantly improves performance in large-scale data processing and adapts to application environments with limited resources and efficient processing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 The figure is an overall flow chart of a short text topic modeling method based on dynamic clustering and word embedding enhancement in an embodiment of the present invention.

[0080] Figure 2 It is a structural diagram of a short text topic modeling system based on dynamic clustering and word embedding enhancement in an embodiment of the present invention.

[0081] Figure 3 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0083] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0084] like Figure 1 As shown, the short text topic modeling method based on dynamic clustering and word embedding enhancement in this embodiment includes the following steps:

[0085] Step 1: Data preprocessing: Collect short text data and perform text cleaning to obtain a short text stream.

[0086] Step 2: Pseudo-document construction: Use the FastStream clustering method to cluster the short text stream and construct a pseudo-document based on the clustering results.

[0087] Specifically, the pseudo-document construction steps are:

[0088] 2.1. Let the short text stream be represented as Among them, d t Represents the short text arriving at time t; initializes the parameters of the FastStream clustering method, including the document collection D (used to store documents assigned to clusters), the vocabulary V, and the word frequency n of each document wd And the word frequency n of each cluster wz wait.

[0089] 2.2. Treat each short text in the short text stream as a document and input it into the document collection in sequence, and use the arrived document d t Update word relationship graph G t =(V,E), where E is the edge set of word co-occurrence relations; edge weight e x,t Represents the co-occurrence frequency of word x and in the document f(x,t), that is, e x,t ∈E,if f(x,t)≥∈,∈ is the word co-occurrence frequency threshold in the word relationship graph, which is used to determine whether the word co-occurrence relationship is valid.

[0090] 2.3. Discover the topic words through the word relationship graph, select the words with a frequency higher than the set word frequency threshold as seed nodes, and calculate the word weight according to its co-occurrence frequency with other words. The formula is as follows:

[0091]

[0092] Among them, c w is the word weight of word w, For the theme m The number of words contained in For the theme m The total number of occurrences of all words in k m To represent the mth topic; the core of discovering topic words through the word relationship graph is to use the co-occurrence relationship between words, combined with the calculation of word frequency and weight, to identify those words that appear repeatedly in multiple documents and have a strong co-occurrence relationship as topic words.

[0093] 2.4. For each document d t , calculate the probability of being assigned to the existing cluster z, when document d t When the probability of being assigned to the existing cluster z exceeds the probability of all other possible assignments to the existing clusters, or exceeds the minimum acceptance probability of cluster z, the document is assigned to the existing cluster z.

[0094] For each document d t , the probability of being assigned to the existing cluster z is:

[0095]

[0096] Among them, z d For document d t The cluster z to which it is assigned, Assign a set of current topics to all documents; α is one of the hyperparameters of the Dirichlet distribution, which is used to control the prior probability of generating new clusters; β is another hyperparameter of the Dirichlet distribution, which is used to control the prior distribution of the probability of words in the vocabulary appearing in the topic; m z is the number of documents in cluster z; n wz is the total number of occurrences of word w in cluster z, that is, the total number of occurrences of word w in all documents assigned to cluster z; n z is the total number of occurrences of all words in cluster z, and i and j represent the position or order of the words in the document or cluster.

[0097] 2.5. For documents that cannot be assigned to existing clusters, t , calculate the probability of creating a new cluster. If the probability of creating a new cluster is greater than the new probability threshold, a new cluster will be created and the documents will be assigned to the new cluster.

[0098] For documents that cannot be assigned to existing clusters, t , the probability of creating a new cluster is:

[0099]

[0100] Among them, z new is the new cluster;

[0101] 2.6. After clustering, pseudo documents are generated based on the clustering results. Each pseudo document contains all the short texts in the cluster to which it belongs, forming a new document set D'.

[0102] Furthermore, in order to avoid generating too many microclusters, when creating a new cluster, the present application introduces cluster activity weights and calculates weight decay by Newton's cooling law:

[0103]

[0104] Among them, l z is the cluster activity weight of the z-th cluster, is the initial active weight when the zth cluster is created, κ is a constant used to control the decay rate of the active weight, and t c is the current time point, u z The time when the zth cluster was last updated;

[0105] For clusters with a cluster activity weight of 0, calculate the probability of merging with other clusters and decide whether to merge or exist separately according to the following formula:

[0106]

[0107] Among them, z c represents the cth cluster being processed, z represents the cluster that can be merged, and z self represents the independent clusters that are not merged, c represents the cluster currently being processed, and n wc represents the total number of occurrences of word w in cluster c. Through a series of rules, documents in inappropriate clusters are identified and reallocated to improve model accuracy and topic concentration.

[0108] Through the above process, FastStream can process short text streams in real time, dynamically generate and update pseudo documents, and achieve efficient short text stream clustering.

[0109] Step 3: Word embedding generation: Use the word embedding model pre-trained by a large corpus to map the words in the pseudo-document into a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix.

[0110] Furthermore, word embedding is used to enhance pseudo-document co-occurrence information, thereby improving the effect of short text topic modeling. The steps for generating word embedding are:

[0111] 3.1. Preprocess the text in the pseudo-document, including removing non-text characters, unifying the format, and segmenting the text into individual words;

[0112] 3.2. Use a word embedding model pre-trained on a large-scale corpus (such as Word2Vec or GloVe) to map each word in the pseudo document into a high-dimensional vector space, generating a high-dimensional vector representation of each word to form a word embedding matrix;

[0113] 3.3. Apply each word embedding vector in the word embedding matrix to each word in the pseudo document, and calculate the similarity between words to form a word co-occurrence matrix. The similarity between words is calculated using cosine similarity, and the formula is:

[0114]

[0115] Among them, sim(v i ,v j ) is the similarity between word i and word j, v i is the word embedding vector of word i, v j is the word embedding vector of word j.

[0116] In this embodiment, the word embedding model adopts the Word2Vec model, and the pre-training process is expressed as follows:

[0117]

[0118] Among them, wO is the target word, w I is the context word, v O and v I are the word embedding vectors of the target word and the context word respectively, ω′ is any word in the vocabulary V, and v ω′ is the word embedding vector of word ω′ in the vocabulary.

[0119] Step 3 improves the effect of topic modeling by implicitly adding word co-occurrence information in pseudo documents. Through these steps, word embedding is effectively applied in pseudo document topic modeling, which solves the problem of short text data sparsity and improves the performance and effect of short text topic modeling.

[0120] Step 4: Pseudo-document topic modeling: Dirichlet distribution is used to model the topic distribution and word distribution of the pseudo-document to obtain a topic model.

[0121] Specifically, the steps of pseudo-document topic modeling are:

[0122] 4.1. Obtain the topic distribution of pseudo documents, represented by Dirichlet distribution: θ d ~Dir(α), where θ d represents the topic distribution of pseudo document d, α is the parameter of Dirichlet distribution;

[0123] 4.2. Use Dirichlet distribution to represent the topic distribution of pseudo documents: θ d ~Dir(α), where θ d represents the topic distribution of pseudo document d, α is the parameter of Dirichlet distribution;

[0124] 4.3. For each word n ​​in the pseudo document d, according to the topic distribution θ of the pseudo document d Choose a topic for each word dn ~θ d , z dn is the topic of word n ​​in pseudo document d;

[0125] 4.4. Use Dirichlet distribution to generate the word distribution of each topic, expressed as: φ k ~Dir(β), where φ k represents the word distribution of topic k, β is the parameter of Dirichlet distribution;

[0126] 4.5. Select topic z for each word n ​​in pseudo document d dn Substitute into the word distribution to get the word distribution of the topic Then generate the distribution of each word n ​​in the pseudo document d

[0127] 4.6. Finally, the topic model is established using pseudo documents enhanced by word embedding, and represented by likelihood function:

[0128]

[0129] in, represents the likelihood function of the entire topic model, w represents the word embedding vector, z represents the topic, l represents the pseudo document set, and α and β are hyperparameters.

[0130] This step forms a complete topic modeling process by embedding the words in each pseudo document and using Dirichlet distribution to model the topic distribution of pseudo documents and the word distribution of each topic. This process enhances the semantic analysis ability of short text data by introducing pre-trained word embedding and pseudo document topic models, effectively solves the data sparsity problem, and improves the effect of topic modeling.

[0131] Step 5, topic distribution inference: Use the Gibbs sampling method on the pseudo document to train the topic model, update the topic assignment of each word in the pseudo document and the topic distribution parameters of the pseudo document until the topic model converges.

[0132] 5.1. Use the Gibbs sampling method to randomly initialize a topic for each word in the pseudo document;

[0133] 5.2. Calculate the conditional probability of each word being assigned to each topic. The calculation formula is:

[0134]

[0135] Among them, p(z dn =k|w dn ,z (-dn) ,α,β) is the conditional probability that word n ​​in pseudo document d is assigned to topic k, z dn is the topic of word n ​​in pseudo document d, w dn is the distribution of word n ​​in pseudo document d; is the number of words assigned to topic k in pseudo document d without including word n; is the total number of all words in pseudo document d without including word n, is the number of words belonging to topic k and being word w without including word n, z (-dn) represents the topics of all words in pseudo document d except word n, V is the vocabulary, and K is the number of topics in the topic model;

[0136] 5.3. Use the conditional probability of each word being assigned to each topic to update the topic of each word through multinomial distribution and input it into the topic model for learning;

[0137] 5.4. Continue to calculate the conditional probability for each word, update the topic and input it into the topic model for learning, that is, iterate steps 5.2-5.4 until the predetermined number of iterations is reached or the model converges;

[0138] 5.5. Obtain the topic assignment result when the predetermined number of iterations is reached or the model converges, and obtain the topic distribution of the pseudo document.

[0139] The iteration of the Gibbs sampling process enables the model to gradually optimize parameter estimation and eventually reach a stable state. Through this method of inference and parameter estimation, the model can accurately capture the topic distribution in pseudo documents, thereby achieving efficient topic modeling for short text data; this process combines pre-trained word embedding vectors and parameter estimation of Dirichlet distribution, which enhances the model's capture and understanding of semantic information and significantly improves the effect of topic modeling.

[0140] Step 6: Short text topic prediction: Get a new short text and generate its word embedding representation, input it into the trained topic model for subject inference, and obtain the topic distribution.

[0141] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0142] Based on the same idea as the short text topic modeling method based on dynamic clustering and word embedding enhancement in the above-mentioned embodiment, the present invention also provides a short text topic modeling system based on dynamic clustering and word embedding enhancement, which can be used to execute the above-mentioned short text topic modeling method based on dynamic clustering and word embedding enhancement. For ease of explanation, in the structural schematic diagram of the embodiment of the short text topic modeling system based on dynamic clustering and word embedding enhancement, only the parts related to the embodiment of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than the illustrated structure, or combine certain components, or arrange the components differently.

[0143] like Figure 2 As shown, another embodiment of the present invention provides a short text topic modeling system based on dynamic clustering and word embedding enhancement, including a data preprocessing module, a pseudo document construction module, a word embedding generation module, a topic modeling module, a distribution inference module and a topic prediction module;

[0144] Among them, the data preprocessing module is used to collect short text data and perform text cleaning to obtain a short text stream;

[0145] The pseudo-document construction module is used to cluster short text streams using the FastStream clustering method and construct pseudo-documents based on the clustering results;

[0146] The word embedding generation module is used to map the words in the pseudo-document into a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix using a word embedding model pre-trained on a large corpus;

[0147] The topic modeling module is used to model the topic distribution and word distribution of pseudo documents using Dirichlet distribution to obtain a topic model;

[0148] The distribution inference module is used to train the topic model using the Gibbs sampling method on the pseudo document, and update the topic assignment of each word in the pseudo document and the topic distribution parameters of the pseudo document until the topic model converges;

[0149] The topic prediction module is used to obtain new short texts and generate their word embedding representations, which are input into the trained topic model for subject inference to obtain the topic distribution.

[0150] It should be noted that the short text topic modeling system based on dynamic clustering and word embedding enhancement of the present invention corresponds one to one with the short text topic modeling method based on dynamic clustering and word embedding enhancement of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the short text topic modeling method based on dynamic clustering and word embedding enhancement are applicable to the embodiment of the short text topic modeling system based on dynamic clustering and word embedding enhancement. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0151] In addition, in the implementation of the short text topic modeling system based on dynamic clustering and word embedding enhancement in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the short text topic modeling system based on dynamic clustering and word embedding enhancement is divided into different program modules to complete all or part of the functions described above.

[0152] See also Figure 3 In one embodiment, an electronic device for implementing a short text topic modeling method based on dynamic clustering and word embedding enhancement is provided. The electronic device may include a first processor, a first memory and a bus, and may also include a computer program stored in the first memory and executable on the first processor, such as a short text topic modeling program.

[0153] Among them, the first memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory can be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the first memory can also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the first memory can also include both an internal storage unit of the electronic device and an external storage device. The first memory can be used not only to store application software and various types of data installed in the electronic device, such as the code of a short text topic modeling program, but also to temporarily store data that has been output or is to be output.

[0154] In some embodiments, the first processor may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules (such as short text topic modeling programs, etc.) stored in the first memory, and calling data stored in the first memory.

[0155] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0156] The short text topic modeling program stored in the first memory of the electronic device is a combination of multiple instructions, and when executed in the first processor, can achieve:

[0157] Step 1: Data preprocessing: collect short text data and perform text cleaning to obtain short text streams;

[0158] Step 2: Pseudo-document construction: Use the FastStream clustering method to cluster the short text stream and construct a pseudo-document based on the clustering results;

[0159] Step 3: Word embedding generation: Use the word embedding model pre-trained by a large corpus to map the words in the pseudo document into a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix;

[0160] Step 4: Pseudo-document topic modeling: Dirichlet distribution is used to model the topic distribution and word distribution of pseudo-documents to obtain a topic model;

[0161] Step 5, topic distribution inference: Use the Gibbs sampling method to train the topic model on the pseudo document, update the topic assignment of each word in the pseudo document and the topic distribution parameters of the pseudo document until the topic model converges;

[0162] Step 6: Short text topic prediction: Get a new short text and generate its word embedding representation, input it into the trained topic model for topic inference, and obtain the topic distribution.

[0163] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0164] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0165] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A short text topic modeling method based on dynamic clustering and word embedding enhancement, characterized in that: The steps include: Step 1: Data preprocessing: collect short text data and perform text cleaning to obtain short text streams; Step 2: Pseudo-document construction: Use the FastStream clustering method to cluster the short text stream and construct a pseudo-document based on the clustering results; Step 3: Word embedding generation: Use the word embedding model pre-trained by a large corpus to map the words in the pseudo document into a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix; Step 4: Pseudo-document topic modeling: Dirichlet distribution is used to model the topic distribution and word distribution of pseudo-documents to obtain a topic model; Step 5, topic distribution inference: Use the Gibbs sampling method to train the topic model on the pseudo document, update the topic assignment of each word in the pseudo document and the topic distribution parameters of the pseudo document until the topic model converges; Step 6: Short text topic prediction: Get a new short text and generate its word embedding representation, input it into the trained topic model for topic inference, and obtain the topic distribution.

2. The short text topic modeling method based on dynamic clustering and word embedding enhancement as claimed in claim 1, characterized in that: The pseudo document construction steps are: Assume that the short text stream is represented as Among them, d t Represents the short text arriving at time t; initializes the parameters of the FastStream clustering method; the parameters include the document set D, the vocabulary V, the word frequency n of each document wd And the word frequency n of each cluster wz ; The document collection is used to store documents assigned to clusters; Treat each short text in the short text stream as a document and input it into the document collection in sequence, and use the document d t Update word relationship graph G t =(V,E), where E is the edge set of word co-occurrence relations; edge weight e x,t Represents the co-occurrence frequency of word x and document t, i.e., e x,t ∈E,if f(x,t)≥∈,∈ is the word co-occurrence frequency threshold in the word relationship graph; The topic is discovered through the word relationship graph. Words with a frequency higher than the set word frequency threshold are selected as seed nodes, and the word weight is calculated based on the co-occurrence frequency with other words. The formula is as follows: Among them, c w is the word weight of word w, For the theme m The number of words contained in For the theme m The total number of occurrences of all words in k m To represent the mth topic; For each document d t , calculate the probability of being assigned to the existing cluster z, when document d t When the probability of being assigned to the existing cluster z exceeds the probability of all other possible assignments to the existing clusters, or exceeds the minimum acceptance probability of cluster z, the document is assigned to the existing cluster z; The document d t The probability of being assigned to an existing cluster z is calculated as: Among them, z d For document d t The cluster z to which it is assigned, Assign a set of current topics to all documents; α is one of the hyperparameters of the Dirichlet distribution, which is used to control the prior probability of generating new clusters; β is another hyperparameter of the Dirichlet distribution, which is used to control the prior distribution of the probability of words in the vocabulary appearing in the topic; m z is the number of documents in cluster z; n wz is the total number of occurrences of word w in cluster z, that is, the total number of occurrences of word w in all documents assigned to cluster z; n z is the total number of occurrences of all words in cluster z, i and j represent the position or order of the word in the document or cluster; For documents that cannot be assigned to existing clusters, t , calculate the probability of creating a new cluster. If the probability of creating a new cluster is greater than the new probability threshold, a new cluster will be created and the documents will be assigned to the new cluster. The calculation formula for the probability of creating a cluster is: Among them, z new is the new cluster; After clustering, pseudo documents are generated based on the clustering results. Each pseudo document contains all the short texts in the cluster to which it belongs, forming a new document set D'.

3. The short text topic modeling method based on dynamic clustering and word embedding enhancement as described in claim 2, characterized in that: When creating a new cluster, the cluster activity weight is introduced and the weight decay is calculated by Newton's cooling law: Among them, l z is the cluster activity weight of the z-th cluster, is the initial active weight when the zth cluster is created, κ is a constant used to control the decay rate of the active weight, and t c is the current time point, u z The time when the zth cluster was last updated; For clusters with a cluster activity weight of 0, calculate the probability of merging with other clusters and decide whether to merge or exist separately according to the following formula: Among them, z c represents the cth cluster being processed, z represents the cluster that can be merged, and z self represents the independent clusters that are not merged, c represents the cluster currently being processed, and n wc represents the total number of occurrences of word w in cluster c.

4. The short text topic modeling method based on dynamic clustering and word embedding enhancement as claimed in claim 1, characterized in that: The word embedding generation steps are: Preprocess the text in the pseudo-documents, including removing non-text characters, unifying the format, and segmenting the text into individual words; Use a word embedding model pre-trained with a large-scale corpus to map each word in the pseudo document into a high-dimensional vector space, generating a high-dimensional vector representation of each word to form a word embedding matrix; Each word embedding vector in the word embedding matrix is ​​applied to each word in the pseudo document, and the similarity between words is calculated to form a word co-occurrence matrix.

5. The short text topic modeling method based on dynamic clustering and word embedding enhancement as described in claim 4, characterized in that: The similarity between the words is calculated using cosine similarity, and the formula is: Among them, sim(v i ,v j ) is the similarity between word i and word j, v i is the word embedding vector of word i, v j is the word embedding vector of word j.

6. The short text topic modeling method based on dynamic clustering and word embedding enhancement as claimed in claim 2, characterized in that: The pseudo-document topic modeling steps are: Use Dirichlet distribution to represent the topic distribution of pseudo documents: θ d ~Dir(α), where θ d represents the topic distribution of pseudo document d, α is the parameter of Dirichlet distribution; For each word n ​​in the pseudo document d, according to the topic distribution θ of the pseudo document d Choose a topic for each word dn ~θ d , z dn is the topic of word n ​​in pseudo document d; The Dirichlet distribution is used to generate the word distribution of each topic, expressed as: k ~Dir(β), where φ k represents the word distribution of topic k, β is the parameter of Dirichlet distribution; Select topic z for each word n ​​in pseudo document d dn Substitute into the word distribution to get the word distribution of the topic Then generate the distribution of each word n ​​in the pseudo document d Finally, the topic model is established using pseudo documents enhanced by word embedding, and represented by the likelihood function: in, represents the likelihood function of the entire topic model, w represents the word embedding vector, z represents the topic, l represents the pseudo document set, and α and β are hyperparameters.

7. The short text topic modeling method based on dynamic clustering and word embedding enhancement as claimed in claim 6, characterized in that: The inferred topic distribution is specifically: Use Gibbs sampling to randomly initialize a topic for each word in the pseudo document; Calculate the conditional probability of each word being assigned to each topic, as follows: Among them, p(z dn =k|w dn ,z (-dn) ,α,β) is the conditional probability that word n ​​in pseudo document d is assigned to topic k, z dn is the topic of word n ​​in pseudo document d, w dn is the distribution of word n ​​in pseudo document d; is the number of words assigned to topic k in pseudo document d without including word n; is the total number of all words in pseudo document d without including word n, is the number of words belonging to topic k and being word w without including word n, z (-dn) represents the topics of all words in pseudo document d except word n, V is the vocabulary, and K is the number of topics in the topic model; Use the conditional probability of each word being assigned to each topic to update the topic of each word through multinomial distribution and input it into the topic model for learning; Continue to calculate the conditional probability for each word, update the topic and input it into the topic model for learning until the predetermined number of iterations is reached or the model converges; The topic assignment result is obtained when a predetermined number of iterations is reached or the model converges, and the topic distribution of the pseudo document is obtained.

8. A short text topic modeling system based on dynamic clustering and word embedding enhancement, characterized by: The system includes a data preprocessing module, a pseudo document construction module, a word embedding generation module, a topic modeling module, a distribution inference module and a subject prediction module; The data preprocessing module is used to collect short text data and perform text cleaning to obtain a short text stream; The pseudo document construction module is used to cluster the short text stream using the FastStream clustering method and construct a pseudo document according to the clustering result; The word embedding generation module is used to map the words in the pseudo document into a high-dimensional vector space to form a word embedding matrix and a word co-occurrence matrix using a word embedding model pre-trained by a large corpus; The topic modeling module is used to model the topic distribution and word distribution of the pseudo document using Dirichlet distribution to obtain a topic model; The distribution inference module is used to train the topic model using the Gibbs sampling method on the pseudo document, and update the topic assignment of each word in the pseudo document and the topic distribution parameters of the pseudo document until the topic model converges; The subject prediction module is used to obtain new short texts and generate their word embedding representations, which are input into the trained topic model for topic inference to obtain topic distribution.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory in communication with the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the short text topic modeling method based on dynamic clustering and word embedding enhancement as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the short text topic modeling method based on dynamic clustering and word embedding enhancement described in any one of claims 1 to 7 is implemented.

Images