Adaptive lda topic model training system based on public opinion real-time data stream

Abstract

The application provides a self-adaptive LDA topic model training system based on public opinion real-time data flow, comprising a data gathering module, a data preprocessing module, a self-adaptive LDA model training module and an incremental LDA model fusion module; the data gathering module is used for extracting and converting and loading the structured and semi-structured data and inputting into a distributed message bus kafka; the data preprocessing module is used for preprocessing the data in the message bus kafka and finally forming a weighted word vector; the self-adaptive LDA model training module is used for training to obtain an LDA model result and merging the training result; the incremental LDA model fusion module is used for fusion training to generate a new round of LDA model. The application is superior to the traditional LDA topic analysis method in accuracy and performance, and is applied to network public opinion field event detection, recommendation, word cloud and retrieval and other practical engineering projects, and creates commercial value.

Description

Technical Field

[0001] This invention belongs to the field of big data distributed computing mining, and in particular relates to an adaptive LDA topic model training system based on real-time public opinion data streams. Background Technology

[0002] With the continuous development of new concepts, technologies, and services such as cloud-edge computing, the Internet of Things, and social media, the global data volume is exploding. How to quickly, accurately, and comprehensively extract valuable information from massive amounts of real-time data has become particularly important. LDA topic models are widely used in topic mining, text classification, text retrieval, and social network analysis. However, traditional text-based LDA models typically use static data for single-machine, offline training. The number of topics and hyperparameter β are basically pre-set a priori parameters, leading to problems such as inability to adapt to dynamic datasets, inaccurate topic classification, and high latency. The basic principle in designing topic analysis methods based on LDA models is to combine distributed computing, model parameter optimization, and incremental model training strategies to improve the response efficiency of topic mining applications, provide timely decision-making support for enterprises, and increase efficiency. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the existing technology by providing an adaptive LDA topic model training system based on real-time public opinion data streams, improving the response efficiency of topic mining applications, realizing dimensionality reduction processing of massive data, providing reliable topic classification data support for further downstream data mining and processing, and ultimately providing timely decision-making support for enterprises to increase efficiency.

[0004] This invention includes a data aggregation module, a data preprocessing module, an adaptive LDA model training module, and an incremental LDA model fusion module;

[0005] The data aggregation module is used to extract, transform, and load structured and semi-structured data, and input it into the distributed message bus Kafka.

[0006] The data preprocessing module is used to preprocess the data in the message bus Kafka and finally form a weighted word vector.

[0007] The adaptive LDA model training module is used to perform optimal model iteration processing on the data in the message bus Kafka, train the LDA model results, and merge the training results.

[0008] The incremental LDA model fusion module is used to perform fusion training to generate a new round of LDA models.

[0009] The structured data includes relational databases, and the full and incremental synchronization operations of data from different relational databases are implemented through JDBC drivers and input into the message bus Kafka; the semi-structured data includes JSON, TXT and custom format data, and the data is input into the message bus Kafka by developing corresponding file parsing or RESTful interfaces.

[0010] The data preprocessing module performs the following steps:

[0011] Step a1: Calculate word weights using TF-IDF, where the normalized word frequency is calculated using the following formula:

[0012]

[0013] Where M represents the total number of documents in the corpus, N represents the total word count after adding up the word counts in each article. i represents the number of words appearing in the i-th document, F represents the number of times a certain word appears in all documents, and TF is the calculated normalized term frequency;

[0014] The inverse document frequency is calculated using the following formula:

[0015]

[0016] Where D+1 represents the number of documents containing a certain word, and IDF is the calculated inverse document frequency;

[0017] The formula for calculating TF-IDF is as follows:

[0018] TF-IDF = TF * IDF (3) The TF-IDF value is the weight of a word in the entire corpus;

[0019] Step a2, calculate word weights:

[0020] The following exponential decay function f(c) is used:

[0021]

[0022] Where c represents the number of cycles that have elapsed since the current time, and the cycle size is set to 1 day;

[0023] Step a3, calculate the word weights after text segmentation:

[0024]

[0025] Among them, TF j -IDF i,j f represents the TF-IDF weight value of the j-th word in the i-th document. i(c) represents the time-depreciation weight value of the i-th document. j This represents the average weight of the j-th feature word in the entire corpus.

[0026] The adaptive LDA model training module is based on the traditional LDA model Gibbs Sampling method, and incorporates feature word weight estimation for the current feature word w. i Theme z i The posterior distribution is calculated; then, the semantic similarity between the word and the top N words of the topic is calculated using the HowNet commonsense knowledge base to dynamically adjust the hyperparameter β of each word; the model results corresponding to different numbers of topics are obtained through iterative training, and the model with the minimum combined perplexity and average similarity between topics is taken as the best LDA model result; the weighted adaptive LDA training model algorithm is integrated into the Flink parallel streaming computing framework to improve the training speed of the model.

[0027] The Gibbs Sampling method based on the traditional LDA model includes: initially, each document randomly belongs to a topic, α and β are prior distribution hyperparameters, Gibbs sampling is used to solve the LDA, α and β are known prior inputs, and the target θ is solved. m,k and θ m,k and These are the document-topic probability distribution and the topic-word probability distribution, respectively.

[0028] The adaptive LDA model training module specifically performs the following steps:

[0029] Step b1, weighted estimation of topic probability distribution using feature words, includes the following steps:

[0030] Step b1-1: Calculate the posterior probability of traditional feature words;

[0031] Step b1-2: Calculate the posterior probability of the topic with the feature word weights;

[0032] Step b2: Improve the LDA model using semantic similarity;

[0033] Step b3: Select the best LDA model;

[0034] Step b4, parallel model training.

[0035] Step b1-1 includes:

[0036] Step b1-1-1, traditional calculation of LDA feature word topic posterior probability values:

[0037]

[0038] in, This indicates the removal of the topic variable corresponding to the i-th feature word; This represents the number of feature words t assigned to topic k. α represents the number of feature words in text m assigned to topic k; k Let β represent the Dirichlet prior distribution of topic k. t Let θ represent the Dirichlet prior distribution of the feature word t; once the topic label to which the feature word belongs is obtained, θ in the LDA model is obtained. m,k and The posterior estimate is p, where p is the probability distribution of the feature word belonging to each topic.

[0039] Step b1-2 includes:

[0040] Step b1-2-1, consider the feature word weights and calculate the posterior probability value of the LDA feature word topic:

[0041]

[0042] Where weight(t,d) represents the weight of each feature word in the document, p weight This represents the probability distribution after considering the weights of the feature words.

[0043] Step b2 includes:

[0044] A hyperparameter β is used for each document to embed the similarity between the current word and each topic, integrating semantic similarity into the topic-word distribution during LDA model training. Calculation in progress;

[0045] The average similarity between the current word and the top N words in the current topic is calculated using HowNet. In each Gibbs sampling iteration, the β value of each word is updated based on the similarity. The top N words are selected to eliminate noise in minimizing clustering, while also covering the high-weighted words in each topic. During the LDA topic model training process, a probability distribution of each word belonging to each topic is formed. Words in each topic are ranked in reverse order of probability, and the top N words are selected as keywords that best represent the topic, thus accelerating the iteration convergence speed. If the similarity of the i-th word increases or decreases compared to the previous iteration, the corresponding β value increases or decreases accordingly. When the similarity increases or decreases by at least 0.1, the β value increases or decreases by 0.001. The maximum β value is 1, and the minimum is 0.001. A larger β value indicates a higher relevance to the current topic, and vice versa.

[0046] The prior parameter vector of β is represented by the following matrix vector:

[0047]

[0048] Among them, wi Let represent the i-th word in the training sample set, N be the total number of words in the training samples, z represent the topic, K represent the number of topics, and β represent the number of topics. ik This represents the β value corresponding to the i-th word belonging to the k-th topic; if the similarity of the i-th word increases or decreases compared to the previous iteration, the corresponding β value increases or decreases accordingly.

[0049] Step b3 includes: calculating the perplexity of the LDA model.

[0050]

[0051] Among them, the denominator It is the sum of words in each document in the training set. In the numerator, p(z|d) represents the probability of a document belonging to each topic, p(w|z) represents the probability of each candidate word belonging to a topic, exp represents the parameter power of the natural number base e, and perplexity is the perplexity of the LDA model training result.

[0052] For LDA models trained with different numbers of topics, calculate the corresponding perplexity. The range of topics with the minimum perplexity is the optimal range of topics.

[0053] The correlation between topic vectors is measured using the standard vector cosine distance. The formula for calculating topic vector correlation is as follows:

[0054]

[0055] Where corre(Z) i Z j ) represents the correlation value between probability distribution vectors of different topics, denoted by β. i and β j Let β represent the probability distribution vectors of the i-th topic in the V-dimensional word space and the j-th topic in the V-dimensional word space, respectively. iv and β jv These represent the probability values ​​of the v-th word in the i-th topic belonging to the current topic and the probability values ​​of the v-th word in the j-th topics belonging to the current topic, respectively.

[0056] The stability of the topic structure is measured using the average similarity among all topics. The formula for the average similarity measure is as follows:

[0057]

[0058] Where avg_corre(structure) represents the average similarity value. This represents the topic probability distribution corre(Z) i Z jThe calculations are performed pairwise and superimposed, where K represents the number of topics. When the combined topic perplexity and average similarity are minimized, the corresponding model is optimal in terms of topic selection.

[0059] Step b4 includes: integrating the methods of steps a1 to a3 and steps b1 to b3 into the Flink parallel streaming computing MapReduce architecture, performing online model training through the Map operator, then aggregating and merging the LDA model vectors of the same topic trained by each node and partition through the Reduce operator, and finally persisting the LDA trained model to HDFS (Hadoop Distributed File System) as the prior parameters of the next round of incremental LDA model.

[0060] The incremental LDA model fusion module performs the following fusion process: First, based on T n-1 The topic-keyword weight vector matrix is ​​generated over time. For each topic, the top X keywords are compressed to form K texts. The keyword weights are set according to matrix normalization, and then based on T... n-1 The topic-document weight vector matrix is ​​generated at each moment, yielding the topic to which each document belongs. The number of documents under each topic used to assign weights to that topic is calculated, and finally, T... n-1 The LDA model generates weighted text data at time T, which is then compared with the data generated at time T. n New texts are fused and trained to form T. n A new LDA model is created at each moment, and the fusion process is continuously repeated to update the model.

[0061] This invention specifically provides an online parallel adaptive LDA model training system, ODA-LDA (Online Distribute Adaptation Topic Analysis Based on LDA). It considers word weights in the corpus, dynamically obtains the optimal number of topics and hyperparameter β, and incrementally expands the LDA model online. The jieba word segmentation corpus and HowNet commonsense knowledge base involved have been accumulated and expanded. Experimental results show that this invention outperforms traditional LDA topic analysis methods in both accuracy and performance.

[0062] Beneficial effects: Compared to traditional text-based LDA models, which typically employ static data for single-machine, offline training, treat feature word weights equally, and use pre-set parameters such as the number of topics and hyperparameter β, this invention considers word weights in the corpus, dynamically obtains the optimal number of topics and hyperparameter β, and performs online model training simultaneously across multiple nodes and partitions. This results in superior accuracy and performance compared to traditional LDA topic analysis methods. Furthermore, it has been applied to practical engineering projects in the field of online public opinion, including event detection, recommendation, word cloud, and retrieval, creating commercial value. Attached Figure Description

[0063] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0064] Figure 1 This is a diagram of the adaptive LDA topic model training architecture for real-time public opinion data streams.

[0065] Figure 2 This is a diagram of the traditional LDA model training framework.

[0066] Figure 3 yes Figure 3 Training framework diagram of weighted and similarity LDA model.

[0067] Figure 4 This is a diagram of the parallel computing architecture of Flink and Kafka.

[0068] Figure 5 This is a graph showing the efficiency difference between incremental and non-incremental methods.

[0069] Figure 6 This is a schematic diagram illustrating the changes in perplexity of LDA models on different topics.

[0070] Figure 7 This is a schematic diagram illustrating the consistency changes of LDA models on different topics.

[0071] Figure 8 This is a diagram comparing the training recall rates of different LDA models.

[0072] Figure 9 This is a diagram showing the comparison of training accuracy of different LDA models.

[0073] Figure 10 This is a diagram comparing the F1 scores of different LDA models.

[0074] Figure 11 This is a diagram comparing the training execution time of different LDA models.

[0075] Figure 12 This is a diagram comparing the F1 scores of different LDA models (the best subject is unknown).

[0076] Figure 13 This is a schematic diagram showing the performance ratio of the three indicators without considering failure degradation.

[0077] Figure 14 This is a schematic diagram showing the performance ratio of three indicators, including failure degradation.

[0078] Figure 15 This is a diagram comparing the training execution time of incremental and non-incremental models.

[0079] Figure 16 This is a diagram showing the performance ratio of three metrics: parallel processing and single-machine operation.

[0080] Figure 17 This is a diagram showing the comparison of execution time with different parallelism levels as the feature words change. Detailed Implementation

[0081] This invention provides an adaptive LDA topic model training system based on real-time public opinion data streams, including a data aggregation module, a data preprocessing module, an adaptive LDA model training module, and an incremental LDA model fusion module;

[0082] Training method architecture as follows Figure 1 As shown, firstly, multi-source heterogeneous data is normalized and aggregated into the Kafka message bus; secondly, the Kafka message bus data undergoes parallel filtering, deduplication (flatmap), transformation (map), and other Flink operators, as well as Jieba word segmentation preprocessing. TF-IDF denoising and weighting are then used to extract the top N feature word weights as candidate weighted word vector datasets; nextly, during model iterative training, dynamic parameter tuning is performed considering the weight of each word and the semantic similarity between words and topics. Each training iteration yields model results corresponding to different numbers of topics, and the model with the minimum combined perplexity and average similarity between topics is taken as the final LDA model result; finally, incremental LDA model fusion is implemented, periodically updating the LDA model in batches to ultimately achieve incremental LDA model training. Compared to traditional text-based LDA model training, which typically employs static data and offline training on a single machine, treats feature word weights equally, and uses pre-set parameters such as the number of topics and hyperparameter β, this invention proposes an online parallel adaptive LDA model training method, ODA-LDA (Online Distribute Adaptation Topic Analysis Based on LDA). This method considers word weights in the corpus, dynamically obtains the optimal number of topics and hyperparameter β, and performs online model training simultaneously across multiple nodes and partitions. Finally, the model results are merged and saved to HDFS. Experimental results show that this invention outperforms traditional LDA topic analysis methods in both accuracy and performance.

[0083] (1) Data aggregation module

[0084] Structured data sources are mainly relational databases, and full and incremental synchronization operations of data from different relational databases can be implemented into the message bus through JDBC drivers; semi-structured data are mainly JSON, TXT or custom format data, and data can be input into the message bus by developing corresponding file parsing or RESTful interfaces.

[0085] (2) Data preprocessing

[0086] Non-compliant data is filtered using a distributed Flink parallel streaming computing component, and unique keys are generated for deduplication according to public opinion uniqueness rules (platform type + publisher + publication time, etc.). Data from the Kafka message bus is processed using Flink's parallel streaming computing framework, including filter, map, and flatmap operators, for filtering, deduplication, transformation, and Jieba word segmentation. TF-IDF is used for noise reduction, weighting, and time-effect decay, ultimately forming weighted word vectors. The top N words by weight are then selected as candidate word groups. The specific weight calculation steps are as follows:

[0087] Step 1: The jieba tool uses TF-IDF to calculate word weights;

[0088] The formula for calculating tf is as follows:

[0089]

[0090] Where M represents the total number of documents in the corpus, N represents the total word count after adding up the word counts in each article. i represents the number of words appearing in the i-th document, F represents the number of times a certain word appears in all documents, and TF is the calculated term frequency;

[0091] The formula for calculating the inverse document frequency (IDF) is as follows:

[0092]

[0093] Where D+1 represents the number of documents containing a certain word, and IDF is the calculated inverse document frequency; the tf-idf calculation formula is as follows:

[0094] TF-IDF = TF*IDF (3) The TF-IDF value is the weight of a word in the entire corpus. It can be seen that the weight value is directly proportional to the number of times it appears in the document and inversely proportional to the number of times it appears in the entire corpus.

[0095] Step 2, calculate word weights for time-related decay.

[0096] Public opinion data has high timeliness requirements, and the timeliness decay factor is used to prevent noise problems caused by stale data in streaming data. This invention adopts a half-life exponential decay function, the formula of which is as follows:

[0097]

[0098] This invention uses a half-life function, where c represents the period compared to the current time. For example, if we take 1 day as a period, the importance of yesterday's data is reduced by half compared to today's data, the day before yesterday's data is 1 / 4 of today's data, and so on.

[0099] Step 3: Calculate the word weights after text segmentation.

[0100]

[0101] Weigh j This represents the average weight of the j-th feature word in the entire corpus. To improve training efficiency, the top N20% of words are selected.

[0102] (3) Text Adaptive LDA Training

[0103] First, based on the traditional LDA model Gibbs Sampling method, feature word weights are added to estimate the current feature word w. i Theme z i The algorithm first calculates the posterior distribution of the LDA model. Then, it dynamically adjusts the hyperparameter β of each word by calculating the semantic similarity between the word and the top N (typically N is 50) words in the HowNet commonsense knowledge base. Next, iterative training yields model results for different numbers of topics, and the model with the minimum combined perplexity and average similarity between topics is taken as the optimal LDA model result. Finally, the weighted adaptive optimal LDA training model algorithm is integrated into the Flink parallel streaming computing framework to improve the model's training speed.

[0104] Based on traditional LDA model research, each document is initially randomly assigned to a topic, and φ and θ represent the topic-word and document-topic probability distributions, respectively. α and β are prior distribution hyperparameters. Related literature indicates that, without loss of generality, all topics can be assigned the same α value, typically 50 / K. This invention uses Gibbs sampling to solve LDA, where α and β are known prior inputs, and the objective function θ is solved. m,k and These are the document-topic probability distribution and the topic-word probability distribution, respectively. Figure 3 The training framework diagram for the weighted similarity LDA model is shown below.

[0105] Specifically, the steps include the following:

[0106] Step 1: Weighted estimation of topic probability distribution using feature words

[0107] (1) Traditional feature word topic posterior probability calculation

[0108] a. LDA feature word topic posterior probability value:

[0109]

[0110] in, This indicates the topic variable corresponding to the i-th feature word to be removed; represents the number of feature words t assigned to topic k, and represents the number of feature words in text m assigned to topic k; represents the Dirichlet prior distribution of topic k, and represents the Dirichlet prior distribution of feature word t; when the topic label to which the feature word belongs is obtained, the posterior estimate of the summation in the LDA model is obtained, where p is the probability distribution of the feature word belonging to each topic. For example... Figure 2 As shown.

[0111] (2) Calculation of feature word weights and topic posterior probabilities

[0112] a. LDA feature word topic posterior probability value:

[0113]

[0114] This invention considers feature word weights, where weight(t,d) represents the weight of each feature word in the document, and pweight represents the probability distribution after considering the feature word weights. The feature word weights weight(t,d) are combined with the posterior estimates of θ and φ in the LDA model. By assigning different weights to different feature words under different topics, the probability of the model generating feature words is changed, such as... Figure 3 As shown.

[0115] Step 2: Improve the LDA model based on semantic similarity

[0116] This invention uses a hyperparameter β for each document to embed the similarity between the current word and each topic. Based on the Tsinghua University HowNet Chinese lexicon, semantic similarity is integrated into the topic-word distribution φ calculation in the LDA model training, thereby improving the speed and accuracy of clustering.

[0117] The average similarity between the current word and the top 50 words in the current topic is calculated using HowNet. In each Gibbs sampling iteration, the β value of each word is updated based on this similarity. The top 50 words are selected to eliminate noise in minimizing clustering, while simultaneously covering almost all important topic words in the corpus. A higher average similarity indicates that the word is an important related word in the corresponding topic, and a higher β value is assigned; conversely, a lower β value is assigned. The level of the β value can guide the LDA clustering process, obtaining more efficient topic clustering results by modifying the probability distribution. The β prior parameter vector can be represented by the following matrix vector:

[0118]

[0119] Where wi represents the i-th word in the training sample set, N is the total number of words in the training samples, z represents the topic, K represents the number of topics, and βik represents the β value corresponding to the i-th word belonging to the k-th topic. If the similarity of the i-th word increases or decreases compared to the previous iteration, the corresponding β value increases or decreases accordingly. During the dynamic adjustment of the β value, when the similarity increases or decreases by at least 0.1, the β value increases or decreases by 0.001 simultaneously. The maximum β value is 1, and the minimum is 0.001.

[0120] Step 3, Optimal LDA Model Selection

[0121] The perplexity of the LDA model is calculated to determine the optimal topic range. Perplexity can be understood as the uncertainty of which topic an article d belongs to after training; this degree of uncertainty is the perplexity. Lower perplexity indicates better clustering performance. The perplexity formula is as follows:

[0122]

[0123] In this model, the denominator is the sum of words in each document in the training set, and the numerator is p(z|d) which represents the probability of a document belonging to each topic, p(w|z) which represents the probability of each candidate word belonging to a topic, exp which represents the parameter raised to the power of the natural number base e, and perplexity which is the perplexity of the LDA model training result. According to relevant literature, theoretical and experimental evidence shows that the interval of the number of topics corresponding to the minimum perplexity is the optimal interval for the number of topics.

[0124] The correlation between topic vectors is measured using the standard vector cosine distance. The formula for calculating topic vector correlation is as follows:

[0125]

[0126] Where corre(Zi,Zj) represents the relevance value between probability distribution vectors of different topics, βi and βj represent the probability distribution vectors of the i-th topic in the V-dimensional word space and the j-th topic in the V-dimensional word space, respectively, and βiv and βjv represent the probability values ​​of the v-th word in the i-th topic and the v-th word in the j-th topics belonging to the current topic, respectively. The average similarity among all topics is used to measure the stability of the topic structure. The formula for the average similarity measure is as follows:

[0127]

[0128] Where avg_corre(structure) represents the average similarity value, which means calculating and superimposing the topic probability distributions corre(Zi,Zj) pairwise. K represents the number of topics. When the combined topic perplexity and average similarity are minimized, the corresponding model has the optimal number of topics. Step 4: Parallel model training.

[0129] Data sources are aggregated into the Kafka message bus, preprocessed using the Flink parallel streaming computing framework, and trained online ODA-LDA models using Map operators. Then, the Reduce process merges the LDA model vectors trained on each node and partition (significantly reducing the number of training samples). Finally, the trained LDA model is persistently saved to HDFS as prior parameters for the next round of incremental LDA model training. The process is as follows: Figure 4 As shown.

[0130] (4) Incremental LDA model fusion

[0131] First, according to T n-1 The topic-word weight vector matrix is ​​formed over time. For each topic, the top 1024 keywords are compressed into K texts, and then based on T... n-1 The topic-document weight vector matrix is ​​generated at any given time. This matrix determines the topic to which each document belongs, calculates the number of documents under each topic to assign weights to each topic, and finally assigns weights to T. n-1 The time-matter model generates weighted text data, which is then compared with T. n T is formed by fusing new texts (each document has a weight of 1) at any given time during training. n A new LDA model is created at each step. This process is then repeated continuously to update the model. An incremental method is compared with traditional methods, for example... Figure 5 As shown. The incremental method trains the LDA model with m+k documents per iteration, where k is the model vector M after the previous training. KxM Where K is the number of topics and N is the top 1024 keywords. Traditional methods require m*t documents for each LDA model training iteration, and the number of training samples increases exponentially with the number of documents.

[0132] Example:

[0133] (1) Experimental data

[0134] This experiment used data crawled from Toutiao (a popular Chinese news and information platform) on the internet, totaling 385,688 data entries. The data was in JSON format and consisted of titles, summaries, manually labeled topics, and tags identified by the LDA model, covering 15 topics. Details of the data are shown in Table 1 below. During the experiment, a portion was randomly selected as the training set, and the remaining portion was used as the test set. The experiment was repeated three times, and the average value was taken as the final result.

[0135] Table 1

[0136]

[0137]

[0138] (2) Experimental Environment

[0139] The experimental environment was mainly divided into single-machine and distributed experimental environments. Except for the parallel computing performance verification which used a cluster environment, all other experiments used single-machine mode.

[0140] a. Standalone environment

[0141] The standalone environment for this invention is a Windows 10 x64 machine environment, and the machine configuration details are shown in Table 2 below.

[0142] Table 2

[0143]

[0144] b. Parallel Environment

[0145] The cluster containing the parallel environment consists of one master node, three compute nodes, and three message queue nodes.

[0146] Table 3

[0147]

[0148] (3) Experimental procedure

[0149] This invention provides an adaptive LDA topic model training method based on real-time public opinion data streams. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

[0150] In all the following experiments, the prior parameters α and β of the LDA model are uniformly set to 2 and 0.01 respectively, the number of iterations is 1000 by default, the number of burn-in iterations is 50 iterations per cycle to update the word-topic probability distribution and document-topic probability distribution, and other prior parameters are determined according to the needs of each experiment. During the experiment, a portion of the data is randomly sampled from each topic, and the remaining portion is used as the test dataset.

[0151] 1. Validation of the optimal number of topics method

[0152] Six topics—culture, sports, car, technology, game, and military—were randomly selected from 15 topics for experimentation. The perplexity and average similarity between topics were compared between the traditional LDA model and the proposed ODA-LDA model training method under different topic numbers. The results are as follows: Figure 6 , 7 As shown, when the initial number of topics k=6, both the traditional LDA model and the ODA-LDA model have the lowest perplexity and average similarity between topics, verifying that the number of topics corresponding to the lowest perplexity and average similarity between topics is the optimal number of topics. Under the same topic conditions, the ODA-LDA model training method has lower perplexity and average similarity between topics compared to the traditional LDA model. Furthermore, the ODA-LDA model reaches convergence in 800 iterations, while the traditional LDA model requires 1000 iterations to reach convergence, indicating that the ODA-LDA model is superior to the traditional LDA model training method in terms of accuracy and convergence speed.

[0153] 2. Comparison of Topic Model Classification Results

[0154] Table 4

[0155]

[0156]

[0157] Five topics—entertainment, sports, car, culture, and game—were randomly selected from 15 topics for experimentation. During the dynamic adjustment of the β value, when the similarity increased or decreased by at least 0.1, the β value increased or decreased by 0.001 simultaneously. The maximum β value was 1, and the minimum was 0.001. Table 4 shows the top 10 keywords generated for different topics (sensitive words are indicated by codes). Traditional LDA and weighted LDA models produced erroneous keywords with low relevance. In contrast, the ODA-LDA and weighted + similarity LDA models proposed in this invention generated the most relevant feature keywords for each topic. Except for the ODA-LDA model proposed in this invention, whose initial optimal number of topics was unknown, the initial optimal number of topics for all others was k=5.

[0158] This experiment compares the recall, accuracy, and F1 score of three LDA models: traditional LDA, feature-weighted LDA, and feature-weighted + similarity LDA. The experimental results are as follows: Figure 8 , 9As shown in Figure 10, the results indicate that performance increases with the increase of training words, eventually stabilizing. This suggests that there are many irrelevant words in the full training samples, and taking the top N TF-IDF word weights is sufficient, which also improves training performance. The feature word weight + similarity LDA model proposed in this invention stabilizes quickly and is optimal in terms of recall, accuracy, and F1 score. This indicates that during training iterations, introducing word weights can attach higher probability distributions to important feature words as much as possible. At the same time, by dynamically adjusting the prior parameter β through the similarity between feature words and current different topics, the probability distribution values ​​of current feature words and similar topics are appropriately increased, improving the feature word probability distribution effect in each iteration. Figure 11 As shown, this invention takes a relatively long time to execute, because the time complexities of the three LDA models are O(SKN), O(SKN+N), and O(S(K(N+V*T) respectively. similar ))), where S represents the number of iterations, K represents the number of topics, N represents the number of feature words, V represents the number of different feature words, and T represents the number of different feature words. similar The calculation time represents the similarity calculation time between each feature word and the top N words of each topic. Analysis of the time complexity of various LDA models shows that the weight + similarity LDA model proposed in this invention has higher time complexity and takes longer, which is consistent with the experimental results. When the initial number of topics k = 3, as shown... Figure 12 As shown, after introducing the adaptive optimal topic identification method ODA-LDA in this invention, the ODA-LDA topic model has the best classification effect in F1 score calculation, which is much higher than the other three models. This indicates that the optimal number of topics was found through the adaptive method, and a better topic model can be obtained.

[0159] 3. Comparison of training effects between incremental and non-incremental models

[0160] The training sample set is first sent to a Kafka message queue, and data is consumed in batches with a 1-minute interval for incremental training and time decay. Figure 13 , 14 As shown, the precision ratio, recall ratio, and F1 score of the ODA-LDA incremental training model proposed in this invention are relatively stable over time compared to the non-incremental model. Incremental training has a smaller sample size, which cannot comprehensively represent the topic distribution in the corpus, and may also suffer from overfitting. Therefore, its performance is worse than non-incremental (full-scale) training. The performance without and with the time-effect decay function is approximately 0.82 and 0.86 of the non-incremental model, respectively. Incremental training with the time-effect decay function shows better results. Figure 15As shown, the time consumption of non-incremental (full) training increases proportionally with the number of training cycles, while the incremental training method remains relatively stable with the number of training cycles. In terms of time consumption, the incremental training method is much more efficient than the non-incremental training method. This is because the time complexity of the incremental method is O(m+k), while the time complexity of the non-incremental method is O(m*T), where m and k represent the number of new documents introduced in each cycle, and k represents the k topic feature vectors after training in each cycle.

[0161] 4. Comparison of training performance between single-machine and parallel models

[0162] Retrieve training data from Kafka, such as Figure 16 As shown, with the progress of the cycle, the number of training samples also increases. The performance of parallel computing compared to single-machine computing gradually increases in terms of accuracy, recall, and F1 score. Initially, due to the smaller number of training samples in parallel computing, the effect may be worse, but it tends to stabilize and become similar later. Figure 17 As shown, the training time increases with the number of feature words in the training samples, which is consistent with the training characteristics of LDA models. Under the same number of feature words in the training samples, the parallelism of Flink is 1, 3, 6, 9, 12, and 15, respectively. The training time decreases with the increase of parallelism, indicating that the training efficiency of LDA models can be improved by increasing the parallelism.

[0163] This invention provides an adaptive LDA topic model training system based on real-time public opinion data streams. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. An adaptive LDA topic model training system based on real-time public opinion data stream, characterized in that, It includes a data aggregation module, a data preprocessing module, an adaptive LDA model training module, and an incremental LDA model fusion module; The data aggregation module is used to extract, transform, and load structured and semi-structured data, and input it into the distributed message bus Kafka. The data preprocessing module is used to preprocess the data in the message bus Kafka and finally form a weighted word vector. The adaptive LDA model training module is used to perform optimal model iteration processing on the data in the message bus Kafka, train the LDA model results, and merge the training results. The incremental LDA model fusion module is used to perform fusion training to generate a new round of LDA models; The adaptive LDA model training module is based on the traditional LDA model Gibbs Sampling method, and incorporates feature word weight estimation for the current feature word w. i Theme z i The posterior distribution is calculated; then, the semantic similarity between the word and the top N words of the topic is calculated using the HowNet commonsense knowledge base to dynamically adjust the hyperparameter β of each word; the model results corresponding to different numbers of topics are obtained through iterative training, and the model with the minimum combined perplexity and average similarity between topics is taken as the best LDA model result; the weighted adaptive LDA training model algorithm is integrated into the Flink parallel streaming computing framework to improve the training speed of the model.

2. The system according to claim 1, characterized in that, The structured data includes relational databases, and the full and incremental synchronization operations of data from different relational databases are implemented through JDBC drivers and input into the message bus Kafka; the semi-structured data includes JSON, TXT and custom format data, and the data is input into the message bus Kafka through the development of file parsing or RESTful interfaces.

3. The system according to claim 2, characterized in that, The data preprocessing module performs the following steps: Step a1: Calculate word weights using TF-IDF, where the normalized word frequency is calculated using the following formula: Where M represents the total number of documents in the corpus, N represents the total word count after adding up the word counts in each article. i denoted by , where represents the number of words appearing in the i-th document, F represents the number of times a word appears in all documents, and TF is the calculated normalized term frequency; The inverse document frequency is calculated using the following formula: Where D+1 represents the number of documents containing a certain word, and IDF is the calculated inverse document frequency; The formula for calculating TF-IDF is as follows: TF-IDF = TF*IDF (3) TF-IDF value is the weight of a word across all corpora; Step a2, calculate word weights: The following exponential decay function f(c) is used: Where c represents the number of cycles that have elapsed compared to the current time; Step a3, calculate the word weights after text segmentation: Among them, TF j -IDF i,j f represents the TF-IDF weight value of the j-th word in the i-th document. i (c) represents the time-depreciation weight value of the i-th document. j This represents the average weight of the j-th feature word in the entire corpus.

4. The system according to claim 3, characterized in that, The Gibbs Sampling method based on the traditional LDA model includes: initially, each document randomly belongs to a topic, α and β are prior distribution hyperparameters, Gibbs sampling is used to solve the LDA, α and β are known prior inputs, and the target θ is solved. m,k and θ m,k and These are the document-topic probability distribution and the topic-word probability distribution, respectively.

5. The system according to claim 4, characterized in that, The adaptive LDA model training module specifically performs the following steps: Step b1, weighted estimation of topic probability distribution using feature words, includes the following steps: Step b1-1: Calculate the posterior probability of traditional feature words; Step b1-2: Calculate the posterior probability of the topic with the feature word weights; Step b2: Improve the LDA model using semantic similarity; Step b3: Select the best LDA model; Step b4, parallel model training.

6. The system according to claim 5, characterized in that, Step b1-1 includes: Step b1-1-1, traditional calculation of LDA feature word topic posterior probability values: in, This indicates the removal of the topic variable corresponding to the i-th feature word; This represents the number of feature words t assigned to topic k. α represents the number of feature words in text m assigned to topic k; k Let β represent the Dirichlet prior distribution of topic k. t Let θ represent the Dirichlet prior distribution of the feature word t; once the topic label to which the feature word belongs is obtained, θ in the LDA model is obtained. m,k and The posterior estimate is p, where p is the probability distribution of the feature word belonging to each topic.

7. The system according to claim 6, characterized in that, Step b1-2 includes: Step b1-2-1, consider the feature word weights and calculate the posterior probability value of the LDA feature word topic: Where weight(t,d) represents the weight of each feature word in the document, p weight This represents the probability distribution after considering the weights of the feature words.

8. The system according to claim 7, characterized in that, Step b2 includes: A hyperparameter β is used for each document to embed the similarity between the current word and each topic, integrating semantic similarity into the topic-word distribution during LDA model training. Calculation in progress; The average similarity between the current word and the top N words in the current topic is calculated using HowNet. In each Gibbs sampling iteration, the β value of each word is updated based on the similarity. The top N words are selected to eliminate noise in minimizing clustering, while also covering the topic words with high weight values ​​in each topic. If the similarity of the i-th word increases or decreases compared to the previous iteration, the corresponding β value is increased or decreased accordingly. The prior parameter vector of β is represented by the following matrix vector: Among them, w i Let represent the i-th word in the training sample set, N be the total number of words in the training samples, z represent the topic, K represent the number of topics, and β represent the number of topics. ik This represents the β value corresponding to the i-th word belonging to the k-th topic; if the similarity of the i-th word increases or decreases compared to the previous iteration, the corresponding β value increases or decreases accordingly. Step b3 includes: calculating the perplexity of the LDA model. Among them, the denominator It is the sum of words in each document in the training set. In the numerator, p(z|d) represents the probability of a document belonging to each topic, p(w|z) represents the probability of each candidate word belonging to a topic, exp represents the parameter power of the natural number base e, and perplexity is the perplexity of the LDA model training result. For LDA models trained with different numbers of topics, calculate the corresponding perplexity. The range of topics with the minimum perplexity is the optimal range of topics. The correlation between topic vectors is measured using the standard vector cosine distance. The formula for calculating topic vector correlation is as follows: Where corre(Z) i Z j ) represents the correlation value between probability distribution vectors of different topics, denoted by β. i and β j Let β represent the probability distribution vectors of the i-th topic in the V-dimensional word space and the j-th topic in the V-dimensional word space, respectively. iv and β jv These represent the probability values ​​of the v-th word in the i-th topic belonging to the current topic and the probability values ​​of the v-th word in the j-th topic belonging to the current topic, respectively. The stability of the topic structure is measured using the average similarity among all topics. The formula for the average similarity measure is as follows: Where avg_corre(structure) represents the average similarity value. This represents the topic probability distribution corre(Z) i Z j The calculations are performed pairwise and superimposed, where K represents the number of topics. When the combined topic perplexity and average similarity are minimized, the corresponding model is optimal in terms of topic selection. Step b4 includes: integrating the methods of steps a1 to a3 and steps b1 to b3 into the Flink parallel streaming computing MapReduce architecture, performing online model training through the Map operator, then merging the LDA model vectors of the same topic trained by each node and partition through the Reduce operator, and finally persisting the LDA trained model to HDFS as the prior parameters of the next round of incremental LDA model.

9. The system according to claim 8, characterized in that, The incremental LDA model fusion module performs the following fusion process: First, based on T n-1 The topic-keyword weight vector matrix is ​​generated over time. For each topic, the top X keywords are compressed to form K texts. The keyword weights are set according to matrix normalization, and then based on T... n-1 The topic-document weight vector matrix is ​​generated at each moment, yielding the topic to which each document belongs. The number of documents under each topic used to assign weights to that topic is calculated, and finally, T... n-1 The LDA model generates weighted text data at time T, which is then compared with the data generated at time T. n New texts are fused and trained to form T. n A new LDA model is created at each moment, and the fusion process is continuously repeated to update the model.

Images