Emergency topic detection method based on HVCH model and BurstNBT model

Abstract

The invention discloses an emergent topic detection method based on an HVCH model and a BurstNBT model. The emergent topic detection method comprises the following steps: S1, preprocessing; s2, clustering the text topics; and S3, sudden topic detection. Compared with a BBTM model, a VSM model and an LDA model, the scheme of the invention can more accurately detect the emergent topic related to the topic. Secondly, experimental results show that the HVCH-BurstNBT model can comprehensively cover sudden themes related to themes compared with a BBTM model, a VSM model and an LDA model. Finally, the F comprehensive value of the HVCH-BurstNBT model is obviously higher than that of the BBTM model, the VSM model, the LDA model and the HVCH-BurstNBT model, and the theme detection performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of burst topic detection, and in particular to a burst topic detection method based on a HVCH model and a Burst_NBT model. Background Art

[0002] Topic detection methods mainly include topic detection methods based on clustering algorithms and topic detection methods based on LDA models. The first type of topic detection method takes into account the statistical characteristics of the text, expands multiple texts in a vectorized form, and then calculates the similarity between texts respectively. According to the similarity value, the text set is clustered to determine the topic (X. Geng, YM Zhang, YH Jiao, et al. A Novel Hybrid Clustering Algorithm for Topic Detection on Chinese Microblogging. IEEE Transactions on Computational Social Systems, 2019, 6 (2): 289-300.). The second type of topic detection method includes a three-layer structure of terms, topics, and texts. The bag-of-words method is used to treat each text as a word frequency vector, so that the text information is converted into digital information that is easy to model (HW Cho. LDA-based Topic Modeling and Analysis on Batch Process Monitoring. Journal of Korea Academia-Industrial Cooperation Society, 2022, 23 (4): 26-33.). At the same time, each text represents the probability distribution of multiple topics, and each topic represents the probability distribution of multiple topic terms. Based on the probability distribution, the topic information hidden in large-scale text sets or corpora can be detected.

[0003] The above topic detection method can better identify the topics in the text set, but the above method still has some problems: (1) It is difficult for the above topic detection method to effectively detect the topics of the Weibo short text set; (2) It is difficult to extract high-quality topics from the Weibo text set. Summary of the invention

[0004] The purpose of the present invention is to solve the defects of the above-mentioned prior art and provide a method for detecting burst topics based on the HVCH model and the Burst_NBT model. The present invention uses a deep neural network as the core of shale layer identification and hierarchical progressive calculation as the implementation method for accurately locating shale layers, and designs a method for detecting burst topics based on the HVCH model and the Burst_NBT model. Relying on this method, the automation of the shale layer identification process can be realized, the objectivity and reliability of the identification results can be improved, and the resolution of shale layer identification can be improved.

[0005] The scheme of the present invention is:

[0006] A burst topic detection method based on the HVCH model and the Burst_NBT model comprises the following steps:

[0007] Step S1, preprocessing;

[0008] Step S2, text topic clustering;

[0009] Step S3, burst topic detection;

[0010] The step S1 preprocessing includes data cleaning, stop word removal and text segmentation;

[0011] The data cleaning filters meaningless content including punctuation marks and emoticons, and extracts two sets of label titles and text content from the text;

[0012] The stop word removal deletes words without semantic information including modal particles, personal pronouns, prepositions and conjunctions;

[0013] The text segmentation converts the text content into terms, generating a minimum granularity set of terms;

[0014] The step of extracting a set of tag titles also includes connecting adjacent words in the original tag titles into a set of meaningful character strings.

[0015] Specifically, step S2 includes the following sub-steps:

[0016] Step S21, obtaining a text probability matrix;

[0017] Step S22, calculating text similarity;

[0018] Step S23, text clustering;

[0019] Step S24, identifying cluster topics.

[0020] Specifically, step S21 includes the following sub-steps:

[0021] Obtain the common heat matrix between terms, calculate the heat of the text using the number of comments and forwardings of the text, and calculate the common heat between terms by combining the number of times any two terms appear in each text;

[0022] Step S211, calculating the topic probability distribution, presetting the number of topics, and using Gibbs sampling to generate the topic probability distribution of the corpus;

[0023] Step S212, generating a topic word probability matrix, using Gibbs sampling to obtain the corresponding word probability distribution under each topic, and expressing it in a matrix form;

[0024] Step S213, calculate the probability distribution of term pairs, use the common heat of term pairs obtained from the co-heat matrix between terms as weights and combine the number of occurrences of term pairs in the microblog text to calculate the probability distribution of term pairs; Step S214, generate a text topic probability matrix, use the Bayesian formula to calculate the main probability distribution of term pairs, combine the term pair probability distribution, calculate the text topic probability distribution and express the probability distribution in the form of a vector matrix;

[0025] The calculation formula is:

[0026]

[0027] in,

[0028] md represents a microblog text, md represents a group of texts, Heat(md) represents the heat value of a text, c represents the number of comments on a text, r represents the number of reposts of a text, -lg represents the negative of the logarithm with base 10, Heat(w i ,w j ) represents the term w i 、w j The common heat between md (w i ) represents the term w i The number of occurrences in md, Num md (w j ) represents the term w j The number of occurrences in md, |Num md | represents the total number of words in md, N represents the total size of the corpus, represents the probability distribution of topic words, n t represents the total number of times a term pair is assigned to topic t, K represents the number of topics, α is a parameter related to the number of topics K, and θ t represents the topic probability distribution, n w|t represents the total number of times the term w is assigned to topic t, β is a parameter set to 0.01, M represents the number of different terms in the corpus, and P(w b|doc) represents the improved text term probability distribution, Num doc (w b ) represents the term pair w b The number of occurrences in the text doc, P(t|doc) represents the probability distribution of the text topic, and P(t|w b ) represents the subject probability distribution of the term pair, P(t) represents the topic probability, and P(w i |t) represents the term w i The probability assigned to topic t, P(w j |t) represents the term w j The probability assigned to topic t, P(w|t) represents the probability distribution of topic words.

[0029] Specifically, the step S22 includes the following sub-steps:

[0030] Step S221, calculating HMBTM similarity, and calculating the similarity between texts according to the probability of multiple topics appearing in different texts in the text topic matrix;

[0031] Step S222, calculating the VSM similarity, selecting the top terms from the acquired topic word matrix as feature terms and improving the tf-idf formula to calculate the feature term weights, and using the feature term weight values ​​to calculate the similarity between the texts;

[0032] Step S223, calculating similarity weights, using a linear weighted method to calculate the similarity between texts;

[0033] The calculation formula is:

[0034]

[0035] Sim(doc i ,doc j )=λSim VSM (doc i ,doc j )+(1-λ)Sim HMBTM (doc i ,doc j );

[0036] Among them, Weight(w i ,doc n ) represents the word w i In the text doc n The weight in tf(w i ) represents the word w i The frequency of occurrence in the text, idf(w i ) represents w iThe inverse text frequency index, N represents the total number of texts in the entire corpus, df(w i ) represents the occurrence of w in the entire corpus i The number of texts in a word; Sim HMBTM (doc i ,doc j ) represents doc i and doc j The HMBTM model similarity, P(t|doc i ) represents topic t in the text doc i The probability in , P(t|doc j ) represents topic t in the text doc j Probability in Sim VSM (doc i ,doc j ) represents doc i and doc j VSM model similarity, Weight ix Indicates that the feature word w is in the text doc i The weight value in Weight iy Indicates that the feature word w is in the text doc j The weight value in, n is the dimension of the VSM model, Sim(doc i ,doc j ) represents doc i and doc j The final similarity, λ represents a weight value greater than 0 and less than 1.

[0037] Specifically, step S23 includes the following sub-steps:

[0038] Step S231, cluster initialization, initialization setting similarity threshold and taking the first text as the first text cluster;

[0039] Step S232, calculate the average similarity, and take the average of the similarities between the newly input text and all the texts in the known cluster as the final similarity between the new text and the cluster. The calculation formula is:

[0040]

[0041] Among them, doc i Represents the i-th text, Clu j represents the jth text cluster, Clu j -doc n Represents the text cluster Clu j The nth text in T Sim (doc i ,Clu j) represents the text doc i With the known text cluster Clu j Similarity; Num represents the text cluster Clu j The total amount of text contained in Sim(doc i ,Clu j -doc n ) represents the text doc i With text cluster j The nth text in Clu j -doc n similarity;

[0042] Step S233, convergence judgment, compares the similarity value of the new text with the known text cluster and the threshold value set by the cluster initialization to determine whether to classify the new text into the text cluster.

[0043] Specifically, step S24 includes the following sub-steps:

[0044] Step S241, solving the text cluster vector, integrating all the texts of a single text cluster into one text and performing vectorization expansion;

[0045] Step S242, solving the topic set vector, integrating all the texts under a single known topic text set into one text and performing vectorization expansion;

[0046] Step S243, calculating the similarity, taking all the terms of a single text cluster as feature items, and using the VSM model to calculate the similarity between the text cluster and each topic set;

[0047] Step S244, summarizing the text topics, and summarizing the text clusters into corresponding topics according to the maximum similarity between the text clusters and the given topic set;

[0048] The calculation formula is:

[0049]

[0050] Among them, weight(w n ) represents the term w n The weight value in the relevant text set, Represents the vector form of the text cluster Clu, Represents the vector form of topic set topic, Sim VSM (Clu,topic) represents the similarity between the text cluster and the topic set, Weight i-Clu Text represents the weight of the term w in the text cluster, Weight i-topic Represents the weight of term w in the topic set.

[0051] Specifically, step S3 includes the following sub-steps:

[0052] Step S31, burst word determination;

[0053] Step S32, burst word clustering;

[0054] Step S33, identifying sudden topics.

[0055] Specifically, the step S31 includes the following sub-steps:

[0056] Step S311, evaluate the burstiness index, and perform burstiness scoring on the frequency of occurrence of words, related users, and related texts in the current time slice;

[0057] Step S312, calculating the burstiness score, linearly weighting the three index scores of the word, and introducing the factor of the word length to obtain the burstiness score of the word;

[0058] Step S313, screening out burst words, sorting each word in ascending order according to its burst score to obtain a burst word set BW, and calculating a burst threshold based on the four-point difference method, and taking the words with a burst score greater than the burst threshold as burst words;

[0059] The calculation formula is:

[0060]

[0061] TBScore(w)=λ1*FBS(w)*log2(length(w))+λ2*UBS(u|w)+λ3*PBS(p|w)

[0062] BW=(TBScore(w1'),TBScore(w'2),TBScore(w'3),TBScore(w'4),...,TBScore(w' n ))

[0063] Q3(BW)=TBScore(w ( ' n+1)*0.75 )

[0064] Q1(BW)=TBScore(w ( ' n+1)*0.25 )

[0065] DifVal(BW)=Q3(BW)-Q1(BW)

[0066] ThVal(BW)=Q3(BW)+1.5*DifVal(BW)

[0067]

[0068] Where FBS(w) represents the burstiness score of the frequency of occurrence of term w in the current time slice n, represents the frequency of occurrence of w in the current time slice n, represents the average of the total frequency of term w in the first m time slices, represents the variance of the frequency of word w in the first m time slices, represents the frequency of occurrence of term w in the previous i-th time slice. If term w has appeared in the set HTword, it is weighted by square. UBS(u|w) represents the burstiness score of users related to term w in the current time slice n. represents the total number of users who mentioned term w in time slice n, represents the average number of users who mentioned term w in the first m time slices, represents the variance of the number of users who mentioned term w in the first m time slices, represents the number of users who mentioned term w in the previous i-th time slice, PBS(p|w) represents the relevant text burstiness score of term w in the current time slice n, represents the total number of comments related to the text that mentions term w in time slice n, represents the average number of comments in the text that mentioned the term w in the previous m time slices, represents the variance of the number of comments in the text that mentions term w in the first m time slices, represents the number of comments in the text that mentions term w in the previous i-th time slice; TBScore(w) represents the burstiness score of term w in the current time slice n after weighted calculation, λ1, λ2, λ3 represent the correlation coefficients, and length(w) represents the length of term w; BW represents the set of bursty terms sorted in ascending order by burstiness score, TBScore(w' n ) represents the score of the nth burst term in ascending order, Q3(BW) represents the burst score of the term in the upper quartile, Q1(BW) represents the burst score of the term in the lower quartile, DifVal(BW) represents the quartile difference of the burst scores of the terms in the burst term set BW; ThVal(BW) represents the threshold of the burst term set BW, SelBurWord(w) represents the screening result of the burst term, the result of 1 represents that the term w is a burst word, and the result of 0 represents that the term w is not a burst word.

[0069] Specifically, the step S32 includes the following sub-steps:

[0070] Step S321, word vectorization, using the word2vec model successfully trained in the skim_gram mode to vectorize each burst word, and use a multi-dimensional vector to represent a single burst word;

[0071] Step S322, term clustering, using the K-means algorithm to randomly select multiple burst words from the burst word set as centroids, and calculate the vector distances between the remaining burst words and the centroids, classify the burst words and the corresponding centroids into one category according to the shortest distance, and obtain multiple burst word clusters under each topic through multiple iterations;

[0072] The calculation formula is:

[0073]

[0074] in, The vector representing the i-th term, v in Representative term w i The vector value in the nth dimension, represents the cluster center of the j-th word cluster, Represents the total number of words in this cluster. The vector representing the terms in this cluster, v jn Represents the vector value of the cluster in the nth dimension, d(w i ,C j ) represents the term w i Specifically, step S33 includes the following sub-steps:

[0075] Step S331, scoring the burst word cluster, using the cumulative frequency score of the burst word cluster, the relevant user score, the relevant comment and the forwarding score to obtain the final score of the burst word cluster;

[0076] Step S332, screening the burst topics, ranking the burst word clusters according to the scores of the burst word clusters, and selecting the burst word clusters with the highest ranking as the final detected burst topics;

[0077] The calculation formula is:

[0078]

[0079] TScore(BWC i )=Fre(BWC i )+UserNum(BWC i )+Post(BWC i );

[0080] Among them, Fre(BWC i ) represents the i-th burst word cluster BWC iThe cumulative frequency score of fre(w) represents the burst word cluster BWC i The frequency of occurrence of the term w in User(BWC i ) represents the i-th burst word cluster BWC i The relevant user rating of , UserNum(w) represents the number of users who mentioned the term w, CT(BWC i ) represents the i-th burst word cluster BWC i ComNum(w) represents the number of comments associated with term w, TraNum(w) represents the number of forwardings associated with term w, and TScore(BWC i ) represents the i-th burst word cluster BWC i Total rating.

[0081] Beneficial effects of the present invention:

[0082] 1. The solution of the present invention detects burst topics related to topics more accurately than the BBTM model, the VSM model and the LDA model.

[0083] 2. The solution of the present invention shows that the HVCH-Burst_NBT model can comprehensively cover the burst topics related to the topic better than the BBTM model, the VSM model and the LDA model.

[0084] 3. The F comprehensive value of the HVCH-Burst_NBT model of the solution of the present invention is significantly higher than that of the BBTM model, the VSM model and the LDA model. The HVCH-Burst_NBT model improves the performance of topic detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0086] Figure 1 It is a technical route flow chart of the present invention;

[0087] Figure 2 It is a schematic diagram comparing the accuracy indicators of all topic detection models of the present invention;

[0088] Figure 3 is a schematic diagram comparing the recall rate indicators of all topic detection models of the present invention;

[0089] Figure 4It is a schematic diagram of the comparison of the F1 comprehensive value indicators of all topic detection models of the present invention. DETAILED DESCRIPTION

[0090] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0091] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below. Obviously, the implementation cases described are part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limiting the scope of the implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0092] This paper proposes a burst topic detection method based on the HVCH model and the Burst_NBT model. First, the burst topic detection method preprocesses the microblog text set to obtain a microblog text term set and extracts a meaningful string set from the microblog tag title. Secondly, the heat of the term is used as the key factor to introduce the Heat Matrix to improve the BTM model, and the VSM model is integrated to calculate the similarity between microblog texts, and the single-pass clustering algorithm is optimized to make it suitable for microblog text clustering, effectively detecting the topic of the microblog text set. Then, the frequency of occurrence, related users, and related texts of the microblog text terms under each topic are linearly weighted to obtain the burst score of the term, and the burst terms are screened out according to the score. The word2vev model is used to vectorize the burst words and cluster them through the K-means algorithm to obtain the burst word clusters. Finally, the burst word clusters are scored and ranked, and the burst word clusters with the highest scores are selected as the detected burst topics. The experimental results show that compared with the BBTM model, the LDA model, and the VSM model, this model improves the three evaluation indicators of precision, recall rate, and F comprehensive value. In summary, the proposed HVCH-Burst_NBT model can effectively solve the data sparsity and data noise problems of short texts, thereby detecting relevant burst topics of various topics.

[0093] 1 Technical Solution

[0094] like Figure 1 As shown, the present invention proposes a burst topic detection method based on the HVCH model and the Burst_NBT model. This model establishes the HVCH model to solve the problem that the traditional model is difficult to detect topics in short text sets, and establishes the Burst_NBT model to effectively filter out meaningless general topics in microblog texts and extract high-quality topics. Figure 1 The framework diagram of the burst topic detection method based on the HVCH model and the Burst_NBT model is given. Figure 1In this paper, the burst topic detection method based on the HVCH model and the Burst_NBT model is divided into three main modules: preprocessing, text topic clustering, and burst topic detection.

[0095] 1.1 Preprocessing

[0096] For microblog text data, text preprocessing is required to reduce data noise and extract valid terms. Preprocessing mainly includes data cleaning, stop word removal and text segmentation. The data cleaning stage is used to filter meaningless content such as punctuation marks and emoticons, and extract two sets of tag titles and text content from the microblog text. The stop word removal stage is used to delete terms without semantic information such as modal particles, personal pronouns, prepositions, conjunctions, etc. The text segmentation stage performs a word segmentation algorithm to convert the text content into terms and generate a minimum granularity set of terms. The tag title set also needs to be processed to connect the terms that are adjacent in the original tag title into a meaningful string set.

[0097] 1.2 Text Topic Clustering

[0098] Text topic clustering uses the HVCH model formed by the fusion of the HMBTM model and the VSM model to calculate the similarity of the microblog text set, and uses the optimized single-pass clustering algorithm to cluster topics. Text topic clustering is divided into text probability matrix acquisition, text similarity calculation, text clustering, and cluster topic identification. The above four modules will be detailed below.

[0099] 1.2.1 Obtaining the Text Probability Matrix

[0100] The acquisition of text probability matrix is ​​to obtain topic word matrix and text topic matrix by using improved BTM model. The acquisition of text probability matrix is ​​divided into five processes: acquisition of inter-term co-heat matrix, calculation of topic probability distribution, generation of topic word probability matrix, calculation of term pair probability distribution, and generation of text topic probability matrix. The acquisition of inter-term co-heat matrix uses the number of comments and forwardings of microblog text to calculate the heat of microblog text and combines the number of times any two terms appear in each text to calculate the common heat between terms. The calculation of topic probability distribution is to pre-set the number of topics and use Gibbs sampling to generate the topic probability distribution of the corpus. The generation of topic word probability matrix is ​​to use Gibbs sampling to obtain the corresponding term probability distribution under each topic and express it in matrix form. The calculation of term pair probability distribution uses the common heat of term pairs obtained by inter-term co-heat matrix as weight and combines the number of times term pairs appear in microblog text to calculate the probability distribution of term pairs. The generation of text topic probability matrix uses Bayes formula to calculate the main probability distribution of term pairs. Combined with the term pair probability distribution, the text topic probability distribution can be calculated and expressed in the form of vector matrix.

[0101] The calculation formula for obtaining the text probability matrix is ​​as follows:

[0102]

[0103] In formula (1), md represents a microblog text, MD represents a group of microblog texts, Heat(md) represents the heat value of a microblog text, c represents the number of comments on a microblog text, r represents the number of reposts of a microblog, Heat(w i ,w j ) represents the term w i 、w j The common heat between md (w i ) represents the term w i Number of occurrences in Weibo md, Num md (w j ) represents the term w j Number of occurrences in Weibo md, |Num md | represents the total number of words in Weibo md, N represents the total size of the corpus, represents the probability distribution of topic words, n t represents the total number of times a term pair is assigned to topic t, K represents the number of topics, α is a parameter related to the number of topics K, and θ t represents the topic probability distribution, n w|t represents the total number of times the term w is assigned to topic t, β is generally a parameter with a value of 0.01, M represents the number of different terms in the corpus, and P(w b |doc) represents the improved text term probability distribution, Num doc (w b ) represents the term pair w b The number of occurrences in the text doc, P(t|doc) represents the probability distribution of the text topic, and P(t|w b ) represents the subject probability distribution of the term pair, P(t) represents the topic probability, and P(w i |t) represents the term w i The probability assigned to topic t, P(w j |t) represents the term w j The probability assigned to topic t, P(w|t) represents the probability distribution of topic words.

[0104] 1.2.2 Text Similarity Calculation

[0105] The text similarity calculation adopts the joint modeling of HMBTM model and VSM model, and uses the linear weighted method to calculate the similarity between microblog texts. The text similarity calculation is divided into three processes: HMBTM similarity calculation, VSM similarity calculation, and similarity weighting. The HMBTM similarity calculation calculates the similarity between texts based on the probability of multiple topics appearing in different texts in the text topic matrix. The VSM similarity calculation selects the top terms from the acquired topic word matrix as feature items and improves the tf-idf formula to calculate the feature item weights, and uses the feature item weight values ​​to calculate the similarity between texts. The weighted similarity passes the linear fusion processing of HMBTM similarity and VSM similarity to obtain the final similarity between texts.

[0106] The text similarity calculation expression is as follows:

[0107]

[0108] In formula (2), Weight(w i ,doc n ) represents the word w i In the text doc n The weight in tf(w i ) represents the word w i The frequency of occurrence in the text, idf(w i ) represents w i The inverse text frequency index, N represents the total number of texts in the entire corpus, df(w i ) represents the occurrence of w in the entire corpus i The number of words in the text. HMBTM (doc i ,doc j ) represents doc i and doc j The HMBTM model similarity, P(t|doc i ) represents topic t in the text doc i The probability in , P(t|doc j ) represents topic t in the text doc j The probability of . VSM (doc i ,doc j ) represents doc i and doc j VSM model similarity, Weight ix Indicates that the feature word w is in the microblog text doc i The weight value in Weight iy Indicates that the feature word w is in the microblog text doc jThe weight value in, n is the dimension of the VSM model, Sim(doc i ,doc j ) represents doc i and doc j The final similarity, λ represents a weight value greater than 0 and less than 1.

[0109] 1.2.3 Text Clustering

[0110] Text clustering sorts texts according to their publishing time and introduces average similarity to improve the single-pass clustering algorithm, making it suitable for topic clustering of microblog text sets. Text clustering is divided into three processes: cluster initialization, average similarity calculation, and convergence judgment. Cluster initialization sets a similarity threshold and takes the first text as the first text cluster. Average similarity calculation takes the average of the similarities between the newly input text and all texts under the known cluster as the final similarity between the new text and the cluster. Convergence judgment compares the similarity value between the new text and the known text cluster with the threshold set by cluster initialization to determine whether to assign the new text to the text cluster.

[0111] The expression for average similarity calculation is as follows:

[0112]

[0113] In formula (3), doc i Represents the i-th text, Clu j represents the jth text cluster, Clu j -doc n Represents the text cluster Clu j The nth text in T Sim (doc i ,Clu j ) represents the text doc i With the known text cluster Clu j Similarity; Num represents the text cluster Clu j The total amount of text contained in Sim(doc i ,Clu j -doc n ) represents the text doc i With text cluster j The nth text Clu j -doc n The similarity.

[0114] 1.2.4 Cluster Topic Identification

[0115] Cluster topic identification calculates the similarity between the vector of a single text cluster and the vector of the text set of the trained topic, so as to classify the text cluster into the corresponding topic for subsequent burst topic detection. Cluster topic identification is divided into four processes: text cluster vector solution, topic set vector solution, similarity calculation, and text topic induction. Text cluster vector solution is to integrate all texts of a single text cluster into one text and vectorize it. Topic set vector solution is to integrate all texts under a single known topic text set into one text and vectorize it. Similarity calculation is to use all terms of a single text cluster as feature items, and use the VSM model to calculate the similarity between the text cluster and each topic set. Text topic induction is to classify the text cluster into the corresponding topic based on the maximum similarity between the text cluster and the given topic set.

[0116] The expression of cluster topic identification is as follows:

[0117]

[0118] In formula (4), weight(w n ) represents the term w n The weight value in the relevant text set, Represents the vector form of the text cluster Clu, Represents the vector form of topic set topic, Sim VSM (Clu,topic) represents the similarity between the text cluster and the topic set, Weight i-Clu Text represents the weight of the term w in the text cluster, Weight i-topic Represents the weight of term w in the topic set.

[0119] 1.3 Burst Topic Detection

[0120] Burst topic detection uses the Burst_NBT model based on enhanced burst word detection to identify the related microblog text clusters under each topic. Burst topic detection is divided into burst word judgment, burst word clustering, and burst topic identification. The following will detail the contents of the above three modules.

[0121] 1.3.1 Burst Word Detection

[0122] The judgment of burst words is to use the historical data of multiple previous time slices as a reference to perform weighted scoring calculations on the relevant indicators of the terms in the current time slice, and select burst words that have mild changes in the historical period but strong changes in the current period according to the burst score of the term and the calculated threshold. The burst score of the term is divided into three processes: burst indicator evaluation, burst score calculation, and burst word screening. The burst indicator evaluation is to perform burst scoring on the frequency of occurrence, related users, and related texts of the words in the current time slice. The burst score calculation linearly weights the three indicator scores of the word, and introduces the term length factor to obtain the burst score of the term. The burst word screening is to sort the burst score of each term in ascending order to obtain the burst term set BW, and calculate the burst threshold based on the four-point difference method, and take the term with a burst score greater than the burst threshold as a burst word.

[0123] The burst word judgment expression is as follows:

[0124]

[0125] In expression (5), FBS(w) represents the burstiness score of the frequency of occurrence of term w in the current time slice n, represents the frequency of occurrence of w in the current time slice n, represents the average of the total frequency of term w in the first m time slices, represents the variance of the frequency of word w in the first m time slices, represents the frequency of occurrence of term w in the previous i-th time slice. If term w has appeared in the set HTword, it is weighted by square. UBS(u|w) represents the burstiness score of users related to term w in the current time slice n. represents the total number of users who mentioned term w in time slice n, represents the average number of users who mentioned term w in the first m time slices, represents the variance of the number of users who mentioned term w in the first m time slices, represents the number of users who mentioned term w in the previous i-th time slice, PBS(p|w) represents the relevant text burstiness score of term w in the current time slice n, represents the total number of comments related to the text that mentions term w in time slice n, represents the average number of comments in the text that mentioned the term w in the previous m time slices, represents the variance of the number of comments in the text that mentions term w in the first m time slices, represents the number of comments in the text that mentioned term w in the previous i-th time slice. TBScore(w) represents the burstiness score of term w in the current time slice n after weighted calculation, λ1, λ2, λ3 represent the correlation coefficients, and length(w) represents the length of term w. BW represents the bursty term set sorted in ascending order by burstiness score, TBScore(w' n ) represents the score of the nth burst word in ascending order, Q3(BW) represents the burst score of the word in the upper quartile, Q1(BW) represents the burst score of the word in the lower quartile, DifVal(BW) represents the quartile difference of the burst score of the word in the burst word set BW. ThVal(BW) represents the threshold of the burst word set BW, SelBurWord(w) represents the screening result of the burst word, the result of 1 represents that the word w is a burst word, and the result of 0 represents that the word w is not a burst word.

[0126] 1.3.2 Burst Word Clustering

[0127] The burst word clustering uses the word2vec model successfully trained in the skim_gram mode to vectorize the burst words, and uses the K-means algorithm to cluster the vector distances between the burst words to obtain multiple burst word clusters. The burst word clustering is divided into two processes: term vectorization and term clustering. Term vectorization uses the word2vec model successfully trained in the skim_gram mode to vectorize each burst word, and use a multi-dimensional vector to represent a single burst word. Term clustering uses the K-means algorithm to randomly select multiple burst words as the centroid in the burst word set, and calculate the vector distance between the remaining burst words and the centroid. The burst words and the corresponding centroid are classified into one category according to the shortest distance, and multiple burst word clusters under each topic are obtained through multiple iterations.

[0128] The burst word clustering expression is as follows:

[0129]

[0130] In expression (6) The vector representing the i-th term, v in Representative term w i The vector value in the nth dimension, represents the cluster center of the j-th word cluster, Represents the total number of words in this cluster. The vector representing the terms in this cluster, v jn Represents the vector value of the cluster in the nth dimension, d(w i ,C j ) represents the term w i .

[0131] 1.3.3 Identification of Emergent Topics

[0132] The burst topic identification is to score the multiple burst word clusters BWC obtained by the burst word clustering module and select the top-ranked burst word clusters as the final detected burst topics according to the final score. The burst topic identification is divided into two processes: burst word cluster scoring and burst topic screening. The burst word cluster scoring is to obtain the final score of the burst word cluster by integrating the cumulative frequency score of the burst word cluster, the relevant user score, the relevant comments and the forwarding score. The burst topic screening is to rank the burst word clusters according to the score of the burst word cluster and select the top-ranked burst word clusters as the final detected burst topic.

[0133] The scoring expression of the burst word cluster is as follows:

[0134]

[0135] In expression (7), Fre(BWC i ) represents the i-th burst word cluster BWC i The cumulative frequency score of fre(w) represents the burst word cluster BWC i The frequency of occurrence of the term w in User(BWC i ) represents the i-th burst word cluster BWC i The relevant user rating of , UserNum(w) represents the number of users who mentioned the term w, CT(BWC i ) represents the i-th burst word cluster BWC i ComNum(w) represents the number of comments associated with term w, TraNum(w) represents the number of forwardings associated with term w, and TScore(BWC i ) represents the i-th burst word cluster BWC i Total rating.

[0136] 1.3 Technical Effect

[0137] In order to verify that the HVCH model and the Burst_NBT model can improve the accuracy of burst topic detection, this experiment implemented four different topic detection models: the BBTM model, the LDA model, the VSM model, and the HVCH-Burst_NBT model, and compared and analyzed the performance of the four models. The BBTM model is a model that improves the BTM model and is applied to burst topic detection. The LDA model is a probabilistic generation model that can be used for large-scale text sets or corpora. The VSM model is a model for text similarity calculation. The HVCH-Burst_NBT model is a model for detecting burst topics that integrates the HVCH model and the Burst_NBT model. This model can effectively solve the data sparsity and noisiness problems of short texts on Weibo.

[0138] This paper studies the detection of burst topics. The experimental text data of this paper comes from the text data of Sina Weibo platform. Through the content of the request header and the content of the keyword, the experiment of this paper uses the get method of the request method to obtain the text data of Weibo on the Weibo platform, including the author of the Weibo, the release time, the number of reposts, the number of comments, the number of likes and the content of the Weibo, totaling more than 10,000 data results. The evaluation indicators include precision, recall rate and F1 comprehensive value. Precision refers to the ratio of the number of burst topics related to the topic detected by the topic detection model to the total number of burst topics detected. This indicator can measure the performance of the topic detection model in correctly detecting burst topics. Recall rate refers to the ratio of the number of burst topics related to the topic detected to the number of all burst topics related to the topic. This indicator can measure the coverage ability of the topic detection model for all related burst topics. F1 comprehensive value refers to a balanced indicator that combines precision and recall rate. This indicator can measure the comprehensive performance of the topic detection model.

[0139] In the experiment, all four topic detection models use the same microblog document data to obtain the experimental results of the three evaluation indicators of the four topic detection models. Figure 2-4 As shown, the comparison of the three evaluation metrics of all four topic detection models is shown separately. Figure 2 It shows that compared with the other three topic detection models, the HVCH-Burst_NBT model can more correctly detect topic-related burst topics. Figure 3 It shows that compared with the other three topic detection models, the HVCH-Burst_NBT model can more comprehensively cover the burst topics related to the topic. Figure 4 It shows that the F1 comprehensive value of the HVCH-Burst_NBT model is significantly higher than that of the other three topic detection models, and the comprehensive performance of this model is better than that of the other three topic detection models.

[0140] The experimental results compare the performance of four topic detection models. First, the experimental results show that the HVCH-Burst_NBT model detects burst topics related to the topic more accurately than the BBTM model, VSM model and LDA model. Second, the experimental results show that the HVCH-Burst_NBT model can comprehensively cover burst topics related to the topic than the BBTM model, VSM model and LDA model. Finally, the experimental results show that the F-comprehensive value of the HVCH-Burst_NBT model is significantly higher than that of the BBTM model, VSM model and LDA model, and the HVCH-Burst_NBT model can improve the performance of topic detection.

[0141] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of the present invention to be protected is defined by the attached claims and their equivalents.

[0142] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and units involved are not necessarily required by this application.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a ROM, a RAM, etc.

[0145] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A burst topic detection method based on HVCH model and Burst_NBT model, characterized in that: The following steps are involved: Step S1, preprocessing; Step S2, text topic clustering; Step S3, burst topic detection; The step S1 preprocessing includes data cleaning, stop word removal and text segmentation; The data cleaning filters meaningless content including punctuation marks and emoticons, and extracts two sets of label titles and body content from the text; The stop word removal deletes words without semantic information including modal particles, personal pronouns, prepositions and conjunctions; The text segmentation converts the text content into terms, generating a minimum granularity set of terms; The step of extracting a set of tag titles also includes connecting adjacent words in the original tag titles into a set of meaningful character strings.

2. According to claim 1, a burst topic detection method based on HVCH model and Burst_NBT model is characterized in that: The step S2 comprises the following sub-steps: Step S21, obtaining a text probability matrix; Step S22, calculating text similarity; Step S23, text clustering; Step S24, identifying cluster topics.

3. The method for detecting burst topics based on the HVCH model and the Burst_NBT model according to claim 2, characterized in that: The step S21 includes the following sub-steps: Obtain the common heat matrix between terms, calculate the heat of the text using the number of comments and forwardings of the text, and calculate the common heat between terms by combining the number of times any two terms appear in each text; Step S211, calculating the topic probability distribution, presetting the number of topics, and using Gibbs sampling to generate the topic probability distribution of the corpus; Step S212, generating a topic word probability matrix, using Gibbs sampling to obtain the corresponding word probability distribution under each topic, and expressing it in a matrix form; Step S213, calculating the probability distribution of term pairs, using the common heat of the term pairs obtained from the co-heat matrix between terms as weights combined with the number of occurrences of the term pairs in the microblog text to calculate the probability distribution of the term pairs; Step S214, generating a text topic probability matrix, using the Bayesian formula to calculate the main probability distribution of the term pair, combining the term pair probability distribution, calculating the text topic probability distribution and expressing the probability distribution in the form of a vector matrix; The calculation formula is: in, md represents a microblog text, md represents a group of texts, Heat(md) represents the heat value of a text, c represents the number of comments on a text, r represents the number of reposts of a text, -lg represents the negative of the logarithm with base 10, Heat(w i ,w j ) represents the term w i 、w j The common heat between md (w i ) represents the term w i The number of occurrences in md, Num md (w j ) represents the term w j The number of occurrences in md, |Num md | represents the total number of words in md, N represents the total size of the corpus, represents the probability distribution of topic words, n t represents the total number of times a term pair is assigned to topic t, K represents the number of topics, α is a parameter related to the number of topics K, and θ t represents the topic probability distribution, n w|t represents the total number of times the term w is assigned to topic t, β is a parameter set to 0.01, M represents the number of different terms in the corpus, and P(w b |doc) represents the improved text term probability distribution, Num doc (w b ) represents the term pair w b The number of occurrences in the text doc, P(t|doc) represents the probability distribution of the text topic, and P(t|w b ) represents the subject probability distribution of the term pair, P(t) represents the topic probability, and P(w i |t) represents the term w i The probability assigned to topic t, P(w j |t) represents the term w j The probability assigned to topic t, P(w|t) represents the probability distribution of topic words.

4. The method for detecting burst topics based on the HVCH model and the Burst_NBT model according to claim 3, characterized in that: The step S22 includes the following sub-steps: Step S221, calculating HMBTM similarity, and calculating the similarity between texts according to the probability of multiple topics appearing in different texts in the text topic matrix; Step S222, calculating the VSM similarity, selecting the top terms from the acquired topic word matrix as feature terms and improving the tf-idf formula to calculate the feature term weights, and using the feature term weight values ​​to calculate the similarity between the texts; Step S223, calculating similarity weights, using a linear weighted method to calculate the similarity between texts; The calculation formula is: Sim(doc i ,doc j )=λSim VSM (doc i ,doc j )+(1-λ)Sym HMBTM (doc i ,doc j ); Among them, Weight(w i ,doc n ) represents the word w i In the text doc n The weight in tf(w i ) represents the word w i The frequency of occurrence in the text, idf(w i ) represents w i The inverse text frequency index, N represents the total number of texts in the entire corpus, df(w i ) represents the occurrence of w in the entire corpus i The number of texts in a word; Sim HMBTM (doc i ,doc j ) represents doc i and doc j The HMBTM model similarity, P(t|doc i ) represents topic t in the text doc i The probability in , P(t|doc j ) represents topic t in the text doc j Probability in Sim VSM (doc i ,doc j ) represents doc i and doc j VSM model similarity, Weight ix Indicates that the feature word w is in the text doc i The weight value in Weight iy Indicates that the feature word w is in the text doc j The weight value in, n is the dimension of the VSM model, Sim(doc i ,doc j ) represents doc i and doc j The final similarity, λ represents a weight value greater than 0 and less than 1.

5. The method for detecting burst topics based on the HVCH model and the Burst_NBT model according to claim 4, characterized in that: The step S23 includes the following sub-steps: Step S231, cluster initialization, initialization setting similarity threshold and taking the first text as the first text cluster; Step S232, calculate the average similarity, and take the average of the similarities between the newly input text and all the texts in the known cluster as the final similarity between the new text and the cluster. The calculation formula is: Among them, doc i Represents the i-th text, Clu j represents the jth text cluster, Clu j -doc n Represents the text cluster Clu j The nth text in T Sim (doc i ,Clu j ) represents the text doc i With the known text cluster Clu j Similarity; Num represents the text cluster Clu j The total amount of text contained in Sim(doc i ,Clu j -doc n ) represents the text doc i With text cluster j The nth text in Clu j -doc n similarity; Step S233, convergence judgment, compares the similarity value of the new text with the known text cluster and the threshold value set by the cluster initialization to determine whether to classify the new text into the text cluster.

6. The method for detecting burst topics based on the HVCH model and the Burst_NBT model according to claim 5, characterized in that: The step S24 includes the following sub-steps: Step S241, solving the text cluster vector, integrating all the texts of a single text cluster into one text and performing vectorization expansion; Step S242, solving the topic set vector, integrating all the texts under a single known topic text set into one text and performing vectorization expansion; Step S243, calculating the similarity, taking all the terms of a single text cluster as feature items, and using the VSM model to calculate the similarity between the text cluster and each topic set; Step S244, summarizing the text topics, and summarizing the text clusters into corresponding topics according to the maximum similarity between the text clusters and the given topic set; The calculation formula is: Among them, weight(w n ) represents the term w n The weight value in the relevant text set, Represents the vector form of the text cluster Clu, Represents the vector form of topic set topic, Sim VSM (Clu,topic) represents the similarity between the text cluster and the topic set, Weight i-Clu Text represents the weight of the term w in the text cluster, Weight i-topic Represents the weight of term w in the topic set.

7. The method for detecting burst topics based on the HVCH model and the Burst_NBT model according to claim 1, characterized in that: The step S3 comprises the following sub-steps: Step S31, burst word determination; Step S32, burst word clustering; Step S33, identifying sudden topics.

8. The method for detecting burst topics based on the HVCH model and the Burst_NBT model according to claim 7, characterized in that: The step S31 includes the following sub-steps: Step S311, evaluate the burstiness index, and perform burstiness scoring on the frequency of occurrence of words, related users, and related texts in the current time slice; Step S312, calculating the burstiness score, linearly weighting the three index scores of the word, and introducing the factor of the word length to obtain the burstiness score of the word; Step S313, screening out burst words, sorting each word in ascending order according to its burst score to obtain a burst word set BW, and calculating a burst threshold based on the four-point difference method, and taking the words with a burst score greater than the burst threshold as burst words; The calculation formula is: TBScore(w)=λ1*FBS(w)*log2(length(w))+λ2*UBS(u|w)+λ3*PBS(p|w) BW=(TBScore(w1'),TBScore(w'2),TBScore(w'3),TBScore(w'4),...,TBScore(w' n )) Q3(BW)=TBScore(w' (n+1)*0.75 ) Q1(BW)=TBScore(w ( ' n+1)*0.25 ) DifVal(BW)=Q3(BW)-Q1(BW) ThVal(BW)=Q3(BW)+1.5*DifVal(BW) Where FBS(w) represents the burstiness score of the frequency of occurrence of term w in the current time slice n, represents the frequency of occurrence of w in the current time slice n, represents the average of the total frequency of term w in the first m time slices, represents the variance of the frequency of word w in the first m time slices, represents the frequency of occurrence of term w in the previous i-th time slice. If term w has appeared in the set HTword, it is weighted by square. UBS(u|w) represents the burstiness score of users related to term w in the current time slice n. represents the total number of users who mentioned term w in time slice n, represents the average number of users who mentioned term w in the first m time slices, represents the variance of the number of users who mentioned term w in the first m time slices, represents the number of users who mentioned term w in the previous i-th time slice, PBS(p|w) represents the relevant text burstiness score of term w in the current time slice n, represents the total number of comments related to the text that mentions term w in time slice n, represents the average number of comments in the text that mentioned the term w in the previous m time slices, represents the variance of the number of comments in the text that mentions term w in the first m time slices, represents the number of comments in the text that mentions term w in the previous i-th time slice; TBScore(w) represents the burstiness score of term w in the current time slice n after weighted calculation, λ1, λ2, λ3 represent the correlation coefficients, and length(w) represents the length of term w; BW represents the set of bursty terms sorted in ascending order by burstiness score, TBScore(w' n ) represents the score of the nth burst term in ascending order, Q3(BW) represents the burst score of the term in the upper quartile, Q1(BW) represents the burst score of the term in the lower quartile, DifVal(BW) represents the quartile difference of the burst scores of the terms in the burst term set BW; ThVal(BW) represents the threshold of the burst term set BW, SelBurWord(w) represents the screening result of the burst term, the result of 1 represents that the term w is a burst word, and the result of 0 represents that the term w is not a burst word.

9. The method for detecting burst topics based on the HVCH model and the Burst_NBT model according to claim 8, characterized in that: The step S32 includes the following sub-steps: Step S321, word vectorization, using the word2vec model successfully trained in the skim_gram mode to vectorize each burst word, and use a multi-dimensional vector to represent a single burst word; Step S322, term clustering, using the K-means algorithm to randomly select multiple burst words from the burst word set as centroids, and calculate the vector distances between the remaining burst words and the centroids, classify the burst words and the corresponding centroids into one category according to the shortest distance, and obtain multiple burst word clusters under each topic through multiple iterations; The calculation formula is: in, Represents the vector of the i-th term, v in Representative term w i The vector value in the nth dimension, represents the cluster center of the j-th word cluster, Represents the total number of words in this cluster. The vector representing the terms in this cluster, v jn Represents the vector value of the cluster in the nth dimension, d(w i ,C j ) represents the term w i .

10. The method for detecting burst topics based on the HVCH model and the Burst_NBT model according to claim 9, characterized in that: The step S33 includes the following sub-steps: Step S331, scoring the burst word cluster, using the cumulative frequency score of the burst word cluster, the relevant user score, the relevant comment and the forwarding score to obtain the final score of the burst word cluster; Step S332, screening the burst topics, ranking the burst word clusters according to the scores of the burst word clusters, and selecting the burst word clusters with the highest ranking as the final detected burst topics; The calculation formula is: TScore(BWC i )=Fre(BWC i )+UserNum(BWC i )+Post(BWC i ); Among them, Fre(BWC i ) represents the i-th burst word cluster BWC i The cumulative frequency score of fre(w) represents the burst word cluster BWC i The frequency of occurrence of the term w in User(BWC i ) represents the i-th burst word cluster BWC i The relevant user rating of , UserNum(w) represents the number of users who mentioned the term w, CT(BWC i ) represents the i-th burst word cluster BWC i ComNum(w) represents the number of comments associated with term w, TraNum(w) represents the number of forwardings associated with term w, and TScore(BWC i ) represents the i-th burst word cluster BWC i Total rating.

Images