Public opinion topic processing method and device based on fast BTM

Abstract

The present invention discloses a method and device for processing public opinion topics based on fast BTM. The method for processing public opinion topics includes: step S1: screening qualified analysis texts in a massive text library; step S2: after preprocessing the analysis texts, performing BTM topic inference to obtain keywords and probabilities of multiple topics; step S3: dividing the topics according to the keywords and probabilities of multiple topics and then merging the topics; step S4: obtaining the text similarity between the text and keywords under the topic according to the merged topics, and selecting the analysis text with the highest similarity as the typical text of the topic.

Description

Technical Field

[0001] The present invention relates to a method and device for processing public opinion topics, and more specifically, to a method and device for processing public opinion topics based on fast BTM. Background Art

[0002] With the development of Internet technology, a large amount of text content has been generated on the Internet, and a large number of spontaneous public opinion on the Internet is concentrated in social media. How to quickly master the theme content in a large number of social media texts, cluster texts on similar topics, and correctly understand the main content discussed on different topics has become an important part of building current big data public opinion analysis capabilities. However, the existing public opinion analysis has the following disadvantages:

[0003] 1) The calculation speed of the existing technology is slow;

[0004] 2) The prior art does not combine similar topics;

[0005] 3) The existing technology does not automatically summarize the topics.

[0006] Therefore, there is an urgent need to develop a method and device for processing public opinion topics that overcome the above-mentioned defects. Summary of the invention

[0007] In view of the above problems, the present invention provides a method for processing public opinion topics based on fast BTM, which includes:

[0008] Step S1: Screening qualified analysis texts in a massive text library;

[0009] Step S2: After preprocessing the analysis text, perform BTM topic inference to obtain keywords and probabilities of multiple topics;

[0010] Step S3: dividing the topics according to the keywords and probabilities of multiple topics and then merging the topics;

[0011] Step S4: Obtain the text similarity between the text and keywords under the theme according to the merged theme, and select the text with the highest similarity as the typical text of the theme.

[0012] In the above-mentioned method for processing public opinion topics, in step S1, qualified analysis texts are selected by setting any keywords, time periods and data source platforms.

[0013] In the above method for processing public opinion topics, step S2 includes:

[0014] Step S21: setting the expected maximum number of topic divisions;

[0015] Step S21: preprocessing the analysis text;

[0016] Step S23: Use the random variational Bayesian inference method to perform BTM topic inference to obtain keywords and probabilities of multiple topics.

[0017] In the above method for processing public opinion topics, step S3 includes:

[0018] Step S31: Obtain the probability that each analysis text belongs to each topic according to the keywords and probabilities of multiple topics, and select the topic with the greatest probability as the topic to which the analysis text belongs;

[0019] Step S32: Calculate the text similarity between the topics based on the keywords of the multiple topics, and merge the topics with similarity higher than a threshold.

[0020] In the above method for processing public opinion topics, step S4 includes:

[0021] Step S41: vectorize the text belonging to the topic using the tf-idf method;

[0022] Step S42: Merge the keywords of the subject into one text;

[0023] Step S43: Calculate the similarity between each text and the topic keyword text using cosine similarity;

[0024] Step S44: sorting the text similarities;

[0025] Step S45: Select the text with the highest similarity as the typical text output of the topic.

[0026] The present invention also provides a public opinion topic processing device based on fast BTM, which includes:

[0027] Screening unit, which screens the analysis texts that meet the conditions in the massive text library;

[0028] A preprocessing and inference unit, which performs BTM topic inference on the analysis text after preprocessing to obtain keywords and probabilities of multiple topics;

[0029] Classification and merging unit: divide the topics according to the keywords and probabilities of multiple topics and then merge the topics;

[0030] The typical text output unit obtains the text similarity between the text and keywords under the topic according to the merged topic, and selects the text with the highest similarity as the typical text of the topic.

[0031] In the above-mentioned public opinion topic processing device, the screening unit selects analysis texts that meet the conditions by setting any keywords, time periods and data source platforms.

[0032] In the above-mentioned public opinion topic processing device, the preprocessing and inference unit includes:

[0033] Set the module and set the expected maximum number of topic divisions;

[0034] Text processing module, which pre-processes the analysis text;

[0035] The inference module uses the random variational Bayesian inference method to perform BTM topic inference to obtain keywords and probabilities of multiple topics.

[0036] In the above-mentioned public opinion topic processing device, the classification merging unit includes:

[0037] The division module obtains the probability of each analysis text belonging to each topic according to the keywords and probabilities of multiple topics, and selects the topic with the largest probability as the topic to which the analysis text belongs;

[0038] The merging module calculates the text similarity between topics based on the keywords of multiple topics and merges the topics with similarity higher than the threshold.

[0039] In the above-mentioned public opinion topic processing device, the typical text output unit includes:

[0040] The vectorization module uses the tf-idf method to vectorize the text belonging to the topic;

[0041] Keyword merging module, which merges the keywords of the topic into one text;

[0042] The similarity calculation module uses cosine similarity to calculate the similarity between each text and the topic keyword text;

[0043] Sorting module, sorting text similarity;

[0044] The selection module selects the text with the highest similarity as the typical text output of the topic.

[0045] In summary, compared with the prior art, the effectiveness of the present invention lies in: through the public opinion processing method and device of the present invention, the calculation speed is significantly improved; at the same time, similar topics can be automatically merged to improve people's cognitive perception of the topic, and the topic can also be automatically summarized.

[0046] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flow chart of the public opinion processing method of the present invention;

[0049] Figure 2 for Figure 1 The step-by-step flow chart of step S2;

[0050] Figure 3 for Figure 1 A step-by-step flow chart of step S3;

[0051] Figure 4 for Figure 1 The step-by-step flow chart of step S4;

[0052] Figure 5 It is a structural schematic diagram of the public opinion processing device of the present invention.

[0053] Wherein, the accompanying drawings are marked as;

[0054] Screening Unit 11

[0055] Preprocessing and inference unit 12

[0056] Setting Module 121

[0057] Text processing module 122

[0058] Inference module 123

[0059] Classification and merging unit 13

[0060] Division module 131

[0061] Merge Module 132

[0062] Typical text output unit 14

[0063] Vectorization Module 141

[0064] Keyword Merging Module 142

[0065] Similarity calculation module 143

[0066] Sorting module 144

[0067] Selection module 145 DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0070] The term "plurality" as used herein includes "two" and "more than two".

[0071] Please refer to Figure 1-Figure 4 , Figure 1 It is a flow chart of the public opinion processing method of the present invention; Figure 2 for Figure 1 The step-by-step flow chart of step S2; Figure 3 for Figure 1 A step-by-step flow chart of step S3; Figure 4 for Figure 1 The step-by-step flow chart of step S4 in FIG. Figure 1-Figure 4 As shown, the public opinion topic processing method based on fast BTM of the present invention includes:

[0072] Step S1: Screening qualified analysis texts in the massive text library.

[0073] Wherein, in the step S1, the analysis text that meets the conditions is selected by setting any keywords, time periods and data source platforms.

[0074] Step S2: After preprocessing the analysis text, perform BTM topic inference to obtain keywords and probabilities of multiple topics.

[0075] Wherein, the step S2 includes:

[0076] Step S21: setting the expected maximum number of topic divisions;

[0077] Step S21: preprocessing the analysis text;

[0078] Step S23: Use the random variational Bayesian inference method to perform BTM topic inference to obtain keywords and probabilities of multiple topics.

[0079] Step S3: Divide the topics according to the keywords and probabilities of multiple topics and then merge the topics.

[0080] Wherein, the step S3 includes:

[0081] Step S31: Obtain the probability that each analysis text belongs to each topic according to the keywords and probabilities of multiple topics, and select the topic with the greatest probability as the topic to which the analysis text belongs;

[0082] Step S32: Calculate the text similarity between the topics based on the keywords of the multiple topics, and merge the topics with similarity higher than a threshold.

[0083] Step S4: According to the merged topic, the text similarity between the text and the keywords under the topic is obtained, and the analysis text with the highest similarity is selected as the typical text of the topic.

[0084] Wherein, the step S4 includes:

[0085] Step S41: vectorize the text belonging to the topic using the tf-idf method;

[0086] Step S42: Merge the keywords of the subject into one text;

[0087] Step S43: Calculate the similarity between each text and the topic keyword text using cosine similarity;

[0088] Step S44: sorting the text similarities;

[0089] Step S45: Select the text with the highest similarity as the typical text output of the topic.

[0090] The public opinion processing method of the present invention is described below with a specific embodiment:

[0091] First, qualified analysis texts are screened according to keywords in the massive text library. In this embodiment, qualified analysis texts are selected by setting any keywords, time periods, and data source platforms.

[0092] Secondly, pre-process the analysis text, set the expected maximum number of topic divisions K, and prepare for topic mining;

[0093] Among them, the preprocessing of the analytical text includes: unified conversion of English to lowercase, conversion of traditional Chinese characters to simplified Chinese, removing characters other than Chinese English numbers, removing expressions, removing urls, removing forward user names, removing text with a cleaned length less than 3, loading user dictionary, word segmentation, filtering stop words, counting word frequency of all words after word segmentation, removing words with a word frequency below 2, removing words with only one word, removing text with only two words after the above processing, and writing text with a space separated into a txt file as input for subject inference, but the present invention is not limited to this.

[0094] Then, the random variational Bayesian inference method is used to perform BTM topic inference to improve the speed of topic inference. The inference method is as follows:

[0095] 1) Randomly initialize N k and N k，w , N k is the sum of the probabilities of all bigrams being assigned to topic K; N k，w is the sum of the probabilities of all bigrams containing word w being assigned to topic K;

[0096] 2) Set the time step t = 1;

[0097] 3) Perform the following operation on all b∈B (where b refers to a bigram and B refers to all bigrams):

[0098] Calculate γ bk , where γ bk Represents the probability of assigning a bigram b to topic k based on observable data:

[0099]

[0100] where α and β are the prior parameters of the Dirichlet distribution, V is the vocabulary size, is the probability that the first word of bigram b is assigned to topic K, is the probability that the second word of the bigram b is assigned to topic K;

[0101] Update N k and N k，w ;

[0102]

[0103]

[0104] where |B| is the size of the bigram;

[0105]

[0106] where τ and κ are the hyperparameters of the Robbins-Monro sequence, ρ t is the weight of each iteration

[0107]

[0108]

[0109] Update time step is t+1;

[0110] End the loop;

[0111] 4) Calculate the probability of topic distribution in the corpus:

[0112] θ k ∝N k +α

[0113] where θ k represents the distribution of topic K;

[0114] 5) Calculate the probability that each word belongs to each topic:

[0115] φ k，w ∝n k，w +β,

[0116] where φ k，w represents the probability that word w belongs to topic k;

[0117] Next, we obtain the keywords and probabilities of K topics, as well as the probability of each article (text) belonging to each topic, and select the topic with the highest probability as the topic to which the article belongs;

[0118] Specifically, after the inference is completed, the probability of each K topic and the probability of each word belonging to each topic are obtained through the formula:

[0119]

[0120] Where P(z|d) represents the distribution of article d on different topics; P(z|b) represents the distribution of bigram b on different topics; P(b|d) represents the distribution of article d on bigrams;

[0121]

[0122]

[0123] Where P(w i |z) represents the distribution of topics on different bigrams; n d (b) represents the frequency of occurrence of bigram b in article d.

[0124] The probability of each text belonging to each topic can be calculated. Finally, for each text, a K-dimensional vector is obtained, each dimension of the vector represents the probability that the text belongs to topic i, and the largest dimension among the K dimensions is selected as the topic classification of the final text.

[0125] Based on the TOPN keywords of K topics, the text similarity between each topic is calculated, a threshold is set, and topics with similarity higher than the threshold are merged. Specifically, the TOPN keywords of each topic are merged into one text, the text is vectorized using the tf-idf method, the cosine similarity between the K topics is calculated, and the topics with similarity higher than the threshold are merged and mapped.

[0126] Finally, based on the merged topic, the text similarity between the text under the topic and the topic's TOPN keywords is calculated, and the N articles with the highest similarity are selected as typical texts of the topic.

[0127] For each of the merged topics:

[0128] 1) Use the tf-idf method to vectorize the text belonging to the topic;

[0129] 2) Combine the topic's top keywords into one text;

[0130] 3) Use cosine similarity to calculate the similarity between each text and the topic keyword text;

[0131] 4) Sort the text similarity;

[0132] 5) Select the top N texts with the highest similarity as the typical text output of the topic.

[0133] Please refer to Figure 5 , Figure 5 Schematic diagram of the structure of the public opinion processing device of the present invention. Figure 5 As shown, the public opinion topic processing device based on fast BTM of the present invention includes:

[0134] A screening unit 11 screens analysis texts that meet the conditions in a massive text library;

[0135] A preprocessing and inference unit 12, after preprocessing the analysis text, performs BTM topic inference to obtain keywords and probabilities of multiple topics;

[0136] A classification and merging unit 13 divides the topics according to the keywords and probabilities of the multiple topics and then merges the topics;

[0137] The typical text output unit 14 obtains the text similarity between the text and the keywords under the theme according to the merged theme, and selects at least one analysis text with the highest similarity as the typical text of the theme.

[0138] The screening unit 11 selects analysis texts that meet the conditions by setting any keywords, time periods and data source platforms.

[0139] Furthermore, the preprocessing and inference unit 12 includes:

[0140] Setting module 121, setting the expected maximum number of topic divisions;

[0141] A text processing module 122 performs preprocessing on the analysis text;

[0142] The inference module 123 uses the random variational Bayesian inference method to perform BTM topic inference to obtain keywords and probabilities of multiple topics.

[0143] Furthermore, the classification merging unit 13 includes:

[0144] A division module 131 obtains the probability that each analysis text belongs to each topic according to the keywords and probabilities of the multiple topics, and selects the topic with the largest probability as the topic to which the analysis text belongs;

[0145] The merging module 132 calculates the text similarity between the topics according to the keywords of the multiple topics, and merges the topics with similarity higher than a threshold.

[0146] Furthermore, the typical text output unit 14 includes:

[0147] A vectorization module 141 vectorizes the text belonging to the topic using a tf-idf method;

[0148] A keyword merging module 142 merges the keywords of the subject into one text;

[0149] A similarity calculation module 143 calculates the similarity between each text and the subject keyword text using cosine similarity;

[0150] A sorting module 144, sorting text similarities;

[0151] The selection module 145 selects at least one text with the highest similarity as a typical text output of the topic.

[0152] In summary, the present invention uses the random variational Bayes inference method to improve the topic inference speed of BTM; at the same time, similar topics inferred by the model are automatically merged to enhance people's cognitive perception of the topic; and a method based on text similarity is used to select typical articles under each topic as topic representatives, thereby automatically summarizing the topic.

[0153] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing public opinion topics based on fast BTM, characterized in that: include: Step S1: Screening qualified analysis texts in a massive text library; Step S2: After preprocessing the analysis text, perform BTM topic inference to obtain keywords and probabilities of multiple topics; Step S3: dividing the topics according to the keywords and probabilities of multiple topics and then merging the topics; Step S4: obtaining the text similarity between the text and the keywords under the theme according to the merged theme, and selecting the text with the highest similarity as the typical text of the theme; Wherein, the step S2 includes: Step S21: setting the expected maximum number of topic divisions; Step S21: preprocessing the analysis text; Step S23: Use random variational Bayesian inference method to perform BTM topic inference to obtain keywords and probabilities of multiple topics; Wherein, the step S3 includes: Step S31: Obtain the probability that each analysis text belongs to each topic according to the keywords and probabilities of multiple topics, and select the topic with the greatest probability as the topic to which the analysis text belongs; Step S32: Calculate the text similarity between the topics based on the keywords of the multiple topics, and merge the topics with similarity higher than a threshold.

2. The method for processing public opinion topics according to claim 1, characterized in that: In the step S1, analysis texts meeting the conditions are selected by setting any keywords, time periods and data source platforms.

3. The method for processing public opinion topics according to claim 1, characterized in that: The step S4 includes: Step S41: vectorize the text belonging to the topic using the tf-idf method; Step S42: Merge the keywords of the subject into one text; Step S43: Calculate the similarity between each text and the topic keyword text using cosine similarity; Step S44: sorting the text similarities; Step S45: Select the text with the highest similarity as the typical text output of the topic.

4. A public opinion topic processing device based on fast BTM, characterized in that: include: Screening unit, which screens the analysis texts that meet the conditions in the massive text library; A preprocessing and inference unit, which performs BTM topic inference on the analysis text after preprocessing to obtain keywords and probabilities of multiple topics; Classification and merging unit: divide the topics according to the keywords and probabilities of multiple topics and then merge the topics; The typical text output unit obtains the text similarity between the text and keywords under the topic according to the merged topic, and selects the text with the highest similarity as the typical text of the topic; Wherein, the preprocessing and inference unit includes: Setting module, setting the expected maximum number of topic divisions; text processing module, pre-processing the analysis text; Inference module, using random variational Bayesian inference method to perform BTM topic inference to obtain keywords and concepts of multiple topics; Wherein, the classification merging unit comprises: The division module obtains the probability of each analysis text belonging to each topic according to the keywords and probabilities of multiple topics, and selects the topic with the largest probability as the topic to which the analysis text belongs; The merging module calculates the text similarity between topics based on the keywords of multiple topics and merges the topics with similarity higher than the threshold.

5. The public opinion topic processing device according to claim 4, characterized in that: The screening unit selects analysis texts that meet the conditions by setting any keywords, time periods and data source platforms.

6. The public opinion topic processing device according to claim 4, characterized in that: The typical text output unit includes: The vectorization module uses the tf-idf method to vectorize the text belonging to the topic; the keyword merging module merges the keywords of the topic into one text; The similarity calculation module uses cosine similarity to calculate the similarity between each text and the topic keyword text; Sorting module, sorting text similarity; The selection module selects the text with the highest similarity as the typical text output of the topic.

Images