Real-time network hot topic mining and analysis and public opinion extraction system

By constructing a high-performance web hotspot crawler system, and combining sliding window block density vision and near block removal methods, we have achieved real-time and efficient discovery and public opinion analysis of web hotspots. This solves the problem of difficulty in extracting structured information and identifying web hotspots in real time in existing technologies, and enables accurate identification and public opinion analysis of web hotspots.

CN117786249BActive Publication Date: 2024-11-15BEIJING XINHUA MULTIMEDIA DATA CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311813831.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-27
Publication Date
2024-11-15
Estimated Expiration
2043-12-27

AI Technical Summary

Technical Problem

Existing technologies are unable to efficiently extract structured information from massive amounts of internet data in real time, identify online hotspots, and lack targeted noise processing and public opinion analysis methods, making it difficult to achieve real-time hot topic discovery and public opinion analysis.

Method used

A web hotspot crawler system was designed, which uses MySQL cluster storage, HBase and ES cluster storage for public opinion documents, Kafka as a message queue, and combines sliding window word block density visual and approximate block clearing methods to extract the main text. Keyword clustering and topic generation are achieved through a chained linear Chinese word segmentation model and keyword election method. The approximation degree hashing method is used to process massive documents.

Benefits of technology

It enables real-time and efficient discovery and public opinion analysis of online hot topics, accurately identifies trending topics, and is versatile and high-performance, capable of capturing trending information across the entire Chinese online media landscape.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117786249B_ABST
    Figure CN117786249B_ABST
Patent Text Reader

Abstract

The network real-time hot topic mining and analysis and public opinion extraction system of the present application can discover hot information for specific topics and fields, and realize a network hot topic crawler system. It ensures the real-time nature of data through a high-performance architecture, uses a chain linear Chinese word segmentation model and a frequency-based keyword selection method to find keywords, measures the popularity of keywords by their relative time density, and uses a keyword induction clustering method based on co-occurrence to realize a keyword-based topic generation strategy, as well as a mass document merging method based on proximity hashing. It uses noise processing of the initial page, text clustering in a specific field, personalized recognition of place names, real-time keyword selection and extraction, fragment and phrase extraction based on word frequency, topic screening and extraction, topic generation based on co-occurrence, and the extinction and merging of topics to accurately and timely capture the entire Chinese network media network hot spots.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to a network hot topic mining and public opinion extraction system, and in particular to a network real-time hot topic mining and analysis and public opinion extraction system, belonging to the technical field of network public opinion extraction. Background Art

[0002] With the rapid development of Internet information technology, the number of hot spots on the Internet is growing explosively. The new Moore's Law describes the rapid development of information in the Internet era. The amount of Internet information doubles almost every month. With the popularization of the Internet, more and more users are beginning to use the Internet as a channel to obtain information and a platform to express their personal opinions. Internet hot spots platforms include personal Weibo, forums such as Tianya, and major news sites. More and more users are beginning to use online social platforms and media platforms, and the speed of information transmission of Internet media far exceeds that of traditional media. As a result, more and more hot events are discovered and promoted in the Internet world. The Internet has become the most important source of information for public opinion.

[0003] Since Internet information does not have a unified structured data storage format, and the amount of information is huge, accompanied by a lot of noise and advertising information, how to extract structured information from massive data and parse network hot spots and extract hot spots is a difficult problem for current real-time hot spot information discovery systems.

[0004] Most of the current research focuses on Weibo, which is limited to a single public opinion platform, and there is no method for discovering real-time hot topics. The problems that need to be solved in the existing technology of real-time hot topic mining and public opinion extraction on the Internet and the key technical difficulties of this application include:

[0005] (1) Due to the rapid spread of Internet information, the amount of information is very large, and an efficient and real-time information collection system is needed to collect related network hot spots. The crawlers in the prior art generally crawl a certain site or the entire network, which is very resource-intensive and may not be able to find network hot spots very efficiently. In addition, since web pages are based on HTML structures and contain a large amount of advertisements and irrelevant information, such as navigation information bars and some advertising text information for product promotion, all information will be included in the HTML page, and these noise information are unnecessary. Therefore, a very high-performance page text extraction method is required to screen the information on the public opinion page. The prior art is unable to discover and process the noise of network hot spots in real time, lacks targeted network hot spot discovery and initial page noise processing methods, and especially lacks an efficient public opinion analysis system in the face of massive network data.

[0006] (2) Many platforms on the Internet, such as Weibo and forums, are important sources of information for hotspot discovery. However, this information is often more colloquial, trendy, and personalized in expression. It has a very rich form of expression, and new words are constantly being born, accompanied by a large number of abbreviations and special symbols, emoticons, links, etc. In the hotspot analysis system, it is not only necessary to analyze Weibo as a platform, but also to obtain network hotspots from various different information platforms. Each platform has its own characteristics and different information formats, which brings great challenges to public opinion analysis. The existing technology lacks a public opinion hot topic analysis and generation system, lacks text clustering in specific fields, personalized recognition of place names, chain linear Chinese word segmentation models, real-time keyword selection and extraction, word frequency fragment and phrase extraction, topic screening and extraction, co-occurrence topic generation, topic extinction and merging methods, and the hotspot extraction sensitivity is low and the error rate is high.

[0007] (3) The prior art lacks a high-performance network hotspot crawler architecture, is unable to capture network hotspots in the network in a timely manner, lacks Mysql clusters to build relational data storage, lacks Hbase clusters and es clusters to store public opinion documents, lacks Kafka as a message queue for message transmission middleware, and is unable to carry huge information traffic and support huge data volumes; the prior art lacks a text extraction method based on sliding window word block density vision, lacks an efficient public opinion hot topic parsing and generation method, lacks a hot topic discovery, merging and extinction strategy, cannot use left and right mutual information and information entropy to extract phrases and select keywords, lacks heat analysis and screening of keywords, cannot use the co-occurrence of keywords to cluster keywords to form a word group, cannot use the number of documents associated with the topic to formulate a topic generation and extinction method, and lacks the use of proximity hashing method to solve the problem of approximate document merging of massive documents. Summary of the invention

[0008] Based on the current hot topic discovery, most of them are targeted at a specific media platform, and it is impossible to achieve real-time hot public opinion discovery. This application designs and implements a hot topic discovery system, and can discover hot information for specific topics and fields. A universal network hotspot crawler system is implemented, which ensures the real-time nature of data through a high-performance architecture, and uses a chain linear Chinese word segmentation model and a frequency-based keyword selection method to find keywords. The relative time density of keywords is used to measure the heat of keywords, and a keyword induction clustering method based on co-occurrence is adopted to implement a keyword-based topic generation strategy, as well as a massive document merging method based on proximity hashing. Through network hotspot discovery, noise processing of the initial page, text clustering in a specific field, personalized recognition of place names, chain linear Chinese word segmentation models, real-time keyword election and extraction, fragment and phrase extraction based on word frequency, topic screening and extraction, topic generation based on co-occurrence, and the extinction and merging of topics, hot events are accurately discovered in real time, hot topics are found, and the system is universal and can capture the entire Chinese network media network hotspot.

[0009] In order to achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0010] The real-time network hot topic mining and analysis and public opinion extraction system includes the real-time network hot topic mining and noise processing, including network hot topic mining and initial page noise processing; the second is the analysis and generation of public opinion hot topic, including pre-processing, text clustering in specific fields, personalized recognition of place names, serial linear Chinese word segmentation model, real-time keyword selection and extraction, fragment and phrase extraction based on word frequency, topic screening and extraction, topic generation based on co-occurrence, and topic extinction and merging;

[0011] S1: Build a high-performance network hotspot crawler architecture to capture network hotspots in a timely manner. Use MySQL cluster to build relational data storage, and use Hbase cluster and es cluster to store public opinion documents. Use Kafka as a message queue and middleware for message transmission. The system carries huge information flow and supports huge data volume.

[0012] S2: Construct a text extraction method based on sliding window word block density vision, which can directly extract the text of theme websites without pre-training models. As a supplement, based on the fact that forum websites all use common templates, a clearing method based on approximate blocks is proposed to extract the text, and the two methods are combined;

[0013] S3: Analysis and generation of hot topics in public opinion. It adopts the strategy of discovering, merging and eliminating hot topics, finds key fragments based on frequency, extracts phrases using left-right mutual information and information entropy, and selects keywords. It performs heat analysis and screening on keywords, clusters keywords using their co-occurrence to form a word group, and summarizes them into a topic. It uses the number of documents associated with the topic to formulate the topic generation and elimination method, and finally uses the proximity hashing method to solve the problem of approximate document merging of massive documents.

[0014] Preferably, the cloud architecture crawler includes the following modules: a page collection module, a task cloud scheduling module, a page cloud storage module, and a page screening module. The page collection module initiates an http request to obtain HTML data. The task cloud scheduling manages new tasks based on cloud computing and distributes tasks to crawler sub-nodes. The page screening module screens pages that do not need to be downloaded, and finally the page cloud storage module stores the data in the cloud for subsequent query and processing.

[0015] Task cloud scheduling and screening strategy: When the network hotspot crawler crawls a news page, it determines whether it is a public opinion page. During the storage process, the URL of the page will be stored. When downloading, it will be determined whether it has been downloaded. If it is a news page that has been downloaded, it will not be downloaded again. Hbase is used to store URLs and is used for duplication checks. After the page collection module parses the HTML page, it reports all URLs to the task cloud scheduling module. After the task cloud scheduling module screens out some duplicate and irrelevant URLs, it releases the URLs to the queue in the form of tasks, and then the page collection cluster competes to obtain the tasks in the queue.

[0016] In addition, all pages are divided into three types: information source seed pages, public opinion pages and other pages. Public opinion pages are the required pages with network hotspots. All real-time public opinion data that need to be analyzed come from public opinion pages. Seed pages are pages published by all network hotspots. All public opinion pages can be found from seed pages, and seed pages are constantly updated and published with new public opinion pages. Seed pages are the entrance to public opinion pages. Other pages are advertising pages or irrelevant pages. When parsing pages, you only need to care about the outbound links of the information source pages. When scheduling tasks in the cloud, if the seed page is obtained, prevent the screening program from filtering it out.

[0017] Preferably, a text extraction method based on sliding window block density vision: The sliding window block density vision method is based on the assumption that the text of the page is relatively close in physical position, based on the inference that most of the text codes are close in position and are mostly text, and the length ratio of hyperlinks is not high, and includes the following steps:

[0018] Step 1: Remove all HTML tags, leaving only the text part and blank information;

[0019] Step 2: Specify a sliding window of size K, start sliding from the initial row, record the position of each window and the length of all text blocks in the window, each sliding window contains all data from row i to row i+K, if there is no text in the window, it is a blank window, otherwise it is a text window;

[0020] Step 3: Merge adjacent blank windows and adjacent text windows;

[0021] Step 4: Select the largest block of text, which is the main text.

[0022] The text extraction method of approximate node removal: the following inferences are used for screening:

[0023] Corollary 1: Forum websites are mostly based on templates;

[0024] Inference 2: Based on the URL, similar sibling pages can be found;

[0025] Inference 3: Most templates have similar strings or tag sequences;

[0026] Inference 4: The main text has significantly more words or punctuation;

[0027] Cache this type of page and use the URL to find its sibling nodes, parse the DOM tree of the page, locate the text part using four features: string, tag sequence, punctuation and text length, and remove redundant DOM nodes;

[0028] Other special treatments: adopt a blacklist strategy, detect the blacklist when crawling, and no longer download if it is a website on the blacklist. Formulate some characteristic URL filters, and all URLs that meet these characteristics will be filtered out.

[0029] Preferably, personalized recognition of place names: extract each text by region, distribute it by region, and extract hot spots by region. For regions that cannot be extracted, treat them as unknown regions;

[0030] 1) Named Entity Recognition: Personalized recognition of Chinese place names is a problem of labeling the state transition of a random sequence and its implicit state sequence. By labeling the text sequence, the boundaries of the named entities are found, all possible named entities are identified, and a place name resolution model is used to recognize Chinese place names. The place name resolution chain is described by a five-tuple.

[0031] (S, O, A, B, π) Formula 1

[0032] (1) Where S represents the set of all states in the model, S = {S 1 ,S2 ,S 3 ,…,S N}, where N is the total number of all states in the set, and each state has a corresponding meaning, and each state can be converted to each other, using q t To represent the state at time t;

[0033] (2) O represents the set of observed values ​​of the actual output of the model, where the observed state is defined as V = {V 1 , V 2 , V 3 ,…,V M}, where M is the number of observations;

[0034] (3) A describes the state transition probability of the model and is a probability matrix, expressed as A = (a ij ), where a ij The probability of transitioning from state i to state j is described by the following formula:

[0035] a ij =p(q t+1 =S j |q t =S i ) 1≤i,j≤N,∑ j a ij =1, a ij ≥0 Formula 2

[0036] (4) B represents the matrix composed of the probabilities of the observed values, B = (b ij ), where b ij represents the probability that the output observation value is i in state j, that is, p = (Vj|Si), where i represents the hidden state, i is less than or equal to the number of states N, and j represents the observation value, which is less than or equal to M;

[0037] (5)π is the initialization probability vector, describing the probability distribution of the initial state, π={π i} represents the probability of being in the i-th state at the initial moment, π i =p(q 1 =S i );

[0038] In the place name resolution model, the implicit state at time t is only related to the previous time t-1, and the observed value at time t is also only constrained and affected by the implicit state at time t;

[0039] By parsing the annotations of the corpus, we obtain the annotated sequence, the transfer matrix, the role dictionary, and the initial state calculation, and calculate the corresponding probability matrices A, B, and π. When the sentence is personalized for place name recognition, word segmentation is performed first, and then the maximum possible hidden state is calculated, and possible place names are identified through dictionary matching.

[0040] Find the hidden state sequence with the highest probability through the observation value sequence, and find the observation sequence hidden state sequence P max , define the local maximum probability δ(i, t) as the maximum probability of reaching state i at time t. The sequence of hidden states corresponding to this value is the path with the maximum probability of reaching state i at time t. Assuming that at the initial moment, the observation value is 0, the local probability is:

[0041] δ(i,1)=π(i)b io Formula 3

[0042] And for time t, for the observation value O t have:

[0043]

[0044] During the calculation process, a pointer for backtracking the path is added, and the optimal path, that is, the optimal hidden state chain solution, can be obtained in one calculation;

[0045] 2) Personalized recognition of place names and feature pattern matching: After finding the corresponding hidden state chain, match the corresponding state and find the following four patterns: CF, CDF, CDEF, and GF, i.e., the pattern of single word / double word / triple word / full ground + place name suffix. Multi-pattern matching consists of three parts: goto table, fail table, and output table. The construction process of the goto table is as follows:

[0046] (1) For each pattern Pi[1,…,j] (1≤i≤m) in P, input it into the automaton in the order of the letters it contains;

[0047] (2) The starting state is D[0]. If the current state machine state is D[p] and there is no new state available for the next letter pi[k], then the total number of states max is increased by 1, and the transition D[p][pi[k]]=max;

[0048] (3) Continue adding the next letter to the state machine;

[0049] The goto table is a dictionary tree structure. The fail table describes what state should be transferred to after the state transfer fails during pattern matching. When building the jump table, find the prefix part of the longest other pattern string in the suffix and build the fail jump table. After matching the pattern string, build the output table, and continue to search for the state according to the suffix and record it in the output.

[0050] Preferably, the chain linear Chinese word segmentation model: given the chain linear Chinese word segmentation model and its observation value sequence, let it be x, find the probability of the implicit state sequence being y, and the feature function f j (y i-1 ,y i , x, i) is a positive real-valued function that describes the associated information of the state transition of a certain y at position i under the condition of a given observation sequence x. j represents the number of the feature function, indicating the label of the jth position. In the sequence labeling problem, the meanings of the four parameters it accepts are the observation sequence x, the current position i, and the state label y of the previous position. i-1 , current position label y i ;

[0051] Taking Chinese word segmentation as a word boundary state sequence labeling problem, we first define several specific tags to mark the text sequence. We use four tags, S, B, M, and E, which correspond to the four roles of single-word, beginning of word, middle of word, and end of word respectively. Single-word words are labeled with S, two-word words are labeled with the label sequence {B, E}, three-word words are labeled with {B, M, E}, and words with more than four words are labeled with the sequence {B, MM, E}. After labeling the text with the trained chain linear Chinese word segmentation model, the text is segmented according to the tags.

[0052] In addition, we also prepare our own feature templates, using macros to define each token of the output data, written as %x[row,col], where col represents the absolute column number and row represents the relative offset. There are two types of templates: U template and B template.

[0053] The total number of feature functions generated by each model is L*N, where N is the number of unique features expanded from a given template, and L is the number of output categories. When using the B template, the system automatically generates a token for the current output combined with the previous token, and the number of features generated can reach L*L*N;

[0054] In feature selection, select feature templates that can improve accuracy, including:

[0055] (1) Unary features, which are activated when the current character or its previous character or next character is a specific value, are represented as a sequence C -1 , C 0 , C1 ;

[0056] (2) Binary feature, which is activated when the current string and the previous string or the next string are specific characters, denoted as sequence C -1 C 0 and C 0 C 1 ;

[0057] (3) Jump feature, which is activated when the previous character and the next character are specific strings, i.e., C -1 C 1 ;

[0058] (4) Ternary feature, which is activated when the current character and the upper and lower characters form a specific string, i.e., C -1 C 0 C 1 ;

[0059] This application also considers the feature of state transition. Since when the state of the previous character is E or S, the state of the current character must be the beginning part of a word, which must be B or s, and cannot be M or E. And when the state of the previous character is B, then the state of the next character may be M or E, but cannot be or B. Therefore, the state of the previous character will affect the probability of the next state. This application adds a state transition template T to the feature template of the training model -1 T 0 .

[0060] Preferably, real-time keyword election and refinement:

[0061] (1) Keyword election: According to the order of text words, construct a window, let all adjacent words in the window vote in favor of this word, and construct a voting word graph of candidate words according to this rule. The steps include:

[0062] 1) For a text D, split it according to sentences: D = [S1, S2, S3,..., Sn];

[0063] 2) Tokenize each sentence, then annotate the part of speech of each word, and filter out candidate words according to the part of speech, retain verbs, nouns, and adjectives, remove auxiliary words, stop words, and meaningless words. Finally, for each sentence Si, obtain a keyword sequence [w1, w2, w3,..., wt];

[0064] 3) Construct a voting word graph for all candidate words, denoted as G = <V, E>, where V represents the set of nodes and E represents the set of edges. For the same sentence Si, all keywords have a directed edge to all words except themselves, that is, for VkESi, an edge <Vp, Vq(pAq)> is added to the E set;

[0065] 4) Finally, use the formula to iterate and vote, calculate the weight of each word, and take the first k as keywords. The formula is as follows:

[0066]

[0067] where ω ji It represents the weight of the edge from node j to node i and is normalized, while d is the damping coefficient, which is 0.85.

[0068] (2) Optimize voting weight assignment: including: 1) whether the keyword appears in the title; 2) the tf-idf value of the keyword;

[0069] Define the weight edge V<i,j> The weight is wa(vi,vj), and its formula is:

[0070]

[0071] Where f(vj) represents the importance of the word. If the word appears in the title, the word has the highest weight. If the characters in the word appear in the title, the weight is relatively lower, but higher than other cases. OUT(Vi) represents all the outgoing nodes pointed to by node i. In addition, a weight is calculated based on tf-idf:

[0072]

[0073] For each voting edge, the weight is the same, we have:

[0074]

[0075] Finally, the weight matrix ω is constructed based on ωa and ωb:

[0076]

[0077] ωij represents the weight score of node i voting for node j, where α, β, and γ are control coefficients that adjust their weight ratios.

[0078] (3) Model optimization for short texts: In the case of short texts, a sliding window approach is used with a window size of 10. Five words are taken forward and five words are taken backward. If five words cannot be taken, as many words as possible are taken. If the word appears multiple times, windows at different positions are calculated, and directed voting edges corresponding to these words are added to the word graph.

[0079] Preferably, segment and phrase extraction based on word frequency:

[0080] (1) Key segment extraction based on word frequency: When extracting hot information each time, obtain a batch of real-time data and calculate the documents in batches. First, divide all documents into single words according to the unary word segmentation method, remove symbols, special characters and stop words, and obtain a word set W;

[0081] The frequency of each word w in the word set is defined as P(w), which is equal to the frequency of all occurrences of the word in the document divided by the number of words in the document. The formula is defined as:

[0082]

[0083] The denominator d represents the document, the len function represents the number of characters in the document, and is also the length of the document. A critical value λ is set so that all words with a probability greater than λ are seed words. After calculating all the words, the seed word set seed is obtained. Each word in the seed is a word with a high repetition frequency in the document. First, some fragments containing high-frequency words are preliminarily selected, and then screening is performed from them;

[0084] Use segmentation characters and stop words to split the text into segments, use bitmap to mark the positions of these text segments, mark them as valid segments, and then filter out all candidate segment sets by merging adjacent valid segments;

[0085] Finally, all candidate segments are screened. If the segment contains the seed word, it is retained, otherwise it is discarded, and a high-frequency segment set is obtained. All segments in this set contain high-frequency words.

[0086] (2) Phrase extraction based on mutual information and left-right information entropy: First, segment each sentence, then remove stop words, and then calculate the frequency of each word for each sentence as a window, and then calculate the frequency of two adjacent words and three words appearing at the same time;

[0087] The mutual information is calculated using the point-to-point mutual information formula:

[0088]

[0089] The higher the probability of two words appearing at the same time, the higher the value of p(x, y). In the same document, the values ​​of p(x) and p(y) are fixed, the higher the mutual information value of the two words, the more related they are, and the greater the possibility of forming a phrase sequence.

[0090] Left-right entropy is used to evaluate phrases. The more word strings that can form phrases in the text, the more likely they will appear in different contexts. Therefore, more uncertain words will appear on the left or right side of the phrase. The left-right entropy is used to calculate the degree of uncertainty of the words on the left and right sides of the phrase. The greater the entropy, the higher the degree of uncertainty and the higher the possibility of forming a phrase. The entropy is calculated using the following formula:

[0091] H(X)=-∑ x∈X P(x)*log(P(x)) Formula 13

[0092] Calculate the left and right entropies separately, that is, calculate the entropy of all left words and the entropy of right words:

[0093] hl(phrase)=-∑p(leftword|phrase)*log 2 p(leftword|phrase)

[0094] hr(phrase)=-∑p(rightword|phrase)*log 2 p(rightword|phrase) formula 14

[0095] The mutual information between points and the entropy of the left and right information are used to score the phrase, and the phrase with the highest score is extracted as the extracted phrase:

[0096] score=mi+hl+hr Formula 15

[0097] After extracting the phrases, the extracted phrases are screened to retain only the phrases containing high-frequency seed words.

[0098] Preferably, the topic is screened and refined: first, a shielding dictionary is set to screen some common noise texts, then place names are screened, and finally a critical value limit is set on the number of documents containing the fragments and the source sites of the documents contained, so as to increase the credibility of the message and ensure the popularity of the message;

[0099] In addition, the time density of the text is calculated to determine whether the topic is a valid topic. The density of information is calculated by the ratio of the number of documents containing the keyword, phrase or fragment in a certain time period to the time. The formula is as follows:

[0100]

[0101] Where x represents the number of hours, and the denominator represents the number of documents containing the word within x hours. The greater the information density, the higher the popularity of the keyword. Relative density is used to reflect the relative change trend of popularity. Relative density is defined as:

[0102]

[0103] Where X is a period of time greater than x, and the density of the word within time x is determined relative to the trend within time X. When a new hot spot appears, the popularity of a related keyword will definitely increase significantly, so this formula can screen out relatively hot words.

[0104] Preferably, topic generation based on co-occurrence: based on the set of filtered keywords and key phrases as the seed for generating topics, a word group is used to define a topic. A word group contains several keywords and key phrases to describe the topic. After obtaining the keywords, calculate in which text each keyword appears, and use bitmap to record all document information containing the keyword, calculate the co-occurrence of each word, and for words with high co-occurrence, divide them into a word group. The co-occurrence of two words is described by the following formula:

[0105]

[0106] Where doc(xy) represents the number of documents containing the two words x and y, and the denominator counts the number of documents containing the word x or the word y. A critical value is set, and when the co-occurrence of two words is greater than the critical value, the two words are merged into one word group;

[0107] Finally, this word group is regarded as a hot topic found, and all related pages are stored in the database as pages of the discovered topic.

[0108] Preferably, the extinction and merging of topics: set a time threshold, and when it exceeds the threshold, the topic is judged to be extinct. To determine whether a topic is active, the number of related posts of the topic is used to determine, and the maximum number of related documents per day for each topic is recorded. When the number of documents on that day is significantly lower than the maximum number of documents, the topic is defined as dead;

[0109] In addition, when generating new topics, we merge topics based on their relevance to old topics, merge similar topics based on regions and keywords, and merge similar texts to avoid repeated calculations.

[0110] Calculate the document similarity, reduce the dimension of the high-dimensional feature vector, map it to a low-dimensional feature vector, and then quantify the similarity between documents through its Hamming distance, so as to eliminate similar pages. The Hamming distance defines the number of different characters in the same position in two strings. In the similarity measurement, it is only necessary to calculate the value obtained by XOR operation of two hash values, the number of bits that are 1. The larger the Hamming distance, the less similar the two texts are, and vice versa. The specific calculation steps are as follows:

[0111] 1) First, segment the text into words to get a word sequence, and then set a weight for each word, which is the frequency of each word. The word weight is set to the value of tf-idf. The more important a word is, the higher its word weight will be.

[0112] 2) Calculate the hash value to obtain the characteristic signature of each word;

[0113] 3) For the signature of each word of fixed length and the weight corresponding to each word, a feature vector W is obtained, where each bit in W corresponds to each bit of the hash value of the word. If it is 1, the feature vector at that position has a positive weight value, and if it is 0, it has a negative weight value;

[0114] 4) Merge the feature vectors of all words, that is, add each feature vector;

[0115] 5) Finally, perform dimensionality reduction, set each position greater than 0 in the feature vector to 1, and each position less than 0 to 0, to obtain a signature value;

[0116] For the signatures of two texts with similar hashes, calculate their Hamming distance. Set a critical value relative to the 64-bit hash. If the Hamming distance is within this critical value, the two texts are considered similar.

[0117] For massive documents, this application maintains an approximate hash signature pool. For each topic or merge, the approximation of the document to the existing article is calculated. For similar documents, they are directly merged and the similar articles of the document are marked, thus avoiding the display of a large number of identical documents.

[0118] Compared with the prior art, the innovations and advantages of this application are:

[0119] (1) Based on the current hot topic discovery, most of them are targeted at a specific media platform, and it is impossible to achieve real-time hot public opinion discovery. This application designs and implements a hot topic discovery system, and can discover hot information for specific topics and fields. A universal network hot spot crawler system is implemented, which ensures the real-time nature of data through a high-performance architecture, and uses a chain linear Chinese word segmentation model and a frequency-based keyword selection method to find keywords. The relative time density of keywords is used to measure the popularity of keywords, and a keyword induction clustering method based on co-occurrence is adopted to implement a keyword-based topic generation strategy, as well as a massive document merging method based on proximity hashing. Through network hot spot discovery, noise processing of the initial page, text clustering in a specific field, personalized recognition of place names, chain linear Chinese word segmentation model, real-time keyword selection and extraction, fragment and phrase extraction based on word frequency, topic screening and extraction, topic generation based on co-occurrence, and the extinction and merging of topics, hot events are accurately discovered in real time, hot topics are found, and the system is universal and can capture the entire Chinese network media network hot spots.

[0120] (2) This application proposes a high-performance network hotspot crawler architecture that can capture network hotspots in the network in a timely manner. It is the basis for real-time hotspot information discovery. It uses Mysql cluster to build relational data storage, and uses Hbase cluster and es cluster to store public opinion documents. Kafka is used as a message queue for message transmission middleware. The system carries a very large amount of information flow and can fully support a huge amount of data. This application proposes a text extraction method based on window density. This method is very flexible and efficient. It does not require pre-training models. It has very good text extraction efficiency for most theme-based websites. The accuracy rate is as high as 97% or more in the actual tested websites. However, this method is not ideal for non-theme websites such as forums. As a supplement, based on the fact that forum-based websites all use general templates, this application proposes a clearing method based on approximate blocks to extract text, which also achieves good results. The combination of these two methods ensures the efficiency and accuracy of text extraction, and generally achieves the expected goal.

[0121] (3) This application proposes a hot topic discovery technology, merging and extinction strategy. Key fragments are found based on frequency, phrases are extracted using left-right mutual information and information entropy, and a chain linear Chinese word segmentation model is used to select keywords in real time. Finally, the keywords are analyzed and screened based on their popularity, and the keywords are clustered using their co-occurrence to form a word group, which is summarized as a topic. This application uses the number of documents associated with the topic to formulate a topic generation and extinction strategy, and finally uses the proximity hashing method to solve the problem of approximate document merging of massive documents. Based on the experimental results, it can be found that this application is very timely in discovering hot spots, can achieve real-time hot spot information discovery, and is sensitive to hot events. A lot of hot spot information is accurately and timely excavated, and it has strong reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Figure 1 This is a diagram of the text length data in the Sohu News page window.

[0123] Figure 2 It is a flow chart for analyzing and generating hot topics in public opinion.

[0124] Figure 3 It is a diagram of place name identification sequence labels and their meanings.

[0125] Figure 4 This is a schematic diagram of the related hot topic information results discovered in the experiment.

[0126] Figure 5 It is a schematic diagram of the real-time calculation results of the experimental simulation topic. DETAILED DESCRIPTION

[0127] The following, in conjunction with the accompanying drawings, further describes the technical solution of the network real-time hot topic mining and analysis and public opinion extraction system provided by the present application, so that technical personnel in the field can better understand the present application and implement it.

[0128] Currently, the discovery of hot topics is mostly targeted at a specific media platform, and it is impossible to achieve real-time hot public opinion discovery. This application designs and implements a hot topic discovery system, and can discover hot information for specific topics and fields. First, a high-performance network hotspot crawler architecture is proposed, which can capture network hotspots in the network in a timely manner. This is the basis for real-time hot information discovery in this application. This application uses Mysql cluster to build relational data storage, and uses Hbase cluster and es cluster to store public opinion documents, and uses Kafka as a message queue as a middleware for message transmission. The system carries a very large amount of information flow and can fully support huge amounts of data.

[0129] Next, a text extraction method based on window density is proposed. This method is very flexible and efficient. It does not require prior model training and has very good text extraction efficiency for most theme-based websites. The accuracy rate is as high as over 96% in the actual tested websites. However, this method is not ideal for non-theme websites such as forums. As a supplement, based on the fact that forum-type websites all use universal templates, this application proposes a clearing method based on approximate blocks to extract text, which also achieves good results. The combination of these two methods ensures the efficiency and accuracy of text extraction, and generally achieves the desired goal.

[0130] Finally, the discovery technology, merging and extinction strategies of hot topics were proposed. Key fragments were found based on frequency, phrases were extracted using left-right mutual information and information entropy, the keywords were analyzed and screened for heat, and the keywords were clustered using the co-occurrence of keywords to form a word group and summarized as a topic. The generation and extinction strategies of topics were formulated using the number of documents associated with the topic, and finally the proximity hashing method was used to solve the problem of merging approximate documents of massive documents.

[0131] It can be found from the final experimental results that the application is very timely in discovering hot spots, can achieve real-time hot spot information discovery, and is sensitive to hot events, digging out a lot of hot spot information.

[0132] 1. Real-time discovery of network hot spots and noise processing

[0133] (I) Discovery of Internet Hotspots

[0134] A large-scale cloud architecture crawler technology based on information sources is used to obtain network hotspots in real time.

[0135] (1) Cloud Architecture Network Hotspot Crawler

[0136] Since the amount of information on the Internet is very large and information is generated every minute and every second, general crawlers cannot meet the needs of real-time acquisition of massive network hotspots. At this time, a cloud-based crawler cluster is urgently needed to crawl information from massive Internet pages.

[0137] The cloud architecture crawler includes the following modules: page collection module, task cloud scheduling module, page cloud storage module, and page filtering module. The page collection module initiates http requests to obtain HTML data. The task cloud scheduling manages new tasks based on cloud computing and distributes tasks to crawler sub-nodes. The page filtering module filters pages that do not need to be downloaded, and finally the page cloud storage module stores the data in the cloud for subsequent query and processing.

[0138] (2) Task cloud scheduling and screening strategy

[0139] Different from the crawler strategy of search engines, network hotspot crawlers focus on public opinion itself and are not interested in the page update strategy, but only in the network hotspots of the page. Therefore, when crawling news pages, network hotspot crawlers need to determine whether they belong to public opinion pages, focusing on news pages or a certain microblog of the user. During the storage process, the URL of the page will be stored, and when downloading, it will be determined whether it has been downloaded. If it is a news page that has been downloaded, it will not be downloaded again. This can greatly save bandwidth and prevent repeated downloading of resources. Hbase is used to store URLs and for duplication checks. The Hbase cloud architecture KV storage system has large data and frequent queries.

[0140] After parsing the HTML page, the page collection module reports all URLs to the task cloud scheduling module. After the task cloud scheduling module filters out some duplicate and irrelevant URLs, it releases the URLs to the queue in the form of tasks, and then the page collection cluster competes to obtain the tasks in the queue.

[0141] In addition, all pages are divided into three types: information source seed pages, public opinion pages and other pages. Public opinion pages are the pages with network hotspots that are required. All real-time public opinion data that need to be analyzed come from public opinion pages. Seed pages are pages where all network hotspots are published. All public opinion pages can be found from seed pages, and seed pages are constantly updated to publish new public opinion pages. Seed pages are the entrance to public opinion pages. Typical seed pages include news homepages, forum homepages, Weibo homepages, and search pages. Other pages are advertising pages or irrelevant pages. After the information source is introduced, a large number of useless downloads can be avoided. Moreover, when parsing the page, you only need to care about the outbound links of the information source page, which greatly reduces irrelevant information and greatly improves download efficiency. When scheduling the task cloud, if the seed page is obtained, it can also prevent the screening program from filtering it out.

[0142] (3) Anti-crawler strategy response

[0143] For many sites, crawlers will occupy a relatively high PV ratio and occupy server resources, so many sites will take certain technical measures to prevent others from obtaining data in batches through crawlers. Anti-crawler strategies include three aspects: user headers, user behavior, and website directory and data loading methods.

[0144] In response to the above anti-crawler strategies, this application also optimizes crawlers:

[0145] 1) For the anti-crawler strategy based on Headers, randomly add the UserAgent field when initiating an http request, and add a reference to the target website domain name to bypass this anti-crawler strategy;

[0146] 2) For the anti-crawler strategy based on user behavior, the IP proxy cluster method is adopted. When the crawler node crawls data, a certain proxy is dynamically used for access. A different IP is used each time the data is crawled. In addition, by simulating login and registering many different user accounts, the user is randomly changed on the page that requires login, thereby bypassing the anti-crawler mechanism based on user behavior;

[0147] 3) For the anti-crawler strategy based on data loading mode, since the crawler of this application is simply an HTML document obtained based on http request, many websites that use JavaScript to dynamically load resources cannot crawl related resources. Since the page information is dynamically generated by js, one way is to add a js engine to the crawler. Currently, some popular js engines include phantomjs, casperjs, etc. However, the introduction of the js engine will bring a very serious problem. Since the js loading content is unpredictable, some sites js operation will take up a lot of resources, so the use of the js engine will greatly reduce the efficiency of the crawler. Some pages even take several minutes to load, which is too wasteful of resources for the crawler, and the efficiency is too low. If the real-time standard is to be achieved, the scale of the page collection node needs to be greatly increased, and the cost increases exponentially. Although the content loaded by ajax is loaded by js, its essence is still http request, and the actual request is still http request. After obtaining the associated data, it is filled into the web page. Therefore, by parsing the js code or using the developer function of the browser, it is very easy to find the address requested by the js code, and the returned data is a json string, which can be easily parsed to obtain information such as the text title and time. Moreover, since the data returned by the ajax request is the latest, this application only needs to set the information source seed to the address corresponding to the ajax request, and visit it at regular intervals to obtain the latest public opinion data of the site.

[0148] (4) Storage of network hot documents

[0149] Since the amount of network hotspot data is very large, a single storage system cannot meet the storage requirements. This system uses Hbase to store the pages and text of public opinion. Since Hbase is a kv database and cannot support better relational queries, a Mysql cluster is used to store the download time, title, and release time of the document for relational queries. In addition, elastic search is used to store all documents for indexing document information to facilitate content-based queries.

[0150] Since the public opinion page has a unique URL page, all storage systems use the URL's hash value as the unique ID storage. Since the hash value is unique, this application only calculates whether the hash of the web page URL exists in the storage system when checking for duplicate pages.

[0151] (II) Noise processing of the initial page

[0152] Since the initial data downloaded by the crawler are all HTML documents, there will be a lot of noise, advertisements, etc. in the page, which will affect the analysis of the public opinion system. Therefore, it is necessary to denoise the page and extract the real public opinion data, that is, the main text part.

[0153] The existing text extraction methods have various shortcomings. For example, the DOM tree-based method has high requirements on the well-formedness of HTML and has a high tree traversal overhead. Moreover, there is no suitable training set for different pages, and the segmentation methods are different, so it is difficult to find a standard method. This system uses a sliding window block density analysis and approximate node removal method to extract the text, achieving a relatively good effect.

[0154] (1) Text extraction method based on sliding window block density vision

[0155] The sliding window block density visual method is based on the assumption that the body of the page is physically close to each other, based on the inference that most of the body codes are close to each other and are mostly text, and the length ratio of hyperlinks is not high. It includes the following steps:

[0156] Step 1: Since the text and tags in HTML are mixed together, this application needs to remove all HTML tags, leaving only the text part and blank information, such as blank lines;

[0157] Step 2: Specify a sliding window of size K, start sliding from the initial row, record the position of each window and the length of all text blocks in the window, each sliding window contains all data from row i to row i+K, if there is no text in the window, it is a blank window, otherwise it is a text window;

[0158] Step 3: Merge adjacent blank windows and adjacent text windows;

[0159] Step 4: Pick out the largest block of text, which is the body of the text;

[0160] This method relies on the density of the text and the length of the text in the window to make decisions, extracting the text in linear time o(n), and has excellent extraction efficiency for thematic websites. Figure 1 There are graphs of window size and line number for several pages, with line number on the horizontal axis and text length on the vertical axis.

[0161] This method randomly captured 1,000 news pages on various topics such as economy and society from mainstream online news media platforms, such as NetEase, Tencent, Sohu, Sina, etc., and conducted a text extraction test with an accuracy rate of up to 97%.

[0162] This method is HTML-independent, does not require the establishment of a DOM tree structure, and has nothing to do with whether the HTML is well-formed. It has low time complexity and only requires scanning the text once to calculate the associated information. It is very efficient and has a good effect.

[0163] (2) Text Extraction Method with Approximate Node Removal

[0164] Since the sliding window word block density method only has a good extraction effect on thematic web pages, it is not very suitable for many relatively sparse pages or pages with less dense text. Most of these websites are forum websites with sparse pages and loose word block density, making it difficult to perform density analysis. However, this type of site is basically based on templates, and then data is filled into the templates. For similar pages under this site, there are many duplicates. This application uses the following inferences for screening:

[0165] Corollary 1: Forum websites are mostly based on templates;

[0166] Inference 2: Based on the URL, similar sibling pages can be found;

[0167] Inference 3: Most templates have similar strings or tag sequences;

[0168] Inference 4: The main text has significantly more words or punctuation;

[0169] This type of page is cached and its sibling nodes are found using the URL. The DOM tree of the page is parsed, the body of the text is located using four features: string, tag sequence, punctuation, and text length, and redundant DOM nodes are removed.

[0170] (3) Other special treatments

[0171] Adopt a blacklist strategy, detect the blacklist during crawling, and stop downloading if it is a website on the blacklist. Formulate some characteristic URL filters, and all URLs that meet these characteristics will be filtered out.

[0172] 2. Analysis and Generation of Hot Topics of Public Opinion

[0173] The process includes the following: first cluster the text, then extract, filter and cluster the keywords and key phrases, and then generate topics. Figure 2 shown.

[0174] (I) Pre-processing

[0175] Due to the complexity of network hot spots, this application calculates the frequency of all characters in the text instead of words, as well as the documents where the character appears, and calculates the corresponding tf-idf value. Based on these calculated data, the approximate probability of occurrence of the character and the relative importance information of a character are obtained. For subsequent queries, if the character has not appeared, their word frequency returns the minimum value and the importance returns the maximum value. In the subsequent public opinion analysis, the tf-idf of the character is used to assist in analyzing the sentence proximity and clearing out words that appear frequently but have no analytical value or meaning.

[0176] 2.2 Text Clustering in Specific Fields

[0177] When a batch of documents arrives, the documents are classified and topics are generated in this batch of texts. User-defined thematic keywords are added to the system, and documents containing keywords are clustered into a batch of documents. Then hot topics are found from these documents. In addition, starting from the region, the regional information contained in the text information is identified, and the text is clustered according to the region.

[0178] In the hot spot analysis for specific topics, user-defined keywords are searched, and any text containing keywords is considered to be a related document of the topic. When analyzing regions, all regions are searched, and all regions related to the document are found according to the hierarchical relationship of regions. For overlapping regional relationships, more detailed regions are used; based on the related hot topics of concern, text clustering in specific fields is achieved.

[0179] 3. Personalized recognition of place names

[0180] Since the outbreak of hot spots is usually regional, this application extracts each text by region, distributes it by region, and extracts hot spots by region. For regions that cannot be extracted, they are treated as unknown regions.

[0181] 1. Named Entity Recognition

[0182] Personalized recognition of Chinese place names is a problem of labeling the state transition of a random sequence and its implicit state sequence. By labeling the text sequence, the boundaries of named entities are found, all possible named entities are identified, and a place name resolution model is used to identify Chinese place names. The place name resolution chain is described by a five-tuple.

[0183] (S, O, A, B, π) Formula 1

[0184] (1) Where S represents the set of all states in the model, S = {S 1 ,S 2 ,S 3 ,…,SN}, where N is the total number of all states in the set, and each state has a corresponding meaning, and each state can be converted to each other, using q t To represent the state at time t;

[0185] (2) O represents the set of observed values ​​of the actual output of the model, where the observed state is defined as V = {V 1 , V 2 , V 3 ,…,V M}, where M is the number of observations;

[0186] (3) A describes the state transition probability of the model and is a probability matrix, expressed as A = (a ij ), where a ij The probability of transitioning from state i to state j is described by the following formula:

[0187] a ij =p(q t+1 =S j |q t =S i ) 1≤i,j≤N,∑ j a ij =1,a ij ≥0 Formula 2

[0188] (4) B represents the matrix composed of the probabilities of the observed values, B = (b ij ), where b ij represents the probability that the output observation value is i in state j, that is, p = (Vj|Si), where i represents the hidden state, i is less than or equal to the number of states N, and j represents the observation value, which is less than or equal to M;

[0189] (5)π is the initialization probability vector, describing the probability distribution of the initial state, π={π i} represents the probability of being in the i-th state at the initial moment, π i =p(q 1 =S i );

[0190] In the place name resolution model, the implicit state at time t is only related to the previous time t-1, and the observed value at time t is only constrained and affected by the implicit state at time t.

[0191] The output of the word is taken as the observation value, and its hidden state is the corresponding word label. The word label is defined as Figure 3 .

[0192] By parsing the annotations of the corpus, we obtain the annotated sequence, the transfer matrix, the role dictionary, and the initial state calculation, and calculate the corresponding probability matrices A, B, and π. When the sentence is personalized for place name recognition, word segmentation is performed first, and then the maximum possible hidden state is calculated, and possible place names are identified through dictionary matching.

[0193] Find the hidden state sequence with the highest probability through the observation value sequence, and find the observation sequence hidden state sequence P max , define the local maximum probability δ(i, t) as the maximum probability of reaching state i at time t. The sequence of hidden states corresponding to this value is the path with the maximum probability of reaching state i at time t. Assuming that at the initial moment, the observation value is 0, the local probability is:

[0194] δ(i,1)=π(i)b io Formula 3

[0195] And for time t, for the observation value O t have:

[0196]

[0197] The use of dynamic programming to solve the hidden state chain reduces a lot of useless calculations and takes the entire sequence into consideration. In the calculation process, with the addition of a pointer for the backtracking path, the optimal path, that is, the optimal hidden state chain solution, can be obtained in one calculation.

[0198] 2. Personalized recognition of place names and feature pattern matching

[0199] After finding the corresponding hidden state chain, we need to match the corresponding state and find the following four modes: CF, CDF, CDEF, and GF, which are single-word / double-word / triple-word / full-ground+place name suffix modes. Multi-mode matching consists of three parts: goto table, fail table, and output table. The construction process of the Goto table is as follows:

[0200] (1) For each pattern Pi[1,…,j] (1≤i≤m) in P, input it into the automaton in the order of the letters it contains;

[0201] (2) The starting state is D[0]. If the current state machine state is D[p] and there is no new state available for the next letter pi[k], then the total number of states max is increased by 1, and the transition D[p][pi[k]]=max;

[0202] (3) Continue adding the next letter to the state machine;

[0203] The goto table is a dictionary tree structure. The fail table describes what state should be transferred to after the state transfer fails during pattern matching. When building the jump table, find the prefix part of the longest other pattern string in the suffix and build the fail jump table. After matching the pattern string, build the output table, and continue to search for the state according to the suffix and record it in the output.

[0204] It solves the efficiency problem of multi-mode matching in the problem of personalized place name recognition and greatly improves the efficiency of personalized place name recognition.

[0205] (IV) Chain Linear Chinese Word Segmentation Model

[0206] Given a chain linear Chinese word segmentation model and its observation sequence, let it be x, find the probability that the implicit state sequence is y, and the feature function f j (y i-1 ,y i , x, i) is a positive real-valued function that describes the associated information of the state transition of a certain y at position i under the condition of a given observation sequence x. j represents the number of the feature function, indicating the label of the jth position. In the sequence labeling problem, the meanings of the four parameters it accepts are the observation sequence x, the current position i, and the state label y of the previous position. i-1 , current position label y i ;

[0207] Taking Chinese word segmentation as a word boundary state sequence labeling problem, we first define several specific tags to mark the text sequence. We use four tags, S, B, M, and E, which correspond to the four roles of single-word, beginning of word, middle of word, and end of word respectively. Single-word words are labeled with S, two-word words are labeled with the label sequence {B, E}, three-word words are labeled with {B, M, E}, and words with more than four words are labeled with the sequence {B, MM, E}. After labeling the text with the trained chain linear Chinese word segmentation model, the text is segmented according to the tags.

[0208] In addition, this application also prepares its own feature template, using macros to define each token of the output data, written as %x[row, col], where col represents the absolute column number and row represents the relative offset. The template has two types: U template and B template.

[0209] The total number of feature functions generated by each model is L*N, where N is the number of unique features expanded from a given template, and L is the number of output categories. When using the B template, the system automatically generates a token for the current output combined with the previous token, and the number of features generated can reach L*L*N.

[0210] In feature selection, select feature templates that can improve accuracy, including:

[0211] (1) Unary features, which are activated when the current character or its previous character or next character is a specific value, are represented as a sequence C -1 , C 0 , C 1 ;

[0212] (2) Binary features, that is, the current string is activated when it is the same as the previous string or the next string, and is represented as a sequence C -1 C 0 and C 0 C 1 ;

[0213] (3) Jump feature, that is, it is activated when the previous character and the next character are a specific string, that is, C -1 C 1 ;

[0214] (4) Ternary feature, that is, it is activated when the current character and the upper and lower characters form a specific string, that is, C -1 C 0 C 1 ;

[0215] The present application also considers the characteristics of state transition. When the state of the previous character is E or S, the state of the current character must be the beginning of the word, so it must be B or S, and cannot be M or E. When the state of the previous character is B, the state of the next character may be M or E, but cannot be M or B. Therefore, the state of the previous character will affect the probability of the next state. The present application adds a state transition template T to the feature template of the training model. -1 T 0 .

[0216] (V) Real-time election extraction of keywords

[0217] (1) Keyword selection

[0218] According to the order of the words in the text, a window is constructed, and all adjacent words in the window are allowed to vote in favor of the word, and a voting word graph of the candidate words is constructed according to the rule. The steps include:

[0219] 1) For a text D, segment it into sentences: D = [S1, S2, S3, …, Sn];

[0220] 2) Segment each sentence, then mark the part of speech of each word, and select candidate words according to the part of speech, retain verbs, nouns, and adjectives, remove auxiliary words, stop words, and meaningless words, and finally obtain the keyword sequence [w1, w2, w3, ..., wt] for each sentence Si;

[0221] 3) Construct a voting word graph for all alternative words, denoted as G = <V, E>, where V represents the set of nodes and E represents the set of edges. For all keywords in the same sentence Si, there is a directed edge from each keyword to every other word (excluding itself). That is, for Vk ∈ Si, an edge <Vp, Vq (p ≠ q)> is added to the set E.

[0222] 4) Finally, use the formula for iterative voting to calculate the weight of each word, and select the top k words as keywords. The formula is as follows:

[0223]

[0224] where ω ji represents the weight of the edge pointing from node j to node i, and is normalized. Here, d is the damping factor, taking 0.85.

[0225] (2) Optimize the assignment of voting weights

[0226] Including: (1) Whether the keyword appears in the title; (2) The tf-idf value of the keyword.

[0227] Define the weight of the weighted edge V<i, j> as wa(vi, vj), and its formula:

[0228]

[0229] where f(vj) represents the importance of the word. If the word appears in the title, it has the highest weight. If a character of the word appears in the title, its weight is relatively lower but higher than other cases. In the formula, OUT(Vi) represents all the out-nodes pointed to by node i (i.e., other nodes in the window). Additionally, a weight is calculated based on tf-idf:

[0230]

[0231] For each voting edge, the weights are the same, and we have:

[0232]

[0233] Finally, construct the weight matrix ω based on ωa and ωb:

[0234]

[0235] ωij represents the weight score of node i voting for j, where α, β, γ are control coefficients to adjust the proportion of their weights.

[0236] (3) Optimize the model for short texts

[0237] In the case of short texts, a sliding window approach is used instead of a sentence-based window segmentation mode. A window size of 10 is used, and 5 words are taken forward and backward. If 5 words cannot be taken, as many words as possible are taken. If the word appears multiple times, windows at different positions are calculated, and voting directed edges corresponding to these words are added to the word graph.

[0238] (VI) Extraction of fragments and phrases based on word frequency

[0239] (1) Key segment extraction based on word frequency

[0240] Hot events are time-sensitive. When a hot event breaks out, a large amount of similar text or text fragments will be generated on the Internet to describe the hot event. Therefore, in the real-time public opinion data stream, there will be some information with a high repetition rate, which appears repeatedly in the text. This is also one of the important information that this application focuses on. Using this feature, this application designs a text fragment extraction method based on word frequency to extract hot fragments in the text.

[0241] Each time when hot information is extracted, a batch of real-time data is obtained, and the documents are calculated according to the batches. First, all documents are divided into single words according to the unary word segmentation method, and symbols, special characters and stop words are removed to obtain a word set W.

[0242] The frequency of each word w in the word set is defined as P(w), which is equal to the frequency of all occurrences of the word in the document divided by the number of words in the document. The formula is defined as:

[0243]

[0244] The denominator d represents the document, the len function represents the number of characters in the document, and is also the length of the document. A critical value λ is set so that all characters with a probability greater than λ are seed characters. After calculating all the characters, the seed character set seed is obtained. Each character in the seed is a character with a high frequency of repetition in the document. First, some fragments containing high-frequency characters are preliminarily selected, and then screening is performed from them.

[0245] Use segmentation characters and stop words to split the text into segments, use bitmap to mark the positions of these text segments, mark them as valid segments, and then filter out all candidate segment sets by merging adjacent valid segments.

[0246] Finally, all candidate segments are screened. If the segment contains the seed word seed, it is retained, otherwise it is discarded, and a high-frequency segment set is obtained. All segments in this set contain high-frequency words.

[0247] (2) Phrase extraction based on mutual information and left-right information entropy

[0248] The extraction of phrases can help to better understand the text. This application realizes the extraction of keywords and some key text fragments, further refines the phrase fragments, and uses the high-frequency seed word set to filter phrases and add them to the key phrase set. First, each sentence is segmented and worded, and then stop words are removed. Then, each sentence is used as a window to calculate the frequency of each word, and then the frequency of two adjacent words and three words appearing at the same time is calculated.

[0249] The mutual information is calculated using the point-to-point mutual information formula:

[0250]

[0251] The higher the probability of two words appearing at the same time, the higher the value of p(x, y). In the same document, the values ​​of p(x) and p(y) are fixed, the higher the mutual information value of the two words, the more related they are, and the greater the possibility of forming a phrase sequence.

[0252] Left-right entropy is used to evaluate phrases. The more word strings that can form phrases in the text, the more likely they will appear in different contexts. Therefore, more uncertain words will appear on the left or right side of the phrase. The left-right entropy is used to calculate the degree of uncertainty of the words on the left and right sides of the phrase. The greater the entropy, the higher the degree of uncertainty and the higher the possibility of forming a phrase. The entropy is calculated using the following formula:

[0253] H(X)=-∑ x∈X P(x)*log(P(x)) Formula 13

[0254] Calculate the left and right entropies separately, that is, calculate the entropy of all left words and the entropy of right words:

[0255] hl(phrase)=-∑p(leftword|phrase)*log 2 p(leftword|phrase)

[0256] hr(phrase)=-∑p(rightword|phrase)*log 2 p(rightword|phrase) formula 14

[0257] The mutual information between points and the entropy of the left and right information are used to score the phrase, and the phrase with the highest score is extracted as the extracted phrase.

[0258] score=mi+hl+hr Formula 15

[0259] After extracting the phrases, the extracted phrases are screened to retain only the phrases containing high-frequency seed words.

[0260] (VII) Topic Screening and Refining

[0261] First, set up a blocking dictionary to filter out some common noise texts, such as "you" and other high-frequency meaningless words, and then filter place names, because this application will refine the place names in advance and distribute public opinion documents according to the place. Finally, set critical value limits on the number of documents containing fragments and the source sites of the documents contained, to increase the credibility of the message and ensure the popularity of the message.

[0262] In addition, the time density of the text is calculated to determine whether the topic is a valid topic. The density of information is calculated by the ratio of the number of documents containing the keyword, phrase or fragment in a certain time period to the time. The formula is as follows:

[0263]

[0264] Where x represents the number of hours, and the denominator represents the number of documents containing the word within x hours. The greater the information density, the higher the popularity of the keyword. Relative density is used to reflect the relative change trend of popularity. Relative density is defined as:

[0265]

[0266] Where X is a period of time greater than x, and the density of the word within time x is determined relative to the trend within time X. When a new hot spot appears, the popularity of a related keyword will definitely increase significantly, so this formula can screen out relatively hot words.

[0267] 8. Topic Generation Based on Co-occurrence

[0268] Based on the set of filtered keywords and key phrases as the seed for producing topics, a word group is used to define a topic. A word group contains several keywords and key phrases to describe the topic. After obtaining the keywords, calculate in which text each keyword appears, and use bitmap to record all document information containing the keyword, calculate the co-occurrence of each word, and for words with high co-occurrence, divide them into a word group. The co-occurrence of two words is described by the following formula:

[0269]

[0270] Where doc(xy) represents the number of documents containing the two words x and y, the denominator calculates the number of documents containing the word x or the word y, and a critical value is set. When the co-occurrence of two words is greater than the critical value, the two words are merged into one word group.

[0271] Finally, this word group is regarded as a hot topic found, and all related pages are stored in the database as pages of the discovered topic.

[0272] 9. The disappearance and merging of topics

[0273] Due to the real-time nature of hot topics, as time goes by, the popularity of a topic will gradually decrease and eventually disappear. Therefore, this application needs to know when a hot topic should disappear.

[0274] The extinction of a topic is related to time, so this application sets a time threshold. When it is greater than this threshold, the topic is judged to be extinct. This can greatly save computing resources and improve system efficiency. To determine whether a topic is active, the number of related posts of the topic is used to determine, and the maximum number of related documents per day for each topic is recorded. When the number of documents on that day is significantly lower than the maximum number of documents, the topic is defined as dead.

[0275] In addition, when generating new topics, topics are merged based on their relevance to old topics, similar topics are merged based on region and keywords, and similar texts are merged to avoid repeated calculations and save computing resources.

[0276] Currently, the document similarity is calculated by reducing the dimension of the high-dimensional feature vector and mapping it to a low-dimensional feature vector. Then, the similarity between documents is quantified by its Hamming distance, thereby eliminating similar pages.

[0277] The Hamming distance defines the number of different characters in the same position in two strings. In the similarity measurement, we only need to calculate the number of bits that are 1 after performing an XOR operation on two hash values. The larger the Hamming distance, the less similar the two texts are, and vice versa. The specific calculation steps are:

[0278] 1) First, segment the text into words to get a word sequence, and then set a weight for each word, which is the frequency of each word. The word weight is set to the value of tf-idf. The more important a word is, the higher its word weight will be.

[0279] 2) Calculate the hash value to obtain the characteristic signature of each word;

[0280] 3) For the signature of each word of fixed length and the weight corresponding to each word, a feature vector W is obtained, where each bit in W corresponds to each bit of the hash value of the word. If it is 1, the feature vector at that position has a positive weight value, and if it is 0, it has a negative weight value;

[0281] 4) Merge the feature vectors of all words, that is, add each feature vector;

[0282] 5) Finally, perform dimensionality reduction, set each position greater than 0 in the feature vector to 1, and each position less than 0 to 0, to obtain a signature value;

[0283] For the signatures of two texts with similar hashes, calculate their Hamming distance. Set a critical value relative to the 64-bit hash. If the Hamming distance is within this critical value, the two texts are considered similar.

[0284] For massive documents, this application maintains an approximate hash signature pool. For each topic or merge, the document's similarity to the existing articles is calculated. Similar documents can be directly merged and the similar articles of the document are marked, so as to avoid displaying a large number of identical documents.

[0285] 3. Experiments and Conclusions

[0286] (I) Experimental environment

[0287] The experiment is divided into three parts: the crawler module, the data storage module, and the public opinion analysis module.

[0288] (1) The crawler is set up on a Linux server, using squid as an IP proxy, and using zookeeper for load balancing and node management. The socks5 protocol is used for the external network, and an external network proxy node is built, with a total of 4 nodes. The crawler download node uses the centos system and a 2-core 2G virtual machine, with a total of 16 crawler nodes and 25 proxy nodes. ELK is used to parse the crawler logs and download calculations, and zabbix is ​​used to monitor and manage each node.

[0289] (2) Data storage uses Hbase as the core storage module and 4 memcache nodes as cache. There are 20 nodes in total. Hbase and Hadoop clusters are deployed together, and each virtual machine has two services, Hbase and Hadoop. Mysql uses a cluster containing 16 nodes, and uses cobar middleware to divide the Mysql cluster into libraries and tables. It is also optimized using Mysql's partitioning technology and partitioning Mysql data according to time. In order to facilitate calculation and search of documents, elasticsearch is used to store all documents, which contains a total of 60 es nodes.

[0290] (II) Experimental results

[0291] 1. Experimental data on public opinion discovery and storage

[0292] At present, the traffic of each node of the network hotspot crawler is about 10-20Mbps. Using zabbix to configure the traffic monitoring system, you can see the traffic information of each crawler node.

[0293] At present, the daily download volume of the network hotspot crawler is about 10,000 to 40,000. It can be seen that the Hbase master node generally does not occupy much resources, the memory usage of the slave node will be very high and the disk throughput is about 100-400KB / s.

[0294] The memory usage of the Es node is very high and the hard disk IO is very high, at 10-20MB / s, and the peak can even reach 100MB / s. The message rate of a single node in the cluster is about 20-100 per second, and the data throughput is 5-10MB / s. At present, the crawler can always work stably, and the anti-crawler strategy has played a very good role. When the anti-crawler mechanism is deployed in the crawler system, it is often encountered that the crawler stops working and cannot crawl data. For example, verification code pages often appear during the crawling process of Weibo, so no useful data can be crawled. Now the download volume is very stable.

[0295] 2. Discovered related hot topic information

[0296] As of April 5, the experiment has been refined into the following topics, and the data can be found at Figure 4 The news reflects the latest developments and hot topics of hot figures very well, and has a very good effect.

[0297] 3. Real-time topic calculation

[0298] The topic data is analyzed and calculated in real time, the generation and extinction time of the topic is calculated, and the related hot information is searched by search engine. The time of the earliest related article or the time of the event is used as the comparison, and the formation time of the topic is obtained manually and compared. Figure 5 , we can see that the system has very good real-time performance and can discover network hot spots and hot topics in a timely manner.

[0299] From the above experimental results, it can be seen that this application can discover hot topic information in a timely manner, and can conduct comprehensive topic analysis for various public opinion platforms.

Claims

1. A real-time network hot topic mining and analysis and public opinion extraction system, characterized by: The first is the real-time discovery and noise processing of network hot spots, including network hot spot discovery and noise processing of the initial page. Network hot spot discovery uses a large-scale cloud architecture crawler technology based on information sources to discover network hot spots in real time, and then the noise processing of the initial page removes the noise of the page to extract the real public opinion data, that is, the text part; The second is the analysis and generation of hot topics in public opinion, including pre-processing, text clustering in specific fields, personalized recognition of place names, chain linear Chinese word segmentation model, real-time keyword election extraction, fragment and phrase extraction based on word frequency, topic screening and extraction, topic generation based on co-occurrence, and topic extinction and merging. First, the pre-processing calculates the word frequency of all characters in the text instead of words, as well as the documents where the word appears, and calculates the corresponding tf-idf value to obtain the probability of occurrence of the word and the relative importance of a word. For subsequent queries, if the word has not appeared, then their word frequency returns the minimum value and the importance returns the maximum value. In public opinion analysis, the tf-idf of the word is used to assist in analyzing the similarity of sentences and remove words with high frequency of occurrence and no analytical value or meaning. Then, we adopt text clustering in a specific field. When a batch of documents arrives, we classify the documents and generate topics in the batch of texts. We add user-defined topic keywords to the system, cluster the documents containing keywords into a batch of documents, and then find hot topics from these documents. In addition, we start from the region, identify the regional information contained in the text information, and cluster the text according to the region; then we identify the place names in a personalized way, extract the region for each text, distribute it according to the region, extract hot spots according to the region, and treat the regions that cannot be extracted as unknown regions; then, under the condition of a given chain linear Chinese word segmentation model and its observation value sequence, let it be x, and find the probability that the implicit state sequence is y, and the characteristic function f j (y i-1 ,y i , x, i) is a positive real-valued function that describes the associated information of the state transition of a certain y at position i under the condition of a given observation sequence x. j represents the number of the feature function, indicating the label of the jth position. In the sequence labeling problem, the meanings of the four parameters it accepts are the observation sequence x, the current position i, and the state label y of the previous position. i-1 , current position label y i ; Then, the real-time election and extraction of keywords is carried out. According to the order of text words, a window is constructed, and all adjacent words in the window are allowed to vote for the word, and a voting word graph of candidate words is constructed according to the rule; Then, fragments and phrases are extracted based on word frequency, and the key fragment extraction based on word frequency and the phrase extraction method based on mutual information and left and right information entropy are adopted respectively; Then filter and refine the topic, set up a shielding dictionary, filter out common noise texts, and then filter out place names, and set critical value limits on the number of documents containing fragments and the source sites of the documents contained; then realize topic generation based on co-occurrence, use the set of filtered keywords and key phrases as the seed for producing topics, and use a word group to define a topic. A word group contains several keywords and key phrases to describe the topic. After obtaining the keywords, calculate in which text each keyword appears, and record all document information containing the keyword, calculate the co-occurrence of each word, and for words with high co-occurrence, classify them into a word group; finally, perform topic extinction and merger, set a time critical value, and judge the extinction of the topic when it is greater than this critical value, use the number of related posts of the topic to determine whether the topic is active, record the maximum number of related documents per day for each topic, and define the topic as dead when the number of documents on the day is lower than the maximum number of documents; The steps for mining and analyzing real-time hot topics on the Internet and extracting public opinion are as follows: S1: Build a high-performance network hotspot crawler architecture to capture network hotspots in a timely manner. Use MySQL cluster to build relational data storage, and use Hbase cluster and es cluster to store public opinion documents. Use Kafka as a message queue and middleware for message transmission. The system carries huge information flow and supports huge data volume. S2: Construct a text extraction method based on sliding window word block density vision, which can directly extract the text of theme websites without pre-training models. As a supplement, based on the common templates used by forum websites, a clearing method based on approximate blocks is used to extract the text, and the two methods are combined; S3: Analysis and generation of hot topics in public opinion. It adopts the strategy of discovering, merging and eliminating hot topics, finds key fragments based on frequency, extracts phrases using left-right mutual information and information entropy, and selects keywords. It performs heat analysis and screening on keywords, clusters keywords using their co-occurrence to form a word group, and summarizes them into a topic. It uses the number of documents associated with the topic to formulate the topic generation and elimination method, and finally uses the proximity hashing method to solve the problem of approximate document merging of massive documents.

2. According to the network real-time hot topic mining and analysis and public opinion extraction system of claim 1, it is characterized by: The cloud architecture crawler includes the following modules: page collection module, task cloud scheduling module, page cloud storage module, and page screening module. The page collection module initiates http requests to obtain HTML data. The task cloud scheduling manages new tasks based on cloud computing and distributes tasks to crawler sub-nodes. The page screening module screens pages that do not need to be downloaded, and finally the page cloud storage module stores the data in the cloud for subsequent query and processing. Task cloud scheduling and screening strategy: When the network hotspot crawler crawls a news page, it determines whether it is a public opinion page. During the storage process, the URL of the page will be stored. When downloading, it will be determined whether it has been downloaded. If it is a news page that has been downloaded, it will not be downloaded again. Hbase is used to store URLs and is used for duplication checks. After the page collection module parses the HTML page, it reports all URLs to the task cloud scheduling module. After the task cloud scheduling module screens out some duplicate and irrelevant URLs, it releases the URLs to the queue in the form of tasks, and then the page collection cluster competes to obtain the tasks in the queue. In addition, all pages are divided into three types: information source seed pages, public opinion pages and other pages. Public opinion pages are the required pages with network hotspots. All real-time public opinion data that need to be analyzed come from public opinion pages. Seed pages are pages published by all network hotspots. All public opinion pages can be found from seed pages, and seed pages are constantly updated and published with new public opinion pages. Seed pages are the entrance to public opinion pages. Other pages are advertising pages or irrelevant pages. When parsing pages, you only need to care about the outbound links of the information source pages. When scheduling tasks in the cloud, if the seed page is obtained, prevent the screening program from filtering it out.

3. According to the network real-time hot topic mining and analysis and public opinion extraction system of claim 1, it is characterized by: Text extraction method based on sliding window block density vision: The sliding window block density vision method is based on the assumption that the text of the page is relatively close in physical position, based on the inference that most of the text codes are close in position and are mostly text, and the length ratio of hyperlinks is not high. It includes the following steps: Step 1: Remove all HTML tags, leaving only the text part and blank information; Step 2: Specify a sliding window of size K, start sliding from the initial row, record the position of each window and the length of all text blocks in the window, each sliding window contains all data from row i to row i+K, if there is no text in the window, it is a blank window, otherwise it is a text window; Step 3: Merge adjacent blank windows and adjacent text windows; Step 4: Pick out the largest block of text, which is the body of the text; The text extraction method of approximate node removal: the following inferences are used for screening: Corollary 1: Forum websites are mostly based on templates; Inference 2: Based on the URL, similar sibling pages can be found; Inference 3: Most templates have similar strings or tag sequences; Inference 4: The main text has significantly more words or punctuation; Cache this type of page and use the URL to find its sibling nodes, parse the DOM tree of the page, locate the text part using four features: string, tag sequence, punctuation and text length, and remove redundant DOM nodes; Other special treatments: adopt a blacklist strategy, detect the blacklist when crawling, and no longer download if it is a website on the blacklist. Formulate some characteristic URL filters, and all URLs that meet these characteristics will be filtered out.

4. According to the network real-time hot topic mining and analysis and public opinion extraction system of claim 1, it is characterized by: Personalized recognition of place names: 1) Named Entity Recognition: Personalized recognition of Chinese place names is a problem of labeling the state transition of a random sequence and its implicit state sequence. By labeling the text sequence, the boundaries of the named entities are found, all possible named entities are identified, and a place name resolution model is used to recognize Chinese place names. The place name resolution chain is described by a five-tuple. (S,O,A,B,π) Formula 1 (1) Where S represents the set of all states in the model, S = {S1, S2, S3, …, S N }, where N is the total number of all states in the set, and each state has a corresponding meaning, and each state can be converted to each other, using q t To represent the state at time t; (2) O represents the set of observed values ​​of the actual output of the model, where the observed state is defined as V = {V1, V2, V3, ..., V M }, where M is the number of observations; (3) A describes the state transition probability of the model and is a probability matrix, expressed as A = (a ij ), where a ii The probability of transitioning from state i to state j is described by the following formula: a ij = p(q t+1 = S j |q t = S i ) 1 ≤ i ≤ N, j ≤ M, ∑ j a ij = 1, a ij ≥ 0 Equation 2 (4) B represents the matrix composed of the probabilities of the observed values, B = (b ij ), where b ij represents the probability that the observed value output at state i is j, that is, p = (Vj Si), where i represents the hidden state, i is less than or equal to the number of states N, and j represents the observed value, which is less than or equal to M; (5)π is the initialization probability vector, describing the probability distribution of the initial state, π={π i } represents the probability of being in the i-th state at the initial moment, π i =p(q1=S i ); In the place name resolution model, the implicit state at time t is only related to the previous time t-1, and the observed value at time t is also only constrained and affected by the implicit state at time t; By parsing the annotations of the corpus, we get the annotated sequence, the transfer matrix, the role dictionary, and the initial state calculation, and calculate the corresponding probability matrix A, B, π. When the sentence is personalized for place name recognition, we first segment the words, then calculate the maximum possible hidden state, and identify possible place names through dictionary matching; Find the hidden state sequence with the highest probability through the observation value sequence, and find the observation sequence hidden state sequence P max , define the local maximum probability δ(i, t) as the maximum probability of reaching state i at time t. The sequence of hidden states corresponding to this value is the path with the maximum probability of reaching state i at time t. Assuming that at the initial moment, the observation value is 0, the local probability is: δ(i, 1) = π(i)bio Formula 3 And for time t, for the observation value O t have: During the calculation process, a pointer for backtracking the path is added, and the optimal path, that is, the optimal hidden state chain solution, can be obtained in one calculation; 2) Personalized recognition of place names and feature pattern matching: After finding the corresponding hidden state chain, match the corresponding state and find the following four patterns: CF, CDF, CDEF, and GF, i.e., the pattern of single word / double word / triple word / full ground + place name suffix. Multi-pattern matching consists of three parts: goto table, fail table, and output table. The construction process of the goto table is as follows: (1) For each pattern Pu[1,…,j] in P, 1≤u≤m, input it into the automaton in the order of the letters it contains; (2) The starting state is D[0]. If the current state machine state is D[p] and there is no new state available for the next letter pu[k], then the total number of states max is increased by 1, and the transition D[p][pu[k]]=max; (3) Continue adding the next letter to the state machine; The goto table is a dictionary tree structure. The fail table describes what state should be transferred to after the state transfer fails during pattern matching. When building the jump table, find the prefix part of the longest other pattern string in the suffix and build the fail jump table. After matching the pattern string, build the output table, and continue to search for the state according to the suffix and record it in the output.

5. According to the network real-time hot topic mining and analysis and public opinion extraction system of claim 1, it is characterized by: Chain linear Chinese word segmentation model: Chinese word segmentation is regarded as a word boundary state sequence labeling problem. First, several specific tags are defined to mark the text sequence. Four tags, S, B, M, and E, are used, corresponding to the four roles of single-word, beginning of word, middle of word, and end of word. Single-word words are labeled with S, two-word words are labeled with the tag sequence {B, E}, three-word words are labeled with {B, M, E}, and words with more than four words are labeled with the sequence {B, MM, E}. After labeling the text with the trained chain linear Chinese word segmentation model, the text is segmented according to the tags. In addition, we also prepare our own feature templates, using macros to define each token of the output data, written as %x[row,col], where col represents the absolute column number and row represents the relative offset. There are two types of templates: U template and B template. The total number of feature functions generated by each model is L*N, where N is the number of unique features expanded from a given template, and L is the number of output categories. When using the B template, the system automatically generates a token for the current output combined with the previous token, and the number of features generated can reach L*L*N; In feature selection, select feature templates that can improve accuracy, including: (1) Unary features, that is, they are activated when the current character or its previous character or next character is a specific value, represented as a sequence C -1 , C0, C1; (2) Binary features, that is, the current string is activated when it is the same as the previous string or the next string, and is represented as a sequence C -1 C0 and C0C1; (3) Jump feature, that is, it is activated when the previous character and the next character are a specific string, that is, C -1 C1; (4) Ternary feature, that is, it is activated when the current character and the upper and lower characters form a specific string, that is, C -1 C0C1; The present application also considers the characteristics of state transition. When the state of the previous character is E or S, the state of the current character must be the beginning of the word, which must be B or S, and cannot be M or E. When the state of the previous character is B, the state of the next character may be M or E, but cannot be or B. Therefore, the state of the previous character will affect the probability of the next state. The present application adds a state transition template T to the feature template of the training model. -1 T0.

6. The real-time network hot topic mining and analysis and public opinion extraction system according to claim 1 is characterized in that: Keyword real-time election extraction: (1) Keyword selection: The steps include: 1) For a text D, segment it into sentences: D = [S1, S2, S3, …, Sn]; 2) Segment each sentence, then mark the part of speech of each word, and select candidate words according to the part of speech, retain verbs, nouns, and adjectives, remove auxiliary words, stop words, and meaningless words, and finally obtain the keyword sequence [w1, w2, w3, ..., wt] for each sentence Si; 3) Construct a voting word graph for all candidate words, represented by G =<V,E> , where V represents the set of nodes and E represents the set of edges. For the same sentence Si, all keywords have a directed edge to words other than themselves, that is, for Vk∈Si,<Vp,Vq> , p≠q, the edge is added to the E set; 4) Finally, use the formula to iterate and vote, calculate the weight of each word, and take the first k as keywords. The formula is as follows: where ω ji represents the weight of the edge from node j to node i, and is normalized, while d is the damping coefficient, which is 0.85; (2) Optimize voting weight assignment: including: 1) whether the key words appear in the title; 2) The tf-idf value of the keyword; Define the weight edge V<i,j> The weight is wa(vi,vj), and its formula is: Where f(vj) represents the importance of the word. If the word appears in the title, the word has the highest weight. If the characters in the word appear in the title, the weight is relatively lower, but higher than other cases. OUT(Vi) represents all the outgoing nodes pointed to by node i. In addition, a weight is calculated based on tf-idf: For each voting edge, the weight is the same, we have: Finally, the weight matrix ω is constructed based on wa, wb, and wc: ωij represents the weight score of node i voting for node j, where α, β, and γ are control coefficients that adjust their weight proportions; (3) Model optimization for short texts: In the case of short texts, a sliding window approach is used with a window size of 10. Five words are taken forward and five words are taken backward. If five words cannot be taken, as many words as possible are taken. If the word appears multiple times, windows at different positions are calculated, and directed voting edges corresponding to these words are added to the word graph.

7. The real-time network hot topic mining and analysis and public opinion extraction system according to claim 1 is characterized in that: Fragment and phrase extraction based on word frequency: (1) Key segment extraction based on word frequency: When extracting hot information each time, obtain a batch of real-time data and calculate the documents in batches. First, divide all documents into single words according to the unary word segmentation method, remove symbols, special characters and stop words, and obtain a word set W; The frequency of each word w in the word set is defined as P(w), which is equal to the frequency of all occurrences of the word in the document divided by the number of words in the document. The formula is defined as: The denominator d represents the document, the len function represents the number of characters in the document, and is also the length of the document. A critical value λ is set so that all words with a probability greater than λ are seed words. After calculating all the words, the seed word set seed is obtained. Each word in the seed is a word with a high repetition frequency in the document. First, some fragments containing high-frequency words are preliminarily selected, and then screening is performed from them; Use segmentation characters and stop words to split the text into segments, use bitmap to mark the positions of these text segments, mark them as valid segments, and then filter out all candidate segment sets by merging adjacent valid segments; Finally, all candidate segments are screened. If the segment contains the seed word, it is retained, otherwise it is discarded, and a high-frequency segment set is obtained. All segments in this set contain high-frequency words. (2) Phrase extraction based on mutual information and left-right information entropy: First, segment each sentence, then remove stop words, and then calculate the frequency of each word for each sentence as a window, and then calculate the frequency of two adjacent words and three words appearing at the same time; The mutual information is calculated using the point-to-point mutual information formula: When the probability of two words appearing at the same time is higher, the value of p(x, y) is higher. When the values ​​of p(x) and p(y) are fixed in the same document, the mutual information value of the two words is higher, and they are more related, and the possibility of forming a phrase sequence is greater. Left-right entropy is used to evaluate phrases. The more word strings that can form phrases in the text, the more likely they will appear in different contexts. Therefore, more uncertain words will appear on the left or right side of the phrase. The left-right entropy is used to calculate the degree of uncertainty of the words on the left and right sides of the phrase. The greater the entropy, the higher the degree of uncertainty and the higher the possibility of forming a phrase. The entropy is calculated using the following formula: H(X)=-∑ x∈X P(x)*log(P(x)) Formula 13 Calculate the left and right entropies separately, that is, calculate the entropy of all left words and the entropy of right words: hl(phrase)=-∑p(leftword|phrase)*log2p(leftword|phrase) hr(phrase)=-∑p(rightword|phrase)*log2p(rightword|phrase) Equation 14 The mutual information between points and the entropy of the left and right information are used to score the phrase, and the phrase with the highest score is extracted as the extracted phrase: score=mi+hl+hr Formula 15 After extracting the phrases, the extracted phrases are screened to retain only the phrases containing high-frequency seed words.

8. The real-time network hot topic mining and analysis and public opinion extraction system according to claim 1 is characterized in that: Topic screening and refinement: Calculate whether a topic is a valid topic by calculating the time density of the text. The density of information is calculated by the ratio of the number of documents containing the keyword, phrase or fragment in a certain time period to the time. The formula is as follows: Where x represents the number of hours, and the denominator represents the number of documents containing the word within x hours. The greater the information density, the higher the popularity of the keyword. Relative density is used to reflect the relative change trend of popularity. Relative density is defined as: Where X is a period of time greater than x, and the density of the word within time x is determined relative to the trend within time X. When a new hot spot appears, the popularity of a related keyword will definitely increase significantly, so this formula can screen out relatively hot words.

9. The real-time network hot topic mining and analysis and public opinion extraction system according to claim 1 is characterized in that: Topic generation based on co-occurrence: Based on the set of filtered keywords and key phrases as the seed for generating topics, a word group is used to define a topic. A word group contains several keywords and key phrases to describe the topic. After obtaining the keywords, calculate in which text each keyword appears, and use bitmap to record all document information containing the keyword, calculate the co-occurrence of each word, and for words with high co-occurrence, divide them into a word group. The co-occurrence of two words is described by the following formula: Where doc(xy) represents the number of documents containing the two words x and y, and the denominator counts the number of documents containing the word x or the word y. A critical value is set, and when the co-occurrence of two words is greater than the critical value, the two words are merged into one word group; Finally, this word group is regarded as a hot topic found, and all related pages are stored in the database as pages of the discovered topic.

10. The real-time network hot topic mining and analysis and public opinion extraction system according to claim 1 is characterized in that: Topic extinction and merging: When generating new topics, merge them based on the relevance between the new and old topics. Merge similar topics based on regions and keywords, and merge similar texts to avoid repeated calculations. Calculate the document similarity, reduce the dimension of the high-dimensional feature vector, map it to a low-dimensional feature vector, and then quantify the similarity between documents through its Hamming distance, so as to eliminate similar pages. The Hamming distance defines the number of different characters in the same position in two strings. In the similarity measurement, it is only necessary to calculate the value obtained by XOR operation of two hash values, the number of bits that are 1. The larger the Hamming distance, the less similar the two texts are, and vice versa. The specific calculation steps are as follows: 1) First, segment the text into words to get a word sequence, and then set a weight for each word, which is the frequency of each word. The word weight is set to the value of tf-idf. The more important a word is, the higher its word weight will be. 2) Calculate the hash value to obtain the characteristic signature of each word; 3) For the signature of each word of fixed length and the weight corresponding to each word, a feature vector W is obtained, where each bit in W corresponds to each bit of the hash value of the word. If it is 1, the feature vector at that position has a positive weight value, and if it is 0, it has a negative weight value; 4) Merge the feature vectors of all words, that is, add each feature vector; 5) Finally, perform dimensionality reduction, set each position greater than 0 in the feature vector to 1, and each position less than 0 to 0, to obtain a signature value; For the signatures of two texts with similar hashes, calculate their Hamming distance. Set a critical value relative to the 64-bit hash. If the Hamming distance is within this critical value, the two texts are considered similar. For massive documents, this application maintains an approximate hash signature pool. For each topic or merge, the approximation of the document to the existing article is calculated. For similar documents, they are directly merged and the similar articles of the document are marked, thus avoiding the display of a large number of identical documents.