Multi-dimensional intelligent extraction system of hot topics from Weibo big data

Abstract

The multi-dimensional intelligent extraction system of hot topics in Weibo big data of the present application proposes a text modeling method based on short text expansion and hidden semantic calculation method, which solves the problems of data noise reduction, insufficient dimensionality reduction and semantic loss in the text modeling method of the prior art for Weibo topic extraction; the present application reorganizes the discussion graph tree to expand the Weibo text, and reduces the dimension of the text vector without losing semantics; the present application proposes a collaborative approximate set method for extracting Weibo topics, and scores the topic heat, incorporates the time data characteristics into the text similarity calculation, proposes the influencing factors and specific methods for calculating the heat value of Weibo topics, and generates a Weibo hot topic scoring list, which improves the accuracy and timeliness of Weibo topics, reduces the loss of topic information, and realizes multi-dimensional intelligent and accurate extraction of hot topics in Weibo big data.

Description

Technical Field

[0001] The present application relates to a system for extracting hot topics from Weibo big data, and more particularly to a system for extracting hot topics from Weibo big data in multiple dimensions, belonging to the technical field of hot topic extraction from social networks. Background Art

[0002] With the rapid development of IT technology and the widespread application of mobile Internet, Internet social networking has gradually and massively entered people's lives, profoundly affecting people's information acquisition, social and lifestyle. Weibo has gradually become a tool for people to obtain and publish information, share status and social interaction due to its advantages of large amount of user data, real-time publishing, strong openness and convenient interaction. Weibo has basically become one of the main sources of online public opinion.

[0003] With the rapid increase in the number of Weibo users, it has become an important source of information and an important channel for publishing information for individuals, enterprises and governments. However, in the face of massive amounts of Weibo data that are updated in real time and published by thousands of users at the same time, how to effectively capture important information that meets the needs from these massive amounts of data in a timely and efficient manner, ensure that the relevant information is accurate and reliable, and not miss important information, so as to grasp the development trend of online public opinion, is an urgent problem to be solved in the Internet era, and it also has very important economic and social significance.

[0004] At present, the commonly used information capture technology is aimed at online news and blogs with long and long online articles. The principle of these technologies is to extract the data contained in these articles through some keywords or high-frequency words, and to capture the current network hot spots by calculating these keywords or high-frequency words. After finding the network hot spots, the development trend of the network hot spots can be grasped in time through real-time tracking of the network hot spots. However, the data released through Weibo has the characteristics of short information, huge data volume, strong interactivity, strong user participation, and extremely fast update speed. Weibo users can spread new topics in real time on Weibo in their own way of expression. Users can push any information released by any user to hot information and become hot topics through searching, commenting and forwarding. These characteristics of Weibo determine that the existing information capture methods are difficult to apply to the capture and analysis of Weibo information. A new method and technical means are urgently needed to realize the information extraction of Weibo.

[0005] In summary, there are still problems with the extraction of hot topics in Weibo in the prior art. The difficulties and problems to be solved in this application are concentrated in the following aspects:

[0006] (1) Although there are many studies on text representation models and topic extraction methods in the prior art, the research objects of traditional topic based on vector space model methods are often long texts such as news media reports, commentary articles, and forum blogs. Weibo has different characteristics from previous news media and commentary articles, which leads to major problems in the application of traditional topic extraction methods. The short text characteristics of Weibo information and text sparsity bring difficulties to semantic information processing. Weibo data is relatively short, and when the VSM method processes data, the larger the amount of text data and the more repeated words between texts, the more conducive it is to measure and extract text similarity. At the same time, there is a large amount of noise data in Weibo, such as user account information, website links, and emoticons, which will also make it difficult to calculate text similarity and affect the effect of topic extraction. The hot spot extraction method of the prior art will have a large deviation when applied to Weibo.

[0007] (2) The amount of information on Weibo is very huge, and the data is highly real-time. There are high requirements for the speed and accuracy of hot topic extraction. To extract from a large amount of messy Weibo information, different users may have very different expressions on the same topic, which generates a huge amount of information. The forwarding and comments between users bring huge data processing requirements. This brings great difficulties to the timeliness and algorithm speed of topic extraction in existing technologies. Due to the short Weibo text and sparse data, it is difficult for existing technologies to directly calculate through ordinary text feature factors. It is impossible to solve the difficulties of text modeling based on vector space model, such as high vector space dimension and loss of semantic information, resulting in strong correlation between words, and the inability to obtain the implicit semantic structure between words. The expression of words and text is inaccurate, and the accuracy of Weibo hot topic extraction is very low.

[0008] (3) Weibo has many characteristics such as originality, repetitiveness, and explosive radial dissemination. The existing technology is unable to timely and efficiently capture important information that meets the needs from the massive amount of Weibo data posted by a large number of users in real time, and follow up on the hot topics of online public opinion. There is a lack of an automatic extraction system for Weibo hot topics, and a lack of text modeling methods based on short text expansion and hidden semantic calculation methods. The existing text modeling methods for Weibo topic extraction have problems such as data noise reduction, insufficient dimensionality reduction, and semantic loss. There is a lack of short text expansion that combines the characteristics of Weibo, and a lack of collaborative approximate set methods for Weibo topic extraction. It is impossible to realize the automatic extraction and scoring display of Weibo hot topics. The accuracy and timeliness of Weibo topic extraction are poor, and a lot of topic information is lost, which cannot meet the functional needs in reality. Summary of the invention

[0009] Since microblog texts are short and data is sparse, it is difficult to calculate directly through ordinary text feature factors. This application makes full use of the characteristics of microblog dissemination and interaction, and proposes a short text expansion method based on the characteristics of microblog publishing and commenting. It solves the difficulties of high vector space dimension and loss of semantic information in text modeling based on vector space model in the prior art. This application models microblog texts based on hidden semantic calculation methods, weakens the correlation between words, obtains the implicit semantic structure between words, expresses words and texts, and reduces the dimension of text vectors. To meet the processing needs of massive microblog information, an improved collaborative approximate set algorithm is proposed, and time data is combined to calculate similarity to improve the accuracy and timeliness of microblog topic extraction. A microblog hot topic automatic extraction system is constructed. By calculating and analyzing the reposted and disseminated content of microblogs, the current social public opinion situation can be efficiently grasped, and the network public opinion can be effectively analyzed and grasped.

[0010] In order to achieve the above technical feature advantages, the technical solutions adopted in this application are as follows:

[0011] The multi-dimensional intelligent extraction system of hot topics from Weibo big data uses the characteristics of Weibo dissemination and interaction to build a hot topic extraction model based on massive Weibo data, text modeling based on short text expansion and hidden semantic calculation methods, and automatic extraction of Weibo hot topics based on collaborative approximate sets and heat scores, and build a corresponding automatic extraction system;

[0012] S1, Weibo text modeling by combining short text expansion and hidden semantics calculation: A short text expansion method is proposed based on the characteristics of Weibo, and the Weibo text is expanded by reorganizing the discussion graph tree. The Weibo text is modeled by hidden semantics calculation method, which weakens the correlation between words, obtains the implicit semantic structure between words, expresses words and texts, and reduces the dimension of text vector without losing semantics.

[0013] S2, extraction of hot topics on Weibo based on collaborative approximate sets and heat scoring, including: improved collaborative approximate set algorithm, text similarity calculation of joint time data, score of factors affecting the heat of Weibo topics, and calculation of the heat value of Weibo topics; integration of hierarchical approximate sets and K-balanced algorithms and improvement, and incorporation of time data characteristics into text similarity calculation; introduction of factors affecting the heat value calculation and implementation methods of topics combined with the characteristics of Weibo texts, and generation of a score list of Weibo hot topics;

[0014] S3, builds a multi-dimensional intelligent extraction system for Weibo hot topics, models the text based on the short text expansion method and the hidden semantic calculation method constructed by the text modeling representation method, and integrates the hot spot extraction method of collaborative approximate sets and heat scoring to improve the extraction accuracy and timeliness, reduce the loss of topic information, and realize the automatic extraction and scoring display of Weibo hot topics.

[0015] Preferably, microblog comment processing based on descriptors:

[0016] (1) Delete reply comments that are not long enough. Here, a certain critical length Length is set, and reply comments that are shorter than Length are removed during microblog data extraction;

[0017] (2) Clean and preprocess the content of Weibo, including removing stop words, emoticons, user accounts, and URLs, and performing Chinese word segmentation on Weibo content;

[0018] (3) Delete comments that are not related to the initial microblog content. From the microblog collection of reply comments, extract comments with descriptors similar to the initial microblog. The specific process for executing this rule is as follows:

[0019] Process 1: Take the descriptive cluster of the original microblog as the initial cluster;

[0020] Process 2: Read one comment from the original Weibo in chronological order;

[0021] Process 3: Calculate the spatial distance between the words in the extracted comments and the initial clusters;

[0022] Step 4: Add the normalized spatial distance between the comment and the initial cluster and calculate the sum; if the result is less than the set critical value, add it to the list of useful comments, if not, remove it;

[0023] Execute process 3 and process 4 again until all reply comments are processed and a valid comment list corresponding to the original microblog is obtained.

[0024] When calculating the spatial distance between the comment and the initial microblog, the text approximation is calculated, and the approximate probability of words a and b is calculated:

[0025]

[0026] P(a|b) is the probability of a appearing when b appears. It is equal to the probability of a and b appearing at the same time divided by the probability of b appearing. Calculate the correlation between the word and the descriptive cluster. Let the distance between word w and cluster C be max{P(c i |ω)c i ∈C}, the specific method is as follows:

[0027]

[0028] If c i In cluster C, then c i If the probability of word w appearing in the text containing word w is high and the spatial distance from word w to cluster C is short, word w is added to cluster C; otherwise, the spatial distance from word w to cluster C is large, word w is not added to cluster C.

[0029] Preferably, the recombinant discussion graph tree expands the Weibo text: In the discussion graph tree of the Weibo, the Weibo content is recombined and expanded to form a long text of the Weibo, including two situations: one is the continuously forwarded Weibo, and the forwarded blog posts are marked; the other is the comment conversation in the reply, and the conversation will point to the label @Weibo blogger;

[0030] The extraction of Weibo text information combines the presentation mode of the Weibo, recombines all the continuous forwards and comment replies under the initial Weibo as the expansion of the content of the Weibo, and thus extracts and calculates the topic. Before recombination, the collected data set Set <blog>Scoring in chronological order to obtain SortedSet <blog>, and then use it as the input of the algorithm, and through the algorithm processing, it is converted into a collection of multiple texts, and the repeated information has been reorganized;

[0031] (1) Reorganize and forward Weibo posts

[0032] Use tags on the page to filter the Set <blog>If the microblog is an initial microblog message or contains a forwarding tag, such microblog information is added to the SortedSet <blog>In the example, the HashValue of the original microblog is used as the key of the textMap, and the reorganized TextSet <blog>After adding them to the corresponding positions and reorganizing them according to the algorithm, the values ​​in textMap are all text sets with reorganized repeated information;

[0033] Understand the scope of the final text set, that is, the initial microblogs that have not been forwarded, and the repeated microblog content that has been forwarded continuously. After processing, mark it as yeD as the initial input set and proceed to the next step of processing;

[0034] (2) Multi-microblog reorganization of dialogue and comments

[0035] After the continuous forwarding microblog algorithm is used to process, the remaining text set yeD contains microblog information with the indicator label @. The multiple microblogs with dialogues and comments are reorganized to construct their own dialogue tree structures. The core is the time order of microblog posting and the microblog information transmission relationship corresponding to the label @.

[0036] First, the microblog information in the yeD constructed in the previous step is arranged in order according to the time of release, and then the entire yeD is searched. For microblogs that do not contain @ pointing to a matching microblog, it is used as the root node to build a dialogue tree, and the newly built tree is added to the discussTreeSet; for forwarded and commented microblogs, the discussTreeSet is searched until: ① the publisher of a microblog in the tree node is the person pointed to by the microblog; ② the release time difference between the two microblogs in all nodes that meet condition ① is the smallest, and then the current microblog information is added as its child node information to the original dialogue tree;

[0037] After being processed by the multi-microblog reorganization algorithm of conversations and comments, the microblog information in yeD is reorganized into a conversation tree, and the conversation trees with only root nodes or very low tree heights are eliminated, that is, microblog information with very few comments or reposts. The microblog information with the same conversation tree structure is reorganized into a text according to the chronological order of release time to obtain meaningful long text information.

[0038] Preferably, microblog text modeling based on hidden semantic calculation: through a large amount of text calculation, a text-word matrix is ​​constructed, and then the hidden semantic structure of words and text is extracted. The specific steps are as follows:

[0039] Let m represent the number of different words in the text, n represent the number of texts, and the word-text co-occurrence matrix is ​​A mxn Indicates that a ij Represents the factor of the i-th word in the j-th text. The text and the word are not very relevant. A mxn At this time, it is a sparse matrix;

[0040] A mxn After decomposition, we get A mx The approximate matrix A k , where k << min(m, n), to achieve dimensionality reduction of the text vector, and the decomposition process is as follows:

[0041] (1) First, construct the co-occurrence matrix A, whose row vectors each correspond to a word, and column vectors each correspond to a text;

[0042] (2) Assume there is a decomposition of matrix A: A = U∑V T , where U and V are orthogonal, ∑ is a diagonal matrix, and we get: AA^T = U∑∑ T U T and A T A = V∑ T ∑V T , ∑∑ T and ∑ T ∑ is diagonal, and U and V respectively cover the eigenvectors of AA T and A T ;

[0043] (3) Let k be the largest singular value. Among U, V and their corresponding singular vectors, k has the smallest approximation error to A. Consider this word and the corresponding text vector as a semantic space. The vector has k entries. Map the vector to a lower space to obtain an approximate matrix to achieve dimensionality reduction of the text vector;

[0044] For the collected microblog text information, the latent semantic computing method is modeled as follows:

[0045] The first step: Construct the co-occurrence matrix of words and texts Suppose there are n microblog texts, which contain m different words, and form a matrix A mxn , a ij is calculated by TF-IDF, and the method is as follows:

[0046]

[0047] tf ij represents the frequency of the word in text j, N represents the total number of texts, n i represents the number of texts containing word i. Since microblog texts are short and a word only appears in a few blogs, A mxn is a sparse matrix and is of high dimension;

[0048] (2) Obtain the co-occurrence matrix A of words and texts mxn After that, on the basis of not losing the semantic relationship of the original text, a k-dimensional semantic space is selected according to k << min(m, n), and low-dimensional word and text vectors are used to express the original space vector, improving the processing speed of a large amount of text sets. In the specific processing process, the SVDLIBC method is used for decomposition to obtain an approximate matrix A mxn of the matrix A k .

[0049] Preferably, an improved collaborative approximate set algorithm: First, perform a preliminary approximate set on the data through a hierarchical approximate set algorithm to find the initial approximate set center C and the initial approximate set number K. Use the approximate set center and the approximate set number as the initial input factors of the K-equilibrium algorithm, and complete the entire approximate set process according to the steps of the K-equilibrium algorithm. Among them, in the calculation of text similarity, a text similarity calculation method combined with time data is used;

[0050] It includes two steps. First, find the initial approximate set center C and the initial approximate set cluster number K through hierarchical approximate set; then use the result as the input factor of the K-equilibrium algorithm and perform approximate set according to the process of the K-equilibrium algorithm;

[0051] The specific process of the improved collaborative approximate set algorithm is as follows:

[0052] Step 1: Find the initial approximate set center C and the initial approximate set cluster number K through the hierarchical approximate set algorithm:

[0053] (1) After text modeling, a matrix is obtained through vectors (i = 1, 2, Λ, n) to represent these n data points to be approximately set;

[0054] (2) Calculate the mutual distances of the n data points, and partially approximate the two data points with the closest distances; among them, in the calculation of text similarity, a text similarity calculation method combined with time data is used;

[0055] (3) Eliminate some isolated points through random sampling: Eliminate clusters smaller than the set critical value when the approximate set is nearly completed and clusters with a class cluster aggregation speed lower than the set critical value during the approximate set process;

[0056] (4) For the central points obtained between the pairwise data in the previous step, that is, obtain the new cluster center, and use F i =(n i , mi), (i = 1, 2, Λ, k) to represent;

[0057] (5) Iteratively execute steps (2), (3), and (4), and calculate the objective function value Q(c) of each layer of class clusters. Select the layer with the smallest Q(c) value. The smaller this value, the more accurate the size of the approximate set number K;

[0058] (6) Calculate F i The mean of the sample data in (i=1, 2, Λ, k) is used as the initial center C of the subsequent K equalization algorithm i (i=1, 2, Λ, k);

[0059] Step 2: Use the initial approximate set center C and the initial approximate set cluster number K as factors of the K-balance algorithm and implement the approximate set:

[0060] (7) Determine whether all the data in matrix A have been processed by approximate set processing in step 1. If so, no further processing is performed. If not, calculate the distances of all cluster centers for the data point and divide it into corresponding clusters according to the minimum distance principle.

[0061] (8) Re-evaluate and update the number, balance, and approximate set characteristics of clusters;

[0062] (9) Iterate steps (7) and (8) until each cluster is no longer updated, that is, the classification result becomes stable.

[0063] Preferably, the text similarity calculation of the combined time data: the time factor is added to the calculation of the text similarity, the specific method is as follows:

[0064]

[0065]

[0066] Among them, S1 and S2 are text vector representations of the hidden semantic calculation method, S1 = {s1_1, s1_2, ..., s1_n}, S2 = {s2_1, s2_2, ..., s2_n}, T(S1) and T(S2) respectively represent the time when the semantic text was published on Weibo (accurate to seconds). The farther the two are apart, the lower the similarity is.

[0067] Preferably, the score of factors influencing the popularity of Weibo topics: the measurement of the popularity of Weibo topics is scored based on the following factors:

[0068] Score 1, user influence: The influence of Weibo users is determined by the number of their followers and whether they are experts in a certain field;

[0069] Rating 2, number of comments: If the blog post is about a topic that many people will pay attention to, it will attract many people to participate in the comments;

[0070] Rating 3, number of reposts;

[0071] Score 4, time characteristics, the time characteristics of the blog post are taken into consideration in the evaluation of the popularity of Weibo topics.

[0072] Preferably, the calculation of the microblog topic heat value: The calculation of the topic heat value takes into account the user's influence, the number of blog comments, the number of reposts, and the publishing time, and constructs the following heat value calculation method:

[0073]

[0074] Among them, λ, μ and η are weight factors, and λ+μ+η=1, H author (i) represents the influence of the user and is defined as follows:

[0075] H author (i)=ζ×Fllowers Formula 7

[0076] In the formula, Fllowers represents the number of followers of the user, ζ is the authority factor, and its value is determined by whether the user is an authority in the relevant field; R(i) represents the number of reposts of blog post i; C(i) represents the number of comments under the original blog post i; H author (j) and C(j) refer to the influence and number of comments of the jth forwarded microblog of original microblog i, respectively;

[0077] After calculating the popularity value of a single microblog message according to the above method, the time factor is included in the calculation of the popularity of the topic. That is, after a microblog message is sent, the impact on the popularity of the topic decreases with the extension of time. This application cools the popularity of microblogs according to this rule based on Newton's cooling law. The cooling method is as follows:

[0078] f'=-α(f(t)-D) Formula 8

[0079] f(t) is a function of the current object temperature with time t as a variable, and its derivative is f'. D represents the ambient temperature, f(t)-D represents the difference between the current object temperature and the ambient temperature, and the parameter α (α>0), -α represents the temperature drop, which is regarded as the parameter value of the cooling speed. Integrating the formula will convert it into:

[0080]

[0081] Where f0 represents the initial temperature, t-t0 is the time interval, and the final formula for the popularity of Weibo topics is as follows:

[0082]

[0083] In the formula, H(topicSet,t) represents the popularity of the topic at time t, t is the current time used in the calculation, n represents the number of blog posts under the topic, and t i The time when the blog post was published, H i It is the initial popularity of Boweni.

[0084] Preferably, the overall architecture design of the multi-dimensional intelligent extraction system for hot topics on Weibo is constructed: from data collection to text segmentation processing, filtering is performed according to the part of speech, the comment text is screened, and the comment information is reorganized to obtain the expanded Weibo text; then VSM is generated according to the expanded text, SVD decomposition is performed on the word text, the Weibo text after semantic calculation is approximated, and the topic heat is calculated and the corresponding page display is performed;

[0085] The overall framework of the system includes: data collection, text preprocessing, hidden semantic calculation, approximate set and popularity calculation, and system display module. The collected Weibo text information is processed through this series of processes to finally obtain the relevant approximate set topics and score their popularity.

[0086] Preferably, each module of the system is designed as follows:

[0087] 1. Data Collection Module

[0088] The platform's data interface is used to obtain initial data, and the extracted microblog content includes initial microblogs, comments and forwarding information, and user information. The interfaces used include the microblog interface, comment interface, and user interface.

[0089] After calling the API, the data will be returned in JSON format. After obtaining the corresponding data, the data will be processed and stored in the local database of this application for subsequent calculation.

[0090] 2. Text preprocessing module

[0091] Clean, filter and reorganize the acquired microblog content and comments, including text selection and cleaning, text segmentation, stop word removal, part-of-speech filtering, comment text screening, and short text expansion;

[0092] 3. Hidden semantic computing module

[0093] The text with higher heat value in the word segmentation and frequency measurement is converted into a word and text matrix. The word frequency feature matrix is ​​modeled using a text vector model. The word and text matrix is ​​decomposed to extract the implicit semantic structure between words, so that the dimension of the text vector can be reduced.

[0094] 4. Approximate set and topic extraction module

[0095] The improved collaborative approximate set algorithm of the present application is used to implement the approximate set, and the time data characteristics are incorporated into the text similarity calculation. The initial approximate set is performed through the hierarchical approximate set algorithm to find the initial approximate set center C and the number of approximate sets K as the initial factors of the K-balanced algorithm, and the entire approximate set process is completed according to the steps of the K-balanced algorithm;

[0096] 5. System display

[0097] A prototype of the system was built using the B / S architecture. The system prototype is divided into data collection, data processing, topic extraction and hot spot calculation, as well as related system interfaces.

[0098] Compared with the existing technology, the above technical solution has the following innovations and advantages:

[0099] First, the topic extraction method of the prior art is to extract and mine information in combination with text content, while Weibo has specific structural features, such as comments, likes, reposts, and mutual attention between users. The present application combines the structural characteristics of Weibo to optimize the modeling of Weibo text, improve the text modeling effect, and enhance the accuracy of topic extraction; the topic extraction method of the prior art is to measure the correlation between words in the text and judge the topic similarity of different texts by a certain frequency of text repetition. The present application models Weibo text based on hidden semantic calculation, taking into account the hidden semantic relationship of words in different contexts, and reducing errors in text approximate sets; the topic extraction method of the prior art does not pay much attention to the time factor. Since the timeliness of Weibo is greatly enhanced compared to traditional media articles, the recent Weibo content can better reflect the latest hot spots than the previous Weibo information. The present application reflects the time factor in the text similarity to truly express the current hot issues and improve the accuracy and timeliness of Weibo topic extraction, which is of great significance and great practical value.

[0100] Second, the amount of microblog data is growing at a high speed. A huge amount of microblog data is accumulated every moment. For a large amount of data, if approximate set calculation is performed directly, strong computing power is required, and accuracy cannot be guaranteed at the same time, because a large amount of information on microblogs is worthless advertisements and records of trivial matters, and there is still less truly valuable data. This application further screens the microblog data, reduces invalid data, and then performs approximate set calculations. In the algorithm, the most suitable approximate set algorithm is found to improve the speed of approximate set calculations, so that the extraction of microblog topics can present the results we need in a timely manner. A set of automatic extraction systems for microblog hot topics is constructed, combined with the characteristics of microblogs, the modeling of microblog texts is optimized, and the approximate set of real-time massive microblog data and the problem of scoring heat are processed. The microblog text is modeled based on short text expansion and hidden semantic calculation methods, and the K-balanced algorithm and the hierarchical approximate set algorithm are integrated to realize the approximate set, and the topic heat value is scored. Finally, it is implemented in the system to meet the functional needs in reality.

[0101] Third, this application proposes a text modeling method based on short text expansion and hidden semantic calculation methods to solve the problems of data noise reduction, insufficient dimensionality reduction, and semantic loss in the text modeling methods of the prior art for extracting Weibo topics. This application proposes a short text expansion method based on the characteristics of Weibo, reorganizes the discussion graph tree to expand the Weibo text, and models the Weibo text through the hidden semantic calculation method, reducing the dimension of the text vector without losing semantics; this application proposes a collaborative approximate set method for extracting Weibo topics, and scores the topic popularity, incorporating the time data characteristics into the text similarity calculation; combined with the characteristics of Weibo text, the application proposes the influencing factors and specific methods for calculating the Weibo topic popularity value, and generates a Weibo hot topic scoring list to improve the accuracy and timeliness of Weibo topics, reduce the loss of topic information, and realize multi-dimensional intelligent and accurate extraction of Weibo big data hot topics.

[0102] Fourth, due to the short text of microblogs and the sparseness of data, it is difficult to directly calculate through ordinary text feature factors. This application makes full use of the characteristics of microblog dissemination and interaction, and proposes a short text expansion method based on the characteristics of microblog publishing and commenting. It solves the difficulties of high vector space dimension and loss of semantic information in text modeling based on the vector space model in the prior art. This application models microblog texts based on hidden semantic calculation methods, weakens the correlation between words, obtains the implicit semantic structure between words, expresses words and texts, and reduces the dimension of text vectors. To meet the processing needs of massive microblog information, an improved collaborative approximate set algorithm is proposed, and time data is combined to calculate similarity to improve the accuracy and timeliness of microblog topic extraction. A microblog hot topic automatic extraction system is constructed. By calculating and analyzing the reprinted and disseminated content of microblogs, the current social public opinion situation can be efficiently grasped, and the network public opinion can be effectively analyzed and grasped. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 It is a specific flow chart of the improved collaborative approximate set algorithm.

[0104] Figure 2 It is the cooling curve diagram of the object under different values ​​of the cooling factor α of the topic.

[0105] Figure 3 This is a framework diagram for building a multi-dimensional intelligent extraction system for hot topics on Weibo.

[0106] Figure 4 It is a flowchart of the Weibo text preprocessing module.

[0107] Figure 5 This is a schematic diagram of the user management interface of the Weibo topic extraction system. DETAILED DESCRIPTION

[0108] In order to make the purpose, characteristic advantages and innovations of this application more obvious, easy to understand and convenient to implement, the specific implementation methods are described in detail below with reference to the accompanying drawings. Those skilled in the art can make similar generalizations without violating the connotation of this application, so this application is not limited by the specific implementation methods disclosed below.

[0109] With the development of social networking and the continuous updating of mobile Internet technology, Weibo has gradually become an important platform for information display and sharing. Weibo's fast transmission speed and wide publicity make it an important source of public opinion and a carrier of communication. By calculating and analyzing the reprinted and disseminated content of Weibo, we can efficiently grasp the current public opinion situation in society and effectively analyze and grasp the network public opinion. However, compared with other media such as blogs and news, Weibo has multiple characteristics such as short content, sparse data, universal participation, and extremely large data volume, which also makes it difficult to extract and analyze Weibo information using traditional methods.

[0110] To address these issues, this application proposes a hot topic extraction method based on massive Weibo data, text modeling based on short text expansion and hidden semantic calculation methods, and automatic extraction of Weibo hot topics based on collaborative approximate sets and heat scores, and at the same time builds a corresponding automatic extraction system.

[0111] (1) Since microblog texts are short and data is sparse, it is difficult to directly calculate them using ordinary text feature factors. Therefore, this application makes full use of the characteristics of microblog dissemination and interaction and proposes a short text expansion method based on the characteristics of microblog publishing and commenting.

[0112] (2) In order to solve the problems of high vector space dimension and loss of semantic information in text modeling based on vector space model commonly used in previous studies, this application models Weibo text based on hidden semantic calculation method, weakens the correlation between words, obtains the implicit semantic structure between words, expresses words and text, and reduces the dimension of text vector.

[0113] (3) In order to meet the needs of processing massive amounts of microblog information, a hierarchical approximate set and K-balanced approximate set method are combined to propose an improved collaborative approximate set algorithm, and time data is combined to calculate similarity. This method can perform hierarchical approximate set on the modeled microblog information, and then determine the initial approximate set center and the number of approximate sets through K-balanced approximate set. It also combines the temporal dimension of hot topics, thereby improving the accuracy and timeliness of microblog topic extraction.

[0114] (4) Construct an automatic extraction system for hot topics on Weibo, and conduct experimental tests on the text modeling method proposed in this application, which is based on the short text expansion method, the hidden semantic calculation method, and the hot topic extraction method that combines collaborative approximate sets and heat scoring. The effectiveness of the model constructed in this application is verified, which can improve the accuracy and timeliness of Weibo topics, reduce the loss of topic information, and meet the functional needs in reality.

[0115] 1. Microblog text modeling based on short text expansion and latent semantics calculation

[0116] Weibo messages are often very brief, but after being forwarded by related comments, they can express a lot of information. The topic extraction methods in the prior art are more suitable for long news reports and commentary articles, but if they are directly applied to the short text phenomenon of Weibo, there will be problems of insufficient data noise reduction and dimensionality reduction, and the extraction of text information is also prone to semantic loss. This application solves this problem through short text expansion and hidden semantic calculation methods.

[0117] (I) Short text expansion of Weibo

[0118] Weibo content is often the interaction and discussion of certain topics between users and their fans. A user can post a Weibo about certain events or issues, and his friends can reply and comment on the Weibo content, or other users can reply and comment after forwarding it. Such a structure expands the original text by extracting relevant reply comments, making it more informative. This can analyze the connotation of the initial topic more widely without introducing additional irrelevant topics, and focuses on the text composed of user comment replies.

[0119] 1. Microblog comment processing based on descriptors

[0120] Replies to Weibo content increase the length of Weibo and expand the content of Weibo. However, due to the large number of Weibo users, there are some meaningless reposts and comments. For example, marketing accounts and zombie users may reply to certain topics in a short period of time. Even some robot accounts usually only reply to some irrelevant content or short likes and support words. There are also simple replies such as "like, support, top" words or emoticons, which express the user's attitude, but if removed, it will not affect the connotation of the Weibo. These comments will affect the quality of topic extraction and reduce the efficiency and accuracy of topic extraction. Therefore, it is necessary to construct reasonable rules to deal with comments for these meaningless problems.

[0121] (1) Delete reply comments that are not long enough. Here, a certain critical length Length is set, and reply comments that are shorter than Length are removed during microblog data extraction;

[0122] (2) Clean and preprocess the content of Weibo, including removing stop words, emoticons, user accounts, and URLs, and performing Chinese word segmentation on Weibo content;

[0123] (3) Delete comments that are not related to the initial microblog content. From the microblog collection of reply comments, extract comments with descriptors similar to the initial microblog. The specific process for executing this rule is as follows:

[0124] Process 1: Take the descriptive cluster of the original microblog as the initial cluster;

[0125] Process 2: Read one comment from the original Weibo in chronological order;

[0126] Process 3: Calculate the spatial distance between the words in the extracted comments and the initial clusters;

[0127] Step 4: Add the normalized spatial distance between the comment and the initial cluster and calculate the sum; if the result is less than the set critical value, add it to the list of useful comments, if not, remove it;

[0128] Execute process 3 and process 4 again until all reply comments are processed and a valid comment list corresponding to the original microblog is obtained.

[0129] When calculating the spatial distance between the comment and the initial microblog, the text approximation is calculated, and the approximate probability of words a and b is calculated:

[0130]

[0131] P(a|b) is the probability of a appearing when b appears. It is equal to the probability of a and b appearing at the same time divided by the probability of b appearing. Calculate the correlation between the word and the descriptive cluster. Let the distance between word w and cluster C be max{P(c i |ω)c i ∈C}, the specific method is as follows:

[0132]

[0133] If c i In cluster C, then c i If the probability of word w appearing in the text containing word w is high and the spatial distance from word w to cluster C is short, word w is added to cluster C; otherwise, the spatial distance from word w to cluster C is large, word w is not added to cluster C.

[0134] 2. Reorganize the discussion graph tree to expand the microblog text

[0135] In order to improve the efficiency of semantic computing and reduce the text storage space, the microblog information is further filtered, screened, and combined before text modeling to meet the needs of subsequent text modeling and similarity calculation. After the initial microblog is published by the user, the object of his attention may comment and forward it. Subsequent users further comment and forward it on this basis, forming a tree structure of the discussion map about the topic. As the spread becomes wider and wider, there are more and more original microblogs related to the same topic, and the related discussions gradually extend and expand, forming a hot topic in a larger range.

[0136] In the discussion graph tree of Weibo, Weibo content is reorganized and expanded to form long text of Weibo, including two situations: one is the continuous forwarding of Weibo. For Weibo content of interest, there are layers of continuous forwarding among Weibo fans, that is, users hope to attract other people's attention for a certain Weibo information and hope to share it with other people. In the Weibo system, forwarded blog posts are marked; the other is the comment dialogue in the reply. Under the same popular Weibo, there will be a lot of replies and mutual comments. In this way, many people participate in a certain Weibo or comment on it, and they will point to the tag @Weibo blogger when talking.

[0137] Microblog text information extraction combines the presentation of microblogs and reorganizes all the continuous forwarding and comment replies under the initial microblog as an extension of the microblog content to extract and calculate topics. Before reorganization, the collected data set Set <blog>Scoring in chronological order to obtain SortedSet <blog>, and then use it as the input of the algorithm, and through the algorithm processing, it is converted into a collection of multiple texts, and the repeated information has been reorganized.

[0138] (1) Reorganize and forward Weibo posts

[0139] Use tags on the page to filter the Set <blog>If the microblog is an initial microblog message or contains a forwarding tag, such microblog information is added to the SortedSet <blog>In the example, the HashValue of the original microblog is used as the key of the textMap, and the reorganized TextSet <blog>After adding them to the corresponding positions and reorganizing them according to the algorithm, the values ​​in textMap are all text sets with reorganized repeated information.

[0140] Combined with the results of the above method, we understand the scope of the final text set, that is, the initial microblogs that have not been forwarded, and the repeated microblog content that has been forwarded continuously. After processing, it is marked as yeD as the initial input set and proceeds to the next step.

[0141] (2) Multi-microblog reorganization of dialogue and comments

[0142] After the continuous forwarding microblog algorithm is used to reorganize the microblogs, the remaining text set yeD contains microblog information with the indicator label @. The multiple microblogs with conversations and comments are reorganized to construct their own dialogue tree structures. The core is the time order of microblog publication and the microblog information transmission relationship corresponding to the label @.

[0143] First, arrange the microblog information in yeD constructed in the previous step in order according to their release time, then query the entire yeD. For microblogs that do not contain @ pointing to a matching microblog, use it as the root node to build a dialogue tree, and add the newly built tree to discussTreeSet; for forwarded and commented microblogs, search discussTreeSet until: ① the publisher of a microblog in the tree node is the person the microblog points to; ② the release time difference between the two microblogs in all nodes that meet condition ① is the smallest, and then add the current microblog information as its child node information to the original dialogue tree.

[0144] After being processed by the multi-microblog reorganization algorithm of conversations and comments, the microblog information in yeD is reorganized into a conversation tree, and the conversation trees with only root nodes or very low tree heights are eliminated, that is, microblog information with very few comments or reposts. The microblog information with the same conversation tree structure is reorganized into a text according to the chronological order of release time to obtain meaningful long text information.

[0145] (II) Microblog text modeling based on hidden semantic computing

[0146] The high dimensionality and semantic relevance of short microblog texts bring difficulties to approximate sets. This application is based on a hidden semantic calculation method to reduce the dimension of the feature space while expressing the semantic information behind the text without loss, thereby modeling the microblog text.

[0147] Through a large amount of text calculations, a text-word matrix is ​​constructed, and then the hidden semantic structure of words and text is extracted. The specific steps are as follows:

[0148] Let m represent the number of different words in the text, n represent the number of texts, and the word-text co-occurrence matrix is ​​A mxn Indicates that a ij The factor representing the i-th word in the j-th text, where the relevance between the text and the word is low, A mxn At this time, it is a matrix with sparse density;

[0149] For A mxn After decomposition, the approximate matrix A of A mx is obtained k , where k << min(m, n), to achieve dimensionality reduction of the text vector. The decomposition process is as follows:

[0150] (1) First, construct the co-occurrence matrix A, whose row vectors each correspond to a word, and the column vectors each correspond to a text;

[0151] (2) Assume there is a decomposition of matrix A: A = U∑V T , where U and V are orthogonal, ∑ is a diagonal matrix, and we get: AAA T = U∑∑ T U T and A T A = V∑ T ∑V T , ∑∑ T and ∑ T ∑ are diagonal, and U and V respectively cover the eigenvectors of AA T and A T A;

[0152] (3) Let k be the largest singular value. Among U, V and their corresponding singular vectors, k has the smallest approximation error to A. Consider this word and the corresponding text vector as a semantic space. The vector has k entries. Map the vector to a lower space to obtain the approximate matrix and achieve dimensionality reduction of the text vector.

[0153] For the collected microblog text information, the hidden semantic computing method is modeled as follows:

[0154] The first step: Construct the co-occurrence matrix A of words and texts m×n = [a ij m×n , assuming there are n microblog texts, which contain m different words, to form a matrix A mxn , a ij is calculated by TF-IDF. The method is as follows:

[0155]

[0156] tf ij ​represents the frequency of a word in text j, N represents the total amount of text, and n i represents the number of texts containing word i. Weibo texts are short, and a word only appears in very few posts. A mxn is a sparse matrix and is of high dimension.

[0157] (2) Obtain the co-occurrence matrix A of words and texts mxn After that, on the basis of not losing the original text semantic relationship, according to k << min(m, n), select a k-dimensional semantic space, use low-dimensional word and text vectors to represent the original space vector, improve the processing speed of the massive text set, and in the specific processing process, use the SVDLIBC method for decomposition to obtain mxn the approximate matrix A of A k .

[0158] II. Extraction of Weibo hot topics based on collaborative approximate sets and popularity scoring

[0159] Users post a large number of Weibo posts every day, generating a massive amount of information data. How to process this massive amount of Weibo data in a timely manner to extract the required hot topics faces great difficulties.

[0160] (I) Collaborative approximate sets of Weibo texts

[0161] 1. Improved collaborative approximate set algorithm

[0162] First, use the hierarchical approximate set algorithm to perform a preliminary approximate set on the data, find the initial approximate set center C and the initial number of approximate sets K, use the approximate set center and the number of approximate sets as the initial input factors of the K-equilibrium algorithm, and complete the entire approximate set process according to the steps of the K-equilibrium algorithm. Among them, in the calculation of text similarity, a text similarity calculation method combined with time data is used.

[0163] It includes two steps. First, find the initial approximate set center C and the initial number of approximate set clusters K through the hierarchical approximate set; then use the result as the input factor of the K-equilibrium algorithm and perform an approximate set according to the process of the K-equilibrium algorithm. As Figure 1 shown.

[0164] The specific process of the improved collaborative approximate set algorithm is as follows:

[0165] Step 1: Find the initial approximate set center C and the initial number of approximate set clusters K through the hierarchical approximate set algorithm:

[0166] (1) After text modeling, obtain matrix A m×n = [a ij m×n , through vector A i [a i1 , a​ i2 , Λ, a ij ] (i=1, 2, Λ, n) represents the set of n data points to be approximated;

[0167] (2) Find the mutual distance of n data points, and partially approximate the data points with the closest distances to each other; in the calculation of text similarity, the text similarity calculation method of time data is combined;

[0168] (3) Eliminate some isolated points through random sampling: when the approximate set is nearly complete, eliminate clusters whose clustering speed is lower than the set critical value and clusters whose clustering speed is lower than the set critical value during the approximate set process;

[0169] (4) Find the center point between the two data points obtained in the previous step, that is, get the new cluster center, and use F i =(n i , m i ), (i=1, 2, Λ, k) to represent;

[0170] (5) Iterate steps (2), (3), and (4), and calculate the objective function value Q(c) of each layer of clusters. Select the layer that minimizes the Q(c) value. The smaller the value, the more accurate the size of the approximate set number K.

[0171] (6) Calculate F i The mean of the sample data in (i=1, 2, Λ, k) is used as the initial center C of the subsequent K equalization algorithm i (i=1, 2, Λ, k);

[0172] Step 2: Use the initial approximate set center C and the initial approximate set cluster number K as factors of the K-balance algorithm and implement the approximate set:

[0173] (7) Determine whether all the data in matrix A have been processed by approximate set processing in step 1. If so, no further processing is performed. If not, calculate the distances of all cluster centers for the data point and divide it into corresponding clusters according to the minimum distance principle.

[0174] (8) Re-evaluate and update the number, balance, and approximate set characteristics of clusters;

[0175] (9) Iterate steps (7) and (8) until each cluster is no longer updated, that is, the classification result becomes stable.

[0176] 2. Text Similarity Calculation for Joint Time Data

[0177] In the process of approximating the set, the calculation of the approximation of the text content is an important step in the approximation set. Since the update speed of Weibo is very fast, people's focus often changes very quickly. It may take only one or two days or even only a few hours for a topic to be no longer popular from its generation. Therefore, in the extraction of hot topics on Weibo, the time factor must be taken as an important consideration. Therefore, this application adopts a text similarity algorithm that combines the time factor and adds the time factor to the calculation of text similarity. The specific method is as follows:

[0178]

[0179]

[0180] Among them, S1 and S2 are text vector representations of the hidden semantic calculation method, S1 = {s1_1, s1_2, ..., s1_n}, S2 = {s2_1, s2_2, ..., s2_n}, T(S1) and T(S2) respectively represent the time when the semantic text was published on Weibo (accurate to seconds). The farther the two are apart, the lower the similarity is.

[0181] (II) Calculation of Weibo Topic Heat

[0182] 1. Score of factors affecting the popularity of Weibo topics

[0183] Topic clusters are obtained by approximately grouping microblog texts. The number of blog posts in a topic cluster only partially reflects the popularity of the topic. It is possible that there are many people paying attention to the field itself. To determine whether a topic is a hot topic, whether it is widely spread, whether it has received participation from many people, and whether it has received widespread attention in the short term, we will judge from the following three aspects:

[0184] (1) Topic participation: Hot topics on Weibo must have attracted a large number of users to participate. Many people pay attention to and comment on the topic. Therefore, the number of participants in the topic, the number of related original blog posts, reposts and comments, including the number of blog posts that have been reposted and commented on multiple times, is counted.

[0185] (2) Participating user groups: A blog post published by a blogger with millions of fans has a completely different influence and popularity from a blog post published by an ordinary user with only a few hundred fans. There are some public figures, celebrities, official Weibo accounts, and bloggers with a large number of fans on Weibo. Since they have a lot of attention, the topics discussed by them are likely to attract the attention and participation of other users in a short period of time, thus forming hot topics for discussion.

[0186] (3) Time when topics are formed: Hot topics on Weibo are formed quickly, but they also disappear quickly because there are always new hot issues that attract people’s attention. The most recent discussion is more likely to be considered the latest hot topic, while the attention to topics that were released earlier can easily dissipate quickly.

[0187] Therefore, this application measures the popularity of Weibo topics based on the following factors:

[0188] Score 1, user influence: The influence of Weibo users is determined by the number of their followers and whether they are experts in a certain field;

[0189] Rating 2, number of comments: When a Weibo user sees a blog post and is interested in it, he or she is likely to comment on it; if the blog post is a topic that many people are interested in, it will attract many people to participate in the comments;

[0190] Score 3, number of reposts: In addition to commenting on blog posts that interest them, people may also repost them so that their followers can also see them. Especially for certain topics, if an authoritative person reposts them, it is easier for other users to repost and comment on them, thus quickly becoming a hot topic of people's attention in a short period of time.

[0191] Rating 4, time characteristics: Even if the number of comments and reposts on two topics is similar, if one reaches this amount in just a few hours, while the other takes a week to reach this amount, the former is more likely to be considered a hot topic. The time characteristics of the blog post are taken into account in the evaluation of the popularity of Weibo topics.

[0192] 2. Calculation of Weibo topic heat value

[0193] The calculation of topic heat value takes into account the user's influence, number of blog comments, number of reposts, and publishing time, and constructs the following heat value calculation method:

[0194]

[0195] Among them, λ, μ and η are weight factors, and λ+μ+η=1, H author (i) represents the influence of the user and is defined as follows:

[0196] H author (i)=ζ×Fllowers Formula 7

[0197] In the formula, Fllowers represents the number of followers of the user, ζ is the authority factor, and its value is determined by whether the user is an authority in the relevant field; R(i) represents the number of reposts of blog post i; C(i) represents the number of comments under the original blog post i; Hautho r C(j) and C(j) refer to the influence and number of comments of the jth forwarded microblog of original microblog i, respectively.

[0198] After calculating the popularity value of a single microblog message according to the above method, the time factor is included in the calculation of the popularity of the topic. That is, after a microblog message is sent, the impact on the popularity of the topic decreases with the extension of time. This application cools the popularity of microblogs according to this rule based on Newton's cooling law. The cooling method is as follows:

[0199]

[0200] f(t) is a function of the current object temperature with time t as a variable, and its derivative is f'. D represents the ambient temperature, f(t)-D represents the difference between the current object temperature and the ambient temperature, and the parameter α (α>0), -α represents the temperature drop, which is regarded as the parameter value of the cooling speed. Integrating the formula will convert it into:

[0201]

[0202] Where f0 represents the initial temperature, t-t0 is the time interval, and the final formula for the popularity of Weibo topics is as follows:

[0203]

[0204] In the formula, H(topicSet,t) represents the popularity of the topic at time t, t is the current time used in the calculation, n represents the number of blog posts under the topic, and t i The time when the blog post was published, H i is the initial popularity of blog post i (without time factor), and this application temporarily takes the value of 0.10. Figure 2 Cooling curve of the reaction object under different values ​​of the cooling factor α of the topic.

[0205] 3. Building a multi-dimensional intelligent extraction system for hot topics on Weibo

[0206] (I) System overall architecture design

[0207] From data collection to text segmentation, filtering is performed according to parts of speech, comment texts are screened, and comment information is reorganized to obtain expanded Weibo texts; then VSM is generated based on the expanded text, SVD decomposition is performed on the word text, and the Weibo texts after semantic calculation are approximated and aggregated, and the topic popularity is calculated and displayed on the corresponding page.

[0208] Figure 3 This is the overall framework design diagram of the system. It includes: data collection, text preprocessing, hidden semantics calculation, approximate set and heat calculation, and system display module. The collected microblog text information is processed through this series of processes to finally obtain the relevant approximate set topics and score their heat.

[0209] (II) Design of each module of the system

[0210] 1. Data Collection Module

[0211] The platform's data interface is used to obtain initial data, and the extracted microblog content includes initial microblogs, comments and forwarding information, and user information. The interfaces used include the microblog interface, comment interface, and user interface.

[0212] After calling the API, the data will be returned in JSON format. After obtaining the corresponding data, the data will be processed and stored in the local database of this application for subsequent calculation.

[0213] 2. Text preprocessing module

[0214] Clean, filter and reorganize the acquired microblog content and comments, including text selection and cleaning, text segmentation, removal of stop words, part-of-speech filtering, comment text screening, and expansion of short texts, such as Figure 4 shown.

[0215] (1) Selection and cleaning of microblog texts

[0216] Microblogs with very low attention do not generate value for hot topics. The method of this application is used to calculate the explicit heat of microblogs. Microblogs with heat values ​​less than a certain critical value are discarded and not processed. Such unpopular microblogs can be abandoned in the preliminary judgment and will not have a substantial impact on the mining of hot topics, but will have a positive help in improving the efficiency and accuracy of subsequent approximate collection work. In addition, the microblog text needs to be cleaned. First, user accounts, emoticons and website links that do not meet the requirements are deleted. After removing these contents, if the number of words in the microblog text is too short and the number of comments and reposts is lower than the critical value, they are also discarded.

[0217] (2) Text segmentation, stop word removal, and part-of-speech filtering

[0218] Remove meaningless content and keep important information of microblog text. Use word segmentation system ICTCLAS to process microblog text. In order to remove meaningless words, reduce unnecessary resource consumption of the system, and improve the quality of approximate sets, the corresponding part-of-speech tagging is obtained after Chinese word segmentation. At this time, other meaningless words, including punctuation marks, auxiliary words, prepositions, and interjections are removed, and nouns and verbs are retained to retain the meaning of the microblog.

[0219] (3) Screening and expansion of microblog texts

[0220] Obtain microblog text word clusters, including the initial blog post and related comment list, and expand the microblog short text method. Process the microblog text word clusters and reorganize them to obtain the number of discussions of the microblog text for subsequent calculation and modeling.

[0221] 3. Hidden semantic computing module

[0222] The text with higher popularity values ​​in the word segmentation and frequency measurement is converted into a word and text matrix. The word frequency feature matrix is ​​modeled using a text vector model, the word and text matrix is ​​decomposed, and the implicit semantic structure between words is extracted, so that the dimension of the text vector can be reduced.

[0223] 4. Approximate set and topic extraction module

[0224] The improved collaborative approximate set algorithm of this application is used to implement approximate sets, and the time data characteristics are incorporated into the text similarity calculation. The hierarchical approximate set algorithm is used to perform preliminary approximate sets, and the initial approximate set center C and the number of approximate sets K are found as the initial factors of the K-balanced algorithm. The entire approximate set process is completed according to the steps of the K-balanced algorithm.

[0225] This application also needs to screen out hot topics and score them. The approximate set results not only form topic clusters, but are also regarded as a collection of feature words. The calculation method of the topic heat value of this application is adopted here, combining the user's influence, number of blog comments, number of reposts, and publication time factors. After calculating the topic heat value, the top topic cluster is scored, which is the current hottest topic.

[0226] In the final topic cluster, find the initial microblog that is closest to the approximate set result, and find the initial blog post whose length is less than a certain set value as the final presentation form of the topic.

[0227] 5. System display

[0228] The prototype of the system was built using the B / S architecture. The system prototype is divided into data collection, data processing, topic extraction and hot spot calculation, as well as related system interfaces. Its functional interface is as follows Figure 5 .< / blog> < / blog> < / blog> < / blog> < / blog> < / blog> < / blog> < / blog> < / blog> < / blog>

Claims

1. A multi-dimensional intelligent extraction system for hot topics in Weibo big data, characterized by: Taking advantage of the characteristics of microblog dissemination and interaction, we built a hot topic extraction model based on massive microblog data, built text modeling based on short text expansion and hidden semantic computing methods, and automatically extracted microblog hot topics based on collaborative approximate sets and heat scores, and built a corresponding automatic extraction system; S1, Weibo text modeling by combining short text expansion and hidden semantics calculation: A short text expansion method is proposed based on the characteristics of Weibo, and the Weibo text is expanded by reorganizing the discussion graph tree. The Weibo text is modeled by hidden semantics calculation method, which weakens the correlation between words, obtains the implicit semantic structure between words, expresses words and texts, and reduces the dimension of text vector without losing semantics. S2, extraction of hot topics on Weibo based on collaborative approximate sets and heat scoring, including: improved collaborative approximate set algorithm, text similarity calculation of joint time data, score of factors affecting the heat of Weibo topics, and calculation of the heat value of Weibo topics; integration of hierarchical approximate sets and K-balanced algorithms and improvement, and incorporation of time data characteristics into text similarity calculation; introduction of factors affecting the heat value calculation and implementation methods of topics combined with the characteristics of Weibo texts, and generation of a score list of Weibo hot topics; Improved collaborative approximate set algorithm: First, the data is preliminarily approximated through the hierarchical approximate set algorithm to find the initial approximate set center C and the initial approximate set number K. The approximate set center and the approximate set number are used as the initial input factors of the K-balanced algorithm. The entire approximate set process is completed according to the steps of the K-balanced algorithm. In the calculation of text similarity, the text similarity calculation method of time data is combined; It includes two steps. First, find the initial approximate set center C and the initial approximate set cluster number K through the hierarchical approximate set. Then, use the result as the input factor of the K-balanced algorithm and perform the approximate set according to the process of the K-balanced algorithm. S3, builds a multi-dimensional intelligent extraction system for Weibo hot topics, models text based on short text expansion method and hidden semantic calculation method constructed by text modeling representation method, and integrates collaborative approximate set and heat scoring hotspot extraction method to improve extraction accuracy and timeliness, reduce topic information loss, and realize automatic extraction and scoring display of Weibo hot topics; Calculate text similarity with combined time data: Add the time factor to the calculation of text similarity. The specific method is as follows: Where S1 and S2 are text vector representations of the hidden semantic calculation method S1 = {s1_1, s1_2, ..., s1_n}, S2 = {s2_1, s2_2, ..., s2_n}, T(S1) and T(S2) represent the time when the semantic text was published on Weibo. The farther the two are apart, the lower the similarity is. Construct the overall architecture design of the multi-dimensional intelligent extraction system for hot topics on Weibo: from data collection to text segmentation, filter according to part of speech, screen the comment text, reorganize the comment information to obtain the expanded Weibo text; then generate VSM based on the expanded text, perform SVD decomposition on the word text, approximate the Weibo text after semantic calculation, calculate the topic popularity and display the corresponding page; The overall framework of the system includes: data collection, text preprocessing, hidden semantic calculation, approximate set and popularity calculation, and system display module, which will collect Weibo text information, finally obtain relevant approximate set topics and score their popularity.

2. According to the multi-dimensional intelligent extraction system of hot topics from Weibo big data according to claim 1, it is characterized in that: Microblog comment processing based on descriptors: (1) Delete reply comments that are not long enough. Here, a certain critical length Length is set, and reply comments that are shorter than Length are removed during microblog data extraction; (2) Clean and preprocess the content of Weibo, including removing stop words, emoticons, user accounts, and URLs, and performing Chinese word segmentation on Weibo content; (3) Delete comments that are not related to the initial microblog content. From the microblog collection of reply comments, extract comments with descriptors similar to the initial microblog. The specific process for executing this rule is as follows: Process 1: Take the descriptive cluster of the original microblog as the initial cluster; Process 2: Read one comment from the original Weibo in chronological order; Process 3: Calculate the spatial distance between the words in the extracted comments and the initial clusters; Step 4: Add the normalized spatial distance between the comment and the initial cluster and calculate the sum; if the result is less than the set critical value, add it to the list of useful comments, if not, remove it; Execute steps 3 and 4 again until all reply comments are processed and a valid comment list corresponding to the original microblog is obtained; When calculating the spatial distance between the comment and the initial microblog, the text approximation is calculated, and the approximate probability of words a and b is calculated: P(a|b) is the probability of a appearing when b appears. It is equal to the probability of a and b appearing at the same time divided by the probability of b appearing. Calculate the correlation between the word and the descriptive cluster. Let the distance between word w and cluster C be max{P(c i |ω)c i ∈C}, the specific method is as follows: If c i In cluster C, then c i If the probability of word w appearing in the text containing word w is high and the spatial distance from word w to cluster C is short, word w is added to cluster C; otherwise, the spatial distance from word w to cluster C is large, word w is not added to cluster C.

3. According to claim 1, the multi-dimensional intelligent extraction system for hot topics in microblog big data is characterized in that: Reorganize the discussion graph tree to expand the microblog text: In the discussion graph tree of Weibo, the microblog content is reorganized and expanded to form the long text of the Weibo, including two situations: one is the continuous forwarding of microblogs, and the forwarded blog posts are marked with @; the other is the comment dialogue in the reply, and the conversation will point to the tag @Weibo blogger; Microblog text information extraction combines the presentation of microblogs and reorganizes all the continuous forwarding and comment replies under the initial microblog as an extension of the microblog content to extract and calculate topics. Before reorganization, the collected data set Set <blog>Scoring in chronological order to obtain SortedSet <blog> , and then use it as the input of the algorithm, and through the algorithm processing, it is converted into a collection of multiple texts, and the repeated information has been reorganized;< / blog> < / blog> (1) Reorganize and forward Weibo posts Use tags on the page to filter the Set <blog>If the microblog is an initial microblog message or contains a forwarding tag, such microblog information is added to the SortedSet <blog>In the example, the HashValue of the original microblog is used as the key of the textMap, and the reorganized TextSet <blog> After adding them to the corresponding positions and reorganizing them according to the algorithm, the values ​​in textMap are all text sets with reorganized repeated information;< / blog> < / blog> < / blog> Understand the scope of the final text set, that is, the initial microblogs that have not been forwarded, and the repeated microblog content that has been forwarded continuously. After processing, mark it as yeD as the initial input set and proceed to the next step of processing; (2) Multi-microblog reorganization of dialogue and comments After the continuous forwarding microblog algorithm is used to process, the remaining text set yeD contains microblog information with the indicator label @. The multiple microblogs with dialogues and comments are reorganized to construct their own dialogue tree structures. The core is the time order of microblog posting and the microblog information transmission relationship corresponding to the label @. First, sort the microblog information in yeD constructed in the previous step according to the order of their release time, then query the entire yeD, and for microblogs that do not contain @ pointing to a match, use it as the root node to build a conversation tree, and add the newly created tree to discussTreeSet; For forwarded and commented microblogs, the discussTreeSet is searched until: ① the publisher of a microblog in the tree node is the person the microblog points to; ② the publishing time difference between the two microblogs in all nodes that meet condition ① is the smallest, and then the current microblog information is added to the original conversation tree as its child node information; After being processed by the multi-microblog reorganization algorithm of conversations and comments, the microblog information in yeD is reorganized into a conversation tree, and the conversation trees with only root nodes or very low tree heights are eliminated, that is, microblog information with very few comments or reposts. The microblog information with the same conversation tree structure is reorganized into a text according to the chronological order of release time to obtain meaningful long text information.

4. According to claim 1, the multi-dimensional intelligent extraction system for hot topics in microblog big data is characterized in that: Microblog text modeling based on hidden semantic computing: Through a large amount of text computing, a text-word matrix is ​​constructed, and then the hidden semantic structure of words and text is extracted. The specific steps are as follows: Let m represent the number of different words in the text, n represent the number of texts, and the word-text co-occurrence matrix is ​​A mxn Indicates that a ij Represents the factor of the i-th word in the j-th text. The text and the word are not very relevant. A mxn At this time, it is a sparse matrix; For A mxn After decomposition, the approximate matrix A mx of A k is obtained, where k << min(m, n), to achieve dimensionality reduction of the text vector. The decomposition process is as follows: (1) First, construct the co-occurrence matrix A. Its row vector Each corresponds to a word, and the column vector All correspond to one text; (2) Assume that there is a decomposition of the matrix A: A = U∑V T , U and V are orthogonal, ∑ is a diagonal matrix, and we get: AAA T =U∑∑ T U T and A T A=V∑ T ∑V T ,∑∑ T and∑ T ∑ Diagonally, U and V cover AA respectively T and A T The eigenvector of A; (3) Let k be the largest singular value. Among U, V and their corresponding singular vectors, the error of k approaching A is the smallest. The word and the corresponding text vector are regarded as a semantic space. There are k entries, and the vector Mapped to a lower space, we get an approximate matrix Realize dimensionality reduction of text vectors; The collected microblog text information is modeled using the hidden semantic computing method as follows: Step 1: Construct a co-occurrence matrix of words and texts Assume there are n microblog texts, each containing m different words, forming a matrix A mxn , a ij It is calculated by TF-IDF as follows: tf ij represents the frequency of word occurrence in text j, N represents the total amount of text, n i Represents the number of texts containing word i. Microblog texts are short. A mxn It is a sparse matrix and has high dimensions; (2) Obtain the co-occurrence matrix A of words and texts mxn After that, on the basis of not losing the semantic relationship of the original text, according to k << min(m, n), select a k-dimensional semantic space, use low-dimensional word and text vectors to express the original space vectors, improve the processing speed of a large number of text sets, and in the specific processing process, use the SVDLIBC method for decomposition to obtain A mxn The approximate matrix A of k .

5. According to claim 1, the multi-dimensional intelligent extraction system for hot topics in microblog big data is characterized in that: The specific process of the improved collaborative approximate set algorithm is as follows: Step 1: Find the initial approximate set center C and the initial approximate set cluster number K through the hierarchical approximate set algorithm: (1) Matrix obtained after text modeling By vector (i=1, 2, Λ, n) represents the set of n data points to be approximated; (2) Find the mutual distance of n data points, and partially approximate the data points with the closest distances to each other; in the calculation of text similarity, the text similarity calculation method of time data is combined; (3) Eliminate some isolated points through random sampling: when the approximate set is nearly complete, eliminate clusters whose clustering speed is lower than the set critical value and clusters whose clustering speed is lower than the set critical value during the approximate set process; (4) Find the center point between the two data points obtained in the previous step, that is, get the new cluster center, and use F i =(n i , m i ), (i=1, 2, Λ, k) to represent; (5) Iterate steps (2), (3), and (4), and calculate the objective function value Q(c) of each layer of clusters. Select the layer that minimizes the Q(c) value. The smaller the value, the more accurate the size of the approximate set number K. (6) Calculate F i The mean of the sample data in (i=1, 2, Λ, k) is used as the initial center C of the subsequent K equalization algorithm i (i=1, 2, Λ, k); Step 2: Use the initial approximate set center C and the initial approximate set cluster number K as factors of the K-balance algorithm and implement the approximate set: (7) Determine whether all the data in matrix A have been processed by approximate set processing in step 1. If so, no further processing is performed. If not, calculate the distances of all cluster centers for the data point and divide it into corresponding clusters according to the minimum distance principle. (8) Re-evaluate and update the number, balance, and approximate set characteristics of clusters; (9) Iterate steps (7) and (8) until each cluster is no longer updated, that is, the classification result becomes stable.

6. According to claim 1, the multi-dimensional intelligent extraction system for hot topics in microblog big data is characterized in that: Score of factors influencing the popularity of Weibo topics: The popularity of Weibo topics is measured and scored based on the following factors: Score 1, user influence: The influence of Weibo users is determined by the number of their followers and whether they are experts in a certain field; Rating 2, number of reviews; Rating 3, number of reposts; Score 4, time characteristics, the time characteristics of the blog post are taken into consideration in the evaluation of the popularity of Weibo topics.

7. According to claim 1, the multi-dimensional intelligent extraction system for hot topics in microblog big data is characterized in that: Calculation of the topic heat value on Weibo: The calculation of the topic heat value takes into account the user's influence, the number of blog comments, the number of reposts, and the time of publication, and constructs the following heat value calculation method: Among them, λ, μ and η are weight factors, and λ+μ+η=1, Hauthor(i) represents the influence of the user, which is defined as follows: H author (i)=ζ×Fllowers Formula 7 In the formula, Fllowers represents the number of followers of the user, ζ is the authority factor, and its value is determined by whether the user is an authority in the relevant field; R(i) represents the number of reposts of blog post i; C(i) represents the number of comments under the original blog post i; H author (j) and C(j) refer to the influence and number of comments of the jth forwarded microblog of original microblog i, respectively; After calculating the popularity value of a single microblog message according to the above method, the time factor is included in the calculation of the popularity of the topic. That is, after a microblog message is sent, the impact on the popularity of the topic decreases with the extension of time. This application cools the popularity of microblogs according to this rule based on Newton's cooling law. The cooling method is as follows: f′=-α(f(t)-D) Formula 8 f(t) is the current object temperature as a function of time t, and its derivative is f ， , D represents the ambient temperature, f(t)-D represents the difference between the current object temperature and the ambient temperature, parameter α (α>0), -α represents the temperature drop, which is regarded as the parameter value of the cooling speed. Integrating the formula will convert it into: Where f0 represents the initial temperature, t-t0 is the time interval, and the final formula for the popularity of Weibo topics is as follows: In the formula, H(topicSet,t) represents the popularity of the topic at time t, t is the current time used in the calculation, n represents the number of blog posts under the topic, and t i The time when the blog post was published, H i It is the initial popularity of Boweni.

8. According to claim 1, the multi-dimensional intelligent extraction system for hot topics in microblog big data is characterized in that: Design of each module of the system: (1) Data Collection Module Use the platform's data interface to obtain initial data and extract the content of the microblog, including the initial microblog, comments and forwarding information, and user information. The interfaces used include the microblog interface, comment interface, and user interface; After calling the API, the data will be returned in JSON format. After obtaining the corresponding data, the data will be processed and stored in the local database of this application for subsequent calculation; (2) Text preprocessing module Clean, filter and reorganize the acquired microblog content and comments, including text selection and cleaning, text segmentation, stop word removal, part-of-speech filtering, comment text screening, and short text expansion; (3) Hidden semantic computing module The text with high heat value in the word segmentation and frequency measurement is converted into a word and text matrix. The word frequency feature matrix is ​​modeled using a text vector model. The word and text matrix is ​​decomposed to extract the implicit semantic structure between words, so that the dimension of the text vector can be reduced. (4) Approximate set and topic extraction module The improved collaborative approximate set algorithm of the present application is used to implement the approximate set, and the time data characteristics are incorporated into the text similarity calculation. The initial approximate set is performed through the hierarchical approximate set algorithm to find the initial approximate set center C and the number of approximate sets K as the initial factors of the K-balanced algorithm, and the entire approximate set process is completed according to the steps of the K-balanced algorithm; (5) System display A prototype of the system was built using the B / S architecture. The system prototype is divided into data collection, data processing, topic extraction and hot spot calculation, as well as related system interfaces.

Images