Hotspot acquisition method and hotspot popularity analysis method for web text data

Abstract

The invention provides a hotspot acquisition method and a hotspot popularity analysis method for web text data, which comprises the following steps: filtering network data according to a hotspot rule, and extracting a plurality of data elements related to hotspots; performing word segmentation processing on the data elements to obtain feature words, and generating data element vectors according to the feature words; a plurality of clustering centers in the data element vectors are determined based on a clustering strategy, similar clustering centers are combined to determine class centers, the class centers are used for representing the hotspots, and the data elements comprise text features and non-text features. According to the invention, the multi-angle data in the data element is fused into one data element vector, so that the hot spot discovery accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network heat ranking, and in particular to a hotspot acquisition method and a hotspot heat analysis method for network text data. Background Art

[0002] In recent years, with the development and popularization of Internet technology, the complexity and diversity of network hot spots have increasingly affected users' ability to quickly and accurately locate effective information. Internet information is characterized by massive, dynamic, and diverse heterogeneity. Due to a large number of correlations and sporadic factors, public opinion has become more complex and changeable, and traditional public opinion monitoring and judgment methods are difficult to work. Massive data has strengthened the "blind man and elephant" effect of public opinion, and extreme views are more easily amplified through the network multiplier effect, thereby strengthening the extreme emotions of public opinion.

[0003] The research on online public opinion hot spots is currently focused on capturing and analyzing online public opinion hot spots, and is gradually developing towards computer automatic discovery and prediction. However, the research on online public opinion hot spot analysis lacks a heat evaluation index system, and often searches for necessary supporting data that conforms to hot events from a large amount of data based on algorithms or models. However, the errors of various algorithms or models in capturing and analyzing hot spots are large, and the correctness of various algorithms or models cannot be verified.

[0004] Therefore, a hotspot acquisition method and a hotspot heat analysis method are needed, which can comprehensively obtain more network data related to hotspots and improve the accuracy of hotspot analysis and hotspot heat arrangement. Summary of the invention

[0005] In view of this, the problem to be solved by the present invention is to provide a hotspot acquisition method and a hotspot thermal analysis method for network text data, which can fuse multi-angle data in a data element into a data element vector and improve the accuracy of hotspot discovery.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for obtaining hot spots of network text data, comprising filtering network data according to hot spot rules and extracting a number of data elements related to the hot spots;

[0008] Data element word segmentation is performed to obtain feature words, and data element vectors are generated based on the feature words;

[0009] Based on the clustering strategy, several cluster centers within the data element vector are determined, similar cluster centers are merged to determine the class center, and the class center is used to represent the hot spot.

[0010] Furthermore, the data elements include text features and non-text features, the text features include keywords, post topics, user names, and mention behavior identifiers, and the non-text features include URLs, images, and videos.

[0011] Furthermore, the filtering of network data includes time filtering: setting a window and a sliding step in time units, extracting data elements that fall within the window period according to the generation time sequence of the data elements, and removing data elements that do not fall within the window period.

[0012] Furthermore, the filtering of network data includes data filtering: calling a translation model to identify data elements according to the language of the data elements, identifying data elements through keyword extraction, natural language processing and semantic analysis, matching the identification results with hot spot rules, and extracting data elements that meet the hot spot rules.

[0013] Furthermore, before executing the step of identifying the data element, a language identification and filtering step is included: the language of the data element is determined based on the CDFS model, the LDIG model of infinite N-Gram and the LDPN model of tower N-Gram, and non-target language information in the data element is deleted.

[0014] Furthermore, the feature word weights are calculated using the tf-idf model, and a data element vector is generated based on the VSM model and the feature word weights.

[0015] Furthermore, the clustering strategy includes setting cluster centers, randomly selecting K points from the data set as initial cluster centers, where K represents a numerical value and is a positive integer;

[0016] Update the cluster center, calculate the distances from the remaining data elements to all cluster centers, add the data element to the class to which the closest cluster center belongs, and calculate the mean of the data elements in each class as the new cluster center;

[0017] Determine the cluster center and repeat the cluster center update step until the cluster center no longer changes or reaches the specified number of iterations to complete the determination of the cluster center.

[0018] Furthermore, the data element vector distance or cluster center similarity is determined by vector cosine similarity.

[0019] Furthermore, the calculation formula for hotspot heat is as follows:

[0020] hot_degree=weight src ×weight focus ×(1-decline),

[0021] Among them, weight s Indicates the weight of the information source, weight focusIt indicates netizens’ attention to the information source, and decline indicates the heat decay factor as the heat decays over time.

[0022] Furthermore, the calculation formula of the information source weight is:

[0023]

[0024] Among them: c represents the damping factor, which is used to represent the probability of randomly selecting other information sources, weight i represents the weight of the user who published data element i, c(i) represents the number of outbound links; N represents the number of documents, B(s) represents the source set, and f(i,s) represents the number of interactions;

[0025] The calculation formula for netizens' attention to information sources is:

[0026]

[0027] Among them: α represents the click constant, β represents the reply constant, T represents the time range, click d Indicates the number of clicks on data source d, reply d Indicates the number of reply data elements d, crawl_time d Indicates the crawl time, pub_time d represents the release time, θ represents the effective duration,

[0028] The calculation formula of the heat decay factor is:

[0029]

[0030] Where: k represents the initial heat number.

[0031] The advantages and positive effects of the present invention are:

[0032] Data elements include text features and non-text features. Text features include keywords, post topics, user names, and mention behavior identifiers. Non-text features include URLs, images, and videos. By identifying the language contained in the data element and calling the corresponding translation model to identify the data element, the accuracy of data source element filtering is improved.

[0033] By setting the CDFS model, the LDIG model of infinite N-Gram and the LDPN model of tower N-Gram to identify the data meta-language, the translation model can be automatically called so that the network data generated by multiple data sources can share a hot spot rule, thereby improving the compatibility of the method.

[0034] Hotspot rules are set by using a large language model, and data elements are identified through keyword extraction, natural language processing and semantic analysis to screen out data elements related to the hotspot rules. The feature words of the data elements are obtained in the form of word segmentation and data element vectors are generated. The data element vectors are clustered to determine the hotspots. The accuracy of hotspot discovery is improved by integrating multi-angle data in the data elements into the data element vectors.

[0035] The heat ranking of different hot spots is determined by the calculation formula for hot spot heat, so as to facilitate timely intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0037] Figure 1 It is an overall flow chart of a method for obtaining hot spots of network text data of the present invention;

[0038] Figure 2 It is a flow chart of determining cluster centers in a method for acquiring hot spots of network text data of the present invention;

[0039] Figure 3 It is an overall flow chart of a hotspot thermal analysis method of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0042] The present invention provides a method for obtaining hot spots of network data streams, such as Figure 1 As shown, it includes filtering network data according to hot spot rules and extracting several data elements related to the hot spot rules.

[0043] Filtering includes time filtering: setting the window and sliding step, and collecting the network data generated in the window after each sliding step. Including setting variables: W represents the size of the window (window length, unit: time), S represents the sliding step of the window (sliding interval, unit: time), t represents the current time, and Ti represents the timestamp of the arrival of the data element.

[0044] When collecting network data, the data stream arrives in time order, and the window slides once every S time. The window range = [tW, t], and the data set D contained in the window is: D = {x|Ti∈[tW, t]}, where x represents the data element.

[0045] When the window is updated, the new window range = [t-W+S, t+S], and the data set contained in the new window is Df: Df = {x|x∈D and F(x) = True}, where: F(x) represents the filtering condition, and only the data elements that satisfy the condition F(x) = True can be retained, and D represents the data elements generated within the window range = [t-W+S, t+S].

[0046] The filtering conditions include: a large number of data elements are continuously generated at different times, data elements with new timestamps are inserted into the cache, and data elements with timestamps not within the window range are removed. An embodiment of the present application is: the time window corresponding to the duration does not exceed the sliding step corresponding to the duration.

[0047] Data elements include text features and non-text features. Text features include keywords, hashtags (words and phrases starting with a pound sign (#) in the post subject), user names, and mentioning behavior identifiers. Non-text features include URLs (Uniform Resource Locator, data element address), images, and videos.

[0048] The acquisition of data elements includes: using the CDFS model, the LDIG model of infinite N-Gram, and the LDPN model of tower N-Gram to jointly determine the language of text features, and filtering out unrecognizable languages, calling the corresponding translation model according to the language, and converting text data into unified text. Non-text data calls the web page recognition model, image recognition model, and video recognition model according to the type, and converts non-text features into text data. This realizes the fusion of different types of data from different data sources.

[0049] Filtering includes data filtering: hot spot rules are set through large language models, hot spot rules include keywords and key semantics, keyword extraction, natural language processing and semantic analysis identify data elements in text format, and retain several data elements related to hot spot rules. The network data collected in the window is filtered to generate a data set, which includes several data elements.

[0050] The data element is segmented to obtain feature words, and the data element vector is generated based on the feature words.

[0051] The data element in text format is segmented to remove invalid words and invalid semantics in the data element to determine the feature words of the data element. The vector space model (VSM) regards the data element as a vector, and each dimension of the vector is a feature word, and its value is the weight of the feature word. All feature words together form a multidimensional space, and each document is a vector or point in the space.

[0052] Based on the VSM model, all data elements are converted into vectors. An embodiment of the present application is: the number of data elements d i Convert to vector where w 1,j is the weight of the feature word W1. The weight of the feature word is generally measured using the tf-idf (term frequency-inverse document frequency) model, that is:

[0053]

[0054] Among them: tf t,d Represents the feature word W t The frequency of occurrence in the data set, Indicates the inverse document frequency of the feature word.

[0055] Based on the clustering strategy, several cluster centers in the data element vector are determined, and similar cluster centers are merged to determine the class center, and the class center is used to represent the hot spot.

[0056] A hot topic is composed of several posts. Each post has a large number of data elements (a large number of user comments) under it. The text features of the data elements are similar and the URLs are the same. A clustering strategy is used to cluster all the data elements under the same post to form several clusters.

[0057] like Figure 2 As shown, the clustering strategy includes setting cluster centers, randomly selecting K points from the data set as initial cluster centers, and K represents a numerical value, which is a positive integer. An embodiment of the present application is that the value of K is higher than the number of posts contained in the data set, but lower than the number of data elements in the data set.

[0058] Update the cluster center, calculate the distances from the remaining data elements to all cluster centers, add the data element to the class to which the closest cluster center belongs, and calculate the mean of the data elements in each class as the new cluster center. One embodiment of the present application is: use vector cosine similarity to represent the distance between the data element and the cluster center.

[0059] Determine the cluster center and repeat the cluster center update step until the cluster center no longer changes or reaches the specified number of iterations to complete the determination of the cluster center.

[0060] When the cluster center represents a hot spot, cosine similarity is used to represent the distance between cluster centers. The cluster center is determined by calculating the distance between all cluster centers, and the cluster center represents the hot spot.

[0061] When hot spots are represented by mixed keywords, all similar cluster centers are regarded as a text, and several keywords with high frequency of occurrence in the text are extracted, and the keywords are used to represent the hot spots.

[0062] When monitoring data flow network data, set several hot spot rules, filter the network streaming data according to the hot spot rules, and use the hot spot trigger method to replace the hot spot rules required to filter the network data to adapt to the simultaneous monitoring of multiple hot spots.

[0063] Hotspot rules are composed of keywords and key semantics. Through language recognition data elements, the translation model can be automatically called so that network data generated by multiple data sources can share one hotspot rule. This can not only adapt to different text data, but also improve the processing efficiency of the system, thereby being able to track several hotspots according to the hotspot rules specified by the user.

[0064] A hotspot thermal analysis method, such as Figure 3 As shown, including the calculation formula of hotspot heat: hot_degree = weight src ×weight focus ×(1-decline), where weight s Indicates the weight of the information source, weight focus It indicates netizens’ attention to the information source, and decline indicates the heat decay factor as the heat decays over time.

[0065] In actual use, a hotspot acquisition method based on a network data stream is used to determine several hotspots and several data elements related to the hotspots. The information source weight, information source attention and heat attenuation factor of the hotspot are calculated based on the data elements. The heat of different hotspots is determined based on a calculation formula based on the hotspot heat, and the heat is arranged in order.

[0066] The information source weight represents the importance of the source of the data element. Based on the principle of PageRank, the weight calculation formula is as follows:

[0067]

[0068] Among them: c represents the damping factor, which is used to represent the probability of randomly selecting other information sources, weight i represents the weight of the user who published the data element i, c(i) represents the number of outbound links; N represents the number of documents published by the user, B(s) represents the source set, and f(i,s) represents the number of interactions between the user and other users.

[0069] Netizens' attention mainly represents the degree of attention paid to the hot topic. It is mainly calculated by the number of clicks and replies of all data elements corresponding to the hot topic. However, since these two values ​​change in real time, we will use statistical methods to calculate the average value (mathematical expectation) of the number of clicks and replies of each document in unit time. The formula is as follows:

[0070]

[0071] Among them: α represents the click constant, β represents the reply constant, T represents the time range (the length of the window), click d Indicates the number of clicks on data source d, reply d Indicates the number of reply data elements d, crawl_time d Indicates the crawling time of the data element, pub_time d represents the release time of the data element, and θ represents the effective release duration of the data element in the window.

[0072] Because hot spots have a strong timeliness, a hot spot attenuation factor is introduced. Generally, newly generated hot spots are more valuable to users. Therefore, Newton's cooling law is used to express the attenuation of hot spot heat. The formula is as follows:

[0073]

[0074] Where: k represents the initial heat value, T in this formula represents the heat constant, and the time range represents the heat constant based on Newton's law of cooling.

[0075] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of this patent.

Claims

1. A method for obtaining hot spots of network text data, characterized in that: It includes filtering network data according to hot spot rules and extracting several data elements related to the hot spot rules; Data element word segmentation is performed to obtain feature words, and data element vectors are generated based on the feature words; Based on the clustering strategy, several cluster centers in the data element vector are determined, and similar cluster centers are merged to determine the class center, and the class center is used to represent the hot spot.

2. The method for obtaining hot spots of network text data according to claim 1, characterized in that: The data elements include text features and non-text features. The text features include keywords, post topics, user names, and mention behavior identifiers. The non-text features include URLs, images, and videos.

3. The method for obtaining hot spots of network text data according to claim 1, characterized in that: The filtering of network data includes time filtering: setting a window and a sliding step in time units, extracting data elements that fall within the window period according to the generation time sequence of the data elements, and removing data elements that do not fall within the window period.

4. The method for obtaining hot spots of network text data according to claim 1, characterized in that: The filtering of network data includes data filtering: calling a translation model to identify data elements according to the language of the data element text features, calling a corresponding recognition model according to the type of non-text features of the data element, converting the non-text features into text data, identifying data elements through keyword extraction, natural language processing and semantic analysis, matching recognition results with hot spot rules, and extracting data elements that meet the hot spot rules.

5. The method for obtaining hot spots of network text data according to claim 4, characterized in that: The data filtering step includes language identification and filtering steps: based on the CDFS model, the LDIG model of infinite N-Gram and the LDPN model of tower N-Gram, the language of the text features is determined, and non-target language information in the data element is deleted.

6. The method for obtaining hot spots of network text data according to claim 1, characterized in that: The feature word weights are calculated through the tf-idf model, and the data element vector is generated based on the VSM model and the feature word weights.

7. The method for obtaining hot spots of network text data according to claim 1, characterized in that: The clustering strategy includes setting cluster centers, randomly selecting K points from the data set as initial cluster centers, where K represents a numerical value and is a positive integer; Update the cluster center, calculate the distances from the remaining data elements to all cluster centers, add the data element to the class to which the closest cluster center belongs, and calculate the mean of the data elements in each class as the new cluster center; Determine the cluster center and repeat the cluster center update step until the cluster center no longer changes or reaches the specified number of iterations to complete the determination of the cluster center.

8. The method for obtaining hot spots of network text data according to claim 1, characterized in that: The data element vector distance or cluster center similarity is determined by vector cosine similarity.

9. A hotspot thermal analysis method, based on a hotspot acquisition method for network text data according to any one of claims 1 to 8, characterized in that: Including the calculation formula of hotspot heat: hot_degree=weight src ×weight focus ×(1-decline), Among them, weight s Indicates the weight of the information source, weight focus It indicates netizens’ attention to the information source, and decline indicates the heat decay factor as the heat decays over time.

10. A hot spot thermal analysis method according to claim 1, characterized in that: The calculation formula of the information source weight is: Among them: c represents the damping factor, which is used to represent the probability of randomly selecting other information sources, weight i represents the weight of the user who published data element i, c(i) represents the number of outbound links; N represents the number of documents, B(s) represents the source set, and f(i,s) represents the number of interactions; The calculation formula for netizens' attention to information sources is: Among them: α represents the click constant, β represents the reply constant, T represents the time range, click d Indicates the number of clicks on data source d, reply d Indicates the number of reply data elements d, crawl_time d Indicates the crawl time, pub_time d represents the release time, θ represents the effective duration, The calculation formula of the heat decay factor is: Where: k represents the initial heat number.

Images