A method for extracting era features of news based on keyword mining

Abstract

The present invention is a method for extracting the characteristics of the times based on keyword mining news. The present invention extracts keywords from a single news article and performs a segmentation operation; weights the segmented keywords through the TF-IDF algorithm and the clustering coefficient; performs a normalization operation based on the TF-IDF weight and the clustering coefficient to obtain the weights of all characteristic words; based on a single news article, a set of characteristic words with a weight greater than 0.0001 is screened to obtain a set of news keywords; preliminary screening and keyword filtering are performed on news keywords, and the importance of keywords is calculated, and characteristic keywords of each period are determined; the heat value of a single news article is calculated, the keyword association is analyzed, and the degree of union of keywords is determined.

Description

Technical Field

[0001] The invention relates to the technical field of news text data set feature extraction, which is a method for extracting era features of news based on keyword mining. Background Art

[0002] In the context of big data, data journalism has emerged. Portals such as NetEase, Sina, Sohu, and Tencent have launched data journalism sections. However, so far, most data journalism is more like digital news, presenting simple static charts to the public, and only providing summarized data information. The information hidden behind the news data is not deeply mined, and the news value and readability are not high enough. Therefore, in-depth data journalism is relatively rare. At the same time, the data in data journalism lacks authority and reliability. This is because some journalists lack an understanding of authoritative data and do not strictly check the data source, resulting in inaccurate or imprecise data analysis or value judgments.

[0003] This technology was developed based on the fact that the news industry is moving towards digitalization and the premise of having a large amount of real and reliable news data. At present, most of the analysis of data news is still at the stage of extracting keywords and doing statistical analysis on keywords. This technology realizes more in-depth data mining based on keywords and can obtain the characteristics of the times contained in news data. Summary of the invention

[0004] The present invention provides a method for extracting era features of news based on keyword mining, and the present invention provides the following technical solutions:

[0005] A method for extracting era features of news based on keyword mining includes the following steps:

[0006] Step 1: Extract keywords from a single news article and perform word segmentation;

[0007] Step 2: Use the TF-IDF algorithm and clustering coefficient to weight the segmented keywords;

[0008] Step 3: Based on the TF-IDF weight and clustering coefficient, normalize the data to obtain the weights of all feature words.

[0009] Step 4: Based on a single news article, filter the feature word set with all values ​​greater than 0.0001 to obtain the news keyword set;

[0010] Step 5: Preliminary screening and keyword filtering of news keywords, calculate the importance of keywords, and determine the characteristic keywords for each period;

[0011] Step 6: Calculate the popularity value of a single news article, analyze the keyword association, and determine the degree of keyword association.

[0012] Preferably, the step 1 specifically includes: for each single news article, using the jieba package as a word segmentation tool, collecting news proper nouns and manual data review, and supplementing the word segmentation maintenance dictionary.

[0013] Preferably, the step 2 is specifically:

[0014] When a word or phrase appears frequently in an article and has a low IDF, the word or phrase has the ability to distinguish categories of the article. The TF-IDF weight calculation method is as follows:

[0015]

[0016]

[0017] TFIDF ij =TF ij *IDF i

[0018] Among them, TF ij represents the frequency of occurrence of feature word i in article j, nij represents the number of occurrences of feature word i in article j, ∑ k n kj Indicates the total number of words in article j; IDF i represents the inverse article frequency of feature word i, |D| represents the total number of articles, and |D i | indicates the number of articles containing feature word i; TFIDF ij Indicates the importance of feature word i in article j.

[0019] Preferably, the step 3 is specifically:

[0020] The clustering coefficient is calculated based on the spatial distribution of the feature words in the article. The article d is segmented. When the position of the feature word in the article d is {p 1 , ..., p n , p end}, n represents the number of times the feature word appears, p k , k≠end indicates the position of the kth occurrence of the feature word, the i-th word of the article d after word segmentation, p end Indicates the end position of the article. The position of the last word in article d. The clustering coefficient calculation method is shown in the following formula:

[0021]

[0022]

[0023]

[0024] Among them, L ij represents the average distance of feature word i in article j; S ij represents the standard deviation of the distance between feature word i in article j; C ij It represents the clustering coefficient of feature word i in article j. The larger the clustering coefficient, the greater the importance of the feature word relative to the article.

[0025] Based on the calculated TF-IDF weight and clustering coefficient, when article j has m feature words, the two indicators are normalized respectively:

[0026]

[0027]

[0028] Assign weights of 0.9 and 0.1 to the two normalized values ​​respectively, and then sum them up. The resulting number is the weight w of feature word i relative to article j. ij :

[0029]

[0030] Preferably, the step 5 is specifically:

[0031] Step 5.1: Based on the extraction operation, get the keywords of each article and the weight w of keyword i relative to article j ij , and keyword i is a keyword in multiple articles and has corresponding weights. The average weight of keyword i in the tth year is defined as the average of all weights of keyword i in the tth year; calculated by the following formula:

[0032]

[0033] Among them, w ij represents the weight of keyword i relative to article j; D ti represents the set of all news articles with keyword i as the keyword in the tth year, |D ti | indicates the total number of news in the collection; represents the average weight of keyword i in the tth year;

[0034] Based on the calculated average keyword weight, sort in descending order and select the top 100 keywords of each year, a total of 4,000 keywords, and use the above keywords as the initial keywords;

[0035] Step 5.2: According to the calculation method of confidence interval in statistics, adjust the parameters and calculate the confidence lower limit of the keyword frequency in year t and the confidence upper limit of the average weight:

[0036]

[0037]

[0038] Among them, MF t , SF t represents the mean and standard deviation of the frequency of 100 keywords in year t; MW t , SW t represents the mean and standard deviation of the average weights of the 100 keywords in year t; F t , W t They represent the lower confidence limit of the keyword frequency in year t and the upper confidence limit of the average weight respectively;

[0039] Combined with multiple experiments, determine the keyword screening threshold:

[0040]

[0041] Step 5.3: The importance is mainly composed of three indicators of keywords: average weight index, frequency share index based on the same year, and frequency share index based on the same keyword;

[0042] Average weight index, which is the result calculated in the preliminary screening

[0043] Based on the frequency share of the same year, it refers to the frequency f of a specific keyword i in the tth year. ti The total frequency of the year ∑ i f ti The ratio of the sum of the frequencies of all keywords in that year F ti :

[0044]

[0045] The frequency ratio index based on the same keyword refers to the frequency ratio of the same specific keyword i in different years t. The formula is as follows:

[0046]

[0047] Among them, f ti represents the frequency of keyword i in year t; represents the highest frequency of keywords in year t;

[0048] Considering that the average weight of low-frequency keywords is often greater than that of high-frequency keywords, the lower confidence limit of keyword frequency and the upper confidence limit of average weight are introduced here to classify and evaluate the importance of keyword i in year t, impor(i, t):

[0049] When fti ≥ F t ,but:

[0050]

[0051] When f ti <F t and but:

[0052]

[0053] When f ti <F t and but:

[0054]

[0055] Based on the calculated importance of the keywords, they are sorted in descending order, and according to actual needs, the top-ranked keywords are selected as the characteristic keywords for each period.

[0056] Preferably, the step 6 is specifically:

[0057] The calculation formula of news heat value is as follows:

[0058]

[0059]

[0060] Among them, new(j, t) represents the popularity value of news article j in year t; KW(j) represents the keyword set of news article j; KW(j, t) represents the subset of the characteristic keyword set in year t, and the characteristic keywords in this set are also the keywords of news article j; impor(i, t) represents the importance of characteristic keyword i in year t; num(j) represents the length of news article j, that is, the total number of words; num(i, j) represents the frequency of occurrence of characteristic keyword i in news article j; σ(i, j) is the coefficient of characteristic keyword i relative to news article j;

[0061] The joint degree of the same keyword pair in the same year is summed up to obtain the joint degree of the keyword pair in that year. The calculation formula is as follows:

[0062]

[0063] cor(i 1 ,i 2 , t) represents keyword i 1 with i 2 The joint degree in year t; J(i 1 ,i 2, t) means in year t, with feature word i 1 、Feature word i 2 A collection of news articles as keywords; new(j) represents the popularity value of news article j; Represents keyword i 1 with i 2 The weight relative to news article j.

[0064] Preferably, when the characteristic keyword i is also the keyword of the article j, the coefficient σ(i, j) is 1, otherwise it is 0.5.

[0065] The present invention has the following beneficial effects:

[0066] In terms of measuring the importance of keywords, the present invention not only considers the average weight of keywords, but also constructs new indicators based on the frequency of keywords: the frequency share index based on the same year and the frequency share index based on the same keyword. The former measures the popularity of keywords in the same time unit, and the latter measures the popularity of keywords in different time units. At the same time, the average weight index also measures the contribution of keywords to news. The importance calculated by combining these three indicators enhances the rationality and reliability of the analysis results.

[0067] In terms of mining the characteristics of the times contained in the news, keywords are important analysis units in this technology. The analysis process is not only limited to the comparison of the importance of a single keyword, but also achieves in-depth mining by calculating the degree of association between keywords. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Flowchart for keyword extraction;

[0069] Figure 2 The calculation process of the importance of keywords. DETAILED DESCRIPTION

[0070] The present invention is described in detail below in conjunction with specific embodiments. Specific embodiment one:

[0072] according to Figure 1 to Figure 2 As shown, the present invention provides a method for extracting era features of news based on keyword mining, and the method for extracting era features of news based on keyword mining includes the following steps:

[0073] Step 1: Extract keywords from a single news article and perform word segmentation;

[0074] The step 1 is specifically as follows: for each single news article, the jieba package is used as a word segmentation tool, and the word segmentation maintenance dictionary is supplemented by collecting news proper nouns and manual data review.

[0075] Step 2: Use the TF-IDF algorithm and clustering coefficient to weight the segmented keywords;

[0076] Step 3: Based on the TF-IDF weight and clustering coefficient, normalize the data to obtain the weights of all feature words.

[0077] Step 4: Based on a single news article, filter the feature word set with all values ​​greater than 0.0001 to obtain the news keyword set;

[0078] Step 5: Preliminary screening and keyword filtering of news keywords, calculate the importance of keywords, and determine the characteristic keywords for each period;

[0079] Step 6: Calculate the popularity value of a single news article, analyze the keyword association, and determine the degree of keyword association.

[0080] The step 2 is specifically as follows:

[0081] When a word or phrase appears frequently in an article and has a low IDF, the word or phrase has the ability to distinguish categories of the article. The TF-IDF weight calculation method is as follows:

[0082]

[0083]

[0084] TFIDF ij =TF ij *IDF i

[0085] Among them, TF ij represents the frequency of occurrence of feature word i in article j, nij represents the number of occurrences of feature word i in article j, ∑ k n kj Indicates the total number of words in article j; IDF i represents the inverse article frequency of feature word i, |D| represents the total number of articles, and |D i | indicates the number of articles containing feature word i; TFIDF ij Indicates the importance of feature word i in article j.

[0086] Preferably, the step 3 is specifically:

[0087] The clustering coefficient is calculated based on the spatial distribution of the feature words in the article. The article d is segmented. When the position of the feature word in the article d is {p 1 , ..., p n , p end}, n represents the number of times the feature word appears, pk, k≠end represents the position of the kth occurrence of the feature word, the i-th word in the article d after word segmentation, p end Indicates the end position of the article. The position of the last word in article d. The clustering coefficient calculation method is shown in the following formula:

[0088]

[0089]

[0090]

[0091] Among them, L ij represents the average distance of feature word i in article j; S ij represents the standard deviation of the distance between feature word i in article j; C ij It represents the clustering coefficient of feature word i in article j. The larger the clustering coefficient, the greater the importance of the feature word relative to the article.

[0092] Based on the calculated TF-IDF weight and clustering coefficient, when article j has m feature words, the two indicators are normalized respectively:

[0093]

[0094]

[0095] Assign weights of 0.9 and 0.1 to the two normalized values ​​respectively, and then sum them up. The resulting number is the weight w of feature word i relative to article j. ij :

[0096]

[0097] Preferably, the step 5 is specifically:

[0098] Step 5.1: Based on the extraction operation, get the keywords of each article and the weight w of keyword i relative to article j ij , and keyword i is a keyword in multiple articles and has corresponding weights. The average weight of keyword i in the tth year is defined as the average of all weights of keyword i in the tth year; calculated by the following formula:

[0099]

[0100] Among them, wij represents the weight of keyword i relative to article j; D ti represents the set of all news articles with keyword i as the keyword in the tth year, |D ti | indicates the total number of news in the collection; represents the average weight of keyword i in the tth year;

[0101] Based on the calculated average keyword weight, sort in descending order and select the top 100 keywords of each year, a total of 4,000 keywords, and use the above keywords as the initial keywords;

[0102] Step 5.2: According to the calculation method of confidence interval in statistics, adjust the parameters and calculate the confidence lower limit of the keyword frequency in year t and the confidence upper limit of the average weight:

[0103]

[0104]

[0105] Among them, MF t , SF t represents the mean and standard deviation of the frequency of 100 keywords in year t; MW t , SW t represents the mean and standard deviation of the average weights of the 100 keywords in year t; F t , W t They represent the lower confidence limit of the keyword frequency in year t and the upper confidence limit of the average weight respectively;

[0106] Combined with multiple experiments, determine the keyword screening threshold:

[0107]

[0108] Step 5.3: The importance is mainly composed of three indicators of keywords: average weight index, frequency share index based on the same year, and frequency share index based on the same keyword;

[0109] Average weight index, which is the result calculated in the preliminary screening

[0110] Based on the frequency share of the same year, it refers to the frequency f of a specific keyword i in the tth year. ti The total frequency of the year ∑ i f ti The ratio of the sum of the frequencies of all keywords in that year F ti :

[0111]

[0112] The frequency ratio index based on the same keyword refers to the frequency ratio of the same specific keyword i in different years t. The formula is as follows:

[0113]

[0114] Among them, fti represents the frequency of keyword i in year t; represents the highest frequency of keywords in year t;

[0115] Considering that the average weight of low-frequency keywords is often greater than that of high-frequency keywords, the lower confidence limit of keyword frequency and the upper confidence limit of average weight are introduced here to classify and evaluate the importance of keyword i in year t, impor(i, t):

[0116] When f ti ≥F t ,but:

[0117]

[0118] When f ti <F t and but:

[0119]

[0120] When f ti <F t and but:

[0121]

[0122] Based on the calculated importance of the keywords, they are sorted in descending order, and according to actual needs, the top-ranked keywords are selected as the characteristic keywords for each period.

[0123] Preferably, the step 6 is specifically:

[0124] The calculation formula of news heat value is as follows:

[0125]

[0126]

[0127] Among them, new(j, t) represents the popularity value of news article j in year t; KW(j) represents the keyword set of news article j; KW(j, t) represents the subset of the characteristic keyword set in year t, and the characteristic keywords in this set are also the keywords of news article j; impor(i, t) represents the importance of characteristic keyword i in year t; num(j) represents the length of news article j, that is, the total number of words; num(i, j) represents the frequency of occurrence of characteristic keyword i in news article j; σ(i, j) is the coefficient of characteristic keyword i relative to news article j;

[0128] The joint degree of the same keyword pair in the same year is summed up to obtain the joint degree of the keyword pair in that year. The calculation formula is as follows:

[0129]

[0130] cor(i 1 ,i 2 , t) represents keyword i 1 with i 2 The joint degree in year t; J(i 1 ,i 2 , t) means in year t, with feature word i 1 、Feature word i 2 A collection of news articles as keywords; new(j) represents the popularity value of news article j; Represents keyword i 1 with i 2 The weight relative to news article j.

[0131] Preferably, when the characteristic keyword i is also the keyword of the article j, the coefficient σ(i, j) is 1, otherwise it is 0.5. Specific embodiment 2:

[0133] The present invention is mainly applied to news text datasets with a long time span and large data volume. The existing combination of news and big data technology is mainly applied in fake news detection, common sense reasoning, etc., and rarely mines potential features through news data. Therefore, we use natural language processing technology to extract keywords from news articles, and then deeply mine the era characteristics contained in the news based on keywords to give full play to the potential value of news data.

[0134] Keyword extraction: The TF-IDF algorithm is one of the most commonly used and effective keyword extraction algorithms. The clustering coefficient measures the degree of clustering of feature words in an article. Based on the news data after word segmentation, this technology takes into account the clustering coefficient on the basis of the TF-IDF weight, and combines the two in a weighted ratio of 9:1 to obtain the final weight of the feature word relative to a specific news article. Finally, the feature word with a final weight greater than 0.0001 is set as the keyword of the news.

[0135] Preliminary screening and filtering of keywords: The purpose of this technology in news applications is to find keywords with characteristics of the times. Considering that different news articles in the same period have different keywords, and different keywords in the same article have different weights, the keyword set of the same period often contains a large number of keywords, and the same keyword usually corresponds to multiple different weights. In order to more specifically characterize the characteristics of the period, it is necessary to select keywords for each period. This technology realizes the automatic selection of keywords through two operations:

[0136] Preliminary screening: In the same period, the same keyword can be used as a keyword in multiple news articles, so it has multiple different weights. We take the average of all weights of the same keyword, which is called the average weight. Then, based on the average weight, we screen out the top 100 keywords in each period.

[0137] Keyword filtering: There is a problem among the keywords obtained through preliminary screening: some keywords have extremely low frequency of occurrence, but their weight in news articles is large, so their average weight is high; while some keywords with high frequency of occurrence have low average weights due to their low weights. Therefore, it is very likely that the average weight of extremely low-frequency keywords will be higher than the average weight of high-frequency keywords in each period. To address this drawback, this technology uses statistical knowledge to calculate the lower confidence limit based on the frequency of keywords and the upper confidence limit based on the average weight, and then designs rules based on these two new indicators to perform secondary screening of keywords.

[0138] Calculation of the importance of keywords: Different periods have different characteristics, which can be reflected by keywords. That is, the more the characteristics of the period are reflected, the higher the importance of the keywords. In view of this feature, we designed an importance calculation formula to measure the importance of the keywords obtained in step 2. In this formula, in addition to the average weight of the keywords, two new indicators are constructed based on frequency: the frequency share index based on the same period and the frequency share index based on the same keyword. The former measures the distribution of keywords in the same period, and the latter measures the distribution of the keyword period and other different periods. Finally, we use the frequency confidence lower limit and average weight confidence upper limit obtained in step 2 to design classification rules, divide the keywords into 3 categories, and set different weights for the three indicators in different categories. The weighted sum of the keyword indicator value and the weight is obtained to obtain the importance value of the keyword in a specific period. Finally, according to the period, the keywords are sorted in descending order according to the importance value. According to specific needs, the top-ranked keywords are used as the characteristic keywords of the corresponding period.

[0139] News popularity: In the same period, different news have different popularity. News with the characteristics of the times is higher than other news, and characteristic keywords usually have a high frequency of occurrence in this part of news. Therefore, in order to quantify the popularity of news, this technology comprehensively considers the frequency of occurrence and importance of characteristic keywords in the news period, the keywords of the news, and the length of the news text, and constructs a formula to calculate the news popularity.

[0140] Correlation between keywords: The change in the correlation between different keywords is the key point to dig deeper into the news. This technology pairs the keywords belonging to the same news with each other, and then calculates the correlation between a group of keyword pairs and the news they belong to by combining the popularity value of the news and the weight of the keyword relative to the news. Two different keywords can appear in different news, so the same group of keyword pairs may have multiple different correlations in the same period. Taking the average of them will give the final correlation between the two keywords in that period.

[0141] Keywords are the main objects of text analysis and mining. In this technology, based on the word segmentation results of news texts, the keywords of news articles are extracted by combining the TF-IDF algorithm and the clustering coefficient indicator.

[0142] For each news text, this technology uses the jieba package as a word segmentation tool. At the same time, in order to improve the accuracy of word segmentation, the word segmentation maintenance dictionary is supplemented by collecting news proper nouns and manual data review, such as maintaining words such as "Western Corridor" and "Pilot Demonstration Zone". The news text after word segmentation is the data basis for subsequent analysis.

[0143] TF-IDF (Term Frequency-Invers Document Frequency) is a weighted technology commonly used in information processing and data mining, and is a statistical method. TF is called term frequency, which indicates the frequency of a keyword in the entire article; IDF is called inverse document frequency, which indicates the inverse of the frequency of a keyword in all articles. It is mainly used to reduce the effect of some common words in all articles that have little impact on the article. The main idea of ​​this technology is: if a word or phrase appears frequently in an article (TF is high) and rarely appears in other articles (IDF is low), it is considered that this word or phrase has a good ability to distinguish categories for the article and is suitable for classification. Its advantage is that it can filter out some common but insignificant words while retaining important words that affect the entire text.

[0144] The TF-IDF weight calculation method is as follows:

[0145]

[0146]

[0147] TFIDF ij =TF ij *IDF i

[0148] Among them, TF ij represents the frequency of occurrence of feature word i in article j, nij represents the number of occurrences of feature word i in article j, ∑ k n kj Indicates the total number of words in article j; IDF i represents the inverse article frequency of feature word i, |D| represents the total number of articles, and |D i | indicates the number of articles containing feature word i; TFIDF ij Indicates the importance of feature word i in article j.

[0149] There is a clustering phenomenon among words in an article, that is, the distance between words decreases exponentially, and the clustering characteristics of words with high relevance to the article are more significant. Similarly, words with different levels of importance should have different clustering characteristics, so indicators that reflect the clustering characteristics of words can be used to judge the importance of entities [1].

[0150] The clustering coefficient is calculated based on the spatial distribution of the feature words in the article. Specifically, the article d is segmented. When the position where the feature word appears in the article d is {p 1 , ..., p n , p end}, n represents the number of times the feature word appears, pk , k≠end indicates the position of the kth occurrence of the feature word, the i-th word of the article d after word segmentation, p end Indicates the end position of the article. The position of the last word in article d. The clustering coefficient is calculated as follows:

[0151]

[0152]

[0153]

[0154] Among them, L ij represents the average distance of feature word i in article j; S ij represents the standard deviation of the distance between feature word i in article j; C ij It represents the clustering coefficient of feature word i in article j. The larger the clustering coefficient, the greater the importance of the feature word relative to the article.

[0155] Based on the calculated TF-IDF weight and clustering coefficient, assuming there are m feature words, the two indicators are normalized respectively:

[0156]

[0157]

[0158] Assign weights of 0.9 and 0.1 to the two normalized values ​​respectively, and then sum them up. The resulting number is the weight w of feature word i relative to article j. ij :

[0159]

[0160] Identify news keywords

[0161] For a single news j, a set of feature words with all values ​​greater than 0.0001 is screened out, and this set is the keyword set of news j.

[0162]

[0163] News reports of the same period can reflect the hot topics or characteristics of the year. Different periods have different hot topics or characteristics. These differences can be reflected through the keywords of the report text. In other words, in this period, the importance of keywords related to hot topics is higher than that of other keywords. The following describes the specific steps for calculating the importance of keywords.

[0164] Step 1: Initial Screening

[0165] Based on the extraction operation in the first part, we can get the keywords of each article and the weight w of keyword i relative to article j. ij , and keyword i is a keyword in multiple articles and has corresponding weights. The average weight of keyword i in the tth year is defined as the average of all weights of keyword i in the tth year; calculated by the following formula:

[0166]

[0167] Among them, w ij represents the weight of keyword i relative to article j; D ti represents the set of all news articles with keyword i as the keyword in the tth year, |D ti | indicates the total number of news in the collection; represents the average weight of keyword i in the tth year;

[0168] Based on the calculated average keyword weight, sort in descending order and select the top 100 keywords for each year, a total of 4,000 keywords. These keywords are used as initial keywords.

[0169] Step 2: Keyword filtering

[0170] In the above steps, the average weight of keywords is calculated. This method has a drawback: keywords with extremely low frequency cannot represent the characteristics of the year, but their average weight may be too high. Because some keywords with low frequency have high weights in the corresponding articles, the calculated average weight is too high; while keywords with high frequency have low weights in some corresponding articles, which reduces the average weight; therefore, it is possible that the average weight of keywords with extremely low frequency is much higher than that of keywords with high frequency. For example, in 2016, "cold air" was only used as a keyword in 4 front-page reports, with an average weight of 0.593, ranking first; while "innovation" appeared as a keyword in 430 front-page reports, with an average weight of 0.054, ranking 81. The average weight of "cold air" is ten times that of "innovation", but "innovation" is much more representative of the characteristics of 2016 than "cold air". Therefore, the invalid keywords of "cold air" need to be filtered.

[0171] To solve this problem, this technology has designed a keyword filtering link. According to the calculation method of confidence interval in statistics, adjust the parameters, calculate the confidence lower limit of the keyword frequency in the tth year and the confidence upper limit of the average weight:

[0172]

[0173]

[0174] Among them, MF t, SF t represents the mean and standard deviation of the frequency of 100 keywords in year t; MW t、 SW t represents the mean and standard deviation of the average weights of the 100 keywords in year t; F t , W t They represent the lower confidence limit of the keyword frequency in year t and the upper confidence limit of the average weight respectively; combined with multiple experiments, the keyword screening threshold is determined:

[0175]

[0176] Step 3: Calculate the importance of your keywords

[0177] The importance is mainly composed of three indicators of keywords: average weight index, frequency share index based on the same year, and frequency share index based on the same keyword.

[0178] Average weight index, i.e. the result calculated in “(1) Preliminary screening”

[0179] Based on the frequency share of the same year, it refers to the frequency f of a specific keyword i in the tth year. ti The total frequency of the year ∑ i f ti The ratio F of the sum of the frequencies of all keywords in that year ti :

[0180]

[0181] The frequency ratio index based on the same keyword refers to the frequency ratio of the same specific keyword i in different years t. The formula is as follows. ti represents the frequency of keyword i in year t; Represents the highest frequency of keywords in year t.

[0182]

[0183] Considering that the average weight of low-frequency keywords is often greater than that of high-frequency keywords, the lower confidence limit of keyword frequency and the upper confidence limit of average weight are introduced here to classify and evaluate the importance of keyword i in year t, impor(i, t):

[0184] If ft i ≥F t ,but:

[0185]

[0186] If f ti <Ft and but:

[0187]

[0188] If f ti <F t and but:

[0189]

[0190] Step 4: Determine the characteristic keywords for each period

[0191] Based on the importance of the keywords calculated in step 3, they are sorted in descending order, and the top-ranked keywords are selected as the characteristic keywords for each period according to actual needs.

[0192] Part 3: Calculating the popularity of news articles

[0193] In the same period, the popularity of different news is different. The higher the popularity of a news event, the more it can reflect the hot spots in Shenzhen that year. This part constructs a news popularity index based on the keywords and related information obtained in the second part to compare the popularity of news released in the same year.

[0194] For year t, there are 60 corresponding feature keywords and their importance values. For news article j, there are corresponding article keywords. The era characteristics of year t are the hot words of year t. Therefore, we can multiply the frequency of occurrence of different characteristic keywords in news articles by the corresponding importance value, and then calculate

[0195] and , to measure the popularity of news in year t. Considering that the characteristic keywords that appear in the news may not be the corresponding article keywords, in this case, the contribution rate of these characteristic keywords to the popularity of the news they belong to is lower than that of other characteristic words. Based on this consideration, for the importance values ​​of characteristic keywords that exist in the news but are not news keywords, only half of them are taken into the calculation of news popularity value.

[0196] At the same time, the number of times a keyword appears in a text is related to the length of the news text to a certain extent: the longer the text, the higher the frequency of keyword appearance. That is, news articles with large differences in length cannot be directly compared. In order to eliminate the impact of text length on the news popularity value, we use the number of words contained in the text after word segmentation to represent the text length and use it as the denominator in the calculation result.

[0197] Based on the above analysis, the calculation formula for the news heat value designed in this section is as follows:

[0198]

[0199]

[0200] Among them, new(j, t) represents the popularity value of news article j in year t; KW(j) represents the keyword set of news article j; KW(j, t) represents the subset of the characteristic keyword set in year t, and the characteristic keywords in this set are also the keywords of news article j; impor(i, t) represents the importance of characteristic keyword i in year t; num(j) represents the length of news article j, that is, the total number of words; num(i, j) represents the frequency of occurrence of characteristic keyword i in news article j; σ(i, j) is the coefficient of characteristic keyword i relative to news article j;

[0201] Part 4: Keyword Correlation Analysis

[0202] There are different degrees of correlation between keywords, and the era information of news is mainly contained in this correlation. Therefore, mining the correlation of news keywords is conducive to exploring the characteristics of different periods.

[0203] According to the first and third parts, each piece of news has corresponding keywords and weights, as well as the corresponding news popularity value. Next, we will use this information to calculate the union between different keywords. Pair the keywords of the same piece of news in pairs to form several groups of keyword pairs; for each group of keyword pairs, sum the weights of the two, and then multiply by the popularity value of the news to get the union degree of this group of keyword pairs relative to the news. Finally, sum the union degrees of the same keyword pair belonging to the same year to get the union degree of the keyword pair in that year. The calculation formula is as follows.

[0204]

[0205] cor(i 1 ,i 2 , t) represents keyword i 1 with i 2 The joint degree in year t; J(i 1 ,i 2 , t) means in year t, with feature word i 1 、Feature word i 2 A collection of news articles as keywords; new(j) represents the popularity value of news article j; Represents keyword i 1 with i 2 The weight relative to news article j.

[0206] The above is only a preferred implementation of a method for extracting the characteristics of the times based on keyword mining news. The protection scope of a method for extracting the characteristics of the times based on keyword mining news is not limited to the above embodiments. All technical solutions under this idea belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, several improvements and changes without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for extracting era features of news based on keyword mining, characterized by: The following steps are involved: Step 1: Extract keywords from a single news article and perform word segmentation; Step 2: Use the TF-IDF algorithm and clustering coefficient to weight the segmented keywords; Step 3: Based on the TF-IDF weight and clustering coefficient, perform normalization to obtain the weights of all keywords; Step 4: According to a single news article, filter all keywords with a value greater than 0.0001 to obtain a news keyword set; Step 5: Preliminary screening and keyword filtering of the news keyword set, and calculation of the importance of keywords, and determination of the characteristic words for each period; The step 5 is specifically as follows: Step 5.1: Based on the extraction operation, get the keywords of each article and the weight w of keyword i relative to article j ij , and keyword i is a keyword in multiple articles and has corresponding weights. The average weight of keyword i in the tth year is defined as the average of all weights of keyword i in the tth year; calculated by the following formula: Among them, w ij represents the weight of keyword i relative to article j; D ti represents the set of all news articles with keyword i as the keyword in the tth year, |D ti | indicates the total number of news in the collection; represents the average weight of keyword i in the tth year; Based on the calculated average keyword weight, sort in descending order and select the top 100 keywords of each year, a total of 4,000 keywords, and use the above keywords as the initial keywords; Step 5.2: According to the calculation method of confidence interval in statistics, adjust the parameters and calculate the confidence lower limit of the keyword frequency in year t and the confidence upper limit of the average weight: Among them, MF t , SF t represents the mean and standard deviation of the frequency of 100 keywords in year t; MW i , SW t represents the mean and standard deviation of the average weights of the 100 keywords in year t; F t , W t They represent the lower confidence limit of the keyword frequency in year t and the upper confidence limit of the average weight respectively; Combined with multiple experiments, determine the keyword screening threshold: Step 5.3: The importance is mainly composed of three indicators of keywords: average weight index, frequency share index based on the same year, and frequency share index based on the same keyword; Average weight index, which is the result calculated in the preliminary screening Based on the frequency share of the same year, it refers to the frequency f of keyword i in the tth year. ti The total frequency of the year Σ i f ti The ratio of the sum of the frequencies of all keywords in that year F ti : The frequency ratio index based on the same keyword refers to the frequency ratio of the same keyword i in different years t. The formula is as follows: Among them, f ti represents the frequency of keyword i in year t; represents the highest frequency of keywords in year t; Considering that the average weight of low-frequency keywords is often greater than that of high-frequency keywords, the lower confidence limit of keyword frequency and the upper confidence limit of average weight are introduced here to classify and evaluate the importance of keyword i in year t, impor(i, t): When f ti ≥F t ,but: When f ti <F t and but: When f ti <F t and but: Based on the calculated importance of keywords, they are sorted in descending order, and the top-ranked keywords are selected as the feature words for each period according to actual needs; Step 6: Calculate the popularity value of a single news article, analyze the keyword association, and determine the degree of keyword association; The step 6 is specifically as follows: The calculation formula of news heat value is as follows: Among them, new(j, t) represents the popularity value of news article j in year t; KW(j) represents the keyword set of news article j; KW(j, t) represents the subset of the feature word set in year t, and the feature words in this set are also the keywords of news article j; impor(i, t) represents the importance of keyword i in year t; num(j) represents the length of news article j, that is, the total number of words; num(i, j) represents the frequency of occurrence of keyword i in news article j; σ(i, j) is the coefficient of keyword i relative to news article j; The joint degree of the same keyword pair in the same year is summed up to obtain the joint degree of the keyword pair in that year. The calculation formula is as follows: cor(i1, i2, t) represents the joint degree of keywords i1 and i2 in year t; J(i1, i2, t) represents the set of news articles with keyword i1 and feature word i2 as keywords in year t; new(j) represents the popularity value of news article j; They represent the weights of keyword i1 and i2 relative to news article j respectively.

2. The method for extracting era features of news based on keyword mining according to claim 1 is characterized by: The step 1 is specifically as follows: for each single news article, the jieba package is used as a word segmentation tool, and the word segmentation maintenance dictionary is supplemented by collecting news proper nouns and manual data review.

3. The method for extracting era features of news based on keyword mining according to claim 2 is characterized by: The step 2 is specifically as follows: When a word or phrase appears frequently in an article and has a low IDF, the word or phrase has the ability to distinguish categories of the article. The TF-IDF weight calculation method is as follows: TFIDF ij =TF ij *IDF i Among them, TF ij represents the frequency of occurrence of keyword i in article j, n ij represents the number of occurrences of keyword i in article j, ∑ k n kj Indicates the total number of words in article j; IDF i represents the inverse article frequency of keyword i, |D| represents the total number of articles, and |D i | indicates the number of articles containing keyword i; TFIDF ij Indicates the importance of keyword i in article j.

4. The method for extracting era features of news based on keyword mining according to claim 3 is characterized by: The step 3 is specifically as follows: The clustering coefficient is calculated based on the spatial distribution of keywords in the article. The article d is segmented. When the position of the keyword in the article d is {p1, ..., p n , p end }, n means the keyword appears n times, p k ,k≠end indicates the position of the kth occurrence of the keyword, p end Indicates the end position of the article. The position of the last word in article d. The clustering coefficient calculation method is shown in the following formula: Among them, L ij represents the average distance of keyword i in article j; S ij represents the standard deviation of the distance between keyword i in article j; C ij It represents the clustering coefficient of keyword i in article j. The larger the clustering coefficient, the greater the importance of the keyword relative to the article. Based on the calculated TF-IDF weight and clustering coefficient, when article j has m keywords, the two indicators are normalized respectively: After normalization, the two weights are assigned as 0.9 and 0.1 respectively, and then the sum is obtained. The number obtained is the weight W of keyword i relative to article j. ij :

Images