A news keyword extraction method and device based on a graph model

Abstract

The application provides an improved news keyword extraction method based on a graph model, obtains internet news text data and performs cleaning processing, obtains candidate words by performing sentence segmentation and word segmentation on the cleaned text data, performs vector representation on the cleaned text, obtains a news hot word library of a specific time period based on a time translation weighting algorithm, and obtains the distribution of text keywords by applying a graph model based on candidate word position information and hot word library information. The application introduces similar real-time news information in a specific time range as supplementary corpus, and uses news hot words for assistance, fully considers the data characteristics of news text, extracts news keywords by using a graph model, improves the accuracy and efficiency of news text keyword extraction, and reduces the dependence on labeled data.

Description

Technical Field

[0001] This invention belongs to the technical field of natural language processing, specifically relating to a method and apparatus for extracting news keywords based on a graph model. Background Technology

[0002] With the rapid development of internet technology, people are bombarded with a massive amount of news information every day. How to quickly extract knowledge from this vast amount of information has become an urgent need for users. Keywords, as a high-level summary of the news topic, can help people quickly understand the main content of a news text. Current keyword extraction methods are mainly divided into two categories: statistical and machine learning-based. Machine learning methods suffer from limitations such as dependence on the size and quality of the training corpus, difficulty in obtaining large-scale manually labeled datasets, and significant influence of the dataset's domain-specific characteristics on extraction results. Graph-based keyword extraction methods are currently widely used, but traditional general graph-based keyword extraction methods often only consider the word co-occurrence features of a single article for weighted calculations, thus limiting the input feature information and resulting in low accuracy. Summary of the Invention

[0003] To further improve the accuracy and efficiency of news text keyword extraction and reduce reliance on labeled data, this invention proposes an improved graph model-based news keyword extraction method. By introducing similar real-time news information within a specific time range as supplementary corpus and utilizing trending news terms, this method fully considers the data characteristics of news texts and uses a graph model to extract news keywords.

[0004] The solution adopted by this invention to solve its technical problem is an improved news keyword extraction method based on a graph model.

[0005] The method specifically includes the following steps:

[0006] Step 1: Obtain internet news text data and perform cleaning processing.

[0007] Step 2: Segment and word segment the text data processed in Step 1 to obtain candidate words.

[0008] Step 3: Represent the text processed in Step 1 as a vector.

[0009] Step 4: Obtain the news hot topic database and popularity for a specific time period based on the time-shift weighted algorithm.

[0010] Step 5: Based on candidate word location information, hot word lexicon information, and similar news texts, apply an improved graph model to calculate the distribution of text keywords.

[0011] Furthermore, in step 1, the news text is acquired from information on a specific news website. The text cleaning process refers to removing webpage tags, special symbols, images, and other information from the acquired raw online text information to obtain the text T to be further processed.

[0012] Furthermore, in step 2, candidate words are obtained by segmenting and dividing the text T into words and sentences through the following steps:

[0013] Step 2.1: Segment the processed text T according to sentence integrity to obtain a sentence set, i.e.: T = {S1, S2, S3, ..., S...} n}

[0014] Step 2.2: Combining the features of the news text, an improved word segmentation algorithm is used to segment and tag the sentence S, and stop words and words with specific parts of speech are removed according to given rules to obtain candidate words, i.e.: S={W1,W2,W3,....,W n}

[0015] Furthermore, the text vector representation model in step 3 is a document representation model obtained by training on news corpora.

[0016] Furthermore, in step 4, the construction of a news hot topic terminology database for a specific time period based on a time-shift weighted algorithm is achieved through the following steps:

[0017] Step 4.1 For the candidate words S after word segmentation in Step 2, calculate the candidate word popularity using the Bayesian mean algorithm. The specific calculation formula is as follows:

[0018]

[0019] Where S(W) i ) is the candidate word W i Popularity, R avg For candidate word W i Average score. F(W) i ,T j ) is the word W i In T j Word frequency over a time period. F avg S represents the average word frequency. avg The average score for all words.

[0020] Where R avg The calculation formula is:

[0021]

[0022] Where F t For the word frequency in the time period t, Ft-1 The word frequency within the previous time period t-1. θ This is the coefficient for adjusting the part of speech.

[0023] Step 4.2 Adjust the weights of candidate words based on part-of-speech, candidate word position features, and co-occurrence relationships in the news hot topic lexicon, and obtain the restart probability of candidate words.

[0024] 4.2.1 Obtain the position weight of candidate words, and the calculation formula is as follows:

[0025]

[0026] Where |S| is the candidate word list, L(W i ) Represents word W i In the candidate word list, combined with the keyword features of the news text, different weights are assigned to candidate words based on their position characteristics, since candidate words in the title and the first sentence of paragraphs are more likely to become keywords.

[0027] 4.2.2 Obtain the popularity weight of candidate words, and the calculation formula is as follows:

[0028]

[0029] Where n is the length of the candidate word list, s(w i H(w) represents the popularity value of candidate word i. i ) represents the popularity weight value of candidate word i.

[0030] 4.2.3 By fusing location information and popularity information, the weights of candidate words are set. These weights are then normalized to obtain the restart probability of the candidate words. The calculation formula is as follows:

[0031]

[0032] Where V(w) i ) is the candidate word W i The weights obtained through weighted summation, P(w) i ) represents the normalized candidate word w i The restart probability, where n is the length of the candidate word list.

[0033] Step 5, based on the location information of candidate words, the news hot word thesaurus information, and similar news texts, applies a graph model to calculate the distribution of text keywords. The specific steps are as follows:

[0034] Step 5.1 Construct a text word graph G = (V, E) based on the candidate word set, where V represents candidate words and E represents the relationship between candidate words.

[0035]

[0036] Among them, adj(w i ) indicates the relationship with candidate word w i The set of co-occurrence relation words, e(w i ,w j ) indicates candidate word w i With candidate word w j The co-occurrence frequency, O(w) j ) indicates that the candidate word is related to w j The sum of the co-occurrence counts of all other candidate words, R(w j ) indicates candidate word w j Rating, P(w) i ) represents the normalized candidate word w i The probability of restarting.

[0037] In particular, a set of documents similar to the target document is used to assist in constructing a word graph, further enhancing the information content of the graph. The calculation formula is as follows:

[0038]

[0039] Where sim(d,d) k ) for the target document d and document set d k Similarity, F(w i ,w j ) is a candidate word w i and candidate word w j The number of times they co-occur.

[0040] Step 5.2 Set the convergence value or number of iterations, calculate and obtain candidate keywords, sort the candidate keywords in reverse order according to the scores, and select the top-n as keywords.

[0041] The beneficial effects of this invention are:

[0042] This invention introduces similar real-time news information within a specific time range as supplementary corpus and uses news hot words as an aid. It fully considers the data characteristics of news texts and uses graph models to extract news keywords, thereby improving the accuracy and efficiency of news text keyword extraction and reducing dependence on labeled data. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] Example 1: This invention introduces a real-time news corpus and trending news terms within a specific time frame. Based on fully considering the data characteristics of news information, it proposes an improved graph model-based news keyword extraction method, which enhances the accuracy and efficiency of keyword extraction from news texts. The invention is further illustrated below with reference to the accompanying drawings and examples:

[0046] like Figure 1 As shown, an improved news keyword extraction method based on a graph model according to the present invention includes the following steps.

[0047] Step 1: Obtain internet news text data and perform cleaning processing.

[0048] Step 2: Segment and word segment the text data processed in Step 1 to obtain candidate words.

[0049] Step 3: Represent the text processed in Step 1 as a vector.

[0050] Step 4: Obtain a database of trending news terms for a specific time period based on a time-shift weighted algorithm.

[0051] Step 5: Based on the candidate word location information and hot word lexicon information, apply a graph model to calculate the distribution of text keywords.

[0052] Furthermore, in step 1, the news text is acquired from information on a specific news website. The text cleaning process refers to removing webpage tags, special symbols, images, and other information from the acquired raw online text information to obtain the text T to be further processed.

[0053] The acquisition of internet news text data employs a distributed architecture, utilizing Scrapy as the framework for the data collection program. The task acquisition module extracts tasks based on the initialized data source and task extraction rules, and writes the parsed tasks to a Kafka collection task. The acquisition module reads tasks from Kafka, performs data collection, preprocessing, and database storage. During implementation, the scheduler can dynamically start and pause some task acquisition or execution nodes based on the task volume in Kafka. News data sources include, but are not limited to, websites of traditional news media and internet news sites.

[0054] The text cleaning process can be based on regular expression string matching technology. By building a rule base and cleaning service, the original text data is processed sequentially through the cleaning service to complete the data cleaning work.

[0055] Furthermore, the text vector representation model in step 3 is a document representation model obtained by training on news corpora.

[0056] Doc2vec, also known as Paragraph Vector, was proposed by Tomas Mikolov based on the word2vec model. It doesn't require a fixed sentence length and can accept sentences of varying lengths as training samples. Doc2vec is an unsupervised learning algorithm used to predict a vector representing different documents. Its structure potentially overcomes the shortcomings of the bag-of-words model. Therefore, in implementation, Doc2vec can be chosen as the text vector representation, using the Sogou News (SogouCS) corpus as training data to construct word vectors, and then using the trained model for text vector representation.

[0057] Example 2: Further, in step 2, candidate words are obtained by segmenting and dividing the text T into words and sentences through the following steps:

[0058] Step 2.1: Segment the processed text T according to sentence integrity to obtain a sentence set, i.e.: T = {S1, S2, S3, ..., S...} n}

[0059] Step 2.2: Segment the sentence using a word segmentation algorithm, and remove stop words and words with specific parts of speech according to given rules to obtain candidate words, i.e.: S={W1,W2,W3,....,W n}

[0060] For Chinese word segmentation, common segmentation algorithms are weak in discovering new words, easily leading to segmentation errors. The quality of segmentation directly affects the effectiveness of subsequent keyword extraction. Therefore, considering the characteristics of news corpora, a specialized segmentation algorithm for news corpora is used to segment the text. Specifically, in the implementation process, open-source segmentation tools such as jieba are used, and a dataset containing features of news corpora such as proper nouns, place names, and organization names is constructed to improve the accuracy of word segmentation.

[0061] For stop word filtering, in addition to considering commonly used natural language processing stop word sets, we can further improve and add new stop words to the stop word database based on the characteristics of news aggregation.

[0062] Example 3: Further, in step 4, the construction of the news hot topic keyword database for a specific time period based on the time-shift weighted algorithm is achieved through the following steps:

[0063] Step 4.1 For the candidate words S after word segmentation in Step 2, calculate the candidate word popularity using the Bayesian mean algorithm. The specific calculation formula is as follows:

[0064]

[0065] Where S(W) i) represents the popularity of candidate words, and W represents the popularity of candidate words. i Average score. F(W) i ,T j ) is the word W i In T j Word frequency over a time period. F avg S represents the average word frequency. avg The average score for all words.

[0066] R avg The calculation formula is as follows: Where F t For the word frequency in the time period t, F t-1 θ represents the word frequency within the previous time period t-1, and θ is the part-of-speech adjustment coefficient. Combining the keyword features of news text, nouns and verbs are more representative of the features of news text. Therefore, different weights are assigned to the part of speech as adjustment coefficients during the calculation process.

[0067] Step 4.2 Adjust the weights of candidate words based on part-of-speech, candidate word position features, and co-occurrence relationships in the news hot topic lexicon, and obtain the restart probability of candidate words.

[0068] 4.2.1 Obtain the position weight of candidate words, and the calculation formula is as follows:

[0069]

[0070] Where |S| is the candidate word list, L(W i ) Represents word W i In the candidate word list, combined with the keyword features of the news text, different weights are assigned to candidate words based on their position characteristics, since candidate words in the title and the first sentence of paragraphs are more likely to become keywords.

[0071] 4.2.2 Obtain the popularity weight of candidate words, and the calculation formula is as follows:

[0072]

[0073] Where n is the length of the candidate word list, s(w i H(w) represents the popularity value of candidate word i. i ) represents the popularity weight value of candidate word i.

[0074] 4.2.3 By fusing location information and sudden word information, the weights of candidate words are set. The weights are then normalized to obtain the restart probability of the candidate words. The calculation formula is as follows:

[0075]

[0076] Where V(w) i ) is the candidate word W iThe weights obtained through weighted summation, P(w) i ) represents the normalized candidate word w i The restart probability, where n is the length of the candidate word list.

[0077] Example 4: Step 5, based on the positional information of candidate words and the news hot word lexicon, uses a graph model to calculate the distribution of text keywords. The specific implementation steps are as follows:

[0078] Step 5.1 Construct a text word graph G = (V, E) based on the candidate word set, where V represents candidate words and E represents the relationship between candidate words. The candidate word score is calculated using the following formula:

[0079]

[0080] Among them, adj(w i ) indicates the relationship with candidate word w i The set of co-occurrence relation words, e(w i ,w j ) indicates candidate words wi With candidate word w j The co-occurrence frequency is specifically calculated within a given co-occurrence window. If the co-occurrence window size k is set, then at most k words will co-occur within a window of size k. This can be adjusted empirically during implementation. O(w j ) indicates that the candidate word is related to w j The sum of the co-occurrence counts of all other candidate words, R(w j ) indicates candidate word w j Rating, P(w) i ) represents the normalized candidate word w i The restart probability is given by λ, where λ is the damping coefficient.

[0081] In particular, by using a set of documents similar to the target document to assist in constructing a word graph, the amount of information in the graph is further enhanced.

[0082]

[0083] sim(d,d k ) for the target document d and document set d k Similarity, F(w i ,w j ) is a candidate word w i and candidate word w j The number of times they co-occur.

[0084] The similarity score can be calculated using cosine similarity based on the text vector representation model obtained in step 3. Specifically, the document set consists of news text data collected over a fixed time period, optionally a day or the most recent few hours. To simplify the computation, a similarity threshold can be specified; only texts with similarity scores greater than the specified threshold are considered as candidate documents for the calculation.

[0085] Step 5.2: Set the convergence value or number of iterations, and use the candidate word calculation formula to iteratively calculate and obtain candidate keywords. Sort the candidate words in reverse order based on the scores, and select the top-n keywords.

[0086] In the formula, λ is the damping coefficient. According to experimental results, with a damping coefficient of 0.85, the value converges to a stable value in about 100 iterations. However, when the damping coefficient is close to 1, the required number of iterations increases sharply, and the sorting becomes unstable. Therefore, 0.85 is chosen here.

[0087] In the specific implementation process, the convergence threshold can be set to 0.00001, and the number of iterations can be set to 50. After the calculation is completed, the candidate words are sorted according to the final score, and the top-n words are selected as the output keywords.

[0088] The basic principles, main features, and advantages of this invention have been described above. Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made without departing from the spirit and scope of the invention, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for extracting news keywords based on a graph model, characterized in that, Includes the following steps: Step 1: Obtain internet news text and perform cleaning processing to obtain the processed text T; Step 2: Perform sentence segmentation and word segmentation on the text T processed in Step 1 to obtain candidate words; Step 3: Represent the text T processed in Step 1 as a vector; Step 4: Obtain the news hot topic keyword database and popularity for a specific time period based on the time-shift weighted algorithm, which is achieved through the following steps: Step 4.1: For the candidate words segmented in Step 2, calculate the candidate word popularity using the Bayesian mean algorithm. The specific calculation formula is as follows: in, Candidate words The popularity Candidate words Average score Candidate words exist Word frequency over time period The average word frequency, The average score for all words; in, The calculation formula is: in, For the word frequency in the time period t, θ represents the word frequency within the previous time period t-1, and θ is the part-of-speech adjustment coefficient. Step 4.2: Adjust the weights of candidate words based on part-of-speech, candidate word position features, and co-occurrence relationships in the news hot topic thesaurus, and obtain the restart probability of candidate words; Step 4.2.1, obtain the position weight of the candidate word, the calculation formula is as follows: in, For the candidate word list, Representative candidate words In the candidate word list, combined with the keyword characteristics of Internet news text, since candidate words in the title and the first sentence of paragraphs are more likely to become keywords, different weights are assigned according to the position characteristics of the candidate words. Step 4.2.2: Obtain the popularity weight of candidate words, calculated using the following formula: Where n is the length of the candidate word list, Candidate words Popularity value Candidate words The popularity weight value; Step 4.2.3: Combine location information and popularity information to obtain the weights of candidate words, normalize the weights, and obtain the restart probability of candidate words. The calculation formula is as follows: in, Candidate words The weights obtained through weighted summation. Candidate words after normalization The restart probability, where n is the length of the candidate word list; Step 5: Based on candidate word location information, hot word thesaurus information, and similar news texts, an improved graph model is applied to calculate the distribution of text keywords. The specific steps are as follows: Step 5.1: Construct a text word graph G = (V, E) based on the candidate word set, where V represents candidate words and E represents the relationships between candidate words. Relationship, in, Candidate words The final score, Indicates with candidate words A set of co-occurrence relation words. Indicate candidate words With candidate words Co-occurrence frequency, Indicate candidate words The sum of the number of times it co-occurs with all other candidate words. Indicate candidate words score, Candidate words after normalization The probability of restarting As the attenuation factor, A set of documents similar to the target document is used to assist in constructing a word graph, thereby increasing the information content of the graph. The calculation formula is as follows: in, For the target document and document collection similarity, Candidate words With candidate words The number of times they co-occur; Step 5.2: Set the convergence value or number of iterations, calculate and obtain candidate keywords, sort the candidate keywords in reverse order according to the scores, and select the top-n as keywords.

2. The news keyword extraction method based on graph model according to claim 1, characterized in that, In step 1, the object of obtaining Internet news text is information from a specific news website. The cleaning process of Internet news text refers to removing webpage tags, special symbols, and image information from the obtained raw online text information to obtain text T to be further processed.

3. The news keyword extraction method based on graph model according to claim 1, characterized in that, In step 2, candidate words are obtained by segmenting and dividing the text T into words and sentences using the following steps: Step 2.1: Segment the processed text T according to sentence integrity, and obtain the sentence set T = {S1, S2, S3, ..., S}. n }; Step 2.2: Combining the features of the news text, the sentence S in the sentence set T is segmented using a word segmentation algorithm. n The process involves word segmentation and part-of-speech tagging, followed by removal of stop words and words with specific parts of speech according to given rules, to obtain a candidate word set S={W1,W2,W3,……,W m } 4. The news keyword extraction method based on graph model according to claim 1, characterized in that, In step 3, the document representation model obtained by training the news corpus is used to represent the text T as a vector.

Images