A lightweight Chinese keyword extraction method based on statistical characteristics and word graph

Abstract

The application discloses a keyword extraction method based on statistical features and word graphs for single Chinese text, which comprises the steps of text preprocessing, calculating features of each word, calculating comprehensive scores of each word, sorting and filtering. The features of the word include word frequency features, position features, distribution span features, sentence frequency features, special word features and word graph scores. The application obtains accurate keyword extraction results and keyword scores with distinction based on statistical features and word graph information. The application has the following advantages: suitable for single Chinese text; lightweight, no model training and additional corpus; widely suitable for texts in different fields.

Description

Technical Field

[0001] The present invention relates to a Chinese keyword extraction method, in particular to a lightweight Chinese keyword extraction method based on statistical features and word graphs, and belongs to the technical field of text mining. Background Art

[0002] With the rapid development of the information age, the amount of text data has exploded. Most articles on the internet lack keyword annotations, yet keywords are crucial for concisely describing, classifying, and retrieving text data. Manual annotation is time-consuming and labor-intensive for tasks like retrieval that require document annotation. Automated annotation can generate keyword lists in real time. For readers, keywords allow them to understand the content of an article without having to read the entire article. Furthermore, keywords can attract users interested in specific topics.

[0003] Keyword extraction technology is divided into unsupervised and supervised methods. Unsupervised methods include statistical feature-based methods and word graph-based methods. Statistical feature-based methods construct features based on text and generate keywords based on these features. For example, the TF-IDF (Term Frequency-Inverse Document Frequency) method scores words using two features: term frequency and inverse document frequency. YAKE (YetAnother Keyword Extractor) gives word scores using multiple features. Word graph-based methods follow the idea of ​​PageRank, considering the co-occurrence of words in a specific window as a connection relationship and assigning a score to each word. Supervised methods use machine learning to train keyword extraction models. For example, KEA (Keyphrase Extraction Algorithm) determines keywords using the Naive Bayes method, and KeyBert determines keywords based on the Bert (Bidirectional Encoder Representation from Transformers) model.

[0004] Existing methods have significantly advanced the automatic tagging of text keywords, achieving some success on specific types of text. However, they also have limitations. For example, TF-IDF requires additional corpus, YAKE's feature design is more suitable for European and American languages ​​and is less effective on Chinese, and the KeyBert method requires a specialized corpus training model. We need a lightweight, accurate, and Chinese-applicable keyword extraction method that meets the following requirements:

[0005] 1. Unsupervised methods: No need to train a proprietary model on specific data.

[0006] 2. Domain- and corpus-independent: Information can be retrieved from a single text without the need to use a domain-specific corpus.

[0007] 3. Suitable for Chinese: Designed according to the language characteristics of Chinese text.

[0008] 4. Fast and lightweight method: directly give results quickly for a single article.

[0009] 5. Accurate keyword scoring and ranking: Not only specific keywords are given, but also the scores of each word are given, and the importance of the words can be clearly distinguished by the scores. Summary of the Invention

[0010] The purpose of the present invention is to provide a lightweight Chinese keyword extraction method based on statistical features and word graphs.

[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0012] A lightweight Chinese keyword extraction method based on statistical features and word graphs includes the following steps:

[0013] Step 1: Text preprocessing, which consists of the following specific steps:

[0014] Step 1-1: Divide the text into text units according to the set sentence symbols;

[0015] Step 1-2: Convert uppercase English letters in each text unit to lowercase English letters.

[0016] Steps 1-3: Use the "jieba" word segmentation to divide each text unit into more than one word, and then filter out the words in each text unit that appear in the stop word list;

[0017] Step 1-4: Special word marking, marking special words in the text unit according to preset rules;

[0018] Step 2: Calculate the features of each word one by one: It consists of the following specific steps:

[0019] Assume that n is the number of text units, tf is the sum of the number of times the current word appears in each text unit, m is the number of text units in which the current word appears, and the set of text unit numbers in which the current word appears is {s1, s2, s3, ..., s m}, where 1≤s1 <s2<s3<……<s m ≤n.

[0020] Step 2-1: Calculate word frequency feature TF:

[0021] TF=tf

[0022] Step 2-2: Calculate the position feature POS:

[0023]

[0024] Where c is a constant, s i The text unit number where the current word appears.

[0025] Step 2-3: Calculate the distribution span feature DS:

[0026]

[0027] Step 2-4: Calculate the sentence frequency feature SF:

[0028]

[0029] Step 2-5: Calculate the special word feature SW:

[0030]

[0031] Where a is a constant.

[0032] Step 2-6: Calculate the word-graph score TRS, which consists of the following steps:

[0033] Step A-1: ​​Calculate a word neighbor connection matrix, where the elements of the word neighbor connection matrix are the frequencies of two words appearing simultaneously within a preset window.

[0034] Step A-2: Construct an undirected graph G based on the word adjacency matrix. Each word is a vertex of the undirected graph G. If two words appear at the same time within a certain preset window, there will be an edge between the two vertices. The frequency of the two words appearing at the same time is used as the weight of the edge.

[0035] Step A-3: Initialize the score of each word and iterate according to the following formula until convergence.

[0036]

[0037] where w i represents the i-th word, the total number of words in each text unit N, α is a constant, E(G) represents the edge of graph G, u(w i ,w j ) represents the weight of the edge connecting the vertices corresponding to the i-th word and the j-th word, TR(w i ) is the score of the i-th word, TR(w j ) is the score of the j-th word.

[0038] Step A-4: Perform min-max normalization on each word score to obtain the word graph score TRS.

[0039]

[0040] Step 3: Calculate the comprehensive score S of each word:

[0041]

[0042] wherein, represents the word frequency average of each word in the text unit.

[0043] Step 4: Sorting and filtering

[0044] Step 4-1: Sort the comprehensive score S of each word.

[0045] Step 4-2: Filter the words using the dictionary, and only keep the words existing in the dictionary.

[0046] Step 4-3: Calculate the Jaccard distance between each word:

[0047]

[0048] wherein w i and w j represent the i-th and j-th words, W i represents the set of characters in the word w i , W j represents the set of characters in the word w j , |W i ∩W j | represents the number of characters appearing in both the i-th and j-th words, |W i ∪W j | represents the number of all characters appearing in the i-th and j-th words.

[0049] Step 4-4: Determine whether the Jaccard distance between each word is less than a preset threshold value, if so, delete the word with a low comprehensive score S, and keep the remaining words and their scores as the output of the key words.

[0050] Further, the sentence symbols in step-1-1 include Chinese and English full stop, question mark, exclamation mark, ellipsis, and semicolon.

[0051] Further, step 1-1 uses "Jieba" segmentation to divide each text unit into more than one word.

[0052] Further, the words in the stop word list in step 1-3 include function words and punctuation marks.

[0053] Further, the preset rule in step 1-4 is that the English words appearing in the Chinese text or the English words and numbers connected by punctuation marks.

[0054] By adopting the technical scheme, the application has the following beneficial effects:

[0055] 1. Based on statistical features and word graph information, the present invention obtains accurate keyword extraction results and discriminative keyword scores.

[0056] 2. The present invention has the following advantages: it is applicable to a single Chinese text; it is lightweight and does not require model training and additional corpus; and it is widely applicable to texts in different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Figure 1 It is a flow chart of the overall method.

[0059] Figure 2 This is the flowchart of candidate word score calculation. DETAILED DESCRIPTION

[0060] In the examples, we selected Chinese texts from different fields to demonstrate how to use the method in this paper to extract keywords. The beginning of each text is shown in the examples.

[0061] Example 1:

[0062] A lightweight Chinese keyword extraction method based on statistical features and word graphs includes the following steps:

[0063] Step 1: Text preprocessing, which consists of the following specific steps:

[0064] Step 1-1: Divide the text into text units according to the set sentence symbols;

[0065] Step 1-2: Convert uppercase English letters in each text unit to lowercase English letters.

[0066] Step 1-3: Use the "Jieba" word segmentation to divide each text unit into more than one word, and then filter out the words in each text unit that appear in the stop word list.

[0067] Step 1-4: Special word marking, marking special words in the text unit according to preset rules,

[0068] Step 2: Calculate the features of each word one by one: It consists of the following specific steps:

[0069] Assume that n is the number of text units, tf is the sum of the number of times the current word appears in each text unit, m is the number of text units in which the current word appears, and the set of text unit numbers in which the current word appears is {s1, s2, s3, ..., s m}, where 1≤s1 <s2<s3<……<s m ≤n.

[0070] Step 2-1: Calculate word frequency feature TF:

[0071] TF=tf

[0072] Step 2-2: Calculate the position feature POS:

[0073]

[0074] Where c is a constant, s i The text unit number where the current word appears.

[0075] Step 2-3: Calculate the distribution span feature DS:

[0076]

[0077] Step 2-4: Calculate the sentence frequency feature SF:

[0078]

[0079] Step 2-5: Calculate the special word feature SW:

[0080]

[0081] Where a is a constant.

[0082] Step 2-6: Calculate the word-graph score TRS, which consists of the following steps:

[0083] Step A-1: ​​Calculate a word neighbor connection matrix, where the elements of the word neighbor connection matrix are the frequencies of two words appearing simultaneously within a preset window.

[0084] Step A-2: Construct an undirected graph G based on the word adjacency matrix. Each word is a vertex of the undirected graph G. If two words appear at the same time within a certain preset window, there will be an edge between the two vertices. The frequency of the two words appearing at the same time is used as the weight of the edge.

[0085] Step A-3: Initialize the score of each word and iterate according to the following formula until convergence.

[0086]

[0087] where w i represents the i-th word, the total number of words in each text unit N, α is a constant, E(G) represents the edge of graph G, u(w i ,w j ) represents the weight of the edge connecting the vertices corresponding to the i-th word and the j-th word, TR(w i ) is the score of the i-th word.

[0088] Step A-4: Perform min-max normalization on each word score to obtain the word graph score TRS.

[0089]

[0090] Step 3: Calculate the comprehensive score S of each word:

[0091]

[0092] in, Represents the mean frequency of each word in the text unit.

[0093] Step 4: Sorting and filtering

[0094] Step 4-1: Sort the comprehensive score S of each word. The lower the score, the more important the word.

[0095] Step 4-2: Use the dictionary to filter the words and only keep the words that exist in the dictionary.

[0096] Step 4-3: Calculate the Jaccard distance between each word:

[0097]

[0098] where w i and w j represents the i-th and j-th words, W i Representation word w i A collection of Chinese characters, W j Representation word w j A collection of Chinese characters, |W i ∩W j | represents the number of words that appear together with the i-th and j-th words, |W i ∪W j | represents the number of all characters that appear in the i-th and j-th words.

[0099] Step 4-4: Determine whether the Jaccard distance between words is less than a preset threshold. If so, delete the words with low comprehensive scores S, and output the remaining words and their scores as keywords.

[0100] The main content of Text 1 is: The Hubble Telescope observed a supermassive black hole. Scientists speculate that gravitational waves generated by the merger of two black holes may have ejected it from the center of the galaxy. As shown in Table 1, the top five keywords generated by this method well represent the content of the article, and the scores also clearly distinguish the importance of different words. In addition, by filtering the vocabulary, the keyword "two" can be removed, which is not a professional term, meeting more professional text annotation needs.

[0101] In this embodiment, text 1 reads, "According to foreign media reports, gravitational waves, under conditions equivalent to the energy produced by 100 million supernovae, may be able to eject supermassive black holes from the centers of distant galaxies. Recently, the Hubble Telescope observed the largest black hole currently located outside the nucleus of a galaxy, with a mass billions of times that of the sun..."

[0102] The sentence symbols in this embodiment include Chinese and English periods, question marks, exclamation marks, ellipsis marks, and semicolons. The words in the stop word list of this embodiment include function words and punctuation marks, such as "的", "吗", ",". In this embodiment, the preset rules are English words appearing in Chinese text or English words and numbers connected by punctuation marks, such as "CAT", "AT", "2 / A~B". In this embodiment, TF=4.

[0103] Text 1 contains a total of 17 text units. The word "gravitational wave" appears four times, appearing in the 1st, 3rd, and 17th text units, that is, in three text units in total. The following features are calculated for the word "gravitational wave".

[0104] In this example, n is 17, and s1, s2, and s3 are 1, 3, and 17, respectively. Substituting these values ​​into the equation yields the position feature POS = 0.2097, the distribution span feature DS = 1, and the sentence frequency feature SF = 0.1765. "Gravitational wave" is not a special word, and its special word feature SW = 0. The final iteration of the word graph score yields TRS = 0.0148.

[0105] In this embodiment, an astronomy dictionary is selected, which contains professional terms such as "black hole", "gravitational wave", and "galactic nucleus". After using the dictionary to filter, words such as "gravitational wave" in the dictionary are retained, while other words that do not exist in the dictionary are filtered out.

[0106] In this embodiment, the two words "galaxy center" and "galactic nucleus" have a Jaccard distance of 0.6; the preset threshold of the Jaccard distance is 0.1. The top 5 keywords and their scores are shown in Table 1.

[0107] In this embodiment, more keyword extraction results are used to demonstrate the applicability of the present invention to different fields, different text lengths, and different types of texts.

[0108] Iron deficiency is common in heart failure patients (30%-80%) and is associated with increased mortality and hospitalization rates. Previous studies have shown that ferric carboxymaltose (FCM) can improve symptoms, quality of life, and exercise tolerance in heart failure patients with iron deficiency. However, further confirmation is needed regarding its impact on clinical outcomes.

[0109] Text 3: During the Mid-Autumn Festival and National Day holidays, the cultural and tourism industries saw a strong recovery, and the national tourism market remained stable and orderly. According to calculations by the Ministry of Culture and Tourism's data center, domestic tourism trips reached 826 million during the eight-day holiday, a year-on-year increase of 71.3% on a comparable basis and a 4.1% increase compared to 2019. Domestic tourism revenue reached 753.43 billion yuan, a year-on-year increase of 129.5% on a comparable basis and a 1.5% increase compared to 2019.

[0110] Text 4: Since the 21st century, the scale of private lending in my country has gradually expanded, its impact has become increasingly profound, and it plays a vital role in China's current economic development. Private lending has both positive and negative effects. On the one hand, the existence and development of private lending has compensated for the shortcomings of formal finance while also driving the development of formal finance...

[0111] The main content of Text 2 is: Ferric carboxymaltose (FCM) does not significantly reduce all-cause mortality or heart failure hospitalization rates in patients with heart failure and iron deficiency. However, a meta-analysis showed that FCM can reduce cardiovascular hospitalization rates and heart failure hospitalization rates, but has no effect on survival rates. As shown in Table 2, the keywords "fcm," "heart failure," "hospitalization," "iron deficiency," and "cardiovascular" well represent the text content, and the score of "fcm" is significantly different from that of other words. The main content of Text 3 is: During the Mid-Autumn Festival and National Day holidays, the domestic tourism market was stable and orderly, with significant growth in the number of tourists and tourism revenue. The keywords and corresponding scores summarize the text content well. The main content of Text 4 is: It analyzes the forms of private lending disputes and the deficiencies in legal regulations, and puts forward suggestions for improving my country's private lending legal regulations, including accelerating private lending legislation, standardizing loan contracts, and standardizing the purpose of loans. It can be seen that this embodiment also has a certain summarizing effect on Text 4, especially the core word "private lending" has obtained a score with obvious differentiation.

[0112] Table 1

[0113]

[0114]

[0115] Table 2

[0116]

Claims

1. A lightweight Chinese keyword extraction method based on statistical features and word graphs, characterized in that: The following steps are involved: Step 1: Text preprocessing, which consists of the following specific steps: Step 1-1: Divide the text into text units according to the set sentence symbols; Step 1-2: Convert the uppercase English letters in each text unit to lowercase English letters; Steps 1-3: Use the "jieba" word segmentation to divide each text unit into one or more words, and then filter out the words in each text unit that appear in the stop word list; Step 1-4: Special word marking, marking special words in the text unit according to preset rules; Step 2: Calculate the features of each word one by one: It consists of the following specific steps: Assume that n is the number of text units, tf is the sum of the number of times the current word appears in each text unit, m is the number of text units in which the current word appears, and the set of text unit numbers in which the current word appears is {s1, s2, s3, ..., s m }, where 1≤s1 <s2<s3<……<s m ≤n; Step 2-1: Calculate word frequency feature TF: TF=tf Step 2-2: Calculate the position feature POS: Where c is a constant, s i The text unit number where the current word appears; Step 2-3: Calculate the distribution span feature DS: Step 2-4: Calculate the sentence frequency feature SF: Step 2-5: Calculate the special word feature SW: Where a is a constant; Step 2-6: Calculate the word-graph score TRS, which consists of the following steps: Step A-1: ​​Calculate a word neighbor connection matrix, where the elements of the word neighbor connection matrix are the frequencies of two words appearing simultaneously within a preset window; Step A-2: Construct an undirected graph G based on the word adjacency matrix. Each word is a vertex of the undirected graph G. If two words appear together within a certain preset window, an edge exists between the two vertices. The frequency of the two words appearing together serves as the weight of the edge. Step A-3: Initialize the score of each word and iterate according to the following formula until convergence; where w i represents the i-th word, the total number of words in each text unit N, α is a constant, E(G) represents the edge of graph G, u(w i ,w j ) represents the weight of the edge connecting the vertices corresponding to the i-th word and the j-th word, TR(w i ) is the score of the i-th word, TR(w j ) is the score of the jth word; Step A-4: Perform min-max normalization on each word score to obtain the word graph score TRS; Step 3: Calculate the comprehensive score S of each word: in, Represents the mean frequency of each word in the text unit; Step 4: Sorting and filtering Step 4-1: Sort the comprehensive score S of each word; Step 4-2: Use the dictionary to filter the words and only keep the words that exist in the dictionary; Step 4-3: Calculate the Jaccard distance between each word: where w i and w j represents the i-th and j-th words, W i Representation word w i A collection of Chinese characters, W j Representation word w j A collection of Chinese characters, |W i ∩W j | represents the number of words that appear together with the i-th and j-th words, |W i ∪W j | represents the number of all characters that appear in the i-th and j-th words; Step 4-4: Determine whether the Jaccard distance between words is less than a preset threshold. If so, delete the words with low comprehensive scores S, and output the remaining words and their scores as keywords.

2. The lightweight Chinese keyword extraction method based on statistical features and word graph according to claim 1 is characterized in that: The sentence separation symbols in step 1-1 include Chinese and English periods, question marks, exclamation marks, ellipsis marks, and semicolons.

3. The lightweight Chinese keyword extraction method based on statistical features and word graph according to claim 1 is characterized in that: Step 1-1 uses the "jieba" word segmentation to divide each text unit into one or more words.

4. The lightweight Chinese keyword extraction method based on statistical features and word graph according to claim 1 is characterized in that: The words in the stop word list in steps 1-3 include function words and punctuation marks.

5. The lightweight Chinese keyword extraction method based on statistical features and word graph according to claim 1 is characterized in that: The preset rules in steps 1-4 are English words appearing in Chinese text or English words and numbers connected by punctuation marks.

Images