Keyword extraction method, device, equipment and storage medium

Abstract

The present application relates to a keyword extraction method, device, equipment and storage medium. The keyword extraction method includes: obtaining an input sample; obtaining a first score for each candidate word in the input sample according to a first preset algorithm; obtaining a second score for each candidate word in the input sample according to a second preset algorithm; performing a setting operation on the first score of each candidate word and the second score of each candidate word to obtain a third score for each candidate word; and determining the keyword of the input sample according to the ranking result of the third score of each candidate word. The solution provided by the present application can improve the precision and recall rate of keyword extraction, so that the extracted keywords can better reflect the meaning of the text content.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a keyword extraction method, device, equipment and storage medium. Background Art

[0002] Keyword extraction is the key to information retrieval, text classification and clustering, and automatic abstract generation, and is an important means to quickly obtain the subject of a document. The extraction of text keywords is to extract the words that are most relevant to the meaning of the text from the entire text, so that viewers can quickly understand the text content through text keywords.

[0003] There are many text keyword extraction algorithms in related technologies, such as open source general algorithms such as the TextRank algorithm, as well as text keyword extraction algorithms based on machine learning models, probability models and deep learning models.

[0004] However, the above algorithms all have the problems of low keyword extraction accuracy and low recall rate. Summary of the invention

[0005] In order to solve or partially solve the problems existing in the related art, the present application provides a keyword extraction method, device, equipment and storage medium, which can improve the accuracy and recall rate of keyword extraction, so that the extracted keywords can better reflect the meaning of the text content.

[0006] The first aspect of the present application provides a keyword extraction method, comprising:

[0007] Get input samples;

[0008] Obtaining a first score for each candidate word in the input sample according to a first preset algorithm;

[0009] Obtaining a second score for each candidate word in the input sample according to a second preset algorithm;

[0010] Performing a setting operation on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word;

[0011] The keyword of the input sample is determined according to the ranking result of the third scores of the candidate words.

[0012] In one embodiment, obtaining the first score of each candidate word in the input sample according to the first preset algorithm includes: obtaining the first score of each candidate word in the input sample according to a preset keyword library and the first preset algorithm;

[0013] The obtaining of the second score of each candidate word in the input sample according to the second preset algorithm includes: obtaining the second score of each candidate word in the input sample according to the set corpus corresponding to the preset keyword library and the second preset algorithm;

[0014] The preset keyword library includes core words and corresponding related words.

[0015] In one embodiment, the core words and the corresponding related words in the preset keyword library are pre-determined in the following manner:

[0016] Filter the words with the highest scores from the set corpus;

[0017] According to the set field, select words related to the set field from the words with the highest scores as core words;

[0018] The core word is searched for synonyms, and part-of-speech filtering and length filtering are performed to select a set number of words related to the core word as corresponding related words.

[0019] In one implementation, determining the keyword of the input sample according to the ranking result of the third scores of the candidate words includes:

[0020] The keyword of the input sample is determined according to the ranking result of the third score of each candidate word and a preset invalid word library.

[0021] In one implementation, performing a setting operation on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word includes:

[0022] A weighted sum operation is performed on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word.

[0023] In one implementation, obtaining an input sample includes:

[0024] Get a sample of preprocessed text content input.

[0025] In one embodiment, the first preset algorithm is a SeedTextRank algorithm, wherein the SeedTextRank algorithm includes weighting the candidate words, and the weighting includes keyword weighting, position weighting, and edge weighting;

[0026] The second preset algorithm is the TFIDF algorithm.

[0027] A second aspect of the present application provides a keyword extraction device, comprising:

[0028] An acquisition module, used to obtain input samples;

[0029] A first operation module, used for obtaining a first score of each candidate word in the input sample according to a first preset algorithm;

[0030] A second operation module, used for obtaining a second score of each candidate word in the input sample according to a second preset algorithm;

[0031] A third operation module, used for performing a set operation on the first score of each candidate word obtained by the first operation module and the second score of each candidate word obtained by the second operation module to obtain a third score of each candidate word;

[0032] The extraction module is used to determine the keyword of the input sample according to the ranking result of the third score of each candidate word obtained by the third operation module.

[0033] In one embodiment, the device further comprises:

[0034] The word library module is used to set a preset keyword library, wherein the preset keyword library includes core words and corresponding related words;

[0035] The first operation module obtains a first score of each candidate word in the input sample according to a preset keyword library of the vocabulary module and a first preset algorithm;

[0036] The second operation module obtains a second score for each candidate word in the input sample according to a set corpus corresponding to a preset keyword library of the vocabulary module and a second preset algorithm.

[0037] A third aspect of the present application provides an electronic device, including:

[0038] Processor; and

[0039] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0040] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0041] The technical solution provided by this application may have the following beneficial effects:

[0042] The technical solution of the present application obtains the first score of each candidate word in the input sample according to the first preset algorithm and obtains the second score of each candidate word in the input sample according to the second preset algorithm; then the first score of each candidate word and the second score of each candidate word are set and operated to obtain the third score of each candidate word; finally, the keyword of the input sample is determined according to the sorting result of the third score of each candidate word. Since the solution of the present application integrates the results of two preset algorithms for comprehensive consideration and finally determines the final score of the candidate word, the score used for sorting reference can more accurately reflect the importance and relevance of the candidate word in the input sample, thereby improving the precision and recall rate of keyword extraction, so that the extracted keywords can better reflect the meaning of the text content.

[0043] Furthermore, the technical solution of the present application may be to obtain a first score for each candidate word in the input sample according to a preset keyword library and a first preset algorithm, obtain a second score for each candidate word in the input sample according to a set corpus corresponding to the preset keyword library and a second preset algorithm, and determine the keyword of the input sample according to the sorting result of the third score of each candidate word and a preset invalid word library. In other words, by using the preset keyword library and the preset invalid word library in the relevant field as the knowledge content input of the algorithm, it is possible to more effectively extract keywords from the text content in the special field, and compared with the general algorithm, the keyword extraction is more accurate and the extraction efficiency is higher.

[0044] Furthermore, in the technical solution of the present application, the first preset algorithm may be a SeedTextRank algorithm, and the second preset algorithm may be a TFIDF algorithm; wherein the SeedTextRank algorithm includes weighted processing of the candidate words, and the weighted processing includes keyword weighting, position weighting, edge weighting, etc.

[0045] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0047] Figure 1 is a flowchart of a keyword extraction method shown in an embodiment of the present application;

[0048] Figure 2 is another flowchart of the keyword extraction method shown in the embodiment of the present application;

[0049] Figure 3 is a schematic diagram of the experimental results of Application Example 1 in the keyword extraction method shown in the embodiment of the present application;

[0050] Figure 4 is a schematic diagram of the experimental results of Application Example 2 in the keyword extraction method shown in the embodiment of the present application;

[0051] Figure 5 is a schematic diagram of the experimental results of Application Example 3 of the keyword extraction method shown in the embodiment of the present application;

[0052] Figure 6 is a schematic diagram of the structure of a keyword extraction device shown in an embodiment of the present application;

[0053] Figure 7 is another structural schematic diagram of a keyword extraction device shown in an embodiment of the present application;

[0054] Figure 8 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0056] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0058] In the related art, the TFIDF algorithm, PageRank algorithm and TextRank algorithm are used to extract keywords, but there are problems such as low precision and low recall rate of keyword extraction. In view of the above problems, the embodiment of the present application provides a keyword extraction method, which can improve the precision and recall rate of keyword extraction, so that the extracted keywords can better reflect the meaning of the text content.

[0059] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0060] Figure 1 It is a flowchart of the keyword extraction method shown in the embodiment of the present application.

[0061] See also Figure 1 , the method comprising:

[0062] S101: Obtain input samples.

[0063] Among them, a preprocessed text content input sample can be obtained.

[0064] S102: Obtain a first score for each candidate word in the input sample according to a first preset algorithm.

[0065] The first score of each candidate word in the input sample can be obtained according to the preset keyword library and the first preset algorithm. The first preset algorithm can be a SeedTextRank algorithm, wherein the SeedTextRank algorithm includes weighted processing of each candidate word, and the weighted processing includes keyword weighting, position weighting, edge weighting, etc.

[0066] The preset keyword library includes core words and corresponding related words. The core words and corresponding related words in the preset keyword library can be determined in advance in the following ways: select words with the highest scores from the set corpus; select words related to the set field from the words with the highest scores as core words according to the set field; perform synonym search on the core words, and perform part of speech filtering and length filtering, and select a set number of words related to the core words as corresponding related words.

[0067] S103. Obtain a second score for each candidate word in the input sample according to a second preset algorithm.

[0068] The second score of each candidate word in the input sample can be obtained according to the set corpus corresponding to the preset keyword library and the second preset algorithm. The second preset algorithm can be a TFIDF algorithm. The set corpus can be, for example, the corpus of a company or industry.

[0069] S104: Perform a setting operation on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word.

[0070] The first score of each candidate word and the second score of each candidate word may be weightedly summed to obtain the third score of each candidate word.

[0071] S105: Determine the keyword of the input sample according to the ranking result of the third score of each candidate word.

[0072] Among them, the keyword of the input sample can be determined according to the ranking result of the third score of each candidate word and a preset invalid word library.

[0073] The invalid words in the preset invalid word library can be determined in advance in the following manner: DF (Document Frequency) features of all words in the set corpus are counted, and words with DF features greater than or equal to a set number are selected as meaningless invalid words.

[0074] It can be seen from this embodiment that the technical solution of the present application obtains the first score of each candidate word in the input sample according to the first preset algorithm and obtains the second score of each candidate word in the input sample according to the second preset algorithm; then the first score of each candidate word and the second score of each candidate word are set and operated to obtain the third score of each candidate word; finally, the keywords of the input sample are determined according to the sorting results of the third scores of each candidate word. Since the solution of the present application integrates the results of the two preset algorithms for comprehensive consideration and finally determines the final score of the candidate word, the score used for sorting reference can more accurately reflect the importance and relevance of the candidate word in the input sample, thereby improving the precision and recall rate of keyword extraction, so that the extracted keywords can better reflect the meaning of the text content.

[0075] Figure 2 It is another flowchart of the keyword extraction method shown in the embodiment of the present application.

[0076] The SeedTextRank_TFIDF algorithm provided in this application is an integrated algorithm of the SeedTextRank algorithm and TFIDF, wherein the SeedTextRank algorithm is an algorithm for integrating core words in a field that is improved on the basis of the open source TextRank algorithm, and the TFIDF algorithm is an open source text feature extraction algorithm. The SeedTextRank_TFIDF algorithm of this application is used to calculate the score of each candidate word in the text, and then sort the scores of the candidate words in descending order, take the top K, and then obtain the K keywords of the text and their weights. The SeedTextRank_TFIDF algorithm of this application optimizes the score calculation method in the TextRank algorithm process.

[0077] See also Figure 2 , the method comprising:

[0078] S201, pre-construct a keyword library and an invalid word library.

[0079] In the present application scheme, a keyword library can be pre-constructed, and the keyword library includes core words and related words. First, the TFIDF algorithm can be used to calculate the words with higher scores in the first set corpus (such as the corpus of a company or industry), such as the top 100 words, and these 100 words are calibrated, and several representative words in the field of new energy vehicles are selected from them, such as "autonomous driving", "NEDC (New European Driving Cycle, New European Cycle Test)", "NGP (Navigation Guided Pilot, Intelligent Navigation Assisted Driving)" and other words in the field of new energy vehicles. The words selected in this step can be called core words; in the second step, the second set corpus (such as Wikipedia corpus and the corpus of this application field) is used to jointly train the word2vec open source word vector model, and the word2vec model is used to search for synonyms of the selected core words, and part of speech filtering and length filtering are performed at the same time, and several representative words in the field of new energy vehicles are selected as corresponding related words. The newly constructed keyword library (or keyword dictionary) contains core words and a set number of related words related to each core word, for example, 5 related words.

[0080] In the present application, an invalid word library (or invalid word dictionary) can also be constructed. The DF (Document Frequency) features of all words in the set corpus are counted, and words with DF features greater than or equal to the set number are selected as meaningless invalid words. For example, the DF features of all words in the application domain corpus are counted, and meaningless words with DF features greater than or equal to 1000 are selected as invalid words and added to the invalid word library.

[0081] S202: Obtain a preprocessed text content input sample.

[0082] The present application can preprocess the input text content to obtain a preprocessed text content input sample. The preprocessing process may include:

[0083] 1) Clean the text to remove HTML (Hyper Text Markup Language) tags, URL (uniform resource locator), punctuation marks, etc.

[0084] 2) Use Jieba word segmentation to segment the text;

[0085] Jieba word segmentation can include three modes: the first is the precise mode, which is to segment the sentences in the sample text most accurately, suitable for text analysis; the second is the full mode, which is to scan all the words that may form words in the sample text sentences, which has the advantages of being fast and efficient, but cannot solve the problem of ambiguity; the third is the search engine mode, which is based on the precise mode and re-segments long words, which has the advantage of being able to improve the recall rate and is suitable for search engine word segmentation. During use, you can flexibly select one of the above three modes according to actual needs to complete the word segmentation operation of the text and divide it into several word segments.

[0086] 3) Use a common stop dictionary to remove stop words;

[0087] The purpose of removing stop words is to consider that most stop words have basically no meaning, so stop words can be removed.

[0088] 4) Part-of-speech filtering: Use Jieba word segmentation to obtain the part-of-speech, and retain the following parts of speech: nouns, common verbs, gerunds, adjectives, adjectives, proper nouns, names of people, names of organizations, place names, time, etc.;

[0089] 5) Perform synonym replacement and use a custom synonym library to replace the corresponding words with common words.

[0090] S203: Obtain the first score of each candidate word in the input sample according to the keyword library and the SeedTextRank algorithm.

[0091] The SeedTextRank algorithm is an improved algorithm based on the open source TextRank algorithm that integrates core domain words. The overall process of the SeedTextRank algorithm is similar to that of the TextRank algorithm. The network is constructed through the adjacent relationship between words, and then the score value of each node is iteratively calculated using PageRank. The score values ​​are sorted from high to low to get the keywords. The TextRank algorithm considers a certain word to have a graph adjacent relationship with the N words in front of it and the N words behind it (similar to the N-gram grammar model), sets a sliding window of length N, and all words in this window are regarded as adjacent nodes of the word node. The word graph constructed by TextRank is an undirected graph. Based on the constructed undirected graph, the TextRank algorithm performs multiple rounds of iterations according to the scoring formula to finally get the score of each word.

[0092] This step uses the SeedTextRank algorithm to calculate the candidate word scores.

[0093] The SeedTextRank algorithm calculates the candidate word scores as follows:

[0094]

[0095] Among them, R(Wi) is the score of the current candidate word, indicating the value of the current new round of iteration, R(Wj) is the score of the word in the window, r(Wi) is the score of the candidate word in the previous iteration, and e(Wj, Wi) is the weight parameter of the edge between Wj and Wi in the word graph.

[0096] Among them, λ, α, and γ are algorithm debugging parameters, which can be set as needed. For example, λ can be a damping coefficient, and its value range can be 0 to 1.

[0097] in, ,express This operator calculates and accumulates the value of all words in the window.

[0098] in, , which means that if the candidate word Wi is in the keyword library, the keyword weight parameter The value is 1, otherwise The value is 0;

[0099] Among them, if the candidate word Wi appears at the beginning of the text, the position weight parameter The value is 1; if the candidate word Wi appears in the highlighted position, the position weight parameter The value is 2; if the candidate word Wi appears in the title position, the position weight parameter The value is 2; otherwise, the position weighting parameter The value is 0.

[0100] Among them, the weight parameter of the edge in the word graph is The calculation formula is as follows:

[0101]

[0102] in, Indicates the position of the word within the window; Indicates the position of the current candidate word. Edge weight parameter Indicates the similarity between sentences. The weight parameters of the edges in the word graph are calculated to consider the distance between the neighbor words and the keywords, so as to reduce the negative impact of the neighbor effect in the algorithm. It should be noted that in the formula for calculating the edge weight parameters in the word graph, some functions that are close to normal distribution can also be used to replace the formula .

[0103] Finally, weighted calculations are performed on the words / keywords in the dominant position, which can be calculated using the following formula:

[0104]

[0105] Among them, λ1, α1, and γ1 are algorithm debugging parameters and can be set as needed.

[0106] The operation of this formula is helpful for capturing words near the words / keywords in an advantageous position during the iteration process, and is also helpful for capturing the words / keywords themselves in an advantageous position.

[0107] S204: Obtain a second score for each candidate word in the input sample according to a set corpus corresponding to the preset keyword library and a TFIDF algorithm.

[0108] This step uses the TFIDF (Term Frequency-Inverse Document Frequency) algorithm to calculate the candidate word score. The IDF in this step is calculated using the corpus of the first set corpus (such as the corpus of a company or industry) corresponding to the preset keyword library, which can reduce the scores of some high-frequency words that are not article-specific.

[0109] TFIDF is a statistical method used to evaluate the importance of a word to a document set or one of the documents in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a document, but it decreases in inverse proportion to the frequency of its appearance in the corpus. TF refers to term frequency, which indicates the frequency of keywords appearing in a document; IDF refers to inverse document frequency, which reflects the prevalence of keywords. The more prevalent a word is (i.e., a large number of documents contain this word), the lower its IDF value is; conversely, the higher its IDF value is.

[0110]

[0111]

[0112] The denominator needs to be added with 1 to avoid the denominator being 0.

[0113]

[0114] Therefore, assuming that p represents a keyword, d represents a document, and D represents the total number of documents, the candidate word score calculation method using the TFIDF algorithm can be as follows:

[0115]

[0116] in:

[0117]

[0118]

[0119] Among them, TF (p, d) represents the frequency of candidate word p in document d, f (p, d) is the number of occurrences of candidate word p, |d| is the number of all words in document d, df (p, d) table contains the total number of documents of candidate word p, |D| is the total number of all documents in the corpus of the first set corpus (such as the corpus of a company or industry), Indicates the name of the first set corpus.

[0120] S205: Perform a weighted sum operation on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word.

[0121] This step uses the SeedTextRank_TFIDF algorithm to calculate the candidate word scores.

[0122] The SeedTextRank_TFIDF candidate word score calculation method is as follows:

[0123]

[0124] In this formula, the candidate word scores obtained by using the SeedTextRank algorithm and the candidate word scores obtained by using the TFIDF algorithm are weighted and summed to obtain the final score of each candidate word.

[0125] Among them, δ is the algorithm debugging parameter, which can be set as needed. Equivalent to the TFIDF algorithm formula .

[0126] S206: Determine the keyword of the input sample according to the ranking result of the third score of each candidate word and the preset invalid word library.

[0127] After the third score of each candidate word is obtained in this step, the candidate words can be sorted in descending order according to the third score, and the top candidate words, such as the top 20, are extracted. In addition, the extracted candidate words are invalidated by referring to the invalid word library, that is, the invalid words in the field are deleted according to the invalid word library. The advantage of deleting these invalid words at the end of this application is that the importance of the words nearby can be retained.

[0128] This application applies the above method to experiment with different input samples.

[0129] For example, Figure 3 Schematic diagram of the experimental results of Application Example 1 of the keyword extraction method shown in the embodiment of the present application. Figure 3In this example 1, the TFIDF algorithm, the TextRank algorithm and the SeedTextRank_TFIDF algorithm of this application are respectively used to extract keywords from the input sample.

[0130] It can be found that the SeedTextRank_TFIDF algorithm of this application recognizes more keywords "ACC (Adaptive Cruise Control)" and "LCC (Lane Centering Control)" than the TFIDF algorithm and TextRank algorithm, and "ACC" and "LCC" are both seed words, that is, important words (keywords) in the input sample. The Precsion (precision) of the SeedTextRank_TFIDF algorithm of this application is 0.2; Recall (recall rate) is 0.57; F1 (F1-Measure, a comprehensive evaluation index) is 0.30, which are higher than the TFIDF algorithm and TextRank algorithm respectively. Precsion (precision rate) indicates how many of the samples predicted as positive are truly positive samples. It refers to the prediction results and is also called the precision rate. Recall (recall rate) indicates how many positive examples in the sample are predicted correctly. It refers to the original sample and is also called the recall rate. Therefore, the present application adopts the SeedTextRank_TFIDF algorithm to extract keywords, improve the precision and recall rate of keyword extraction, and make the extracted keywords better reflect the meaning of the text content.

[0131] For example, Figure 4 Schematic diagram of the experimental results of Application Example 2 of the keyword extraction method shown in the embodiment of the present application. Figure 4 In Example 2, the TFIDF algorithm, the TextRank algorithm and the SeedTextRank_TFIDF algorithm of this application are used to extract keywords from the input sample.

[0132] It can be found that the SeedTextRank_TFIDF algorithm of this application recognizes more keywords "Xi'an", "activity" and "test drive" than the TFIDF algorithm and TextRank algorithm, and "activity" and "test drive" are both seed words, and "Xi'an" is located at the beginning of the sentence and the title, that is, these three words are important words in the input sample. The Precsion of the SeedTextRank_TFIDF algorithm of this application is 0.5; Recall is 0.59; F1 is 0.54, which are higher than the TFIDF algorithm and TextRank algorithm respectively. Therefore, this application adopts the SeedTextRank_TFIDF algorithm to extract keywords, improve the precision and recall rate of keyword extraction, so that the extracted keywords can better reflect the meaning of the text content.

[0133] For example, Figure 5 Schematic diagram of the experimental results of Application Example 3 in the keyword extraction method shown in the embodiment of the present application. Figure 5 In Example 3, the TFIDF algorithm, the TextRank algorithm and the SeedTextRank_TFIDF algorithm of this application are respectively used to extract keywords from the input sample.

[0134] It can be found that the SeedTextRank_TFIDF algorithm of the present application recognizes more keywords "winter", "car use" and "experience" than the TFIDF algorithm and the TextRank algorithm, and "winter", "car use" and "experience" are all located in the title, that is, these three words are important words in the input sample. The Precsion of the SeedTextRank_TFIDF algorithm of the present application is 0.35; Recall is 0.7; F1 is 0.47, which are higher than the TFIDF algorithm and the TextRank algorithm respectively. Therefore, the present application adopts the SeedTextRank_TFIDF algorithm to extract keywords, improve the precision and recall rate of keyword extraction, so that the extracted keywords can better reflect the meaning of the text content.

[0135] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a keyword extraction device, an electronic device and corresponding embodiments.

[0136] Figure 6 It is a structural schematic diagram of a keyword extraction device shown in an embodiment of the present application.

[0137] See also Figure 6 A keyword extraction device 60 includes: an acquisition module 61, a first operation module 62, a second operation module 63, a third operation module 64, and an extraction module 65.

[0138] The acquisition module 61 is used to acquire input samples.

[0139] The first operation module 62 is used to obtain the first score of each candidate word in the input sample according to a first preset algorithm. The first preset algorithm can be a SeedTextRank algorithm, wherein the SeedTextRank algorithm includes weighting each candidate word, and the weighting includes keyword weighting, position weighting, and edge weighting.

[0140] The second operation module 63 is used to obtain a second score of each candidate word in the input sample according to a second preset algorithm. The second preset algorithm may be a TFIDF algorithm.

[0141] The third operation module 64 is used to perform a set operation on the first score of each candidate word obtained by the first operation module 62 and the second score of each candidate word obtained by the second operation module 63 to obtain a third score of each candidate word. The third operation module 64 can perform a weighted sum operation on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word.

[0142] The extraction module 65 is used to determine the keyword of the input sample according to the ranking result of the third score of each candidate word obtained by the third operation module 64. The extraction module 65 can determine the keyword of the input sample according to the ranking result of the third score of each candidate word and the preset invalid word library.

[0143] It can be seen from this embodiment that the technical solution of the present application obtains the first score of each candidate word in the input sample according to the first preset algorithm and obtains the second score of each candidate word in the input sample according to the second preset algorithm; then the first score of each candidate word and the second score of each candidate word are set and operated to obtain the third score of each candidate word; finally, the keywords of the input sample are determined according to the sorting results of the third scores of each candidate word. Since the solution of the present application integrates the results of the two preset algorithms for comprehensive consideration and finally determines the final score of the candidate word, the score used for sorting reference can more accurately reflect the importance and relevance of the candidate word in the input sample, thereby improving the precision and recall rate of keyword extraction, so that the extracted keywords can better reflect the meaning of the text content.

[0144] Figure 7 is another structural diagram of a keyword extraction device shown in an embodiment of the present application. Figure 7 A keyword extraction device 60 includes: an acquisition module 61, a first operation module 62, a second operation module 63, a third operation module 64, an extraction module 65, a vocabulary module 66, and a preprocessing module 67.

[0145] The acquisition module 61, the first operation module 62, the second operation module 63, the third operation module 64, and the extraction module 65 can be found in Figure 6 Description in .

[0146] The word library module 66 is used to set a preset keyword library, wherein the preset keyword library includes core words and corresponding related words;

[0147] The first operation module 62 obtains a first score of each candidate word in the input sample according to the preset keyword library of the word library module 66 and the first preset algorithm;

[0148] The second operation module 63 obtains a second score for each candidate word in the input sample according to the set corpus corresponding to the preset keyword library of the vocabulary module 66 and the second preset algorithm.

[0149] Among them, the core words and corresponding related words in the preset keyword library can be determined in advance in the following way: select words with top scores from the set corpus; select words related to the set field from the words with top scores as core words according to the set field; search for synonyms of the core words, and perform part of speech filtering and length filtering to select a set number of words related to the core words as corresponding related words.

[0150] First, the word library module 66 can use the TFIDF algorithm to calculate the words with higher scores in the first set corpus (such as the corpus of a company or industry), such as the top 100 words, and mark these 100 words, and select several representative words in the field of new energy vehicles, such as "autonomous driving", "NEDC", "NGP" and other words in the field of new energy vehicle content. The words selected in this step can be called core words; second, use the second set corpus (such as Wikipedia corpus and the corpus of this application field) to jointly train the word2vec open source word vector model, use the word2vec model to search for synonyms of the selected core words, and perform part of speech filtering and length filtering at the same time, and select several representative words in the field of new energy vehicles as corresponding related words. The newly constructed keyword library (or keyword dictionary) contains core words and a set number of related words related to each core word, such as 5 related words.

[0151] In the present application, the word library module 66 can also construct an invalid word library (or invalid word dictionary). The DF (Document Frequency) features of all words in the set corpus are counted, and words with DF features greater than or equal to the set number are selected as meaningless invalid words. For example, the DF features of all words in the application field corpus are counted, and the meaningless words with DF features greater than or equal to 1000 are selected as invalid words and added to the invalid word library.

[0152] The preprocessing module 67 is used to preprocess the input sample; the acquisition module 61 acquires the text content input sample preprocessed by the preprocessing module 67.

[0153] The preprocessing process of the preprocessing module 67 may include:

[0154] 1) Clean the text and remove HTML (Hyper Text Markup Language) tags, URLs (uniform resource locators), punctuation marks, etc.; 2) Use Jieba word segmentation to segment the text; 3) Use a common stop dictionary to remove stop words; 4) Perform part-of-speech filtering: Use Jieba word segmentation to obtain the part-of-speech, and retain the following parts of speech: nouns, common verbs, gerunds, adjectives, nouns, proper nouns, names of people, names of organizations, place names, time, etc.; 5) Perform synonym replacement and use a custom synonym library to replace the corresponding words with common words.

[0155] It can be found that the technical solution of the present application can be to obtain the first score of each candidate word in the input sample according to the preset keyword library and the first preset algorithm, obtain the second score of each candidate word in the input sample according to the set corpus corresponding to the preset keyword library and the second preset algorithm, and determine the keyword of the input sample according to the sorting result of the third score of each candidate word and the preset invalid word library. In other words, by using the preset keyword library and the preset invalid word library in the relevant field as the knowledge content input of the algorithm, it is more effective to extract keywords from the text content of the special field, and compared with the general algorithm, the keyword extraction is more accurate and the extraction efficiency is higher.

[0156] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0157] Figure 8 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0158] See also Figure 8 , the electronic device 1000 includes a memory 1010 and a processor 1020 .

[0159] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0160] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0161] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0162] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0163] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0164] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A keyword extraction method, characterized in that: include: Get input samples; A first score for each candidate word in the input sample is obtained according to a preset keyword library and a first preset algorithm; wherein the preset keyword library includes core words and corresponding related words, and the first preset algorithm is a SeedTextRank algorithm, which is an algorithm for integrating domain core words improved on the basis of the TextRank algorithm, and is used to perform weighted processing on each candidate word, and the weighted processing includes keyword weighting, position weighting, and edge weighting; wherein the keyword weighting is used to perform weighting according to whether the candidate word appears in the preset keyword library, the position weighting is used to perform weighting according to the position of the candidate word in the input sample, and the edge weighting is used to perform weighting according to the adjacent relationship between the candidate word and the neighboring word; Obtaining a second score for each candidate word in the input sample according to a second preset algorithm; Performing a setting operation on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word; The keyword of the input sample is determined according to the ranking result of the third scores of the candidate words.

2. The method according to claim 1, characterized in that: The method of obtaining the second score of each candidate word in the input sample according to the second preset algorithm includes: obtaining the second score of each candidate word in the input sample according to a set corpus corresponding to a preset keyword library and the second preset algorithm.

3. The method according to claim 2, characterized in that The core words and corresponding related words in the preset keyword library are pre-determined in the following manner: Filter the words with the highest scores from the set corpus; According to the set field, select words related to the set field from the words with the highest scores as core words; The core word is searched for synonyms, and part-of-speech filtering and length filtering are performed to select a set number of words related to the core word as corresponding related words.

4. The method according to claim 1, characterized in that The step of determining the keyword of the input sample according to the ranking result of the third scores of the candidate words includes: The keyword of the input sample is determined according to the ranking result of the third score of each candidate word and a preset invalid word library.

5. The method according to claim 1, characterized in that The step of performing a setting operation on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word includes: A weighted sum operation is performed on the first score of each candidate word and the second score of each candidate word to obtain a third score of each candidate word.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining of input samples comprises: Get a sample of preprocessed text content input.

7. The method according to any one of claims 1 to 5, characterized in that: The second preset algorithm is the TFIDF algorithm.

8. A keyword extraction device, characterized in that: include, An acquisition module, used to obtain input samples; The word library module is used to set a preset keyword library, wherein the preset keyword library includes core words and corresponding related words; A first operation module, used for obtaining a first score of each candidate word in the input sample according to a preset keyword library and a first preset algorithm; the first preset algorithm is a SeedTextRank algorithm, which is an algorithm integrating domain core words improved on the basis of the TextRank algorithm, and is used for weighting each candidate word, wherein the weighting includes keyword weighting, position weighting, and edge weighting; wherein the keyword weighting is used for weighting according to whether the candidate word appears in the preset keyword library, the position weighting is used for weighting according to the position of the candidate word in the input sample, and the edge weighting is used for weighting according to the adjacent relationship between the candidate word and the neighboring word; A second operation module, used for obtaining a second score of each candidate word in the input sample according to a second preset algorithm; A third operation module, used for performing a set operation on the first score of each candidate word obtained by the first operation module and the second score of each candidate word obtained by the second operation module to obtain a third score of each candidate word; The extraction module is used to determine the keyword of the input sample according to the ranking result of the third score of each candidate word obtained by the third operation module.

9. The device according to claim 8, characterized in that The device also includes: The second operation module obtains a second score for each candidate word in the input sample according to a set corpus corresponding to a preset keyword library of the vocabulary module and a second preset algorithm.

10. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having executable codes stored thereon, and when the executable codes are executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1 to 7.

Images