Method, device, equipment and storage medium for extracting short text keywords

Abstract

The present application relates to the field of artificial intelligence, and discloses a method, device, equipment and storage medium for extracting short text keywords, so as to improve the accuracy of short text keyword extraction. The present application method includes: obtaining candidate keywords from a short text to be processed; obtaining target annotation information with continuous sequence numbers in the candidate keywords and target candidate keywords corresponding to the target annotation information to obtain information to be processed; creating N-gram grammatical information according to the information to be processed, and annotating the N-gram grammatical information to the target candidate keywords to obtain annotated keywords; merging the annotated keywords to obtain candidate phrases; analyzing the candidate phrases according to preset phrase part-of-speech splicing rules to obtain target candidate phrases; when it is analyzed that the target candidate phrases have a substring relationship, deleting the target candidate phrases corresponding to the substring relationship to obtain a modified candidate phrase; splicing the modified candidate phrases with cross characters to obtain the target keyword.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a method, device, equipment and storage medium for extracting keywords from short texts. Background Art

[0002] Keyword extraction from short texts is an important research direction in text mining, and plays an important role in automatic question answering, topic tracking, information retrieval, and product recommendation. However, only a small number of the massive Internet documents are annotated with keywords, so how to label short texts with accurate keywords has become an important issue in the field of information extraction.

[0003] The existing short text keyword extraction is to segment and tag the short text, use the term frequency-inverse document frequency (TF-IDF) algorithm or the graph-based sorting algorithm TextRank algorithm or the topic model (latent dirichlet allocation, LDA) to obtain the candidate keywords of the short text, and then recombine or sort the candidate keywords to obtain the final keywords.

[0004] Since the TF-IDF algorithm is limited to the case of repeated word occurrence, it is more suitable for long texts than for short texts. The TextRank algorithm is limited to using graph nodes as candidate words and graph edges as co-occurrence relationships between words, so it is more suitable for long texts than for short texts. The LDA topic model can achieve relatively good results when mining long text topics represented by traditional news documents, but it is not suitable for short texts. The supervised classification model represented by decision trees and neural networks requires a large number of labeled samples for keyword extraction from short texts. In addition, due to the small number of words, sparse features and monotonous word information in short texts, the current unsupervised keyword extraction technology based on deep learning cannot be well applied to short texts. Therefore, the accuracy of keyword extraction from short texts is low. Summary of the invention

[0005] The present application provides a method, apparatus, device and storage medium for extracting keywords from short texts, which are used to extract keywords from short texts by combining grammatical structure and part-of-speech splicing filtering rules, thereby improving the accuracy of short text keyword extraction.

[0006] A first aspect of an embodiment of the present application provides a method for extracting keywords from a short text, comprising:

[0007] Perform keyword pre-extraction on the short text to be processed to obtain candidate keywords;

[0008] Obtain target annotation information of the candidate keyword and the target candidate keyword corresponding to the target annotation information to obtain information to be processed, wherein the target annotation information is annotation information with a continuous sequence number;

[0009] Creating N-gram information according to the information to be processed, and marking the N-gram information to the target candidate keyword to obtain a marked keyword;

[0010] Merge the annotated keywords according to the target annotated information and the N-gram information to obtain candidate phrases;

[0011] Analyze the candidate phrases according to preset phrase part-of-speech splicing rules to obtain target candidate phrases;

[0012] When it is analyzed that the target candidate phrase has a substring relationship, the target candidate phrase corresponding to the substring relationship is deleted to obtain a modified candidate phrase, the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase;

[0013] The modified candidate phrases with cross characters are concatenated to obtain target keywords, wherein the cross characters indicate that the end word of one modified candidate phrase is the same as the beginning word of the other modified candidate phrase in the two modified candidate phrases.

[0014] Optionally, in a first implementation manner of the first aspect of the embodiment of the present application, the step of merging the annotation keywords according to the target annotation information and the N-gram information to obtain a candidate phrase includes:

[0015] Obtaining the part of speech of the marked keyword, and filtering the marked keyword according to a preset filtering rule and the part of speech to obtain a target marked keyword;

[0016] Comparing the values ​​of the N-gram information of the target marked keyword, and obtaining the target word corresponding to the N-gram information with the largest value;

[0017] According to the target sequence number in the target annotation information, obtain the keywords in the target annotation keywords that are sorted before the target word to obtain the keywords to be merged;

[0018] The keyword to be merged and the target word are merged according to the target sequence number to obtain a candidate phrase.

[0019] Optionally, in a second implementation of the first aspect of the embodiment of the present application, analyzing the candidate phrases according to a preset phrase part-of-speech concatenation rule to obtain a target candidate phrase includes:

[0020] By analyzing whether the part of speech of the candidate phrase has at least one of the preset parts of speech, a first phrase to be analyzed and a second phrase to be analyzed are obtained, wherein the preset parts of speech include verbs, gerunds and morphemes, the first phrase to be analyzed is a candidate phrase whose part of speech has at least one of the preset parts of speech, and the second phrase to be analyzed is a candidate phrase whose part of speech is not any of the preset parts of speech;

[0021] Analyze the first phrase to be analyzed according to the part of speech after the preset part of speech is connected and the first preset rule to obtain a target candidate phrase;

[0022] The second phrase to be analyzed is analyzed according to a second preset rule and a target part of speech to obtain a target candidate phrase, wherein the target part of speech is the part of speech of each part of speech of the second phrase to be analyzed and the part of speech after the parts of speech are connected.

[0023] Optionally, in a third implementation manner of the first aspect of the embodiment of the present application, when it is analyzed that the target candidate phrase has a substring relationship, the target candidate phrase corresponding to the substring relationship is deleted to obtain a modified candidate phrase, the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase, including:

[0024] Classifying the target candidate phrases to obtain main target candidate phrases and / or sub-target candidate phrases, wherein the sub-target candidate phrases are composed of words in the main target candidate phrases;

[0025] Taking the main target candidate phrase as a parent node and the length of the sub-target candidate phrase as a branch condition;

[0026] Obtaining a substring relationship recognition tree according to the main target candidate phrase, the subtarget candidate phrase and the branch condition;

[0027] Retrieving the position of the target candidate phrase in the substring relationship recognition tree;

[0028] If the target candidate phrase is found to be located at a parent node in the substring relationship identification tree, then deleting the target candidate phrases corresponding to all child nodes in the substring relationship identification tree to obtain a revised candidate phrase;

[0029] If the target candidate phrase is retrieved and located at a candidate subnode in the substring relationship recognition tree, the target candidate phrase corresponding to the target subnode is deleted to obtain a revised candidate phrase, wherein the target subnode is a subnode after the candidate subnode.

[0030] Optionally, in a fourth implementation of the first aspect of the embodiment of the present application, the modified candidate phrases with crossed characters are concatenated to obtain a target keyword, wherein the crossed characters indicate that the end word of one modified candidate phrase is the same as the start word of the other modified candidate phrase in the two modified candidate phrases, including:

[0031] When it is recognized that the terminal word of one of the two modified candidate phrases is the same as the initial word of the other modified candidate phrase, the modified candidate phrases corresponding to the terminal word and the initial word are concatenated to obtain an initial keyword;

[0032] Calculating the term frequency-inverse text frequency index TF-IDF value of the initial keyword, and determining the candidate keyword according to the TF-IDF value;

[0033] The candidate keywords are screened according to the preset phrase part-of-speech concatenation rule to obtain target keywords.

[0034] Optionally, in a fifth implementation of the first aspect of the embodiment of the present application, calculating the term frequency-inverse text frequency index TF-IDF value of the initial keyword and determining the candidate keyword according to the TF-IDF value includes:

[0035] Obtaining a dictionary and identifying keywords in the field to which the short text to be processed belongs, and obtaining reference data;

[0036] Calculate the term frequency-inverse text frequency index TF-IDF value of the initial keyword in the reference data to obtain a value to be calculated;

[0037] Calculate the weighted average of the values ​​to be calculated according to the preset weight values ​​to obtain a preset threshold value;

[0038] The difference between the value to be calculated and the preset threshold is calculated, and when the difference is greater than or equal to 0, the corresponding initial keyword is used as a candidate keyword.

[0039] Optionally, in a sixth implementation of the first aspect of the embodiment of the present application, performing keyword pre-extraction processing on the short text to be processed to obtain candidate keywords includes:

[0040] The initial short text to be processed, which has been processed by removing spaces, is segmented by a word segmentation tool to obtain a plurality of initial words;

[0041] Calling a preset part-of-speech tagging model to perform part-of-speech tagging on the multiple initial words to obtain a short text to be processed;

[0042] Identify whether the short text to be processed has a part of speech vacancy;

[0043] When it is identified that the short text to be processed does not have a part-of-speech vacancy, filtering the short text to be processed to obtain candidate keywords, wherein the filtering process includes reserved word filtering process, stop word filtering process and part-of-speech filtering process;

[0044] When it is identified that the short text to be processed has a part-of-speech vacancy, the short text to be processed with the part-of-speech vacancy is subjected to a part-of-speech filling process to obtain a filling word, and the filling word is subjected to the filtering process to obtain a candidate keyword.

[0045] The second aspect of the embodiment of the present application provides a device for extracting short text keywords, which has the function of implementing the method for extracting short text keywords provided in the first aspect above. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the unit can be software and / or hardware.

[0046] The device for extracting short text keywords comprises:

[0047] An extraction unit, used for performing keyword pre-extraction processing on the short text to be processed to obtain candidate keywords;

[0048] An acquisition unit, configured to acquire target annotation information of the candidate keyword and a target candidate keyword corresponding to the target annotation information, to obtain information to be processed, wherein the target annotation information is annotation information with a continuous sequence number;

[0049] A creating unit, configured to create N-gram information according to the information to be processed, and annotate the N-gram information to the target candidate keyword to obtain an annotated keyword;

[0050] A merging unit, configured to merge the marked keywords according to the target marked information and the N-gram information to obtain candidate phrases;

[0051] An analysis unit, configured to analyze the candidate phrases according to a preset phrase part-of-speech concatenation rule to obtain a target candidate phrase;

[0052] a deleting unit, configured to, when analyzing that the target candidate phrase has a substring relationship, delete the target candidate phrase corresponding to the substring relationship to obtain a modified candidate phrase, wherein the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase;

[0053] The concatenation processing unit is used to concatenate the modified candidate phrases with cross characters to obtain a target keyword, wherein the cross characters indicate that the end word of one modified candidate phrase is the same as the beginning word of the other modified candidate phrase in the two modified candidate phrases.

[0054] Optionally, in a first implementation of the second aspect of the embodiment of the present application, the merging unit is specifically used to:

[0055] Obtaining the part of speech of the marked keyword, and filtering the marked keyword according to a preset filtering rule and the part of speech to obtain a target marked keyword;

[0056] Comparing the values ​​of the N-gram information of the target marked keyword, and obtaining the target word corresponding to the N-gram information with the largest value;

[0057] According to the target sequence number in the target annotation information, obtain the keywords in the target annotation keywords that are sorted before the target word to obtain the keywords to be merged;

[0058] The keyword to be merged and the target word are merged according to the target sequence number to obtain a candidate phrase.

[0059] Optionally, in a second implementation of the second aspect of the embodiment of the present application, the analysis unit is specifically used to:

[0060] By analyzing whether the part of speech of the candidate phrase has at least one of the preset parts of speech, a first phrase to be analyzed and a second phrase to be analyzed are obtained, wherein the preset parts of speech include verbs, gerunds and morphemes, the first phrase to be analyzed is a candidate phrase whose part of speech has at least one of the preset parts of speech, and the second phrase to be analyzed is a candidate phrase whose part of speech is not any of the preset parts of speech;

[0061] Analyze the first phrase to be analyzed according to the part of speech after the preset part of speech is connected and the first preset rule to obtain a target candidate phrase;

[0062] The second phrase to be analyzed is analyzed according to a second preset rule and a target part of speech to obtain a target candidate phrase, wherein the target part of speech is the part of speech of each part of speech of the second phrase to be analyzed and the part of speech after the parts of speech are connected.

[0063] Optionally, in a third implementation of the second aspect of the embodiment of the present application, the deleting unit is specifically configured to:

[0064] Classifying the target candidate phrases to obtain main target candidate phrases and / or sub-target candidate phrases, wherein the sub-target candidate phrases are composed of words in the main target candidate phrases;

[0065] Taking the main target candidate phrase as a parent node and the length of the sub-target candidate phrase as a branch condition;

[0066] Obtaining a substring relationship recognition tree according to the initial target candidate phrase, the subtarget candidate phrase and the branch condition;

[0067] Retrieving the position of the target candidate phrase in the substring relationship recognition tree;

[0068] If the target candidate phrase is found to be located at a parent node in the substring relationship identification tree, then deleting the target candidate phrases corresponding to all child nodes in the substring relationship identification tree to obtain a revised candidate phrase;

[0069] If the target candidate phrase is retrieved and located at a candidate subnode in the substring relationship recognition tree, the target candidate phrase corresponding to the target subnode is deleted to obtain a revised candidate phrase, wherein the target subnode is a subnode after the candidate subnode.

[0070] Optionally, in a fourth implementation of the second aspect of the embodiment of the present application, the splicing processing unit further includes:

[0071] The recognition subunit is used for, when recognizing that the terminal word of one of the two modified candidate phrases is the same as the initial word of the other modified candidate phrase, concatenating the modified candidate phrases corresponding to the terminal word and the initial word to obtain an initial keyword;

[0072] A calculation subunit, used to calculate the term frequency-inverse text frequency index TF-IDF value of the initial keyword, and determine the candidate keyword according to the TF-IDF value;

[0073] The screening subunit is used to screen the candidate keywords according to the preset phrase part-of-speech splicing rule to obtain the target keyword.

[0074] Optionally, in a fifth implementation of the second aspect of the embodiment of the present application, the computing subunit is specifically configured to:

[0075] Obtaining a dictionary and identifying keywords in the field to which the short text to be processed belongs, and obtaining reference data;

[0076] Calculate the term frequency-inverse text frequency index TF-IDF value of the initial keyword in the reference data to obtain a value to be calculated;

[0077] Calculate the weighted average of the values ​​to be calculated according to the preset weight values ​​to obtain a preset threshold value;

[0078] The difference between the value to be calculated and the preset threshold is calculated, and when the difference is greater than or equal to 0, the corresponding initial keyword is used as a candidate keyword.

[0079] Optionally, in a sixth implementation of the second aspect of the embodiment of the present application, the extraction unit is specifically used to:

[0080] The initial short text to be processed, which has been processed by removing spaces, is segmented by a word segmentation tool to obtain a plurality of initial words;

[0081] Calling a preset part-of-speech tagging model to perform part-of-speech tagging on the multiple initial words to obtain a short text to be processed;

[0082] Identify whether the short text to be processed has a part of speech vacancy;

[0083] When it is identified that the short text to be processed does not have a part-of-speech vacancy, filtering the short text to be processed to obtain candidate keywords, wherein the filtering process includes reserved word filtering process, stop word filtering process and part-of-speech filtering process;

[0084] When it is identified that the short text to be processed has a part-of-speech vacancy, the short text to be processed with the part-of-speech vacancy is subjected to a part-of-speech filling process to obtain a filling word, and the filling word is subjected to the filtering process to obtain a candidate keyword.

[0085] A third aspect of an embodiment of the present application provides a device for extracting short text keywords, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for extracting short text keywords described in any one of the above-mentioned embodiments when executing the computer program.

[0086] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method for extracting short text keywords described in any of the above embodiments.

[0087] Compared with the prior art, the technical solution provided in the embodiment of the present application obtains candidate keywords from the short text to be processed; obtains the target annotation information with continuous sequence numbers in the candidate keywords and the target candidate keywords corresponding to the target annotation information to obtain the information to be processed; creates N-gram information according to the information to be processed, and annotates the N-gram information to the target candidate keywords to obtain the annotated keywords; merges the annotated keywords to obtain candidate phrases; analyzes the candidate phrases according to the preset phrase part-of-speech splicing rules to obtain the target candidate phrases; when it is analyzed that the target candidate phrases have a substring relationship, the target candidate phrases corresponding to the substring relationship are deleted to obtain the modified candidate phrases; the modified candidate phrases with cross-words are spliced ​​to obtain the target keywords. The embodiment of the present application combines the grammatical structure with the part-of-speech splicing filtering rules to extract keywords from short texts, thereby improving the accuracy of keyword extraction from short texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A schematic diagram of an embodiment of a method for extracting short text keywords in an embodiment of the present application;

[0089] Figure 2 A schematic diagram of an embodiment of a preset phrase part-of-speech splicing rule in an embodiment of the present application;

[0090] Figure 3 A schematic diagram of another embodiment of the method for extracting short text keywords in an embodiment of the present application;

[0091] Figure 4 A schematic diagram of an embodiment of a device for extracting keywords from short texts in an embodiment of the present application;

[0092] Figure 5 A schematic diagram of another embodiment of the device for extracting short text keywords in an embodiment of the present application;

[0093] Figure 6 This is a schematic diagram of an embodiment of a device for extracting short text keywords in an embodiment of the present application. DETAILED DESCRIPTION

[0094] The embodiments of the present application provide a method, apparatus, device and storage medium for extracting keywords from short texts, which are used to extract keywords from short texts by combining grammatical structure and part-of-speech splicing filtering rules to improve the accuracy of short text keyword extraction.

[0095] In order to enable those skilled in the art to better understand the present application, the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0096] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device including a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of the modules that appear in the present application is only a logical division, and there may be other division methods when implemented in practical applications, for example, multiple modules can be combined into or integrated in another system, or some features can be ignored or not executed.

[0097] See also Figure 1 , a flowchart of a method for extracting short text keywords provided in an embodiment of the present application, the following is an example of a method for extracting short text keywords provided by the present application, the method is executed by a computer device, the computer device can be a server or a terminal, and the present application does not limit the type of the execution subject, specifically including:

[0098] 101. Perform keyword pre-extraction processing on the short text to be processed to obtain candidate keywords.

[0099] The server performs keyword pre-extraction processing on the short text to be processed to obtain candidate keywords, specifically including: performing word segmentation processing on the initial short text to be processed after deleting spaces through a word segmentation tool to obtain multiple initial words; calling a preset part-of-speech tagging model to perform part-of-speech tagging on the multiple initial words to obtain the short text to be processed; identifying whether there are part-of-speech vacancies in the short text to be processed; when it is identified that there are no part-of-speech vacancies in the short text to be processed, filtering the short text to be processed to obtain candidate keywords, the filtering processing including reserved word filtering processing, stop word filtering processing and part-of-speech filtering processing; when it is identified that there are part-of-speech vacancies in the short text to be processed, performing part-of-speech filling processing on the short text to be processed with part-of-speech vacancies to obtain filling words, and filtering the filling words to obtain candidate keywords.

[0100] The server analyzes the domain to which the initial short text to be processed belongs, obtains the dictionary of the domain, and performs word segmentation on the initial short text to be processed according to the dictionary of the domain through the two-way matching word segmentation algorithm and machine learning algorithm in the word segmentation tool, obtains multiple initial words, identifies the parts of speech of the multiple initial words, and calls the preset part-of-speech tagging model to tag the parts of speech on the multiple initial words to obtain the short text to be processed, wherein the word segmentation tool can be ANSJ. By combining the machine learning algorithm and the dictionary to perform word segmentation and part-of-speech tagging on the initial short text to be processed, on the one hand, the word segmentation accuracy can be improved, and on the other hand, the domain adaptability can be improved. The Jieba word segmentation algorithm can also be used to perform efficient word graph scanning on the initial short text to be processed based on the prefix dictionary, generate a directed acyclic graph consisting of all possible word formation situations of Chinese characters in the sentence, dynamically plan to find the maximum probability path, find the maximum segmentation combination based on word frequency, and for unregistered words, a hidden Markov model HMM model based on the word formation ability of Chinese characters is used, and the Viterbi algorithm is used to perform word segmentation on the short text to be processed.

[0101] The graph-based sorting algorithm TextRank algorithm or the keyword extraction algorithm (rapid automatic keyword extraction, RAKE) is used to extract keywords from the short text to be processed after word segmentation and part-of-speech tagging to obtain pre-extracted keywords, and the preset stop word list and reserved word list are called to screen the pre-extracted keywords, and the keywords in the stop word list and reserved word list are deleted from the pre-extracted keywords, and the keywords corresponding to pronouns, locative words, conjunctions, prepositions and adverbs are deleted to obtain candidate keywords. The keywords that have undergone keyword pre-extraction are filled with part-of-speech vacancies and filtered with reserved words, stop words and part-of-speech to improve the quality of keywords and improve the efficiency of subsequent processing.

[0102] 102. Obtain target annotation information of the candidate keyword and the target candidate keyword corresponding to the target annotation information to obtain information to be processed, wherein the target annotation information is annotation information with a continuous sequence number.

[0103] The server obtains target annotation information of the candidate keyword and the target candidate keyword corresponding to the target annotation information, and obtains information to be processed. The target annotation information is annotation information with a continuous sequence number.

[0104] When the server is performing part-of-speech tagging, it sorts the segmented words of the short text to be processed from left to right in a continuous order, and marks the sorted sequence number on the segmented words. Analyze whether the sequence number in the tagging information of the candidate keywords is continuous. If it is continuous, obtain the corresponding tagging information and candidate keywords. For example: the candidate keywords are based on, words, grammar, model, segmented words and methods, and the sequence numbers in the tagging information based on, words, grammar, model, segmented words and methods are 1, 2, 3, 4, 5 and 6 respectively. Then the continuous tagging information obtained is "1, 2, 3, 4 and 5" or "2, 3, 4, 5 and 6", and the corresponding candidate keywords are "based on, words, grammar, model and segmented words" or "words, grammar, model, segmented words and methods".

[0105] 103. Create N-gram information according to the information to be processed, and annotate the N-gram information to the target candidate keyword to obtain the annotated keyword.

[0106] The server creates N-gram grammatical information according to the information to be processed, and annotates the N-gram grammatical information to the target candidate keyword to obtain the annotated keyword.

[0107] The server calculates the probability of candidate keywords corresponding to the continuous annotation information: w represents the candidate keyword corresponding to the continuous annotation information, and m represents the mth candidate keyword corresponding to the continuous annotation information. The estimated n-grams of the candidate keywords corresponding to the continuous annotation information that have not appeared are smoothed by the back-off smoothing algorithm to obtain N-gram information, and the N-gram information is annotated to the candidate keywords corresponding to the continuous annotation information to obtain annotated candidate keywords, where N is 5.

[0108] 104. Combine and annotate keywords according to the target annotation information and the N-gram information to obtain candidate phrases.

[0109] The server merges the annotated keywords according to the target annotated information and the N-gram information to obtain the candidate phrases. Specifically, the process includes: obtaining the part of speech of the annotated keywords, and filtering the annotated keywords according to the preset filtering rules and the part of speech to obtain the target annotated keywords; comparing the values ​​of the N-gram information of the target annotated keywords to obtain the target words corresponding to the N-gram information with the largest value; obtaining the keywords in the target annotated keywords that are ranked before the target words according to the target sequence number in the target annotated information to obtain the keywords to be merged; merging the keywords to be merged and the target words according to the target sequence number to obtain the candidate phrases.

[0110] The server sorts and combines the candidate keywords with the largest probability value in the order of the keyword annotation sequence number according to the sequence number of the continuous annotation information and the probability value of the N-gram information to obtain candidate phrases, wherein the candidate keywords will be filtered according to the preset filtering rules before combination, and the candidate keywords that meet the preset filtering rules can be sorted and combined. On the one hand, the preset filtering rules are set to the same part of speech of the combined multiple candidate keywords. For example: the candidate keywords are "skin, mucosa, lymph nodes and syndrome", and the parts of speech of "skin, mucosa, lymph nodes and syndrome" are all noun n, which can be combined into "skin mucosal lymph node syndrome"; the candidate keywords are "sister, sit and car", and the parts of speech of "sister, sit and car" are noun n, verb v and noun n respectively, and the parts of speech are different and cannot be combined. On the other hand, the number of candidate keywords corresponding to the continuous annotation information must be greater than 2. The above examples are only for illustration, and the actual operation and accuracy of their content are not considered. Through filtering processing, candidate keywords corresponding to continuous annotation information and N-gram information with the largest value are merged to improve the accuracy of candidate phrases, reduce the number of operations for generating candidate phrases, and improve operation efficiency.

[0111] 105. Analyze the candidate phrases according to the preset phrase part-of-speech splicing rules to obtain the target candidate phrases.

[0112] The server analyzes the candidate phrases according to the preset phrase part-of-speech splicing rule to obtain the target candidate phrase, specifically including: obtaining a first phrase to be analyzed and a second phrase to be analyzed by analyzing whether the part-of-speech of the candidate phrase contains at least one of the preset parts-of-speech, wherein the preset parts-of-speech include verbs, gerunds and morphemes, the first phrase to be analyzed is a candidate phrase whose part-of-speech contains at least one of the preset parts-of-speech, and the second phrase to be analyzed is a candidate phrase whose part-of-speech is not any of the preset parts-of-speech; analyzing the first phrase to be analyzed according to the parts-of-speech after the preset parts-of-speech are connected and the first preset rule to obtain the target candidate phrase; analyzing the second phrase to be analyzed according to the second preset rule and the target part-of-speech to obtain the target candidate phrase, wherein the target part-of-speech is the part-of-speech after the parts-of-speech of the second phrase to be analyzed are connected.

[0113] The server performs multiple layers of filtering and retention processing on the candidate phrases’ parts of speech according to the preset phrase part-of-speech concatenation rules to obtain the final target candidate phrases. The validity and accuracy of the target candidate phrases are ensured by using multiple layers of preset phrase part-of-speech concatenation rules. Among them, the first layer of analysis of the preset phrase part-of-speech concatenation rules is to analyze whether there are verbs, gerunds, and grammatical morphemes in the candidate phrases’ parts of speech. For candidate phrases that do not contain verbs, gerunds, and grammatical morphemes, thirteen layers of filtering and retention processing are performed, including analyzing and judging the parts of speech after the parts of speech are connected; for candidate phrases that contain verbs, gerunds, and grammatical morphemes, twelve layers of filtering and retention processing are performed, including analyzing and judging the parts of speech of the candidate phrases and the parts of speech after the parts of speech are connected. The specific preset phrase part-of-speech concatenation rules are as follows: Figure 2 shown. Figure 2 This is only an example, and the actual operation and accuracy of its content are not considered. The candidate phrases are filtered and retained by the preset phrase part-of-speech splicing rules, and a relatively simple algorithm is used to achieve a light model, high operating efficiency and high keyword extraction accuracy.

[0114] 106. When it is analyzed that a substring relationship exists in the target candidate phrase, the target candidate phrase corresponding to the substring relationship is deleted to obtain a modified candidate phrase, where the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase.

[0115] When the server analyzes that the target candidate phrase has a substring relationship, the target candidate phrase corresponding to the substring relationship is deleted to obtain a modified candidate phrase, wherein the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase. Specifically, it includes: classifying the target candidate phrase to obtain a main target candidate phrase and / or a subtarget candidate phrase, wherein the subtarget candidate phrase is composed of words in the main target candidate phrase; taking the main target candidate phrase as a parent node and taking the length of the subtarget candidate phrase as a branch condition; obtaining a substring relationship recognition tree according to the main target candidate phrase, the subtarget candidate phrase and the branch condition; if the target candidate phrase is retrieved and located at a parent node in the substring relationship recognition tree, then deleting the target candidate phrase corresponding to all child nodes in the substring relationship recognition tree to obtain a modified candidate phrase; if the target candidate phrase is retrieved and located at a candidate child node in the substring relationship recognition tree, then deleting the target candidate phrase corresponding to the target child node to obtain a modified candidate phrase, wherein the target child node is a child node after the candidate child node.

[0116] After obtaining the target candidate phrase, the server analyzes whether the target candidate phrase has a substring relationship. If the target candidate phrase is located at a child node of the substring relationship recognition tree (i.e., there is a substring relationship), the target candidate phrase corresponding to the child node is deleted to correct the target candidate phrase to obtain a corrected candidate phrase; if the target candidate phrase is located at a parent node of the substring relationship recognition tree (i.e., there is no substring relationship), all the target candidate phrases are directly used as corrected candidate phrases. For example: the string of the main target candidate phrase is ldfujioengkleim, and the strings of the subtarget candidate phrases that have a substring relationship with the main target candidate phrase are engkle, ldfujioengkleim, and dfujioen. With ldfujioengkleim as the parent node, the length of engkle is 6, the length of ldfujioengkleim is 15, and the length of dfujioen is 8, then 6, 15, and 8 are used as branch conditions, and engkle, ldfujioengkleim, and dfujioen are child nodes to obtain a substring relationship recognition tree. The above examples are only for illustration, and the actual operation and accuracy of their contents are not considered. The substring relationship recognition tree is used to identify whether a substring relationship exists, and the target candidate phrase corresponding to the substring relationship is deleted to improve processing efficiency and ensure the accuracy of the modified candidate phrase.

[0117] 107. Concatenate the modified candidate phrases with crossover characters to obtain target keywords, wherein the crossover characters indicate that the end word of one modified candidate phrase is the same as the beginning word of the other modified candidate phrase in two modified candidate phrases.

[0118] The server concatenates the modified candidate phrases with crossover characters to obtain the target keyword, wherein the crossover character indicates that the end word of one modified candidate phrase is the same as the beginning word of the other modified candidate phrase in two modified candidate phrases.

[0119] The server analyzes whether there are any correction candidate phrases with crossover characters in the correction candidate phrases. If there are any crossover characters, the server recombines the two correction candidate phrases with crossover characters according to the information marked on the correction candidate phrases to obtain a more accurate target phrase (i.e., target keyword). By splicing the correction candidate phrases with crossover characters, the accuracy of keyword extraction of the short text is improved.

[0120] In the embodiment of the present application, the accuracy of keyword extraction from short texts is improved by combining grammatical structure with part-of-speech splicing filtering rules.

[0121] See also Figure 3Another embodiment of the method for extracting short text keywords in the embodiment of the present application includes:

[0122] 301. Perform keyword pre-extraction processing on the short text to be processed to obtain candidate keywords.

[0123] 302. Obtain target annotation information of the candidate keyword and the target candidate keyword corresponding to the target annotation information to obtain information to be processed, wherein the target annotation information is annotation information with a continuous sequence number.

[0124] 303. Create N-gram information according to the information to be processed, and annotate the N-gram information to the target candidate keyword to obtain an annotated keyword.

[0125] 304. Combine and annotate keywords according to the target annotation information and the N-gram information to obtain candidate phrases.

[0126] 305. Analyze the candidate phrases according to the preset phrase part-of-speech splicing rule to obtain the target candidate phrases.

[0127] 306. When it is analyzed that a substring relationship exists in the target candidate phrase, the target candidate phrase corresponding to the substring relationship is deleted to obtain a modified candidate phrase, where the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase.

[0128] In the embodiment of the present application, the methods of 301 to 306 can refer to 101 to 106 and will not be repeated here.

[0129] 307. When it is identified that the terminal word of one of the two modified candidate phrases is the same as the initial word of the other modified candidate phrase, the modified candidate phrases corresponding to the terminal word and the initial word are concatenated to obtain an initial keyword.

[0130] When the server recognizes that the terminal word of one of the two modified candidate phrases is the same as the initial word of the other modified candidate phrase, the modified candidate phrases corresponding to the terminal word and the initial word are concatenated to obtain the initial keyword.

[0131] When the server recognizes that the end word of the first modified candidate phrase is the same as the beginning word of the second modified candidate phrase, it means that there are cross words between the first modified candidate phrase and the second modified candidate phrase, and the first modified candidate phrase and the second modified candidate phrase are concatenated to obtain the initial keyword, and the cross word is used to indicate that the end word of the first modified candidate phrase in the modified candidate phrase is the same as the beginning word of the second modified candidate phrase in the modified candidate phrase. For example: the first modified candidate phrase is "congenital heart" and the second modified candidate phrase is "heart disease", then there are cross words between the first modified candidate phrase and the second modified candidate phrase, and the cross word is "heart", and the first modified candidate phrase and the second modified candidate phrase are concatenated to obtain "congenital heart disease", and "congenital heart disease" is the initial keyword.

[0132] 308. Calculate the term frequency-inverse text frequency index TF-IDF value of the initial keyword, and determine the candidate keyword according to the TF-IDF value.

[0133] The server calculates the word frequency-inverse text frequency index TF-IDF value of the initial keyword, and determines the candidate keyword according to the TF-IDF value, specifically including: obtaining the dictionary and recognition keywords in the field to which the short text to be processed belongs, and obtaining reference data; calculating the word frequency-inverse text frequency index TF-IDF value of the initial keyword in the reference data, and obtaining the value to be calculated; calculating the weighted average of the value to be calculated according to the preset weight value, and obtaining the preset threshold; calculating the difference between the value to be calculated and the preset threshold, and when the difference is greater than or equal to 0, taking the corresponding initial keyword as the candidate keyword.

[0134] After the server obtains the initial keyword, there may be multiple initial keywords. In order to obtain more accurate short text keywords that are more suitable for the corresponding field and more in line with the user's search intention, the word frequency-inverse text frequency index TF-IDF value (i.e., the value to be calculated) of the initial keyword is calculated, and a preset threshold is set according to the TF-IDF value, and a judgment is made as to whether the difference between the TF-IDF value and the preset threshold is greater than or equal to 0 to determine the candidate keyword. For example: there are three initial keywords, namely A, B and C. The TF-IDF value corresponding to A (i.e., the value to be calculated) is 0.74, the TF-IDF value corresponding to B (i.e., the value to be calculated) is 0.80, and the TF-IDF value corresponding to C (i.e., the value to be calculated) is 0.95. The preset threshold is 0.75. The difference between A's value to be calculated 0.74 and the preset threshold 0.75 is -0.01, the difference between B's value to be calculated 0.80 and the preset threshold 0.75 is 0.05, and the difference between C's value to be calculated 0.95 and the preset threshold 0.75 is 0.20. Then the candidate keywords are B and C.

[0135] 309. The candidate keywords are screened according to the preset phrase part-of-speech concatenation rule to obtain the target keyword.

[0136] The server screens the candidate keywords according to the preset phrase part-of-speech splicing rules to obtain the target keywords.

[0137] After the server obtains the candidate keywords, in order to obtain more accurate keywords that meet the grammatical structure and part-of-speech splicing filtering rules among the candidate keywords, the server uses the following method to obtain the candidate keywords: Figure 2 The preset phrase part-of-speech splicing rules are used to perform a series of judgments and screening on the candidate keywords to obtain the unregistered words in the corresponding fields of the candidate keywords, that is, the final target keywords. By performing a series of screening processes on the candidate keywords, the unregistered words in the exclusive field are output, so as to improve the accuracy of keyword extraction for short texts.

[0138] The method for extracting short text keywords in the embodiment of the present application plays an important role in the fields of automatic question answering, topic tracking, information retrieval and product recommendation. For example, in the field of information retrieval, it is necessary to extract keywords from the input retrieval information to obtain keywords, and then use the keywords to retrieve the data to be retrieved to obtain the target retrieval data. In the keyword extraction from the input retrieval information to obtain keywords, the grammatical structure, part-of-speech splicing filtering rules and substring relationships in the embodiment of the present application can be combined with the grammatical structure, part-of-speech splicing filtering rules and substring relationships of the input short text retrieval information to extract keywords from the short text to obtain the corrected candidate phrases of the retrieval information, and the corrected candidate phrases are spliced, TF-IDF value calculated and screened according to the preset phrase part-of-speech splicing rules to obtain the target keywords. The method for extracting short text keywords in the embodiment of the present application can combine words and characters in the retrieval information of the short text without relying on an external dictionary, and accurately output the unregistered words in the exclusive field corresponding to the retrieval information, so as to give the retrieval information of the short text a keyword tag with accurate meaning, thereby improving the accuracy of keyword extraction of the short text, thereby improving the retrieval efficiency and retrieval accuracy of the retrieval information in the field of information retrieval.

[0139] The above describes the method for extracting short text keywords in the embodiment of the present application. The following describes the device for extracting short text keywords in the embodiment of the present application. Figure 4 , an embodiment of the device for extracting short text keywords in the embodiment of the present application includes:

[0140] An extraction unit 401 is used to perform keyword pre-extraction processing on the short text to be processed to obtain candidate keywords;

[0141] An acquisition unit 402 is used to acquire target annotation information of a candidate keyword and a target candidate keyword corresponding to the target annotation information to obtain information to be processed, wherein the target annotation information is annotation information with a continuous sequence number;

[0142] A creating unit 403 is used to create N-gram information according to the information to be processed, and annotate the N-gram information to the target candidate keyword to obtain an annotated keyword;

[0143] A merging unit 404 is used to merge the annotated keywords according to the target annotated information and the N-gram information to obtain candidate phrases;

[0144] An analysis unit 405 is used to analyze the candidate phrases according to a preset phrase part-of-speech splicing rule to obtain a target candidate phrase;

[0145] A deleting unit 406 is used to delete the target candidate phrase corresponding to the substring relationship when analyzing that the target candidate phrase has a substring relationship, so as to obtain a modified candidate phrase, wherein the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase;

[0146] The concatenation processing unit 407 is used to concatenate the modified candidate phrases with cross characters to obtain the target keyword, wherein the cross character indicates that the end word of one modified candidate phrase is the same as the beginning word of the other modified candidate phrase in two modified candidate phrases.

[0147] In the embodiment of the present application, the accuracy of keyword extraction from short texts is improved by combining grammatical structure with part-of-speech splicing filtering rules.

[0148] See also Figure 5 Another embodiment of the device for extracting short text keywords in the embodiment of the present application includes:

[0149] An extraction unit 401 is used to perform keyword pre-extraction processing on the short text to be processed to obtain candidate keywords;

[0150] An acquisition unit 402 is used to acquire target annotation information of a candidate keyword and a target candidate keyword corresponding to the target annotation information to obtain information to be processed, wherein the target annotation information is annotation information with a continuous sequence number;

[0151] A creating unit 403 is used to create N-gram information according to the information to be processed, and annotate the N-gram information to the target candidate keyword to obtain an annotated keyword;

[0152] A merging unit 404 is used to merge the annotated keywords according to the target annotated information and the N-gram information to obtain candidate phrases;

[0153] An analysis unit 405 is used to analyze the candidate phrases according to a preset phrase part-of-speech splicing rule to obtain a target candidate phrase;

[0154] A deleting unit 406 is used to delete the target candidate phrase corresponding to the substring relationship when analyzing that the target candidate phrase has a substring relationship, so as to obtain a modified candidate phrase, wherein the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase;

[0155] The identification subunit 4071 is used for, when it is identified that the terminal word of one of the two modified candidate phrases is the same as the initial word of the other modified candidate phrase, concatenating the modified candidate phrases corresponding to the terminal word and the initial word to obtain an initial keyword;

[0156] A calculation subunit 4072 is used to calculate the term frequency-inverse text frequency index TF-IDF value of the initial keyword, and determine the candidate keyword according to the TF-IDF value;

[0157] The screening subunit 4073 is used to screen the candidate keywords according to the preset phrase part-of-speech splicing rules to obtain the target keywords.

[0158] Optionally, the extraction unit 401 is specifically used to: perform word segmentation processing on the initial short text to be processed after deleting spaces through a word segmentation tool to obtain multiple initial words; call a preset part-of-speech tagging model to perform part-of-speech tagging on the multiple initial words to obtain the short text to be processed; identify whether there are part-of-speech vacancies in the short text to be processed; when it is identified that there are no part-of-speech vacancies in the short text to be processed, filter the short text to be processed to obtain candidate keywords, and the filtering processing includes reserved word filtering processing, stop word filtering processing and part-of-speech filtering processing; when it is identified that there are part-of-speech vacancies in the short text to be processed, perform part-of-speech filling processing on the short text to be processed with part-of-speech vacancies to obtain filler words, and filter the filler words to obtain candidate keywords.

[0159] Optionally, the merging unit 404 is specifically used to: obtain the part of speech of the marked keyword, and filter the marked keyword according to preset filtering rules and part of speech to obtain a target marked keyword; compare the values ​​of the N-grammatical information of the target marked keyword to obtain the target word corresponding to the N-grammatical information with the largest value; according to the target sequence number in the target marking information, obtain the keyword in the target marked keyword that is sorted before the target word to obtain the keyword to be merged; merge the keyword to be merged and the target word according to the target sequence number to obtain a candidate phrase.

[0160] Optionally, the analysis unit 405 is specifically used to: obtain a first phrase to be analyzed and a second phrase to be analyzed by analyzing whether the part of speech of the candidate phrase exists at least one of the preset parts of speech, the preset parts of speech include verbs, gerunds and verb morphemes, the first phrase to be analyzed is a candidate phrase whose part of speech exists at least one of the preset parts of speech, and the second phrase to be analyzed is a candidate phrase whose part of speech is not any of the preset parts of speech; analyze the first phrase to be analyzed according to the parts of speech after the preset parts of speech are connected and the first preset rule to obtain a target candidate phrase; analyze the second phrase to be analyzed according to the second preset rule and the target part of speech to obtain a target candidate phrase, the target part of speech is the part of speech after the parts of speech of the second phrase to be analyzed are connected.

[0161] Optionally, the deleting unit 406 is specifically used to: classify the target candidate phrases to obtain main target candidate phrases and / or sub-target candidate phrases, wherein the sub-target candidate phrases are composed of words in the main target candidate phrases; take the main target candidate phrase as the parent node and take the length of the sub-target candidate phrase as the branch condition; obtain a substring relationship recognition tree according to the main target candidate phrases, the sub-target candidate phrases and the branch condition; retrieve the position of the target candidate phrase in the substring relationship recognition tree; if the target candidate phrase is retrieved and is located at the parent node in the substring relationship recognition tree, then delete the target candidate phrases corresponding to all child nodes in the substring relationship recognition tree to obtain a revised candidate phrase; if the target candidate phrase is retrieved and is located at the candidate child node in the substring relationship recognition tree, then delete the target candidate phrase corresponding to the target child node to obtain a revised candidate phrase, wherein the target child node is the child node after the candidate child node.

[0162] Optionally, the calculation subunit 4072 is specifically used to: obtain the dictionary and identify keywords in the field to which the short text to be processed belongs, and obtain reference data; calculate the word frequency-inverse text frequency index TF-IDF value of the initial keyword in the reference data to obtain the value to be calculated; calculate the weighted average of the value to be calculated according to the preset weight value to obtain the preset threshold; calculate the difference between the value to be calculated and the preset threshold, and when the difference is greater than or equal to 0, use the corresponding initial keyword as a candidate keyword.

[0163] The device for extracting short text keywords in the embodiment of the present application plays an important role in the fields of automatic question answering, topic tracking, information retrieval and product recommendation. For example, in the field of information retrieval, it is necessary to extract keywords from the input retrieval information to obtain keywords, and then use the keywords to retrieve the data to be retrieved to obtain the target retrieval data. In the keyword extraction of the input retrieval information to obtain keywords, the retrieval information input as a short text can be extracted from the short text by combining the grammatical structure, part-of-speech splicing filtering rules and substring relationships of the embodiment of the present application to obtain the corrected candidate phrases of the retrieval information, and the corrected candidate phrases are spliced, TF-IDF value calculated and screened according to the preset phrase part-of-speech splicing rules to obtain the target keywords. The device for extracting short text keywords in the embodiment of the present application can combine words and characters in the retrieval information of the short text without relying on an external dictionary, and accurately output the unregistered words in the exclusive field corresponding to the retrieval information, so as to give the retrieval information of the short text a keyword tag with accurate meaning, thereby improving the accuracy of keyword extraction of the short text, thereby improving the retrieval efficiency and retrieval accuracy of the retrieval information in the field of information retrieval.

[0164] above Figures 4 to 5 The apparatus for extracting short text keywords in the embodiment of the present application is described in detail from the perspective of modular functional entities. The device for extracting short text keywords in the embodiment of the present application is described in detail from the perspective of hardware processing.

[0165] Figure 6 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 601 (for example, one or more processors) and a memory 609, and one or more storage media 608 (for example, one or more mass storage devices) storing application programs 607 or data 606. Among them, the memory 609 and the storage medium 608 may be short-term storage or permanent storage. The program stored in the storage medium 608 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the sign-in management device. Furthermore, the processor 601 may be configured to communicate with the storage medium 608 to execute a series of instruction operations in the storage medium 608 on the device 600 for extracting short text keywords.

[0166] The device 600 for extracting short text keywords may also include one or more power supplies 602, one or more wired or wireless network interfaces 603, one or more input and output interfaces 604, and / or one or more operating systems 605, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 6 The device structure for extracting short text keywords shown in the figure does not constitute a limitation on the device for extracting short text keywords, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The processor 601 may execute the functions of the extraction unit 401, the acquisition unit 402, the creation unit 403, the merging unit 404, the analysis unit 405, the deletion unit 406, the splicing processing unit 407, the identification subunit 4071, the calculation subunit 4072, and the screening subunit 4073 in the above-mentioned embodiment.

[0167] Combine the following Figure 6 The following is a detailed introduction to the various components of the device for extracting short text keywords:

[0168] The processor 601 is the control center of the device for extracting short text keywords, and can be processed according to the method for extracting short text keywords. The processor 601 uses various interfaces and lines to connect the various parts of the entire device for extracting short text keywords, and executes various functions of the device for extracting short text keywords and processes data by running or executing software programs and / or modules stored in the memory 609, and calling data stored in the memory 609, thereby realizing the function of improving the accuracy of short text keyword extraction. The storage medium 608 and the memory 609 are both carriers for storing data. In the embodiment of the present application, the storage medium 608 can refer to an internal memory with a small storage capacity but a fast speed, and the memory 609 can be an external memory with a large storage capacity but a slow storage speed.

[0169] The memory 609 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing of the device 600 for extracting short text keywords by running the software programs and modules stored in the memory 609. The memory 609 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (obtaining the target annotation information of the candidate keyword and the target candidate keyword corresponding to the target annotation information, obtaining the information to be processed, the target annotation information is the annotation information with a continuous sequence number, etc.); the data storage area can store data created according to the use of the sign-in management device (pre-extracting keywords for the short text to be processed, obtaining candidate keywords, etc.). In addition, the memory 609 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. The method program for extracting short text keywords provided in the embodiment of the present application and the received data stream are stored in the memory, and when needed, the processor 601 calls from the memory 609.

[0170] When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., an optical disk), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in the embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0176] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for extracting keywords from short text, characterized in that: include: Perform keyword pre-extraction on the short text to be processed to obtain candidate keywords; Obtain target annotation information of the candidate keyword and the target candidate keyword corresponding to the target annotation information to obtain information to be processed, wherein the target annotation information is annotation information with a continuous sequence number; Creating N-gram information according to the information to be processed, and marking the N-gram information to the target candidate keyword to obtain a marked keyword; Merge the annotated keywords according to the target annotated information and the N-gram information to obtain candidate phrases; Analyze the candidate phrases according to preset phrase part-of-speech splicing rules to obtain target candidate phrases; When it is analyzed that the target candidate phrase has a substring relationship, the target candidate phrase corresponding to the substring relationship is deleted to obtain a modified candidate phrase, the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase; The modified candidate phrases with cross characters are concatenated to obtain target keywords, wherein the cross characters indicate that the end word of one modified candidate phrase is the same as the beginning word of the other modified candidate phrase in the two modified candidate phrases.

2. The method according to claim 1, characterized in that The step of combining the marked keywords according to the target marked information and the N-gram information to obtain candidate phrases includes: Obtaining the part of speech of the marked keyword, and filtering the marked keyword according to a preset filtering rule and the part of speech to obtain a target marked keyword; Comparing the values ​​of the N-gram information of the target marked keyword, and obtaining the target word corresponding to the N-gram information with the largest value; According to the target sequence number in the target annotation information, obtain the keywords in the target annotation keywords that are sorted before the target word to obtain the keywords to be merged; The keyword to be merged and the target word are merged according to the target sequence number to obtain a candidate phrase.

3. The method according to claim 1, characterized in that The step of analyzing the candidate phrases according to a preset phrase part-of-speech concatenation rule to obtain a target candidate phrase includes: By analyzing whether the part of speech of the candidate phrase has at least one of the preset parts of speech, a first phrase to be analyzed and a second phrase to be analyzed are obtained, wherein the preset parts of speech include verbs, gerunds and morphemes, the first phrase to be analyzed is a candidate phrase whose part of speech has at least one of the preset parts of speech, and the second phrase to be analyzed is a candidate phrase whose part of speech is not any of the preset parts of speech; Analyze the first phrase to be analyzed according to the part of speech after the preset part of speech is connected and the first preset rule to obtain a target candidate phrase; The second phrase to be analyzed is analyzed according to a second preset rule and a target part of speech to obtain a target candidate phrase, wherein the target part of speech is the part of speech of each part of speech of the second phrase to be analyzed and the part of speech after the parts of speech are connected.

4. The method according to claim 1, characterized in that: When it is analyzed that the target candidate phrase has a substring relationship, the target candidate phrase corresponding to the substring relationship is deleted to obtain a modified candidate phrase, wherein the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase, including: Classifying the target candidate phrases to obtain main target candidate phrases and / or sub-target candidate phrases, wherein the sub-target candidate phrases are composed of words in the main target candidate phrases; Taking the main target candidate phrase as a parent node and the length of the sub-target candidate phrase as a branch condition; Obtaining a substring relationship recognition tree according to the main target candidate phrase, the subtarget candidate phrase and the branch condition; Retrieving the position of the target candidate phrase in the substring relationship recognition tree; If the target candidate phrase is found to be located at a parent node in the substring relationship identification tree, then deleting the target candidate phrases corresponding to all child nodes in the substring relationship identification tree to obtain a revised candidate phrase; If the target candidate phrase is retrieved and located at a candidate subnode in the substring relationship recognition tree, the target candidate phrase corresponding to the target subnode is deleted to obtain a revised candidate phrase, wherein the target subnode is a subnode after the candidate subnode.

5. The method according to any one of claims 1 to 4, characterized in that: The step of concatenating the modified candidate phrases with cross characters to obtain a target keyword, wherein the cross characters indicate that the end word of one modified candidate phrase is the same as the beginning word of the other modified candidate phrase in the two modified candidate phrases, comprises: When it is recognized that the terminal word of one of the two modified candidate phrases is the same as the initial word of the other modified candidate phrase, the modified candidate phrases corresponding to the terminal word and the initial word are concatenated to obtain an initial keyword; Calculating the term frequency-inverse text frequency index TF-IDF value of the initial keyword, and determining the candidate keyword according to the TF-IDF value; The candidate keywords are screened according to the preset phrase part-of-speech concatenation rule to obtain target keywords.

6. The method according to claim 5, characterized in that The step of calculating the term frequency-inverse text frequency index TF-IDF value of the initial keyword and determining the candidate keyword according to the TF-IDF value includes: Obtaining a dictionary and identifying keywords in the field to which the short text to be processed belongs, and obtaining reference data; Calculate the term frequency-inverse text frequency index TF-IDF value of the initial keyword in the reference data to obtain a value to be calculated; Calculate the weighted average of the values ​​to be calculated according to the preset weight values ​​to obtain a preset threshold value; The difference between the value to be calculated and the preset threshold is calculated, and when the difference is greater than or equal to 0, the corresponding initial keyword is used as a candidate keyword.

7. The method according to claim 1, characterized in that The keyword pre-extraction process of the short text to be processed to obtain candidate keywords includes: The initial short text to be processed, which has been processed by removing spaces, is segmented by a word segmentation tool to obtain a plurality of initial words; Calling a preset part-of-speech tagging model to perform part-of-speech tagging on the multiple initial words to obtain a short text to be processed; Identify whether the short text to be processed has a part of speech vacancy; When it is identified that the short text to be processed does not have a part-of-speech vacancy, filtering the short text to be processed to obtain candidate keywords, wherein the filtering process includes reserved word filtering process, stop word filtering process and part-of-speech filtering process; When it is identified that the short text to be processed has a part-of-speech vacancy, the short text to be processed with the part-of-speech vacancy is subjected to a part-of-speech filling process to obtain a filling word, and the filling word is subjected to the filtering process to obtain a candidate keyword.

8. A device for extracting keywords from short text, characterized in that: The device comprises: An extraction unit, used for performing keyword pre-extraction processing on the short text to be processed to obtain candidate keywords; An acquisition unit, configured to acquire target annotation information of the candidate keyword and a target candidate keyword corresponding to the target annotation information, to obtain information to be processed, wherein the target annotation information is annotation information with a continuous sequence number; A creating unit, configured to create N-gram information according to the information to be processed, and annotate the N-gram information to the target candidate keyword to obtain an annotated keyword; A merging unit, configured to merge the marked keywords according to the target marked information and the N-gram information to obtain candidate phrases; An analysis unit, configured to analyze the candidate phrases according to a preset phrase part-of-speech concatenation rule to obtain a target candidate phrase; a deleting unit, configured to, when analyzing that the target candidate phrase has a substring relationship, delete the target candidate phrase corresponding to the substring relationship to obtain a modified candidate phrase, wherein the target candidate phrase includes a main target candidate phrase and / or a subtarget candidate phrase, and the substring relationship indicates that the subtarget candidate phrase is composed of words in the main target candidate phrase; The concatenation processing unit is used to concatenate the modified candidate phrases with cross characters to obtain a target keyword, wherein the cross characters indicate that the end word of one modified candidate phrase is the same as the beginning word of the other modified candidate phrase in the two modified candidate phrases.

9. A device for extracting keywords from short text, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for extracting short text keywords as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the method for extracting short text keywords as claimed in any one of claims 1 to 7.

Images