Text keyword weight evaluation method and device

Abstract

The invention discloses a text keyword weight evaluation method and device, and the method comprises the steps: obtaining a to-be-analyzed text, recognizing words in the to-be-analyzed text, and filtering the recognized words to obtain candidate words; positioning text sentences where the candidate words are located, and analyzing semantic importance degrees of the candidate words in the text sentences; key words in the candidate words are screened out according to the semantic importance degree, and the semantic weight of each key word is determined according to the semantic importance degree; obtaining multi-dimensional features of each keyword, and determining feature weights of the keywords according to the multi-dimensional features; determining a comprehensive weight of each keyword according to the semantic weight and the feature weight of each keyword; according to the keyword weight analysis method and device, the semantic weight and the feature weight of the keyword are comprehensively considered, the importance of the keyword can be evaluated more comprehensively, the problem that the accuracy is low due to the fact that the keyword weight is determined only depending on a single feature can be avoided, and therefore the accuracy of keyword weight analysis in the keyword extraction process is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of keyword extraction in documents, and in particular to a text keyword weight evaluation method and device. Background Art

[0002] With the rapid development of the Internet, the amount of information that users can access has exploded. Every day, users need to spend a lot of time browsing a large amount of text content to find the information they are interested in. However, as the amount of information increases, users spend more and more time on information retrieval. At present, storing millions of web pages and hundreds of thousands of documents has become the norm. In the field of natural language processing (NLP), identifying and analyzing keywords in text is an important task. Keywords can provide key information of the text and help us understand the theme, sentiment and importance of the text. By using keywords, we can better describe the theme of the text, improve the efficiency and accuracy of text processing, and provide users with a better information experience.

[0003] However, when faced with massive amounts of unlabeled Internet information, traditional automated methods usually only consider surface features such as word frequency and word correlation, but fail to fully utilize the semantic information of words. This results in unsatisfactory accuracy in keyword weight analysis. Therefore, how to accurately evaluate keyword weight has become a technical problem that needs to be urgently solved by technicians in this field. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a text keyword weight evaluation method and device, comprising:

[0005] Obtain the text to be analyzed, identify the words in the text to be analyzed, and filter the identified words to obtain candidate words;

[0006] Locating the text sentence where the candidate word is located, and analyzing the semantic importance of the candidate word in the text sentence;

[0007] Filtering out keywords from the candidate words according to the semantic importance, and determining the semantic weight of each keyword according to the semantic importance;

[0008] Acquire multidimensional features of each of the keywords, and determine feature weights of the keywords according to the multidimensional features;

[0009] The comprehensive weight of each keyword is determined according to the semantic weight and feature weight of each keyword.

[0010] Furthermore, the step of obtaining the text to be analyzed, identifying words in the text to be analyzed, and filtering the identified words to obtain candidate words includes:

[0011] Obtaining the text to be analyzed and preprocessing the text to be analyzed, wherein the preprocessing includes removing special characters, punctuation marks, and letter conversion;

[0012] Use the word segmentation tool to segment the preprocessed text to be analyzed to obtain multiple single words;

[0013] By comparing multiple single words using a stop word list, words without actual meaning in the multiple single words are excluded to obtain pre-selected candidate words;

[0014] The frequency of occurrence of the pre-selected candidate words in the text to be analyzed is calculated, and the pre-selected candidate words with a frequency lower than a preset value are filtered out, and the remaining words are the candidate words.

[0015] Furthermore, the locating the text sentence where the candidate word is located and analyzing the semantic importance of the candidate word in the text sentence includes:

[0016] Use the sentence segmentation tool to segment the text to be analyzed into sentences to obtain multiple text sentences;

[0017] Traversing each of the text sentences, determining whether each of the text sentences contains a preset number of candidate words, and locating the text sentences containing the preset number of candidate words;

[0018] The semantic importance between each of the candidate words and other words in the text sentence is calculated, and the semantic importance of the candidate word in the text sentence is measured according to the semantic importance of the candidate word.

[0019] Furthermore, the calculating of the semantic importance between each candidate word and other words in the text sentence, and measuring the semantic importance of the candidate word in the text sentence according to the semantic importance of the candidate word, includes:

[0020] Vectorizing each of the candidate words and other words in the text sentence through a word vector model to obtain candidate word vectors and other word vectors respectively;

[0021] The cosine values ​​of the angles between the candidate word vector and other word vectors are calculated in sequence, the cosine values ​​are used to represent the semantic importance of the candidate word, and the importance of the candidate word in the text sentence is measured according to the semantic importance of the candidate word.

[0022] Furthermore, the step of selecting keywords from the candidate words according to the semantic importance and determining the semantic weight of each keyword according to the semantic importance includes:

[0023] Obtaining the semantic importance between the candidate word vector and other word vectors, and calculating an average value based on the semantic importance;

[0024] Select the candidate words whose semantic importance is greater than a preset value as keywords;

[0025] The semantic importance of the keyword is obtained, and the semantic weight of each keyword is determined according to a mapping relationship between the semantic importance of the keyword and a preset semantic weight.

[0026] Furthermore, the acquiring of the multidimensional features of each of the keywords and determining the feature weight of the keywords according to the multidimensional features includes:

[0027] Obtaining multidimensional features of each of the keywords in the text to be tested, wherein the multidimensional features include word frequency, average word span, word length, and inverse document frequency;

[0028] Normalizing the multidimensional features to obtain a word frequency value, a word average span value, a word length value, and an inverse document frequency value;

[0029] Assigning word frequency values, word average span values, word length values, and inverse document frequency values ​​by an expert assignment method, and determining weight coefficients corresponding to the word frequency values, word average span values, word length values, and inverse document frequency values, respectively, according to the assignment results;

[0030] The feature weight of the keyword is determined based on the word frequency value, the word average span value, the word length value and the inverse document frequency value and their corresponding weight coefficients.

[0031] Furthermore, the characteristic weight calculation formula of the keyword is:

[0032] S=α*A+β*B+γ*C+δ*D,

[0033] Among them, S is the feature weight value of the keyword, α is the weight coefficient corresponding to the word frequency value, A is the word frequency value, β is the weight coefficient corresponding to the word average span value, B is the word average span value, γ is the weight coefficient corresponding to the word length value, C is the word length value, δ is the weight coefficient corresponding to the inverse document frequency value, and D is the inverse document frequency value.

[0034] Furthermore, determining the comprehensive weight of each keyword according to the semantic weight and feature weight of each keyword includes:

[0035] The semantic weight and feature weight of the keyword are added together to obtain the comprehensive weight of the keyword.

[0036] The present invention also provides a text keyword weight evaluation device, comprising:

[0037] The recognition module is used to obtain the text to be analyzed, identify the words in the text to be analyzed, and filter the identified words to obtain candidate words;

[0038] An analysis module, used to locate the text sentence where the candidate word is located, and analyze the semantic importance of the candidate word in the text sentence;

[0039] A screening module, used for screening out keywords from the candidate words according to the semantic importance, and determining the semantic weight of each keyword according to the semantic importance;

[0040] A determination module, used to obtain the multi-dimensional features of each of the keywords, and determine the feature weight of the keyword according to the multi-dimensional features;

[0041] The calculation module is used to determine the comprehensive weight of each keyword according to the semantic weight and feature weight of each keyword.

[0042] Compared with the prior art, the text keyword weight evaluation method and device of the embodiment of the present invention have the following beneficial effects:

[0043] The present invention can accurately extract keywords from the text by performing word recognition and filtering on the text to be analyzed, thereby preventing some irrelevant or unimportant words from being selected as keywords;

[0044] By analyzing the semantic importance of candidate words in text sentences, the present invention can more accurately screen out keywords with higher semantic importance, and can ensure that the selected keywords are more relevant to the text theme and importance;

[0045] The present invention adopts a method based on semantic analysis, which can better capture the semantic relevance between keywords and ensure that the selected keywords are more coherent and relevant when expressing the text theme and content;

[0046] The present invention can more comprehensively evaluate the importance of keywords by comprehensively considering the semantic weights and feature weights of keywords, and can avoid the problem of low accuracy caused by relying solely on a single feature to determine keyword weights, thereby improving the accuracy of keyword weight analysis in the keyword extraction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the process structure of the text keyword weight evaluation method in an embodiment of the present invention;

[0048] Figure 2 It is a schematic diagram of the composition of a text keyword weight evaluation device in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0050] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the platform or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0051] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0052] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0053] like Figure 1 As shown, in an embodiment of the present application, a text keyword weight evaluation method is provided, including: S100: obtaining a text to be analyzed, and identifying words in the text to be analyzed, and filtering the identified words to obtain candidate words; S200: locating the text sentence where the candidate word is located, and analyzing the semantic importance of the candidate word in the text sentence; S300: screening out keywords in the candidate words according to the semantic importance, and determining the semantic weight of each keyword according to the semantic importance; S400: obtaining multidimensional features of each keyword, and determining the feature weight of the keyword according to the multidimensional features; S500: determining the comprehensive weight of each keyword according to the semantic weight and feature weight of each keyword.

[0054] Furthermore, the present invention can accurately extract keywords in the text by performing word recognition and filtering on the text to be analyzed, thereby avoiding some irrelevant or unimportant words from being selected as keywords; the present invention can more accurately screen out keywords with higher semantic importance by analyzing the semantic importance of candidate words in text sentences, thereby ensuring that the selected keywords are more relevant to the text theme and importance; the present invention adopts a method based on semantic analysis, which can better capture the semantic correlation between keywords, thereby ensuring that the selected keywords are more coherent and relevant when expressing the text theme and content; the present invention can more comprehensively evaluate the importance of keywords by comprehensively considering the semantic weights and feature weights of keywords, thereby avoiding the problem of low accuracy caused by relying solely on a single feature to determine the keyword weight, thereby improving the accuracy of keyword weight analysis in the keyword extraction process.

[0055] In an embodiment of the present application, a text keyword weight evaluation method is provided, wherein the text to be analyzed is obtained, and words in the text to be analyzed are identified, and candidate words are obtained after filtering the identified words, including: obtaining the text to be analyzed, and preprocessing the text to be analyzed, wherein the preprocessing includes removing special characters, punctuation marks, and letter conversion; using a word segmentation tool to segment and split the preprocessed text to be analyzed to obtain multiple single words; by using a stop word list to compare the multiple single words, words without practical meaning in the multiple single words are excluded to obtain preselected candidate words; calculating the frequency of occurrence of the preselected candidate words in the text to be analyzed, filtering the preselected candidate words with a frequency lower than a preset value, and the remaining words are candidate words.

[0056] In an embodiment of the present application, a text keyword weight evaluation method is provided, which locates the text sentence where the candidate word is located and analyzes the semantic importance of the candidate word in the text sentence, including: using a sentence segmentation tool to segment the text to be analyzed according to sentences to obtain multiple text sentences; traversing each of the text sentences, determining whether each of the text sentences contains a preset number of candidate words, and locating the text sentences containing the preset number of candidate words; calculating the semantic importance between each of the candidate words and other words in the text sentence, and measuring the semantic importance of the candidate words in the text sentence according to the semantic importance of the candidate words.

[0057] In an embodiment of the present application, a text keyword weight evaluation method is provided, wherein the semantic importance between each candidate word and other words in the text sentence is calculated, and the semantic importance of the candidate word in the text sentence is measured according to the semantic importance of the candidate word, including: vectorizing each candidate word and other words in the text sentence through a word vector model to obtain candidate word vectors and other word vectors respectively; calculating the cosine value of the angle between the candidate word vector and the other word vector in turn, using the cosine value to represent the semantic importance of the candidate word, and measuring the importance of the candidate word in the text sentence according to the semantic importance of the candidate word.

[0058] In an embodiment of the present application, a text keyword weight evaluation method is provided, which filters out keywords from the candidate words according to the semantic importance, and determines the semantic weight of each keyword according to the semantic importance, including: obtaining the semantic importance between the candidate word vector and other word vectors, and calculating the average value according to the semantic importance; filtering out the candidate words whose semantic importance is greater than a preset value as keywords; obtaining the semantic importance of the keywords, and determining the semantic weight of each keyword according to the mapping relationship between the semantic importance of the keywords and the preset semantic weights.

[0059] In an embodiment of the present application, a text keyword weight evaluation method is provided, wherein the multidimensional features of each keyword are obtained, and the feature weights of the keywords are determined according to the multidimensional features, including: obtaining the multidimensional features of each keyword in the text to be tested, the multidimensional features including word frequency, average word span, word length and inverse document frequency; normalizing the multidimensional features to obtain word frequency values, average word span values, word length values ​​and inverse document frequency values; assigning the word frequency values, average word span values, word length values ​​and inverse document frequency values ​​by an expert assignment method, and determining the weight coefficients corresponding to the word frequency values, average word span values, word length values ​​and inverse document frequency values ​​according to the assignment results; and jointly determining the feature weights of the keywords according to the word frequency values, average word span values, word length values ​​and inverse document frequency values ​​and their corresponding weight coefficients.

[0060] In an embodiment of the present application, a text keyword weight evaluation method is provided, and the characteristic weight calculation formula of the keyword is:

[0061] S=α*A+β*B+γ*C+δ*D,

[0062] Among them, S is the feature weight value of the keyword, α is the weight coefficient corresponding to the word frequency value, A is the word frequency value, β is the weight coefficient corresponding to the word average span value, B is the word average span value, γ is the weight coefficient corresponding to the word length value, C is the word length value, δ is the weight coefficient corresponding to the inverse document frequency value, and D is the inverse document frequency value.

[0063] In an embodiment of the present application, a text keyword weight evaluation method is provided, which determines the comprehensive weight of each keyword based on the semantic weight and feature weight of each keyword, including: adding the semantic weight and feature weight of the keyword to obtain the comprehensive weight of the keyword.

[0064] like Figure 2 As shown, in an embodiment of the present application, a text keyword weight evaluation device is provided, including: an identification module, used to obtain a text to be analyzed, and identify words in the text to be analyzed, and filter the identified words to obtain candidate words; an analysis module, used to locate the text sentence where the candidate word is located, and analyze the semantic importance of the candidate word in the text sentence; a screening module, used to screen out the keywords in the candidate words according to the semantic importance, and determine the semantic weight of each of the keywords according to the semantic importance; a determination module, used to obtain the multidimensional features of each of the keywords, and determine the feature weight of the keywords according to the multidimensional features; a calculation module, used to determine the comprehensive weight of each of the keywords according to the semantic weight and feature weight of each of the keywords.

[0065] In summary, an embodiment of the present invention provides a text keyword weight evaluation method and device, which includes: obtaining a text to be analyzed, and identifying words in the text to be analyzed, and filtering the identified words to obtain candidate words; locating the text sentence where the candidate words are located, and analyzing the semantic importance of the candidate words in the text sentence; screening out keywords in the candidate words according to the semantic importance, and determining the semantic weight of each keyword according to the semantic importance; obtaining multidimensional features of each keyword, and determining the feature weight of the keyword according to the multidimensional features; determining the comprehensive weight of each keyword according to the semantic weight and feature weight of each keyword; the present invention can more comprehensively evaluate the importance of keywords by comprehensively considering the semantic weight and feature weight of keywords, and can avoid the problem of low accuracy caused by relying solely on a single feature to determine the keyword weight, thereby improving the accuracy of keyword weight analysis in the keyword extraction process.

[0066] Finally, it should be noted that it is obvious that those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0067] The above is only an example of implementation of the present invention, but it cannot be used to limit the scope of the present invention. Any structural changes made according to the present invention, as long as they do not lose the essence of the present invention, should be regarded as falling within the scope of protection of the present invention and being restricted. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and related instructions of the platform described above can refer to the corresponding process in the aforementioned platform embodiment, and will not be repeated here.

[0068] The term "comprises" or any other similar term is intended to cover a non-exclusive inclusion such that a process, platform, article, or apparatus / platform that includes a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, platform, article, or apparatus / platform.

[0069] So far, the technical solutions of the present invention have been described in conjunction with the further embodiments shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A text keyword weight evaluation method, characterized in that: include: Obtain the text to be analyzed, identify the words in the text to be analyzed, and filter the identified words to obtain candidate words; Locating the text sentence where the candidate word is located, and analyzing the semantic importance of the candidate word in the text sentence; Filtering out keywords from the candidate words according to the semantic importance, and determining the semantic weight of each keyword according to the semantic importance; Acquire multidimensional features of each of the keywords, and determine feature weights of the keywords according to the multidimensional features; The comprehensive weight of each keyword is determined according to the semantic weight and feature weight of each keyword.

2. A text keyword weight evaluation method according to claim 1, characterized in that: The step of obtaining the text to be analyzed, identifying words in the text to be analyzed, and filtering the identified words to obtain candidate words includes: Obtaining the text to be analyzed and preprocessing the text to be analyzed, wherein the preprocessing includes removing special characters, punctuation marks, and letter conversion; Use the word segmentation tool to segment the preprocessed text to be analyzed to obtain multiple single words; By comparing multiple single words using a stop word list, words without actual meaning in the multiple single words are excluded to obtain pre-selected candidate words; The frequency of occurrence of the pre-selected candidate words in the text to be analyzed is calculated, and the pre-selected candidate words with a frequency lower than a preset value are filtered out, and the remaining words are the candidate words.

3. A text keyword weight evaluation method according to claim 1, characterized in that: The locating the text sentence where the candidate word is located and analyzing the semantic importance of the candidate word in the text sentence includes: Use the sentence segmentation tool to segment the text to be analyzed into sentences to obtain multiple text sentences; Traversing each of the text sentences, determining whether each of the text sentences contains a preset number of candidate words, and locating the text sentences containing the preset number of candidate words; The semantic importance between each of the candidate words and other words in the text sentence is calculated, and the semantic importance of the candidate word in the text sentence is measured according to the semantic importance of the candidate word.

4. A text keyword weight evaluation method according to claim 3, characterized in that: The calculating the semantic importance between each candidate word and other words in the text sentence, and measuring the semantic importance of the candidate word in the text sentence according to the semantic importance of the candidate word, includes: Vectorizing each of the candidate words and other words in the text sentence through a word vector model to obtain candidate word vectors and other word vectors respectively; The cosine values ​​of the angles between the candidate word vector and other word vectors are calculated in sequence, the cosine values ​​are used to represent the semantic importance of the candidate word, and the importance of the candidate word in the text sentence is measured according to the semantic importance of the candidate word.

5. A text keyword weight evaluation method according to claim 4, characterized in that: The step of selecting keywords from the candidate words according to the semantic importance and determining the semantic weight of each keyword according to the semantic importance includes: Obtaining the semantic importance between the candidate word vector and other word vectors, and calculating an average value based on the semantic importance; Select the candidate words whose semantic importance is greater than a preset value as keywords; The semantic importance of the keyword is obtained, and the semantic weight of each keyword is determined according to a mapping relationship between the semantic importance of the keyword and a preset semantic weight.

6. A text keyword weight evaluation method according to claim 1, characterized in that: The acquiring of the multidimensional features of each of the keywords and determining the feature weight of the keywords according to the multidimensional features includes: Obtaining multidimensional features of each of the keywords in the text to be tested, wherein the multidimensional features include word frequency, average word span, word length, and inverse document frequency; Normalizing the multidimensional features to obtain a word frequency value, a word average span value, a word length value, and an inverse document frequency value; Assigning word frequency values, word average span values, word length values, and inverse document frequency values ​​by an expert assignment method, and determining weight coefficients corresponding to the word frequency values, word average span values, word length values, and inverse document frequency values, respectively, according to the assignment results; The feature weight of the keyword is determined based on the word frequency value, the word average span value, the word length value and the inverse document frequency value and their corresponding weight coefficients.

7. A text keyword weight evaluation method according to claim 6, characterized in that: The characteristic weight calculation formula of the keyword is: S=α*A+β*B+γ*C+δ*D, Among them, S is the feature weight value of the keyword, α is the weight coefficient corresponding to the word frequency value, A is the word frequency value, β is the weight coefficient corresponding to the word average span value, B is the word average span value, γ is the weight coefficient corresponding to the word length value, C is the word length value, δ is the weight coefficient corresponding to the inverse document frequency value, and D is the inverse document frequency value.

8. A text keyword weight evaluation method according to claim 1, characterized in that: Determining the comprehensive weight of each keyword according to the semantic weight and feature weight of each keyword includes: The semantic weight and feature weight of the keyword are added together to obtain the comprehensive weight of the keyword.

9. A text keyword weight evaluation device, characterized in that: include: The recognition module is used to obtain the text to be analyzed, identify the words in the text to be analyzed, and filter the identified words to obtain candidate words; An analysis module, used to locate the text sentence where the candidate word is located, and analyze the semantic importance of the candidate word in the text sentence; A screening module, used for screening out keywords from the candidate words according to the semantic importance, and determining the semantic weight of each keyword according to the semantic importance; A determination module, used to obtain the multi-dimensional features of each of the keywords, and determine the feature weight of the keyword according to the multi-dimensional features; The calculation module is used to determine the comprehensive weight of each keyword according to the semantic weight and feature weight of each keyword.

Images